[07:32:42] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 67236 [07:33:31] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [07:33:51] MySQL slave on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [07:34:50] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [07:35:25] ralf * Re: [Toolserver-l] Future of the toolserver [07:36:52] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 552815 MB (10% inode=45%): [07:40:41] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [07:41:51] APT on mayapple is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [07:41:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:49:10] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2154.000000 [07:57:52] /aux0 on hemlock is CRITICAL: Connection refused by host [07:58:51] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 552766 MB (10% inode=45%): [08:03:41] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:03:42] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:03:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:04:41] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:07:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:11:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:13:51] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [08:15:10] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3624.000000 [08:15:50] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [08:16:52] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 73832 [08:16:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:30:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:31:10] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 70588 [08:32:50] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [08:34:02] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [08:40:40] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [08:42:51] APT on mayapple is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [08:42:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:53:51] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 552718 MB (10% inode=45%): [09:03:40] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:03:40] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:03:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:04:41] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:08:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:11:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:12:25] Platonides * Re: [Toolserver-l] Future of the toolserver [09:13:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [09:15:10] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5906.000000 [09:16:53] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 77350 [09:16:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:30:21] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:31:10] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 73953 [09:32:51] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [09:33:52] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:34:51] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 552683 MB (10% inode=45%): [09:35:01] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 82011 [09:35:52] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:39:10] Load avg. on hemlock is CRITICAL: Connection refused by host [09:39:11] /home on hemlock is CRITICAL: Connection refused by host [09:39:20] Environment IPMI on hemlock is CRITICAL: Connection refused by host [09:40:10] Load avg. on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:40:10] /home on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:40:20] Environment IPMI on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:40:41] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [09:41:51] CAM on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:42:01] / on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:42:52] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:42:52] APT on mayapple is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [09:42:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:43:01] SSH on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:43:25] DeltaQuad Wikipedia * Re: [Toolserver-l] Future of the toolserver [09:46:57] SMF on web.amaranth is CRITICAL -> SMF on web.AmonAmarth is CRITICAL :) [09:54:02] SSH on hemlock is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [09:54:02] / on hemlock is OK: DISK OK - free space: / 6349 MB (31% inode=89%): [09:54:10] Load avg. on hemlock is OK: OK - load average: 0.61, 0.40, 0.36 [09:54:10] /home on hemlock is OK: DISK OK - free space: /home 19846 MB (39% inode=87%): [09:54:29] Environment IPMI on hemlock is OK: ok: temperature ok fan ok voltage ok chassis ok [09:54:51] /tmp on hemlock is OK: DISK OK - free space: / 6346 MB (31% inode=89%): [09:54:52] CAM on hemlock is OK: OK - cam detected no new errors [10:02:15] @replag [10:03:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:04:40] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:04:40] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:04:40] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:04:52] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 552469 MB (10% inode=45%): [10:08:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:11:49] @replag [10:11:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:13:52] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:13:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [10:15:10] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7202.000000 [10:17:52] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 81010 [10:17:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:18:50] CAM on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:19:00] / on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:19:09] Load avg. on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:19:21] /home on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:19:21] Environment IPMI on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:19:26] @replag [10:19:43] where's the bot [10:19:51] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:20:01] SSH on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:22:21] @replag [10:31:00] SMTP on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:31:10] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 77362 [10:31:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:32:52] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [10:35:01] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 85556 [10:35:52] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 552397 MB (10% inode=45%): [10:35:52] /tmp on hemlock is OK: DISK OK - free space: / 6341 MB (31% inode=89%): [10:35:52] SSH on hemlock is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [10:37:51] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:38:00] SMTP on hemlock is OK: SMTP OK - 0.916 sec. response time [10:40:52] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [10:41:10] Load avg. on hemlock is OK: OK - load average: 0.29, 0.44, 0.39 [10:41:21] /home on hemlock is OK: DISK OK - free space: /home 19919 MB (39% inode=87%): [10:42:52] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 552394 MB (10% inode=45%): [10:42:52] APT on mayapple is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [10:42:52] CAM on hemlock is OK: OK - cam detected no new errors [10:42:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:43:01] / on hemlock is OK: DISK OK - free space: / 6321 MB (31% inode=89%): [10:43:20] Environment IPMI on hemlock is OK: ok: temperature ok fan ok voltage ok chassis ok [10:46:26] Andre Koopal * Re: [Toolserver-l] Future of the toolserver [10:52:52] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [11:03:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:05:40] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:05:41] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:05:41] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:09:51] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:11:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:12:50] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 552334 MB (10% inode=45%): [11:13:50] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 105275 MB (10% inode=99%): [11:14:51] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [11:15:10] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7545.000000 [11:16:51] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107524 MB (11% inode=99%): [11:18:52] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 84636 [11:18:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:22:25] Danny B. * Re: [Toolserver-l] Future of the toolserver [11:31:09] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 80890 [11:32:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:32:51] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [11:36:41] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 89208 [11:40:51] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [11:43:50] APT on mayapple is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [11:43:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:04:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:06:40] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:06:41] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:06:41] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:07:25] Katie Chan * [Toolserver-l] Fwd: Re: [Wikimediauk-l] Fwd: Future of the toolserver [12:09:50] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:12:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:12:52] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 552268 MB (10% inode=45%): [12:14:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [12:15:10] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3670.000000 [12:16:26] Delphine Ménard * Re: [Toolserver-l] Future of the toolserver [12:18:51] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 88137 [12:18:51] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:31:10] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 84415 [12:32:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:32:52] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [12:35:25] Andre Koopal * Re: [Toolserver-l] Future of the toolserver [12:37:01] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 92807 [12:40:51] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [12:43:52] APT on mayapple is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [12:43:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:57:09] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3593.000000 [13:03:25] DaB. * Re: [Toolserver-l] Future of the toolserver [13:04:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:06:40] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:06:41] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:06:41] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:09:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:10:49] <`fox`> hi! I'd like to build a python web app that uses the toolserver database. Is it possible to access the db from a third-party server? [13:13:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:13:51] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 552127 MB (10% inode=45%): [13:15:51] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [13:16:26] Ariel T. Glenn * Re: [Toolserver-l] Future of the toolserver [13:16:26] Thehelpfulone * Re: [Toolserver-l] Future of the toolserver [13:18:52] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 91606 [13:18:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:21:25] Thehelpfulone * Re: [Toolserver-l] Future of the toolserver [13:26:10] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3606.000000 [13:31:09] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 87896 [13:32:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:33:51] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [13:37:01] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 96385 [13:38:10] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3596.000000 [13:40:52] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [13:43:10] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3609.000000 [13:44:51] APT on mayapple is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [13:44:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:00:26] Merlissimo * Re: [Toolserver-l] Future of the toolserver [14:05:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:07:40] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:07:40] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:07:40] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:09:52] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 104977 MB (10% inode=99%): [14:10:51] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:10:52] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 109892 MB (11% inode=99%): [14:11:10] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3243.000000 [14:13:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:14:50] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 551962 MB (10% inode=45%): [14:15:51] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [14:18:52] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 95112 [14:18:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:31:10] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 91423 [14:32:19] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:32:25] Matthew Bowker * Re: [Toolserver-l] Future of the toolserver [14:33:51] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [14:37:01] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 99952 [14:40:51] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [14:44:51] APT on mayapple is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [14:44:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:58:10] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1739.000000 [15:05:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:07:40] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:07:40] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:07:40] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:10:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:13:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:14:51] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 551700 MB (10% inode=45%): [15:15:51] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [15:19:51] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 98623 [15:19:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:31:10] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 94873 [15:32:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:33:51] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [15:37:01] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 103516 [15:40:52] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [15:42:03] * Danny_B|backup considers changing topic to "RIP Toolserver" :-( [15:45:51] APT on mayapple is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [15:45:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:06:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:07:40] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:07:40] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:07:40] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:11:50] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:13:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:14:52] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 551070 MB (10% inode=45%): [16:15:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [16:19:51] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 102135 [16:20:51] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:31:10] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 98391 [16:32:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:33:53] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [16:37:00] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 107095 [16:40:53] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [16:42:10] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 39133 MB (9% inode=99%): [16:45:51] APT on mayapple is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [16:45:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:54:30] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:54:30] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:54:30] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:56:01] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 80857 MB (8% inode=98%): [16:56:01] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 80857 MB (8% inode=98%): [16:56:02] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 39116 MB (9% inode=99%): [17:01:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 125946 MB (20% inode=99%): [17:04:30] Federico Leva (Nemo) * Re: [Toolserver-l] Future of the toolserver [17:06:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:07:40] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:07:40] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:07:40] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:11:50] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:13:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:15:52] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 550859 MB (10% inode=45%): [17:16:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [17:19:51] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 105632 [17:20:50] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:27:01] Free Memory on damiana is WARNING: WARNING - 6.5% (545556 kB) free! [17:31:20] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 101843 [17:32:30] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:34:51] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [17:37:10] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 110668 [17:41:52] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [17:45:00] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (416940 kB) free! [17:45:52] APT on mayapple is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [17:45:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:46:01] Free Memory on damiana is WARNING: WARNING - 5.2% (431724 kB) free! [17:50:01] Free Memory on damiana is CRITICAL: CRITICAL - 4.8% (398504 kB) free! [17:54:10] @replag [17:54:16] Platonides: s3-rr-a: 1d 7h 2m 3s [+0.99 s/s]; s3-user: 1d 7h 2m 6s [+0.99 s/s]; s6-rr-a: 1d 4h 39m 25s [+0.96 s/s]; s6-user: 1d 4h 39m 28s [+0.96 s/s]; s7-rr-a: 1d 5h 54m 0s [+0.97 s/s]; s7-user: 1d 5h 54m 0s [+0.97 s/s] [17:54:32] why is it so high? [18:00:26] were the servers rebooted and the replication stopped? [18:02:01] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 13767 MB (1% inode=95%): [18:02:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 116638 MB (19% inode=99%): [18:06:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:07:40] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:07:41] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:08:40] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:41:51] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [18:45:52] APT on mayapple is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [18:45:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:49:26] Erik Moeller * Re: [Toolserver-l] Future of the toolserver [18:54:30] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:55:01] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 80682 MB (8% inode=98%): [19:02:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 110333 MB (18% inode=99%): [19:06:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:07:40] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:08:41] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:08:41] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:11:51] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:13:00] Free Memory on damiana is CRITICAL: CRITICAL - 3.0% (248756 kB) free! [19:14:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:16:51] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 550553 MB (10% inode=45%): [19:17:51] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [19:19:52] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 112661 [19:21:51] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:31:20] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 108722 [19:32:30] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:33:57] @replag [19:35:52] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [19:36:31] Krinkle: s3-rr-a: 1d 8h 41m 4s [+1.00 s/s]; s3-user: 1d 8h 42m 46s [+1.00 s/s]; s6-rr-a: 1d 6h 16m 33s [+0.98 s/s]; s6-user: 1d 6h 17m 23s [+0.99 s/s]; s7-rr-a: 1d 7h 33m 46s [+0.95 s/s]; s7-user: 1d 7h 33m 46s [+0.95 s/s] [19:36:51] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106918 MB (10% inode=99%): [19:37:10] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 117798 [19:41:52] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [19:45:51] APT on mayapple is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [19:45:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:55:20] MySQL on z-dat-s6-a is CRITICAL: Cant connect to MySQL server on z-dat-s6-a (146) [19:57:29] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [19:58:02] MySQL on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [20:01:20] MySQL on z-dat-s6-a is OK: Uptime: 21 Threads: 5 Questions: 182 Slow queries: 0 Opens: 64 Flush tables: 1 Open tables: 49 Queries per second avg: 8.666 [20:02:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 102897 MB (16% inode=99%): [20:06:29] MySQL on z-dat-s7-a is OK: Uptime: 46 Threads: 4 Questions: 1056 Slow queries: 0 Opens: 168 Flush tables: 1 Open tables: 161 Queries per second avg: 22.956 [20:06:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:07:40] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:09:40] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:09:40] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:10:27] Merlissimo * Re: [Toolserver-l] Future of the toolserver [20:11:01] MySQL on z-dat-s3-a is OK: Uptime: 35 Threads: 3 Questions: 88 Slow queries: 0 Opens: 56 Flush tables: 1 Open tables: 49 Queries per second avg: 2.514 [20:11:51] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:13:01] Free Memory on damiana is CRITICAL: CRITICAL - 2.9% (239876 kB) free! [20:14:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:17:51] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 550452 MB (10% inode=45%): [20:17:51] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [20:19:51] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 109867 MB (11% inode=99%): [20:20:50] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 109540 [20:21:51] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:28:30] Tim Landscheidt * Re: [Toolserver-l] Future of the toolserver [20:31:20] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 106257 [20:32:30] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:36:51] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [20:37:10] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 120464 [20:42:51] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [20:46:51] APT on mayapple is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [20:46:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:02:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 108386 MB (17% inode=99%): [21:02:27] Hersfold Wikipedia * Re: [Toolserver-l] Future of the toolserver [21:07:40] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:07:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:09:40] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:09:41] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:12:51] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:13:00] Free Memory on damiana is CRITICAL: CRITICAL - 2.0% (165292 kB) free! [21:13:25] DaB. * Re: [Toolserver-l] Future of the toolserver [21:14:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:17:51] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 550317 MB (10% inode=45%): [21:17:51] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [21:20:52] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 84675 [21:21:51] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:29:25] Platonides * Re: [Toolserver-l] Future of the toolserver [21:29:58] lol @ domas image [21:31:21] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 98377 [21:32:30] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:36:51] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [21:37:10] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 107555 [21:39:25] Federico Leva (Nemo) * Re: [Toolserver-l] Future of the toolserver [21:42:52] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [21:46:52] APT on mayapple is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [21:46:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:02:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 100198 MB (16% inode=99%): [22:07:40] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:07:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:09:40] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:10:40] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:12:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:14:01] Free Memory on damiana is CRITICAL: CRITICAL - 1.9% (159596 kB) free! [22:14:26] Platonides * Re: [Toolserver-l] Future of the toolserver [22:14:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:17:52] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 550207 MB (10% inode=45%): [22:17:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [22:20:52] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 75581 [22:21:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:23:25] Ryan Lane * Re: [Toolserver-l] Future of the toolserver [22:31:25] Ryan Lane * Re: [Toolserver-l] Future of the toolserver [22:32:20] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 98405 [22:32:29] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:37:10] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 109002 [22:37:51] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [22:42:51] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [22:46:51] APT on mayapple is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [22:46:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:58:25] DaB. * Re: [Toolserver-l] Future of the toolserver [23:02:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 90122 MB (14% inode=99%): [23:07:40] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:07:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:09:40] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:10:40] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:13:51] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:14:01] Free Memory on damiana is CRITICAL: CRITICAL - 1.6% (130740 kB) free! [23:15:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:18:50] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 548961 MB (10% inode=45%): [23:18:51] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [23:20:51] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 78813 [23:23:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:32:21] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 101312 [23:32:29] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:37:11] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 112432 [23:37:52] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [23:42:52] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [23:46:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:47:51] APT on mayapple is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates).