[00:02:16] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:02:44] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:04:53] /sql on ptolemy is CRITICAL: (Service Check Timed Out) [00:16:46] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 81024 MB (13% inode=99%): [00:16:46] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:16:55] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:17:05] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:17:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:17:25] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:17:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:18:55] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 547847 MB (10% inode=45%): [00:19:05] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [00:19:36] Free Memory on damiana is OK: OK - 84.7% (7100688 kB) free. [00:20:56] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 81560 [00:22:55] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:26:25] Ryan Lane * Re: [Toolserver-l] Future of the toolserver [00:27:12] I just had qcronsub throw me an error… http://pastebin.com/raw.php?i=dGYRG9Yq [00:27:27] Anyone know if that is a problem on my end or the toolserver? [00:32:36] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:33:06] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 102981 [00:38:06] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 115130 [00:38:26] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [00:38:41] legoktm: can you post the complete command line with all arguements? [00:39:22] qcronsub -l h_rt=0:20:00 -l virtual_free=50M -b y -wd /home/legoktm/rewrite -l arch=sol -N til -o ~/public_html/r_til_watcher.log /home/legoktm/python/bin/python /home/legoktm/rewrite/pwb.py /home/legoktm/rewrite/r_til_watcher.py [00:43:26] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [00:44:31] could it be that "til" has a special measning in python? legoktmcan you test it choosing a different job name? [00:44:58] Sure [00:45:52] Resubmitted it using qsub with the name "testing" [00:47:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:48:26] APT on mayapple is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [00:50:06] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2174.000000 [00:50:10] legoktm: this error won't occur if you use qsub, that's only possible if you use qcronsub [00:50:16] s5 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2185.000000 [00:50:26] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2191.000000 [00:50:26] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2196.000000 [00:50:36] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2208.000000 [00:50:41] @replag [00:50:54] Merlissimo: s1-rr-a: 36m 57s [-]; s1-user: 36m 57s [-]; s2-user: 39m 5s [-]; s2-user-c: 36m 56s [-]; s3-rr-a: 1d 8h 8m 24s [-]; s3-user: 1d 8h 8m 24s [-]; s4-rr-a: 36m 56s [-]; s4-user: 36m 56s [-] [00:50:55] Merlissimo: s5-rr-a: 37m 0s [-]; s5-user: 37m 0s [-]; s5-user-c: 36m 56s [-]; s6-rr-a: 1d 4h 46m 17s [-]; s6-user: 1d 4h 46m 29s [-]; s7-rr-a: 23h 3m 42s [-]; s7-user: 23h 3m 42s [-] [00:50:57] Oh. Can I run qcronsub from my shell without braking anything? [00:52:01] it only submits the job it is not already running. [00:52:58] and the failure you posted happend while the script was testing if the job is already running [00:53:54] qcronsub is not anything special for cron, it simply adds the "already running" test feature [00:56:37] Sun Grid Engine execd on yarrow is UNKNOWN: Execution timeout exceeded [00:57:16] SSH on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:58:13] Merlissimo: Oh. It seems "til" was already running (according to qstat) [00:58:31] What does "Eqw" mean for the state? q means quitting, but E? [00:58:42] job error [00:59:04] q is queued [01:00:15] w waiting ; q queued; r running ; R re-; E error; s suspended; d deleted; D disabled [01:00:26] alright [01:00:39] thanks for the help Merlissimo [01:00:50] you can delete the error using qmod -cj [01:01:08] Oh, I just deleted the job [01:01:11] I'll do that next time [01:01:44] deleting is ok, too. then you only have to resubmit the job ;-) [01:02:23] after the name change qcronsub is running now without errors? [01:04:35] Yes it worked fine thanks [01:07:34] then the name value must be escaped within the soure code if the jsv script. i'll check this tomorrow [01:14:05] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3613.000000 [01:14:17] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3625.000000 [01:14:26] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3631.000000 [01:14:26] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3636.000000 [01:15:16] s5 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2973.000000 [01:16:26] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3546.000000 [01:16:47] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 73251 MB (12% inode=99%): [01:16:47] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:16:56] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:17:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:17:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:17:16] s5 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1308.000000 [01:17:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:17:36] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1387.000000 [01:17:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:18:56] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 547759 MB (10% inode=45%): [01:19:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [01:20:46] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3439 [01:20:56] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 84523 [01:22:56] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:26:06] SSH on yarrow is OK: SSH OK - OpenSSH_5.5p1 Debian-6+squeeze2 (protocol 2.0) [01:26:26] Sun Grid Engine execd on yarrow is OK: Host and Queues Ok [01:31:26] Erik Moeller * Re: [Toolserver-l] Future of the toolserver [01:32:36] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:34:06] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 105598 [01:37:46] MySQL slave on rosemary is OK: Uptime: 14379617 Threads: 36 Questions: 6293485474 Slow queries: 1913718 Opens: 246323 Flush tables: 6 Open tables: 3842 Queries per second avg: 437.667 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1746 [01:38:06] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 118281 [01:38:26] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1678.000000 [01:38:26] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [01:38:26] @replag [01:38:28] Krinkle: s1-rr-a: 27m 57s [-0.19 s/s]; s1-user: 27m 57s [-0.19 s/s]; s2-user: 1h 26m 50s [+1.00 s/s]; s2-user-c: 1h 24m 41s [+1.00 s/s]; s3-rr-a: 1d 8h 51m 45s [+0.91 s/s]; s3-user: 1d 8h 51m 46s [+0.91 s/s]; s5-user-c: 1h 24m 43s [+1.00 s/s]; s6-rr-a: 1d 5h 23m 20s [+0.78 s/s] [01:38:29] Krinkle: s6-user: 1d 5h 23m 20s [+0.77 s/s]; s7-rr-a: 23h 45m 8s [+0.87 s/s]; s7-user: 23h 45m 8s [+0.87 s/s] [01:40:21] Hm.. only 1 commons right now (on s5), we used to have more commons copies, right? (not that I need one) [01:42:25] Hersfold * Re: [Toolserver-l] Future of the toolserver [01:43:25] Why is nightshade asking for password on ssh? [01:43:26] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [01:43:36] http://bit.ly/toolserverLast [01:47:25] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:48:26] APT on mayapple is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [02:08:26] Chris Grant * Re: [Toolserver-l] Future of the toolserver [02:10:25] Ryan Lane * Re: [Toolserver-l] Future of the toolserver [02:14:07] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7213.000000 [02:14:26] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7236.000000 [02:16:46] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 67301 MB (11% inode=99%): [02:16:46] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:16:56] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:17:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:17:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:17:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:17:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:19:05] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [02:19:46] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 67067 MB (10% inode=99%): [02:19:56] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 546887 MB (10% inode=45%): [02:20:56] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 87738 [02:22:56] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:28:26] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3398.000000 [02:32:36] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:33:06] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106537 MB (10% inode=99%): [02:34:06] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 108640 [02:36:06] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 111079 MB (11% inode=99%): [02:38:06] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3435.000000 [02:38:06] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 121802 [02:38:26] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [02:43:25] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [02:47:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:49:26] APT on mayapple is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [02:51:26] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1611.000000 [02:54:06] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1784.000000 [03:16:47] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:16:56] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:17:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:17:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:17:25] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:17:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:19:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [03:19:47] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 60725 MB (9% inode=99%): [03:20:56] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 545102 MB (10% inode=45%): [03:21:56] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 91100 [03:22:56] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:32:36] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:33:06] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:34:06] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 111807 [03:38:06] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [03:38:26] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [03:38:46] Free Memory on turnera is CRITICAL: CRITICAL - 2.6% (214388 kB) free! [03:39:07] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 125361 [03:39:47] Free Memory on turnera is OK: OK - 16.0% (1336972 kB) free. [03:43:26] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [03:47:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:49:26] APT on mayapple is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [04:16:47] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:17:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:17:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:17:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:17:56] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:18:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:19:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [04:19:46] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 55033 MB (9% inode=99%): [04:20:56] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 545498 MB (10% inode=45%): [04:21:55] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 94692 [04:22:56] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:32:36] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:34:06] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 114934 [04:37:06] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106828 MB (10% inode=99%): [04:38:26] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [04:39:06] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 128793 [04:43:26] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [04:47:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:49:26] APT on mayapple is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [04:50:06] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107297 MB (11% inode=99%): [04:51:06] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106140 MB (10% inode=99%): [04:56:36] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 38170 MB (9% inode=99%): [04:56:46] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 80307 MB (8% inode=98%): [04:59:06] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 109277 MB (11% inode=99%): [05:03:25] Krinkle * Re: [Toolserver-l] Future of the toolserver [05:06:47] Free Memory on turnera is WARNING: WARNING - 6.3% (528552 kB) free! [05:08:46] Free Memory on turnera is CRITICAL: CRITICAL - 3.7% (307684 kB) free! [05:16:46] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:17:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:17:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:17:56] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:18:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:19:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [05:19:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:19:46] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 45844 MB (7% inode=99%): [05:20:55] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 545648 MB (10% inode=45%): [05:21:56] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 98245 [05:22:57] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:32:37] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:34:07] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 118132 [05:38:26] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [05:39:06] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 132273 [05:39:25] Federico Leva (Nemo) * Re: [Toolserver-l] Future of the toolserver [05:43:26] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [05:47:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:49:26] APT on mayapple is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [05:52:47] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 76412 MB (12% inode=99%): [06:01:26] Ryan Lane * Re: [Toolserver-l] Future of the toolserver [06:02:57] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 10280 MB (0% inode=94%): [06:08:46] Free Memory on turnera is CRITICAL: CRITICAL - 2.0% (164720 kB) free! [06:16:46] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:17:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:17:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:17:56] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:18:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:19:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [06:19:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:20:15] @replag [06:20:18] Tanvir: s2-user: 14s [-0.31 s/s]; s3-rr-a: 1d 13h 24m 42s [+0.97 s/s]; s3-user: 1d 13h 24m 43s [+0.97 s/s]; s6-rr-a: 1d 9h 33m 52s [+0.89 s/s]; s6-user: 1d 9h 33m 52s [+0.89 s/s]; s7-rr-a: 1d 4h 13m 45s [+0.95 s/s]; s7-user: 1d 4h 13m 45s [+0.95 s/s] [06:20:56] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 545515 MB (10% inode=45%): [06:21:56] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 101715 [06:22:56] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:29:25] Federico Leva (Nemo) * Re: [Toolserver-l] Future of the toolserver [06:32:36] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:34:06] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 121600 [06:38:26] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [06:40:06] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 135824 [06:43:26] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [06:47:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:49:26] APT on mayapple is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [06:52:46] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 80266 MB (13% inode=99%): [06:55:47] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 80372 MB (8% inode=98%): [07:08:46] Free Memory on turnera is CRITICAL: CRITICAL - 2.1% (173556 kB) free! [07:09:46] Free Memory on turnera is WARNING: WARNING - 5.7% (473700 kB) free! [07:15:46] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (418572 kB) free! [07:16:47] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:17:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:17:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:18:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:18:56] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:19:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [07:20:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:20:56] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 545401 MB (10% inode=45%): [07:21:06] Load avg. on adenia is WARNING: WARNING - load average: 16.77, 14.97, 9.36 [07:21:57] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 105136 [07:22:56] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:28:06] Load avg. on adenia is OK: OK - load average: 11.98, 14.65, 11.43 [07:31:46] Free Memory on turnera is WARNING: WARNING - 5.3% (446340 kB) free! [07:32:36] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:33:46] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (421504 kB) free! [07:34:06] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 125029 [07:38:27] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [07:40:07] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 139362 [07:43:26] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [07:47:27] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:49:26] APT on mayapple is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [07:53:46] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 79719 MB (13% inode=99%): [08:08:46] Free Memory on turnera is WARNING: WARNING - 6.2% (519816 kB) free! [08:12:46] Free Memory on turnera is OK: OK - 7.4% (618200 kB) free. [08:16:47] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:17:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:18:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:18:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:18:56] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:20:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [08:20:25] Andre Koopal * Re: [Toolserver-l] Future of the toolserver [08:20:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:20:56] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 545272 MB (10% inode=45%): [08:21:56] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 108632 [08:22:56] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:23:47] Free Memory on turnera is WARNING: WARNING - 6.1% (511500 kB) free! [08:27:25] Andre Koopal * Re: [Toolserver-l] Future of the toolserver [08:32:36] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:32:46] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (422428 kB) free! [08:34:06] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 128462 [08:37:25] Platonides * Re: [Toolserver-l] Future of the toolserver [08:38:25] Pavel Richter * Re: [Toolserver-l] Future of the toolserver [08:38:26] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [08:41:06] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 142987 [08:43:26] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [08:47:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:49:26] APT on mayapple is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [08:52:47] Free Memory on turnera is WARNING: WARNING - 5.8% (488872 kB) free! [08:54:47] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 79399 MB (13% inode=99%): [09:11:46] Free Memory on turnera is CRITICAL: CRITICAL - 4.3% (363560 kB) free! [09:17:15] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:17:46] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:18:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:18:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:18:56] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:20:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [09:20:56] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 545031 MB (10% inode=45%): [09:21:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:21:56] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 112145 [09:22:56] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:30:47] Free Memory on turnera is WARNING: WARNING - 5.2% (431952 kB) free! [09:31:46] Free Memory on turnera is CRITICAL: CRITICAL - 4.9% (411380 kB) free! [09:32:37] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:32:47] Free Memory on turnera is WARNING: WARNING - 6.0% (503840 kB) free! [09:34:06] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 131781 [09:38:26] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [09:41:06] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 146525 [09:43:26] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [09:47:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:49:26] APT on mayapple is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [09:54:47] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 79068 MB (12% inode=99%): [10:03:25] DaB. * Re: [Toolserver-l] Future of the toolserver [10:05:46] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (420696 kB) free! [10:06:46] Free Memory on turnera is WARNING: WARNING - 5.1% (425000 kB) free! [10:07:46] Free Memory on turnera is CRITICAL: CRITICAL - 4.8% (405700 kB) free! [10:17:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:17:46] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:18:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:18:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:18:56] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:20:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [10:20:25] DaB. * Re: [Toolserver-l] Future of the toolserver [10:20:56] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 544904 MB (10% inode=45%): [10:21:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:21:56] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 115635 [10:22:56] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:31:25] DaB. * Re: [Toolserver-l] Future of the toolserver [10:32:26] @replag [10:32:27] jem-: s3-rr-a: 1d 17h 32m 49s [+0.98 s/s]; s3-user: 1d 17h 32m 50s [+0.98 s/s]; s6-rr-a: 1d 13h 32m 23s [+0.95 s/s]; s6-user: 1d 13h 32m 23s [+0.95 s/s]; s7-rr-a: 1d 8h 17m 34s [+0.97 s/s]; s7-user: 1d 8h 17m 34s [+0.97 s/s] [10:32:46] Arggh [10:32:46] Free Memory on turnera is WARNING: WARNING - 5.4% (454192 kB) free! [10:33:35] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:34:06] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 135226 [10:34:46] Free Memory on turnera is CRITICAL: CRITICAL - 4.9% (409204 kB) free! [10:36:46] /sql on thyme is WARNING: DISK WARNING - free space: /sql 199163 MB (20% inode=99%): [10:38:26] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [10:38:46] Free Memory on turnera is WARNING: WARNING - 5.1% (424736 kB) free! [10:39:46] /sql on thyme is OK: DISK OK - free space: /sql 212732 MB (22% inode=99%): [10:41:07] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 150053 [10:42:25] Sumurai8 (DD) * Re: [Toolserver-l] Future of the toolserver [10:43:26] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [10:47:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:49:26] DaB. * Re: [Toolserver-l] Future of the toolserver [10:49:26] APT on mayapple is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [10:55:47] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 78488 MB (12% inode=99%): [11:03:46] Free Memory on turnera is CRITICAL: CRITICAL - 3.0% (252548 kB) free! [11:07:47] Free Memory on turnera is WARNING: WARNING - 5.5% (461384 kB) free! [11:17:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:18:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:18:46] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:18:56] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:19:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:20:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [11:20:56] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 544769 MB (10% inode=45%): [11:21:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:21:56] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 119078 [11:22:56] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:24:46] Free Memory on turnera is OK: OK - 7.1% (596324 kB) free. [11:33:36] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:35:06] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 136018 [11:38:26] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [11:41:06] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 153558 [11:43:26] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [11:47:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:49:26] APT on mayapple is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [11:54:46] Free Memory on turnera is WARNING: WARNING - 6.5% (547544 kB) free! [11:55:47] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 78100 MB (12% inode=99%): [12:13:26] Andre Koopal * Re: [Toolserver-l] Future of the toolserver [12:17:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:18:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:18:46] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:18:56] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:19:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:19:46] /sql on thyme is WARNING: DISK WARNING - free space: /sql 197079 MB (20% inode=99%): [12:20:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [12:21:47] Free Memory on turnera is CRITICAL: CRITICAL - 4.8% (400520 kB) free! [12:21:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:21:56] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 543157 MB (10% inode=45%): [12:21:56] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 122470 [12:22:56] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:31:47] Free Memory on turnera is WARNING: WARNING - 5.4% (455348 kB) free! [12:33:37] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:35:05] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 138560 [12:38:26] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [12:42:06] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 157152 [12:43:26] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [12:43:46] Free Memory on turnera is CRITICAL: CRITICAL - 4.6% (381712 kB) free! [12:45:46] /sql on thyme is OK: DISK OK - free space: /sql 211401 MB (22% inode=99%): [12:47:25] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:49:26] APT on mayapple is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [12:55:47] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 77559 MB (12% inode=99%): [12:56:47] Free Memory on turnera is WARNING: WARNING - 5.3% (443852 kB) free! [12:58:47] Free Memory on turnera is OK: OK - 7.1% (592548 kB) free. [13:05:58] @replag [13:06:04] Merlissimo: s2-user: 3m 39s [+0.01 s/s]; s3-rr-a: 1d 20h 2m 53s [+0.98 s/s]; s3-user: 1d 20h 2m 57s [+0.98 s/s]; s6-rr-a: 1d 14h 58m 7s [+0.56 s/s]; s6-user: 1d 14h 58m 8s [+0.56 s/s]; s7-rr-a: 1d 10h 41m 26s [+0.94 s/s]; s7-user: 1d 10h 41m 26s [+0.94 s/s] [13:17:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:18:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:18:46] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:18:56] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:19:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:20:07] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [13:21:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:21:56] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 543679 MB (10% inode=45%): [13:22:56] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 125695 [13:23:06] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106863 MB (10% inode=99%): [13:23:56] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:24:26] MZMcBride * Re: [Toolserver-l] Future of the toolserver [13:24:46] Free Memory on turnera is CRITICAL: CRITICAL - 4.5% (379416 kB) free! [13:31:46] Free Memory on turnera is WARNING: WARNING - 5.2% (436992 kB) free! [13:33:06] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107562 MB (11% inode=99%): [13:34:36] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:34:46] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (418672 kB) free! [13:35:06] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 141917 [13:35:46] Free Memory on turnera is WARNING: WARNING - 5.2% (431676 kB) free! [13:37:06] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106909 MB (10% inode=99%): [13:38:26] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [13:42:06] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 160697 [13:43:26] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [13:47:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:49:26] APT on mayapple is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [13:50:26] Merlissimo, how are jobs in error state cleared? [13:50:46] Free Memory on turnera is CRITICAL: CRITICAL - 4.6% (382708 kB) free! [13:51:02] qcmod -cj clears the error state or you could delete the job using qdel [13:55:06] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107724 MB (11% inode=99%): [13:56:07] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106754 MB (10% inode=99%): [13:56:46] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 77137 MB (12% inode=99%): [13:58:46] /sql on thyme is WARNING: DISK WARNING - free space: /sql 199002 MB (20% inode=99%): [13:59:47] Platonides: ups, qmod ... [14:02:46] /sql on thyme is OK: DISK OK - free space: /sql 212247 MB (22% inode=99%): [14:11:26] Merlissimo * [Toolserver-l] Reasons for not migrating to Tool Lab [14:13:06] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107258 MB (11% inode=99%): [14:15:23] Merlissimo, I mean this morning I had a job at Eqw [14:15:25] Platonides * Re: [Toolserver-l] Future of the toolserver [14:15:33] I didn't kill it, but now it isn't there [14:15:44] are they automatically reaped after some hours? [14:16:24] yes some queues and jobs failed because login check failed because of broken ldap [14:16:40] i cleared all error stats some hours ago [14:16:50] ah, ok [14:16:53] it was you [14:17:04] yes, it had failed with: can't get password entry for user "platonides". Either the user does not exist or NIS error! [14:17:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:17:53] i could not login yesterday, i think that was slready the ldap error. NIS is caching data for 24 hours [14:18:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:18:33] maybe that was the reason the servers were disconnecting me on login [14:18:46] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:18:56] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:19:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:20:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [14:21:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:21:56] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 543548 MB (10% inode=45%): [14:22:56] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 128887 [14:23:56] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:34:36] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:35:06] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 145080 [14:38:26] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [14:41:06] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 108617 MB (11% inode=99%): [14:42:06] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 164215 [14:43:26] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [14:47:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:49:25] APT on mayapple is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [14:50:46] Free Memory on turnera is CRITICAL: CRITICAL - 2.0% (171528 kB) free! [14:56:47] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 76628 MB (12% inode=99%): [14:59:46] /sql on thyme is WARNING: DISK WARNING - free space: /sql 187888 MB (19% inode=99%): [15:06:06] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106833 MB (10% inode=99%): [15:09:29] Daniel Schwen * Re: [Toolserver-l] Future of the toolserver [15:18:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:18:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:18:47] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:18:56] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:20:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [15:20:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:21:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:21:55] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 541966 MB (10% inode=45%): [15:22:56] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 132260 [15:23:46] /sql on thyme is OK: DISK OK - free space: /sql 217420 MB (22% inode=99%): [15:23:56] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:34:36] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:35:06] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 148002 [15:35:07] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107752 MB (11% inode=99%): [15:38:26] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [15:42:06] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 167731 [15:43:25] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [15:47:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:49:06] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106905 MB (10% inode=99%): [15:49:26] APT on mayapple is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [15:50:46] Free Memory on turnera is CRITICAL: CRITICAL - 2.1% (172272 kB) free! [15:53:07] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 108450 MB (11% inode=99%): [15:56:47] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 76289 MB (12% inode=99%): [16:18:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:18:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:18:47] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:18:55] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:20:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [16:21:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:21:56] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 542346 MB (10% inode=45%): [16:22:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:22:56] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 135496 [16:23:56] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:35:06] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 151316 [16:35:35] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:38:26] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [16:42:06] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 171279 [16:43:26] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [16:47:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:49:26] APT on mayapple is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [16:50:46] Free Memory on turnera is CRITICAL: CRITICAL - 2.1% (176984 kB) free! [16:56:36] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 38396 MB (9% inode=99%): [16:56:46] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 79956 MB (8% inode=98%): [16:56:46] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 75747 MB (12% inode=99%): [17:16:26] MZMcBride * Re: [Toolserver-l] Reasons for not migrating to Tool Lab [17:18:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:18:25] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:18:47] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:18:56] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:20:05] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [17:21:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:21:56] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 542185 MB (10% inode=45%): [17:22:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:22:56] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 138743 [17:24:56] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:35:06] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 154648 [17:35:36] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:38:26] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [17:42:06] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 174810 [17:43:26] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [17:47:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:49:26] APT on mayapple is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [17:50:46] Free Memory on turnera is CRITICAL: CRITICAL - 1.8% (153632 kB) free! [17:56:26] Tim Landscheidt * Re: [Toolserver-l] Reasons for not migrating to Tool Lab [17:56:47] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 75436 MB (12% inode=99%): [18:02:56] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 14086 MB (1% inode=95%): [18:18:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:18:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:18:46] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:18:56] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:20:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [18:21:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:21:56] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 541936 MB (10% inode=45%): [18:22:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:22:55] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 141919 [18:24:56] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:26:26] Ryan Lane * Re: [Toolserver-l] Reasons for not migrating to Tool Lab [18:33:25] Daniel Schwen * Re: [Toolserver-l] Reasons for not migrating to Tool Lab [18:33:53] does anybody know, how long are backups of ~ kept? [18:35:05] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 158234 [18:35:36] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:38:25] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [18:41:26] Erik Moeller * Re: [Toolserver-l] Reasons for not migrating to Tool Lab [18:42:06] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 178338 [18:43:26] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [18:47:25] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:48:25] Ryan Lane * Re: [Toolserver-l] Reasons for not migrating to Tool Lab [18:49:26] APT on mayapple is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [18:50:46] Free Memory on turnera is CRITICAL: CRITICAL - 2.2% (185540 kB) free! [18:56:25] Ryan Lane * Re: [Toolserver-l] Future of the toolserver [18:56:47] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 79872 MB (8% inode=98%): [18:56:47] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 75023 MB (12% inode=99%): [19:14:53] is login.toolserver.org deprecated? [19:15:01] what is the recommended server name? [19:17:30] dschwen: in order to log through SSH ? [19:17:46] yeah [19:18:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:18:15] I've had login.ts.org in my .ssh/config since forever [19:18:24] willow.toolserver.org or nightsade.toolserver.org [19:18:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:18:32] why? [19:18:46] login is an alias for willow [19:18:46] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:18:56] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:20:01] https://wiki.toolserver.org/view/Login_servers says that "Previously users were encouraged to use the alias login.toolserver.org. This has been deprecated." [19:20:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [19:21:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:21:57] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 541824 MB (10% inode=45%): [19:22:25] Ryan Lane * Re: [Toolserver-l] Future of the toolserver [19:22:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:22:56] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 145251 [19:24:16] PING on amaranth is WARNING: PING WARNING - Packet loss = 0%, RTA = 202.00 ms [19:24:26] PING on web.amaranth is WARNING: PING WARNING - Packet loss = 0%, RTA = 204.00 ms [19:24:55] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:35:06] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 161827 [19:35:36] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:38:26] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [19:39:26] PING on web.amaranth is OK: PING OK - Packet loss = 0%, RTA = 158.00 ms [19:40:15] PING on amaranth is OK: PING OK - Packet loss = 0%, RTA = 156.00 ms [19:42:06] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 181876 [19:43:26] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [19:47:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:49:26] APT on mayapple is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [19:50:46] Free Memory on turnera is CRITICAL: CRITICAL - 2.0% (170816 kB) free! [19:56:46] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 74588 MB (12% inode=99%): [20:18:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:18:25] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:18:47] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:18:56] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:20:05] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [20:21:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:21:56] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 541617 MB (10% inode=45%): [20:22:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:22:56] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 148651 [20:24:56] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:35:06] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 165102 [20:35:36] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:38:27] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [20:42:07] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 185422 [20:43:25] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [20:47:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:49:26] APT on mayapple is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [20:50:46] Free Memory on turnera is CRITICAL: CRITICAL - 1.8% (153828 kB) free! [20:56:47] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 74330 MB (12% inode=99%): [21:06:46] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2151 [21:07:36] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2200.000000 [21:16:26] MZMcBride * Re: [Toolserver-l] Reasons for not migrating to Tool Lab [21:18:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:18:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:18:56] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:19:47] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:20:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [21:21:56] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 541478 MB (10% inode=45%): [21:22:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:22:56] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 152087 [21:23:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:24:56] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:35:06] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 168342 [21:35:37] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1700.000000 [21:35:37] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:35:47] MySQL slave on rosemary is OK: Uptime: 14451498 Threads: 8 Questions: 6335911858 Slow queries: 1930447 Opens: 251102 Flush tables: 6 Open tables: 3839 Queries per second avg: 438.425 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1673 [21:38:26] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [21:39:26] Platonides * Re: [Toolserver-l] Future of the toolserver [21:42:01] sigh, i desperately need an admin :-/ [21:42:20] A Toolserver root, you mean? [21:42:23] Or a local project admin? [21:43:02] yup, ts root [21:43:07] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 189036 [21:43:26] Platonides * Re: [Toolserver-l] Future of the toolserver [21:43:26] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [21:47:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:49:26] APT on mayapple is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [21:50:46] Free Memory on turnera is CRITICAL: CRITICAL - 2.0% (170888 kB) free! [21:56:46] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 73685 MB (12% inode=99%): [21:57:25] Ryan Lane * Re: [Toolserver-l] Future of the toolserver [21:59:44] replication is not running, right? [22:02:13] @replag [22:02:34] Brooke: s1-rr-a: 18m 51s [+0.02 s/s]; s1-user: 18m 50s [+0.02 s/s]; s2-user: 11s [-0.00 s/s]; s3-rr-a: 2d 4h 49m 49s [+0.98 s/s]; s3-user: 2d 4h 49m 50s [+0.98 s/s]; s6-rr-a: 1d 23h 10m 38s [+0.92 s/s]; s6-user: 1d 23h 10m 54s [+0.92 s/s]; s7-rr-a: 1d 18h 51m 50s [+0.92 s/s] [22:02:35] Brooke: s7-user: 1d 18h 51m 50s [+0.92 s/s] [22:07:25] Ryan Lane * Re: [Toolserver-l] Reasons for not migrating to Tool Lab [22:18:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:18:56] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:19:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:20:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [22:20:46] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:21:56] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 541090 MB (10% inode=45%): [22:22:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:22:56] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 155424 [22:23:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:24:56] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:30:25] Hersfold * Re: [Toolserver-l] Reasons for not migrating to Tool Lab [22:35:06] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 171595 [22:36:36] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:38:25] Platonides * Re: [Toolserver-l] Reasons for not migrating to Tool Lab [22:38:26] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [22:43:06] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 192572 [22:43:26] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [22:47:27] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:49:26] APT on mayapple is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [22:50:47] Free Memory on turnera is CRITICAL: CRITICAL - 2.4% (198348 kB) free! [22:52:25] Ryan Lane * Re: [Toolserver-l] Reasons for not migrating to Tool Lab [22:56:47] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 73079 MB (11% inode=99%): [23:08:25] Ryan Lane * Re: [Toolserver-l] Reasons for not migrating to Tool Lab [23:18:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:18:55] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:19:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:20:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [23:20:46] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:21:56] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 539744 MB (10% inode=45%): [23:22:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:22:56] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 158672 [23:24:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:24:56] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:35:07] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 174596 [23:37:36] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:38:26] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [23:39:25] Tim Landscheidt * Re: [Toolserver-l] Reasons for not migrating to Tool Lab [23:43:06] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 196074 [23:43:25] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [23:47:27] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:49:26] APT on mayapple is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [23:50:47] Free Memory on turnera is CRITICAL: CRITICAL - 2.0% (165416 kB) free! [23:57:46] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 72713 MB (11% inode=99%):