[00:04:04] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 786385.000000 [00:24:14] / on hemlock is WARNING: DISK WARNING - free space: / 4183 MB (20% inode=89%): [00:46:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 29600.000000 [00:47:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 788196 [00:47:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:52:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 465785 MB (8% inode=42%): [00:53:04] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 29468 [00:53:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:54:23] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:55:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:59:44] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [00:59:54] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [01:04:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 788773.000000 [01:25:13] / on hemlock is WARNING: DISK WARNING - free space: / 4181 MB (20% inode=89%): [01:47:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 27698.000000 [01:47:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 790406 [01:47:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:53:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 465718 MB (8% inode=42%): [01:53:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:54:03] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 27610 [01:54:24] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:56:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:57:06] Where can I write a 500+ MB file to local disk that is large enough? [01:59:44] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [01:59:54] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [02:04:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 791054.000000 [02:08:24] Dispenser: user-store doesn't count as local? [02:08:26] Or your home dir? [02:09:27] They're both on the NAS, something like /tmp/ but without the risk that it'll fill before I'm done [02:10:04] Put it in memory. (o; [02:10:46] It needs to be on disk for MySQL's LOAD DATA INFILE [02:12:33] So far only nightshade has a small /tmp < 1 GB [02:23:22] oh I was looking at /var/tmp not /tmp [02:26:13] / on hemlock is WARNING: DISK WARNING - free space: / 4178 MB (20% inode=89%): [02:47:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25473.000000 [02:47:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 792670 [02:47:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:53:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:54:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 463670 MB (8% inode=42%): [02:54:23] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:55:04] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 25116 [02:56:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:59:43] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [02:59:53] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [03:04:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 793287.000000 [03:10:13] / on hemlock is OK: DISK OK - free space: / 4207 MB (21% inode=89%): [03:19:54] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 39873 MB (9% inode=99%): [03:42:53] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23942 MB (5% inode=99%): [03:47:24] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22538.000000 [03:47:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 795078 [03:47:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:51:53] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 54570 MB (13% inode=99%): [03:53:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:54:24] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 463647 MB (8% inode=42%): [03:55:03] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 21969 [03:55:23] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:57:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:59:43] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [03:59:53] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [04:04:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 795763.000000 [04:47:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 797743 [04:47:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:48:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18161.000000 [04:53:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:54:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 463509 MB (8% inode=42%): [04:55:03] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 17819 [04:55:24] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:57:14] /sql on thyme is WARNING: DISK WARNING - free space: /sql 110725 MB (11% inode=99%): [04:57:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:59:32] @replag [04:59:32] Dispenser: s1-rr-a: 4h 55m 33s [-0.73 s/s]; s1-user: 1w 2d 5h 46m 15s [+0.67 s/s]; s3-rr-a: 1m 22s [+0.00 s/s]; s3-user: 1m 22s [+0.00 s/s]; s5-rr-a: 23s [+0.00 s/s]; s5-user: 23s [+0.00 s/s] [04:59:43] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [04:59:53] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [05:04:14] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 798549.000000 [05:43:24] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 39450 MB (3% inode=98%): [05:43:33] Load avg. on willow is CRITICAL: CRITICAL - load average: 13.66, 105.42, 79.47 [05:47:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 800482 [05:47:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:48:24] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15586.000000 [05:50:13] dschwen: it's /* SLOW_OK */ not /* SLOW OK */ [05:53:23] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:54:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:55:03] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 15199 [05:55:24] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 463408 MB (8% inode=42%): [05:55:24] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:56:53] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 97160 MB (9% inode=98%): [05:57:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:59:43] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [05:59:53] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [06:03:21] @replag [06:03:21] Earwig: s1-rr-a: 4h 4m 31s [-0.80 s/s]; s1-user: 1w 2d 6h 32m 5s [+0.72 s/s]; s2-user: 45s [+0.00 s/s]; s3-rr-a: 29s [-0.01 s/s]; s3-user: 29s [-0.01 s/s]; s7-rr-a: 28s [-0.08 s/s]; s7-user: 28s [-0.08 s/s] [06:04:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 801150.000000 [06:11:34] Load avg. on willow is WARNING: WARNING - load average: 7.37, 8.64, 19.56 [06:20:34] Load avg. on willow is CRITICAL: CRITICAL - load average: 48.48, 23.69, 20.28 [06:21:33] Load avg. on willow is WARNING: WARNING - load average: 24.49, 21.19, 19.60 [06:34:34] Load avg. on willow is OK: OK - load average: 7.84, 11.20, 14.55 [06:47:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 802896 [06:47:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:48:24] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11706.000000 [06:54:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:55:03] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11152 [06:55:24] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 463348 MB (8% inode=42%): [06:55:24] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:57:53] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 97658 MB (10% inode=98%): [06:58:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:59:43] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [06:59:53] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [07:05:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 803515.000000 [07:47:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 805221 [07:47:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:48:23] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7224.000000 [07:54:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:55:03] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6801 [07:55:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 463220 MB (8% inode=42%): [07:56:23] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:58:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:59:43] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [08:00:53] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [08:05:14] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 805835.000000 [08:47:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 807471 [08:47:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:48:23] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3594.000000 [08:49:04] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3584 [08:50:33] Load avg. on willow is WARNING: WARNING - load average: 12.45, 18.46, 15.72 [08:52:34] Load avg. on willow is OK: OK - load average: 7.07, 14.48, 14.55 [08:54:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:55:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 463121 MB (8% inode=42%): [08:56:24] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:58:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:59:44] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [09:00:53] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [09:05:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 808133.000000 [09:20:23] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1795.000000 [09:21:03] MySQL slave on rosemary is OK: Uptime: 9137019 Threads: 11 Questions: 3657090090 Slow queries: 667002 Opens: 15610 Flush tables: 6 Open tables: 711 Queries per second avg: 400.249 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1724 [09:47:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 809865 [09:47:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:54:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:55:24] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 463002 MB (8% inode=42%): [09:56:23] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:58:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:59:43] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [10:00:54] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [10:05:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 810567.000000 [10:47:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 812186 [10:47:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:54:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:55:24] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 462885 MB (8% inode=42%): [10:57:23] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:58:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:59:43] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [11:00:53] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [11:05:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 812840.000000 [11:16:13] /sql on thyme is CRITICAL: DISK CRITICAL - free space: /sql 104544 MB (10% inode=99%): [11:17:13] /sql on thyme is WARNING: DISK WARNING - free space: /sql 113633 MB (11% inode=99%): [11:47:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 814672 [11:47:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:54:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:56:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 462744 MB (8% inode=42%): [11:57:23] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:59:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:59:43] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [12:00:53] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [12:05:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 815345.000000 [12:48:33] Load avg. on willow is CRITICAL: CRITICAL - load average: 11.06, 34.80, 29.25 [12:48:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 816972 [12:48:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:54:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:56:24] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 462434 MB (8% inode=42%): [12:57:23] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:58:33] Load avg. on willow is WARNING: WARNING - load average: 7.61, 12.33, 19.46 [12:59:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:59:43] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [13:00:53] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [13:03:43] APT on yarrow is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:03:53] APT on nightshade is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:05:14] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 817481.000000 [13:08:14] / on hemlock is WARNING: DISK WARNING - free space: / 4193 MB (20% inode=89%): [13:12:34] Load avg. on willow is OK: OK - load average: 11.83, 11.77, 14.97 [13:42:04] DaB. * [Toolserver-l] Please move more longrunning-tasks to linux [13:48:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 818930 [13:48:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:52:42] DaBPunkt: Can't locate POE.pm in @INC (@INC contains: /etc/perl /usr/local/lib/perl/5.10.1 /usr/local/share/perl/5.10.1 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.10 /usr/share/perl/5.10 /usr/local/lib/site_perl .) [13:52:50] (nightshade) [13:53:01] i tried to run the bot which runs on willow now [13:53:48] Danny_B|backup: I will look at package that is [13:53:51] @replag [13:53:52] russblau: s1-rr-a: 33s [-0.52 s/s]; s1-user: 1w 2d 11h 31m 17s [+0.59 s/s]; s2-user: 6m 34s [+0.01 s/s]; s2-user-c: 2m 52s [+0.00 s/s]; s3-rr-a: 60s [+0.00 s/s]; s3-user: 60s [+0.00 s/s]; s5-user-c: 2m 52s [+0.00 s/s] [13:54:21] libpoe-perl, AFAIS [13:55:10] DaBPunkt, hello; now that rosemary is back to normal while thyme is still in bad shape, would it make sense to move the user databases over to rosemary and point sql-s1-user to rosemary for a few weeks until thyme recovers? [13:55:12] i don't understand it at all, i'm just reporting the error i've got from the bot i run ;-) [13:55:33] but thanks for care anyway [13:55:42] let me know when i should try again [13:55:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:56:24] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 462323 MB (8% inode=42%): [13:56:25] Danny_B|backup: ok, try now [13:57:11] DaBPunkt: danke schon [13:57:21] russblau: that would cause some hours of total downtime for tools using sql-s1-user [13:58:12] Danny_B|backup: no problem. please notice that there is "schon" and "schön" (schoen) ;) [13:58:14] DaBPunkt: I see, but those tools are fairly useless anyway with lag time > 9 days [13:58:23] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:58:40] russblau: that's a point yes. I will think about that during the weekend [13:58:54] DaBPunkt: thanks [14:00:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:01:30] DaBPunkt: so did i understand properly, you want us to move all screen stuff to nightshade? [14:03:15] Danny_B|backup: no. You should move all your screen-stuff to SGE. But you should test first if the stuff is running under linux too [14:04:08] * Danny_B|backup never understood moving to sge :-/ [14:04:37] Danny_B|backup: how do you run your stuff for normal? [14:04:53] just run [14:05:14] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 819404.000000 [14:05:21] Danny_B|backup: give me an example [14:05:29] i just run a command [14:05:41] script in screen [14:05:52] if it's supposed to run long [14:06:02] or just command in ssh window [14:06:06] (putty) [14:07:11] Danny_B|backup: is it a script (like "./run.sh") or do you run a command (like "perl someperlprogramm.py")? [14:08:13] / on hemlock is WARNING: DISK WARNING - free space: / 4190 MB (20% inode=89%): [14:48:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 820548 [14:48:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:55:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:56:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 462239 MB (8% inode=42%): [14:58:24] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:00:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:05:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 820984.000000 [15:09:13] / on hemlock is WARNING: DISK WARNING - free space: / 4187 MB (20% inode=89%): [15:47:34] Load avg. on willow is CRITICAL: CRITICAL - load average: 15.61, 34.58, 26.52 [15:49:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 822544 [15:49:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:54:33] Load avg. on willow is WARNING: WARNING - load average: 6.52, 14.20, 19.61 [15:55:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:56:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 462066 MB (8% inode=42%): [15:58:23] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:00:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:04:34] Load avg. on willow is OK: OK - load average: 9.83, 10.56, 14.77 [16:05:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 823063.000000 [16:09:13] / on hemlock is WARNING: DISK WARNING - free space: / 4185 MB (20% inode=89%): [16:48:34] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:49:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 824339 [16:49:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:53:34] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [16:56:24] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 461914 MB (8% inode=42%): [16:56:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:58:23] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:00:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:05:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 824798.000000 [17:09:13] / on hemlock is WARNING: DISK WARNING - free space: / 4182 MB (20% inode=89%): [17:43:24] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 43479 MB (3% inode=98%): [17:49:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 826309 [17:49:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:49:53] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 38011 MB (9% inode=99%): [17:56:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 461792 MB (8% inode=42%): [17:56:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:56:54] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 90493 MB (9% inode=98%): [17:58:23] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:00:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:05:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 826779.000000 [18:09:14] / on hemlock is WARNING: DISK WARNING - free space: / 4180 MB (20% inode=89%): [18:15:53] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24421 MB (5% inode=99%): [18:49:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 828476 [18:49:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:56:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 461683 MB (8% inode=42%): [18:56:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:57:53] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 99683 MB (10% inode=98%): [18:58:23] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:00:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:05:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 829066.000000 [19:08:36] @replag [19:08:37] Earwig: s1-user: 1w 2d 14h 20m 13s [+0.54 s/s] [19:09:14] / on hemlock is WARNING: DISK WARNING - free space: / 4177 MB (20% inode=89%): [19:10:54] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 63890 MB (15% inode=99%): [19:22:13] /sql on thyme is CRITICAL: DISK CRITICAL - free space: /sql 106003 MB (10% inode=99%): [19:31:13] /sql on thyme is WARNING: DISK WARNING - free space: /sql 109311 MB (11% inode=99%): [19:49:44] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 830887 [19:49:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:56:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 461627 MB (8% inode=42%): [19:56:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:58:23] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:00:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:05:14] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 831444.000000 [20:10:13] / on hemlock is WARNING: DISK WARNING - free space: / 4175 MB (20% inode=89%): [20:49:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 833121 [20:49:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:56:24] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 461515 MB (8% inode=42%): [20:57:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:58:23] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:00:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:05:14] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 833744.000000 [21:10:14] / on hemlock is WARNING: DISK WARNING - free space: / 4172 MB (20% inode=89%): [21:34:23] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:34:23] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:34:54] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 99490 MB (10% inode=98%): [21:34:54] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 99490 MB (10% inode=98%): [21:42:53] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 41885 MB (10% inode=99%): [21:44:23] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:44:23] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:44:23] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:46:29] Lost connection to MySQL server at 'reading initial communication packet', system error: 0 [21:46:54] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 41722 MB (10% inode=99%): [21:49:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 835381 [21:49:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:56:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 461437 MB (8% inode=42%): [21:58:23] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:58:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:00:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:03:54] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 99592 MB (10% inode=98%): [22:03:54] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 99591 MB (10% inode=98%): [22:05:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 835922.000000 [22:10:13] / on hemlock is WARNING: DISK WARNING - free space: / 4170 MB (20% inode=89%): [22:50:44] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 837661 [22:50:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:56:24] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 461287 MB (8% inode=42%): [22:58:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:59:23] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:00:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:00:53] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24472 MB (5% inode=99%): [23:05:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 838196.000000 [23:10:13] / on hemlock is WARNING: DISK WARNING - free space: / 4167 MB (20% inode=89%): [23:50:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 839661 [23:50:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:56:24] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 459873 MB (8% inode=42%): [23:58:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:59:23] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output