[00:05:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1049564 [00:08:53] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23579 MB (5% inode=99%): [00:13:34] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 409696 MB (7% inode=39%): [00:15:04] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 96384 MB (9% inode=98%): [00:15:13] /sql on thyme is CRITICAL: DISK CRITICAL - free space: /sql 99459 MB (10% inode=99%): [00:16:33] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:19:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:28:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1050334.000000 [00:31:04] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:39:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:05:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1051485 [01:05:53] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 50783 MB (12% inode=99%): [01:13:34] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 411353 MB (7% inode=39%): [01:16:33] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:20:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:28:33] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1052097.000000 [01:31:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:39:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:40:22] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:40:34] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:41:23] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 95833 MB (9% inode=98%): [01:41:23] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 95833 MB (9% inode=98%): [01:54:23] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:54:33] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:59:43] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:59:53] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:00:03] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 95813 MB (9% inode=98%): [02:00:04] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 95813 MB (9% inode=98%): [02:00:04] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:00:34] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [02:00:43] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [02:00:44] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [02:06:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1053063 [02:13:23] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:13:34] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:13:34] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 410926 MB (7% inode=39%): [02:14:54] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 95633 MB (9% inode=98%): [02:15:04] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 95627 MB (9% inode=98%): [02:16:34] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:20:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:27:23] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:27:34] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:28:44] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1053713.000000 [02:31:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:39:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:46:53] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 95921 MB (9% inode=98%): [02:47:04] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 95921 MB (9% inode=98%): [03:06:44] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1054594 [03:14:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 408310 MB (7% inode=39%): [03:16:33] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:20:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:29:44] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1055104.000000 [03:31:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:39:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:06:50] MySQL slave on thyme is CRITICAL: (Return code of 139 is out of bounds) [04:12:19] Sun Grid Engine execd on nightshade is UNKNOWN: Error with qhost: error: commlib error: got select error (Connection refused) [04:12:40] Sun Grid Engine execd on yarrow is UNKNOWN: Error with qhost: error: commlib error: got select error (Connection refused) [04:14:50] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 408707 MB (7% inode=39%): [04:16:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:17:19] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:17:20] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:18:20] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 96089 MB (9% inode=98%): [04:18:50] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 96070 MB (9% inode=98%): [04:21:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:27:20] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:27:20] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:29:59] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1056803.000000 [04:31:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:40:00] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:46:50] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 96261 MB (9% inode=98%): [04:46:50] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 96261 MB (9% inode=98%): [05:02:11] anyone around? [05:02:12] error: unable to send message to qmaster using port 536 on host "turnera-bge0": got send error [05:02:20] when submitting jobs [05:02:39] Earwig: it is 7am... [05:02:53] 1am where I am [05:02:59] * jeremyb too [05:03:02] where are you? [05:03:16] I live in NY, currently in GA right now [05:03:22] but that seems kind of beside the point :P [05:03:23] * jeremyb is brooklyn [05:04:20] this is kind of important? was hoping someone was around and could tell me the problem was at least known [05:04:49] or I may have to just run the tasks in screen w/o sge's help [05:05:12] Earwig: the people that can deal with it are essentially all in 7:05am time atm [05:05:16] i think [05:05:33] right (for DaB., mainly) [05:05:39] but um [05:05:41] no, all of them [05:05:53] who is there besides DaB.? [05:06:00] nosy, daniel [05:06:07] all germany [05:06:10] AFAIK [05:06:14] right [05:06:43] although i guess she's around less now [05:06:50] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1057685 [05:07:25] nacht! [05:07:29] :( [05:08:39] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:09:18] i wonder if there's still a working "notify the admins of a problem" email address [05:09:36] maybe one should be established sooner than later [05:09:45] there's the mailing list... [05:09:59] sure but that's not what i mean [05:12:08] I'm confused [05:12:13] I thought we had some redundancy here [05:12:19] Sun Grid Engine execd on nightshade is UNKNOWN: Error with qhost: error: commlib error: got select error (Connection refused) [05:12:23] mmmmmokay [05:12:29] that's basically the issue [05:12:39] Sun Grid Engine execd on yarrow is UNKNOWN: Error with qhost: error: commlib error: got select error (Connection refused) [05:12:45] and that one [05:13:26] Earwig: redundancy how? [05:13:37] redundancy in the job... host things [05:13:45] maybe there is [05:13:52] i don't really understand how it all works [05:13:54] will try submit.toolserver.org again [05:14:00] * jeremyb runs away [05:14:08] oh, I know how it works [05:14:09] > poorly [05:14:11] * Earwig runs too [05:14:33] clematis - nope! [05:14:50] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 409099 MB (7% inode=39%): [05:16:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:21:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:30:20] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1058326.000000 [05:31:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:33:09] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [05:40:00] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:45:39] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 33853 MB (2% inode=98%): [06:06:49] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1059246 [06:10:31] is qsub/qconsub down for other people? [06:10:37] yes [06:10:42] definitely down [06:10:47] no one seems to know why [06:11:16] :( [06:12:19] Sun Grid Engine execd on nightshade is UNKNOWN: Error with qhost: error: commlib error: got select error (Connection refused) [06:12:40] Sun Grid Engine execd on yarrow is UNKNOWN: Error with qhost: error: commlib error: got select error (Connection refused) [06:14:50] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 411227 MB (7% inode=39%): [06:16:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:17:03] legoktm: I was just running jobs manually [06:17:24] as in just putting the raw command in your crontab? [06:17:31] no [06:17:31] or like manually with a screen? [06:17:33] that [06:17:34] :P [06:17:45] I have only one job [06:17:50] earwigbot is self-contained [06:18:03] Well I only have one job…except it runs every hour. [06:21:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:28:22] @replag [06:28:23] Earwig: s1-rr-a: 4m 18s [-]; s1-user: 1w 5d 6h 23m 22s [-]; s2-user: 11s [-]; s3-rr-a: 46s [-]; s3-user: 46s [-] [06:30:30] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1059860.000000 [06:31:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:31:40] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [06:31:50] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [06:36:59] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 40919 MB (10% inode=99%): [06:40:00] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:43:05] Krinkle * [Toolserver-l] JSV stderr: File "/sge/GE/bin/sol-amd64/qjobtest" [06:48:36] There we go again.. [06:56:10] Is anything being done regarding the SGE outage? - or does it depend on something else? Do those that should know know - aside from whether or not it is fixable on the short term [06:57:23] I talked with Earwig about it earlier, and according to the logs it said they couldn't contact any admins [06:57:35] um [06:57:55] I'm the first one who brought it up AFAIK [06:58:01] no one was around [06:58:30] ^what he said [06:58:38] I'm just running my bot under screen right now [06:58:40] sadly enough [06:59:12] by the way, either of you know when the default-architecture switch from sol to * will occur? [07:01:08] "Solaris will be the default for SGE until 30. [07:01:08] September 2012, after that the default will be switched to any available os (I [07:01:09] will send a reminder or 2 somewhen before)." [07:01:22] From DaB's email on Jul 14th [07:01:26] thanks [07:01:28] I've got time then [07:03:59] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23647 MB (5% inode=99%): [07:04:50] I'm off. [07:04:53] good luck, guys [07:06:50] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1060690 [07:11:42] shit [07:11:42] setting -m n (no email) in the qcronsub entry doesn't even help [07:12:04] the error here is from cron itself, not from cronie (which sets cron) or from qcronsub (which would be executed) [07:12:11] since qcronsub can't even be reached [07:12:20] Sun Grid Engine execd on nightshade is UNKNOWN: Error with qhost: error: commlib error: got select error (Connection refused) [07:12:24] neither can qsub or qstat [07:12:39] exactiy [07:12:39] Sun Grid Engine execd on yarrow is UNKNOWN: Error with qhost: error: commlib error: got select error (Connection refused) [07:13:01] so if you disable your cronie/crontab mail that might work? [07:14:50] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 410275 MB (7% inode=39%): [07:16:11] I don't want to disable e-mail for cron in general. Its useful [07:16:15] and quite seldom [07:16:41] Only when I make a syntax error or when Toolserver executable [07:16:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:16:51] Only when I make a syntax error or when Toolserver's executables are themselves broken. [07:16:55] Both cases, I want to know. [07:17:05] Krinkle * Re: [Toolserver-l] JSV stderr: File "/sge/GE/bin/sol-amd64/qjobtest" [07:17:19] Well your other option would be to just temporarily disable that one job? [07:19:28] right [07:19:56] well, the thing is. I've got about 13 long-running things. On 3 different account (me, and 2 mmt accounts) [07:20:39] what caused it to break though [07:20:50] did someone make a change to SGE and then go ofline [07:20:53] did it break tiself? [07:21:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:21:59] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 46288 MB (11% inode=99%): [07:30:29] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1060928.000000 [07:31:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:31:40] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [07:31:50] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [07:32:26] I'm setting up a Gmail filter for now to filter these out [07:40:00] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:42:14] Krinkle: are you using qcronsub on sbmit? [07:42:27] Yes [07:42:46] the way the documentation recommends, the way you helped me set it up [07:43:05] i have changes qcronsub for solaris so tha tit only execudes exit 0 about 15 minutes ago [07:43:19] also note that these errors aren't coming from qcronsub's execution. The thing it self crashes at run time [07:43:47] no matter what arguments I pass it [07:43:57] qmaster on tunera crashed and clematis schadow master does not take over [07:44:29] ok [07:44:35] can you do a master switch>/ [07:44:38] to fix this a root must restart the master [07:44:47] oh, that would work too [07:45:29] but i am still searching ehy clematis shadowd does not take over, but without master a cannot change any config or testing [07:46:13] .. the fact that the master is the master is stored on the master? [07:46:16] error happend 4:02 utc [07:46:19] that sounds like a design flaw [07:47:44] i am a age operator only. for me changing config files is only possible by sending command to the master and after right check the master process changes the config files on disk [07:48:35] okay, so you can only do it from the master but in general those config files are synced to the other servers and could be done (with the right permissions) from another server as well [07:48:43] e.g. one can assign a new master when the master is down [07:49:15] if you have have file permissions, yes [07:49:20] ok [07:51:31] but i am still investigating why shadowd does not take over the master. this should normally happens after 5 minutes of master timeout [08:06:50] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1061132 [08:12:20] Sun Grid Engine execd on nightshade is UNKNOWN: Error with qhost: error: commlib error: got select error (Connection refused) [08:12:40] Sun Grid Engine execd on yarrow is UNKNOWN: Error with qhost: error: commlib error: got select error (Connection refused) [08:14:50] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 409267 MB (7% inode=39%): [08:16:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:21:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:24:00] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 42059 MB (10% inode=99%): [08:30:30] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1061423.000000 [08:31:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:31:39] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [08:31:50] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [08:39:59] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:06:49] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1061993 [09:09:59] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23674 MB (5% inode=99%): [09:12:19] Sun Grid Engine execd on nightshade is UNKNOWN: Error with qhost: error: commlib error: got select error (Connection refused) [09:12:40] Sun Grid Engine execd on yarrow is UNKNOWN: Error with qhost: error: commlib error: got select error (Connection refused) [09:14:50] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 408168 MB (7% inode=39%): [09:16:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:21:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:27:20] Sun Grid Engine execd on nightshade is OK: Host and Queues Ok [09:27:39] Sun Grid Engine execd on yarrow is OK: Host and Queues Ok [09:30:30] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1062478.000000 [09:31:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:31:39] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [09:31:49] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [09:38:00] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 50193 MB (12% inode=99%): [09:40:00] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:06:51] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1063186 [10:14:51] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 407424 MB (7% inode=39%): [10:16:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:21:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:30:29] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1063697.000000 [10:31:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:31:40] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [10:31:50] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [10:39:37] Any TS admin around? [10:39:46] Daniel_WMDE, available for a quick help? [10:39:59] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:01:05] Marlen Caemmerer * Re: [Toolserver-l] JSV stderr: File "/sge/GE/bin/sol-amd64/qjobtest" [11:06:49] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1064243 [11:14:50] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 406755 MB (7% inode=39%): [11:16:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:21:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:30:29] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1064609.000000 [11:31:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:31:40] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [11:31:50] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [11:40:00] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:50:20] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:50:20] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:52:09] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 95677 MB (9% inode=98%): [11:52:10] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 95677 MB (9% inode=98%): [11:55:52] hello [12:04:05] DaB. * Re: [Toolserver-l] JSV stderr: File "/sge/GE/bin/sol-amd64/qjobtest" [12:06:50] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1065348 [12:14:49] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 406640 MB (7% inode=39%): [12:16:00] /sql on thyme is CRITICAL: DISK CRITICAL - free space: /sql 98471 MB (10% inode=99%): [12:16:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:21:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:30:30] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1066026.000000 [12:31:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:31:40] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [12:31:50] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [12:39:59] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:06:50] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1066370 [13:14:50] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 406529 MB (7% inode=39%): [13:16:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:21:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:30:30] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1066722.000000 [13:31:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:31:39] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [13:31:49] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [13:40:00] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:15:45] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 406381 MB (7% inode=39%): [14:17:44] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:22:15] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:30:45] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1068254.000000 [14:31:25] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:32:14] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [14:32:24] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [14:38:25] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 40433 MB (9% inode=99%): [14:40:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:03:25] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23738 MB (5% inode=99%): [15:07:15] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1068808 [15:15:45] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 406351 MB (7% inode=39%): [15:17:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:22:15] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:22:24] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 49637 MB (12% inode=99%): [15:26:51] anyone have a minute to write a quick js tool? [15:31:56] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 45914 MB (11% inode=99%): [15:31:56] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:32:40] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [15:32:50] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [15:35:18] APT on nightshade is OK: APT OK: 0 packages available for upgrade (0 critical updates). [15:35:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1069199 [15:35:19] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:35:19] APT on yarrow is OK: APT OK: 0 packages available for upgrade (0 critical updates). [15:35:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:35:33] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 45912 MB (11% inode=99%): [15:40:43] SRaid on mayapple is CRITICAL: NRPE: Command check_mdadm not defined [15:40:53] SRaid on nightshade is CRITICAL: NRPE: Command check_mdadm not defined [15:40:53] SRaid on yarrow is CRITICAL: NRPE: Command check_mdadm not defined [15:41:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:46:42] SRaid on mayapple is OK: OK md0 status=[UUUU]. [16:19:25] DaBPunkt: using arch=lx i get this error: /usr/bin/python: error while loading shared libraries: libc.so.6: failed to map segment from shared object: Cannot allocate memory [16:19:46] (with qcronsub on willow) [16:26:51] alchimista: why do you try to start qcronsub on willow? [16:28:05] DaBPunkt: it used to be on submit, but to prevent have several crontabs i was tryng to put it all together [16:28:36] than put it all together on submit ;) [16:29:44] * DaBPunkt wonders what needs all the memory on willow [16:30:21] so it can be all on submit? it used to be just for longrun [16:31:00] no. It is for starting (submit) all SGE-tasks. The SGE-system will decide than, where the script will run [16:31:27] but what about nonlongrun tasks? can start there too? [16:31:45] alchimista: that is a "nonlogrun task"? [16:32:04] wich come with the error? No.. it's a longrun [16:33:19] alchimista: as long as it uses SGE (qcronsub or cronsub): put it on submit [16:33:59] no matter if the task will run for 1s or until the end of the universe [16:34:22] (we can not guratee it will run that long, but we try ;)) [16:34:43] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 406094 MB (7% inode=39%): [16:34:53] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1070196.000000 [16:34:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:35:06] ok, so i'll join qcro*n and cron* on submit instead of willow. But i thought submit was about to get off for some time [16:35:12] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1070202 [16:35:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:35:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:35:36] DaBPunkt: ok, so i'll join qcro*n and cron* on submit instead of willow. But i thought submit was about to get off for some time [16:36:01] DaBPunkt: one other thing, how can i remove vim temp files? they're bugging me [16:37:07] alchimista: submit WAS also a sge-exec-host for some time (because we had too few servers); that was changed. [16:37:57] alchimista: did you check "man vim"? [16:38:21] (you can delete every file with rm of course, but maybe there is a config-value for what you need) [16:40:24] I have to leave for a moment [16:40:53] SRaid on nightshade is CRITICAL: NRPE: Command check_mdadm not defined [16:40:53] SRaid on yarrow is CRITICAL: NRPE: Command check_mdadm not defined [16:41:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:53:43] re [17:00:53] SRaid on yarrow is OK: OK md0 status=[UU]. [17:34:43] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 405860 MB (7% inode=39%): [17:34:53] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1071202.000000 [17:34:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:35:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1071210 [17:35:13] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:35:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:40:53] SRaid on nightshade is CRITICAL: NRPE: Command check_mdadm not defined [17:41:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:45:43] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 37627 MB (3% inode=98%): [18:34:43] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 405824 MB (7% inode=39%): [18:34:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:35:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1072720.000000 [18:35:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1072723 [18:35:13] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:35:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:40:53] SRaid on nightshade is CRITICAL: NRPE: Command check_mdadm not defined [18:41:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:48:32] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 41645 MB (10% inode=99%): [19:03:14] Hi. [19:03:28] Kai ist nicht zufällig grade hier, oder? [19:03:32] (Kai Krüger) [19:14:33] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23778 MB (5% inode=99%): [19:25:29] apmon [19:25:37] ist Kai [19:27:51] und nicht da# [19:28:14] Hab mir grad deinen Wikiminiatlas angeguckt - sieht ja schon nett aus [19:28:29] (nur, warum du das in 'nen iframe verpackst, versteh ich nicht, aber was solls) [19:34:43] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 405480 MB (7% inode=39%): [19:34:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:35:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1074152.000000 [19:35:12] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1074157 [19:35:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:35:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:38:33] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 35922 MB (8% inode=99%): [19:38:43] weil ich xmlhttp requests an den toolserver schicken muss [19:39:03] das kann man natuerlich mittlerweile auch cross-domain machen [19:39:14] aber vor 7 Jahren war das noch nicht etabliert [19:39:44] und crossdomain ist nicht so effizient [19:40:12] aber das läuft doch beides auf dem toolserver, wo ist da das x-dom-problem? [19:40:40] ich have nur kein x-dom problem WEIL ich einen iframe verwende [19:40:52] SRaid on nightshade is CRITICAL: NRPE: Command check_mdadm not defined [19:41:06] sobald du den äußeren frame anderswo einsetzt [19:41:11] sonst waere ich in der *.wiki[mp]edia.org domain [19:41:21] ach, du meinst die demo [19:41:31] ;) ja, ok - verstanden [19:41:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:41:42] WikiMiniAtlas ist aber fuer Wikipedia/Commons gedacht [19:41:49] klar [19:41:53] und auf diversen Proekten per default aktiviert [19:42:01] bei der demo hab ichs halt nicht verstanden, denn da wär's nicht notwendig [19:42:20] noe, aber einfacher [19:42:28] klar [19:42:43] ...spätestens, wenn der iframe anderswo schon existiert oder später existieren muss [19:42:45] dann muss ich nix aendern, sondern emuliere direkt die standard anwedungs situation [19:43:12] kenne ich Dich aus de.wp? [19:43:46] vermutlich nicht [19:43:52] komme eigentlich eher aus dem osm-umfeld ;) [19:44:05] ok. woher kennst Du den WMA? [19:44:14] hab 'nen account auf wp.de (jongleur1983), aber sonst... [19:44:32] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23186 MB (5% inode=99%): [19:44:35] naja - wir haben uns gerade quasi per Mail unterhalten ;) [19:44:40] ach [19:44:42] ich bin der Lalm-Typ :P [19:44:50] oh! [19:44:52] cool [19:45:13] stimmt... in dem mailwechsel bin ich nicht als jongleur aufgetreten; sorry ;) [19:46:22] kann man von LALM schon was sehen? [19:46:34] gaaanz wenig, und nicht das aktuellste [19:46:56] wo denn? [19:47:26] toolserver.org/lalm [19:47:56] und dann unten im text auf "an einem zufälligen ort starten" (der ist nichtmal zufällig, aber was solls) [19:48:31] die Beschreibung, die du dann kriegst, hab ich hier lokal schon in schönerer Textform, aber daran arbeite ich noch; und dafür muss die Datenbank-API für die mapnik-DB noch angepasst werden; wie gesagt - hier hab ich 'ne snapshot-DB [19:48:50] ist das dann wie Zork? [19:49:11] ja, so ähnlich [19:49:19] aber ohne story, dafür mit echten Daten ;) [19:49:25] Und ohne Grue [19:49:28] :-D [19:49:42] hab zork glaub ich nie gespielt ;) kenne nur den Namen :P [19:49:58] aber text-adventures hab ich durchaus schon gespielt, ja. [19:50:05] interessantes projekt [19:50:19] ein Unterschied ist, dass wir nicht in erster Linie 'ne Kommandozeile einbauen wie bei textadventures, sondern schon Hypertext als Basis nehmen [19:50:45] na, das gibts bei neueren textadventures auch ;-) [19:51:05] wenn der Text also sagt, "Sie befinden sich auf der Lessingstraße. 100 Meter voraus geht links die Göthestraße und rechts die Schillerstraße weiter", dann kann man Göthestraße und Schillerstraße anklicken, um da weiterzunavigieren [19:51:10] d.h. du analysierst die rohdaten und generierst daraus textbeschreibungen [19:51:18] genau [19:51:30] das ist aber anspruchsvoll [19:51:43] dazu soll dann irgendwann noch 'n Routenplaner kommen, der dir zusätzlich 'ne direkte Zielführung gibt und sowas [19:51:46] ich weiß ;) [19:51:49] wenn du mitmachen willst :P [19:51:52] wenn dann nich "zu ihrerr Linken sehen sie das Kaffee Dingens" uns wo weiter kommt [19:51:59] nich->noch [19:52:38] ja, das ist geplant [19:53:06] (nur nicht mit der Formulierung "sehen sie" - das sind Blinde zwar gewohnt, aber wenn wir uns schon vor allem an diese Zielgruppe richten...) [19:53:48] nach Möglichkeit gleich auch noch alle anderen Daten, die wir in OSM haben: "Zur linken befindet sich das Kaffee Dingens. Es hat noch eine halbe Stunde geöffnet." [19:53:51] oder sowas ;) [19:54:21] au weia [19:54:31] ist das Dein Vollzeitprojekt? [19:54:44] fast [19:54:58] neben Masterarbeit und 9,5h Werksstudent ;) [19:56:55] ach so - und weshalb ich sagte, dein Atlas interessiert uns insbesondere: weiter auf der Agenda stehen dann visuelle Karten für bestimmte Sehbeeinträchtigungen; also mit verstärktem Kontrast, oder für besonders hohe Blendempfindlichkeit, rot-grün-Blindheit und so weiter [19:57:28] das lässt sich halt nur mit clientseitigem Rendering sinnvoll umsetzen, weil die jeweilige Zielgruppe zu speziell ist und das vorrendern einer bitmap-Karte nicht praktikabel ist [20:03:33] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 54292 MB (13% inode=99%): [20:09:24] jo [20:09:44] ich werde auch noch auswechselbare render styles anbieten [20:10:17] habe da aber eher an Hervorhebung bestimmter Objekte gedacht, wie z.B. Bruecken, Tunnel, U-Bahnen etc [20:10:58] mein Renderer braucht auch noch etwas Arbeit. layers werden noch ignoriert und tunnel nicht extra gestyled [20:11:16] alles nix Weltbewegendes, aber braucht halt Arbeit [20:12:31] Prinzipiell laesst sich so ein client-side renderer aber recht schnell aus dem Boden stampfen [20:12:41] bei mir hats etwa vier Tage gedauert [20:12:55] wie dynamisch ist der bei dir? [20:13:03] kennst du kothic.js? [20:13:04] wie meinst Du das? [20:13:07] jaja [20:13:17] hat mir jemand am dritten Tag gezeigt :-) [20:13:33] naja: hast du 'ne Möglichkeit, Renderingregeln einfach anzupassen, oder steht das im Code? [20:13:34] fand es aber etwas sluggish [20:13:45] beides :-) [20:13:48] sind die json-tiles statisch, oder passen die sich an den rendering-stil an? [20:13:57] ist im Code, aber als datenstruktur [20:14:06] nicht als logik [20:14:08] da lässt sich glaub ich extrem viel optimieren und tunen ;) [20:14:12] das ist beides getrennt [20:14:22] ich koennte styles so auch als JSON nachlasbar machen [20:14:33] sagt dir MapCSS was? :D [20:14:43] kothic hat auch einen etwas anderen Anspruch als ich [20:14:47] nee [20:15:27] mapcss.org - "Standard", an Cascading Stylesheets angelehnt mit dem Ziel, sowas wie CSS für Karten zu bekommen [20:15:52] jaja, macht man [20:16:12] waerend da noch standardisiert wird programmiere ich lieber schonmal ;-) [20:16:17] idealvorstellung meinerseits im Zuge des clientseitigen Renderings wäre sozusagen, dass man lokal im browser den Stil der Karte überschreiben kann wie das mit CSS schon funktioniert [20:16:20] ;) [20:16:28] nee, ist kein "wir machen einen standard und dann"-Standard [20:16:39] nee, ist ja im Prinzip alles toll und interessant [20:16:49] sondern eher ein lebender standard, immer mal wieder kommt jemand und implementiert was neues, schlägt das dann vor und so weiter [20:16:55] und es ist auch super, dass es "alles schon gibt" [20:17:06] aber jeder macht seinen eigenen kleinen Teil [20:17:06] aber es gibt eben eine Teilmenge, die zum Beispiel Potlatch und JOSM gemeinsam unterstützen [20:17:17] (client side tiles, 3d buildings, etc.) [20:17:26] jo [20:17:33] und ich brauche es zusammengefuegt [20:17:42] da muss ichs halt selber machen [20:17:55] ausserdem sind wiegesagt die Ziele unterschiedlich [20:18:05] OSM ist recht viel mit sich selbst beschaeftigt [20:18:08] ;) ich wollt dich ja nur drauf hinweisen, falls du es noch nicht kennst - denn nichts ist ärgerlicher, als am ende sowas zu sehen und dann zu merken, dass man dämliche Arbeit wirklich doppelt gemacht hat [20:18:13] will heisse, auf die Karte fokussiert [20:18:24] teilweise [20:18:32] WikiMiniAtlas soll aber auf Wikipedia ausgerichtet sein [20:18:43] na ja [20:19:05] ;) du meinst dieses Projekt, das stark mit sich selbst beschäftigt ist, auf dieses Lexikon und die paar Varianten davon fokussiert? [20:19:31] (ok, jetzt muss ich vorsichtig werden - ich hoffe, da hat jeder den Smiley gesehen) [20:20:02] haha [20:20:08] genau das ;-) [20:20:38] na ja, also mapcss scheint mir nicht ideal fuer client side rendering zu sein [20:20:48] ich denke, das kommt darauf an [20:20:58] einerseits kann man selektoren ueber jquery nutzen, wenn man die Daten als XML docs im speicher hat [20:21:06] aber man muss die regeln erstmal parsen [20:21:12] hmm [20:21:47] da hack ich mir lieber schnell was zusammen, macht mehr Spass! :-) [20:21:51] ;) [20:22:08] schliesslich hab ich Programmieren aufm C64 gelernt :-D [20:22:16] okay, da war ich noch nicht dabei [20:23:57] wobei ich nach html immerhin noch turbo pascal auf der konsole - mit 'nem bisschen inline assembler gelernt habe [20:24:19] (und ja, ich weiß, dass html keine programmiersprache ist) [20:26:07] aber JavaScript! [20:26:31] ;) damals, als html4 noch neu war, war javascript auch keine programmiersprache [20:27:03] och, klar doch [20:27:14] oder die win95-Eingabeaufforderung war eine Kommandozeile ;) [20:27:16] man konnt schon lange lauftexte in die Statuszeile setzen damit! [20:27:25] eben... ;) [20:34:42] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 406802 MB (7% inode=39%): [20:34:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:35:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1075469.000000 [20:35:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1075473 [20:35:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:35:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:40:53] SRaid on nightshade is CRITICAL: NRPE: Command check_mdadm not defined [20:41:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:49:43] /aux0 on hemlock is OK: DISK OK - free space: /aux0 603074 MB (11% inode=49%): [21:19:39] nacht ts [21:34:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:35:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1076739.000000 [21:35:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1076743 [21:35:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:35:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:40:53] SRaid on nightshade is CRITICAL: NRPE: Command check_mdadm not defined [21:41:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:30:33] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 40838 MB (10% inode=99%): [22:34:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:35:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1077563.000000 [22:35:12] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1077566 [22:35:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:35:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:40:52] SRaid on nightshade is CRITICAL: NRPE: Command check_mdadm not defined [22:41:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:59:33] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23287 MB (5% inode=99%): [23:21:34] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 49726 MB (12% inode=99%): [23:34:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:35:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1078411.000000 [23:35:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1078411 [23:35:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:35:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:40:53] SRaid on nightshade is CRITICAL: NRPE: Command check_mdadm not defined [23:41:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:52:33] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 95145 MB (9% inode=98%): [23:52:33] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 95145 MB (9% inode=98%):