[00:00:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:16:04] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:18:23] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [00:30:07] Sumana Harihareswara * [Toolserver-l] HTML5 coming to Wikimedia sites 17 September; please check scripts [00:37:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:41:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:46:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:58:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [00:59:54] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 27458 MB (4% inode=99%): [01:00:04] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:16:03] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:18:22] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [01:37:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:41:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:46:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:54:55] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 87122 MB (8% inode=98%): [01:54:55] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 87122 MB (8% inode=98%): [01:58:55] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [01:59:54] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 25668 MB (4% inode=99%): [02:00:04] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:17:03] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:18:23] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [02:37:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:41:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:46:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:59:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [02:59:54] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 26972 MB (4% inode=99%): [03:00:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:17:03] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:18:22] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [03:38:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:41:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:47:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:00:04] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:00:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [04:00:54] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 26782 MB (4% inode=99%): [04:18:03] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:18:23] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [04:38:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:41:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:47:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:49:34] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24018 MB (5% inode=99%): [05:00:04] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:00:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [05:00:54] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 26651 MB (4% inode=99%): [05:18:03] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:18:34] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [05:30:34] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 29542 MB (7% inode=99%): [05:38:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:41:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:48:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:48:54] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 10370 MB (0% inode=94%): [06:00:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [06:00:54] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 26500 MB (4% inode=99%): [06:01:04] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:18:34] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [06:19:04] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:38:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:41:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:48:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:00:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [07:00:54] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 26280 MB (4% inode=99%): [07:01:04] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:14:34] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3466 [07:19:33] MySQL slave on z-dat-s3-a is OK: Uptime: 1713052 Threads: 20 Questions: 1394295638 Slow queries: 111044 Opens: 19820344 Flush tables: 1 Open tables: 16384 Queries per second avg: 813.924 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1561 [07:20:03] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:38:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:41:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:49:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:00:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [08:00:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 26081 MB (4% inode=99%): [08:01:13] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:07:07] emijrp * [Toolserver-l] ssh and sftp not working [08:13:24] nosy: i can't connect via ssh (putty) neither to willow nor to nightshade. any known issues? [08:13:49] Danny_B: just saw the email on the list - i am checking [08:13:55] Danny_B|webchat: both v6 and v4 worksforme [08:14:09] it always ends up with authenticating the key [08:14:11] nightshade works for me too [08:14:14] and then it times out [08:15:43] the error on willow is " Sep 6 08:14:39 willow sshd[3670]: [ID 800047 auth.crit] fatal: tilde_expand_filename: No such uid 507" but it is hard to find out what exactly this means [08:17:28] oh...i can login from outside into willow... [08:17:41] Danny_B|webchat: what about your? [08:17:48] nope [08:18:00] trying both nightshade and willow [08:18:28] what about ipv4? [08:18:30] they dropped about 30 mins ago and since then i can't log in [08:18:36] looks like you are coming via v6 [08:18:44] do i? [08:18:53] that would be weird [08:19:10] so far i know i have only ipv4 assigned [08:20:04] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:21:04] so i forced v4 and it's also freezing on authentication [08:21:35] it always lets me fill the username then writes it's authenticating the key and then it times out [08:22:13] i think i want to simply reboot willow [08:24:55] Sun Grid Engine execd on willow is CRITICAL: Connection refused by host [08:25:35] uh ?! [08:25:48] all logged in via willow? [08:25:53] now willow is "ssh_exchange_identification: Connection closed by remote host" [08:26:04] I didn't think nemo_bis was a bot too :D [08:28:21] o_o [08:28:34] lost the backlog now [08:29:08] seems to work now [08:29:43] Danny_B|webchat: what about you? can you login? [08:30:06] still freezing on authentication :-/ [08:30:14] wow... [08:30:20] what is any other server i can try? [08:33:22] Danny_B|webchat: wolfsbane maybe? [08:33:35] @reboot on willow obviously still does not work :-/ [08:33:45] jimmyxu: will try, thx [08:34:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:35:19] wolfsbane doesn't even send a "login as" prompt :-/ [08:35:33] danny_b: i was wrong - you dont use v6. [08:35:45] ah, now it sends [08:35:52] but i cannot see any login attempts from you now [08:36:03] but again - freeze on authenticating [08:36:10] @wolfsbane [08:36:31] now trying nightshade [08:37:54] Free Memory on damiana is WARNING: WARNING - 7.0% (586540 kB) free! [08:37:58] and willow [08:38:08] and everything times out on authentication [08:38:23] my guess - any issues with /home? [08:39:54] Free Memory on damiana is OK: OK - 7.1% (590792 kB) free. [08:39:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:46:44] \o/ finally on nightshade at least [08:46:54] Free Memory on damiana is WARNING: WARNING - 6.7% (558536 kB) free! [08:47:47] and wolfsbane now [08:48:04] and willow as well [08:48:06] yay! [08:49:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:49:10] did not do anything...strange... [09:00:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [09:01:53] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 25890 MB (4% inode=99%): [09:02:04] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:03:07] any eta about zws->apache ? [09:18:34] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24451 MB (5% inode=99%): [09:20:04] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:27:55] Free Memory on damiana is CRITICAL: CRITICAL - 4.9% (409004 kB) free! [09:28:54] Free Memory on damiana is WARNING: WARNING - 5.1% (428912 kB) free! [09:30:55] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (418720 kB) free! [09:34:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:40:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:42:34] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 29399 MB (7% inode=99%): [09:50:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:51:03] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:51:42] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 29291 MB (7% inode=99%): [09:55:54] Free Memory on damiana is CRITICAL: CRITICAL - 4.6% (381920 kB) free! [10:00:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [10:01:54] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 25656 MB (4% inode=99%): [10:02:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:09:10] Merlissimo: on qcronsub, is there any way of add -o -e output.out, instead of adding the two statments separatedly? [10:09:51] -j y -o output.out mergen the err stream into the output stream [10:15:32] oops, i've used -r y instead of -j y [10:20:03] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:23:48] Merlissimo: -r y is not an argument right? I'm tryng to figure out what he is doing in my cronie [10:25:09] -r y ist an argument to, it enabbled rescheduling. e.g. like the willow outage today, the job gets rescheduled on onother host after 1 hour [10:28:53] Ah, ok, it's not on https://wiki.toolserver.org/view/Job_scheduling [10:31:22] yes because it is enabled by default on queue configs [10:34:14] so it's not really needed? [10:34:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:40:04] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2172.000000 [10:40:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:50:04] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1789.000000 [10:50:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:55:55] Free Memory on damiana is CRITICAL: CRITICAL - 4.1% (346432 kB) free! [11:00:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [11:01:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 25407 MB (4% inode=99%): [11:02:04] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:14:54] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 44662 MB (10% inode=99%): [11:20:03] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:26:55] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 121523 MB (12% inode=99%): [11:34:56] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:40:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:50:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:51:54] /sql on z-dat-s7-a is OK: DISK OK - free space: /sql 45712 MB (11% inode=99%): [11:56:53] Free Memory on damiana is CRITICAL: CRITICAL - 3.7% (311836 kB) free! [11:59:05] DaB. * Re: [Toolserver-l] ssh and sftp not working [12:00:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [12:01:54] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 25126 MB (4% inode=99%): [12:02:13] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:13:06] hello all [12:14:05] Maarten Dammers * [Toolserver-l] Willow reboot? [12:14:50] DaBPunkt: Hi, why was willow rebooted? [12:15:14] multichill: I just came online [12:15:36] And could you bump the resources for erfgoed at sql? Getting loads of: [12:15:37] _mysql_exceptions.OperationalError: (1226, "User 'erfgoed' has exceeded the 'max_user_connections' resource (current value: 30)") [12:16:20] multichill: "sql" as in "where the user-databases are" or as in "everywhere"? [12:16:31] As in sql.toolserver.org [12:16:42] sure [12:17:30] (somewhen someone has to tell me what "erfgoed" means… sound swedish) [12:17:55] Looks like the tools are very popular because it appears to be quite overloaded. Erfgoed is heritage in Dutch [12:19:25] increased. But there are several >500s queris of erfgoed running at the moment. Is there no way for optimization? [12:19:53] all from the webservers AFAIS [12:19:59] Hmm, I thought I killed off most of the heavy ones [12:20:04] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:21:30] DaBPunkt: It looks like the database gets stuck and all the queries are waiting [12:21:59] because of a writing operation? [12:22:39] Write is only once a day in the middle of the night. it's all read now [12:33:38] multichill: AFAIS the longest-running query is run by prolineserver on the erfgoed-db [12:33:54] Would that block other queries? [12:34:40] it should not [12:34:48] multichill: it was rebooted because of problems with ssh [12:34:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:35:47] Automagicly? [12:41:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:48:34] the syslog noticed only the reboot itself – no reason afais. I was not online at tjis time, maybe nosy? [12:49:23] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:50:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:52:11] nosy did [12:52:38] [10:22] i think i want to simply reboot willow [12:52:47] [10:24] Sun Grid Engine execd on willow is CRITICAL: Connection refused by host [12:54:04] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [12:56:54] Free Memory on damiana is CRITICAL: CRITICAL - 3.4% (282376 kB) free! [13:00:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [13:01:54] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 24864 MB (4% inode=99%): [13:02:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:08:51] DaBPunkt: what would happen if a non-member posts to the mailing list? discarded or queued somewhere or else? [13:09:19] should be rejected [13:10:05] okay I'll resend [13:12:05] Jimmy Xu * [Toolserver-l] Ortelius is missing one SSHFP RR [13:21:04] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:35:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:41:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:45:04] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2068.000000 [13:50:53] Load avg. on adenia is WARNING: WARNING - load average: 15.28, 13.32, 7.69 [13:51:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:54:55] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 89157 MB (9% inode=98%): [13:54:55] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 89157 MB (9% inode=98%): [13:56:54] Free Memory on damiana is CRITICAL: CRITICAL - 3.2% (266552 kB) free! [14:00:55] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [14:02:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 24814 MB (4% inode=99%): [14:03:13] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:04:41] * Danny_B|webchat is looking forward the day when @reboot will work again [14:17:55] Load avg. on adenia is OK: OK - load average: 13.97, 14.99, 13.92 [14:21:03] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:35:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:36:04] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1760.000000 [14:42:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:51:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:57:55] Free Memory on damiana is CRITICAL: CRITICAL - 3.0% (253780 kB) free! [15:00:55] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [15:02:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 24559 MB (4% inode=99%): [15:04:13] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:07:24] are there problems with jira? [15:07:37] it is rejecting a password I'm sure that is right [15:10:46] nosy, maybe the "issue with Jira authentication" you mentioned? [15:11:24] but it didn' t need to sync anything for me :/ [15:12:04] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1802.000000 [15:13:03] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1718.000000 [15:21:03] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:35:04] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2060.000000 [15:35:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:39:03] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1742.000000 [15:42:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:52:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:58:55] Free Memory on damiana is CRITICAL: CRITICAL - 3.0% (251576 kB) free! [16:00:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [16:02:54] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 24187 MB (3% inode=99%): [16:03:03] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2005.000000 [16:05:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:14:03] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1754.000000 [16:21:03] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:25:08] @replag [16:25:08] multichill: s1-rr-a: 28m 5s [+0.21 s/s]; s1-user: 28m 5s [+0.21 s/s]; s2-user: 2h 14m 1s [+0.27 s/s]; s2-user-c: 25m 19s [-0.14 s/s]; s3-rr-a: 22s [-0.01 s/s]; s3-user: 22s [-0.01 s/s]; s5-user-c: 25m 19s [-0.14 s/s] [16:34:03] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2068.000000 [16:34:33] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2090 [16:35:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:42:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:53:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:58:55] Free Memory on damiana is CRITICAL: CRITICAL - 3.1% (259604 kB) free! [17:01:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [17:02:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 23988 MB (3% inode=99%): [17:05:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:07:49] @replag [17:07:49] liangent: s1-rr-a: 43m 3s [+0.35 s/s]; s1-user: 43m 3s [+0.35 s/s]; s2-user: 1h 51m 21s [-0.53 s/s]; s3-rr-a: 10s [-0.00 s/s]; s3-user: 10s [-0.00 s/s] [17:21:03] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:29:03] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1773.000000 [17:29:33] MySQL slave on rosemary is OK: Uptime: 12708726 Threads: 14 Questions: 5255577894 Slow queries: 1527160 Opens: 150915 Flush tables: 6 Open tables: 3755 Queries per second avg: 413.540 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1740 [17:35:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:42:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:48:55] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 15139 MB (1% inode=95%): [17:54:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:59:54] Free Memory on damiana is CRITICAL: CRITICAL - 3.1% (259756 kB) free! [18:01:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [18:03:54] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 23666 MB (3% inode=99%): [18:06:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:13:33] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2078 [18:14:03] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2079.000000 [18:21:03] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:35:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:42:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:55:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:59:55] Free Memory on damiana is CRITICAL: CRITICAL - 2.8% (238172 kB) free! [19:01:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [19:03:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 23403 MB (3% inode=99%): [19:07:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:08:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3604.000000 [19:08:32] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3626 [19:22:03] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:35:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:42:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:55:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:59:55] Free Memory on damiana is CRITICAL: CRITICAL - 2.9% (240996 kB) free! [20:01:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [20:04:54] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 23109 MB (3% inode=99%): [20:07:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:08:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6142.000000 [20:08:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6168 [20:22:04] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:28:54] Free Memory on damiana is OK: OK - 13.8% (1158084 kB) free. [20:35:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:42:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:56:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:58:22] so, does pagelinks table also contain template invocations? [21:01:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [21:04:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 22872 MB (3% inode=99%): [21:08:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9057.000000 [21:08:13] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:08:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9084 [21:17:03] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1841.000000 [21:19:03] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1780.000000 [21:22:03] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:36:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:42:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:52:33] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 29019 MB (7% inode=99%): [21:56:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:02:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [22:05:54] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 22609 MB (3% inode=99%): [22:08:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12120 [22:09:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12147.000000 [22:09:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:12:03] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1984.000000 [22:22:04] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:24:04] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1788.000000 [22:36:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:39:09] hey, who have an access to toolserver logs? [22:39:45] I mean which group of users. Everyone with account, a special group? [22:42:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:47:00] Herr_Kriss: what kind of logs? [22:48:45] Betacommand: "Of course, you should use POST requests, otherwise the passwords will show up in the toolerver logs.." - it's about TUSC [22:49:10] http://tools.wikimedia.de/~magnus/tusc.php?check=1&botmode=1&user=USERNAME&language=LANGUAGE&project=PROJECT&password=TUSC_PASSWORD [22:49:36] that would be the webserver access logs and any user has access to that [22:49:50] you should never, ever put passwords in a GET request as a matter of basic security [22:49:52] and let's say that somebody will post there his password, is it rather safe or not at all? [22:49:54] ok, thanks [22:49:55] those are saved in all kinds of places [22:50:04] that applies everywhere, not just TS [22:50:06] felicity: I know, but thanks anyway [22:56:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:03:55] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [23:05:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 22372 MB (3% inode=99%): [23:07:54] where is the list of accounts? I'm searching for pl.wiki users with accounts. I know their nicks but I'd need to have a list to cpmpare. [23:09:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15231.000000 [23:09:04] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:09:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 15259 [23:22:03] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:27:55] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 120383 MB (12% inode=99%): [23:36:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:42:56] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:56:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output