[00:02:20] hi [00:03:45] I'm having problems with rewrite.script. I have a working (under apache) htacces ( http://toolserver.org/~fale/htaccess ) and what I think whould be the right "conversion" to rewrite.script ( http://toolserver.org/~fale/rewrite.script ) but it does not work and I have no idea about why :( [00:07:30] fixed :) I was putting it in $HOME/public_html/ istead of $HOME/ [01:18:53] Hi, I think I found an important error to report [01:26:13] huh? [01:27:04] well, I'm not sure about this [01:29:48] huh: one sec, getting you the place to email [01:30:10] Betacommand: nvm, I see it escapes quotes [01:30:18] so you can't do much damage [01:33:26] huh: ts-admins-at-toolserver.org [01:41:26] Betacommand: meh, it also escapes "/" so I think it's fine [08:13:04] [[Special:Log/newusers]] create 10 * Steel1943 * (New user account) [11:47:58] anyone around that could write me an online list generated from toolserver DBs of en wiki articles with the most interwiki links? [11:49:26] similar to http://en.wikipedia.org/wiki/Wikipedia_talk:Database_reports#Data_request but I just want a list of articles instead of counts [11:49:41] and to be able to request a list of the top 1000 or so linked articles from a webpage :) [12:09:36] addshore: not right now [12:09:50] I noticed :P [12:10:10] addshore: with the introduction of wikidata the iwlinks table is screwed up [12:10:18] I also noticed that :P [12:11:16] see http://en.wikipedia.org/wiki/User:Addshore/most_interwiki#Stats xD [12:12:05] addshore: there is a bug for schema change to fix this [12:12:31] =] [12:14:10] * Betacommand goes to find the bug [12:23:40] addshore: I cant find the bug... [12:24:11] I'll have a hunt for it in a few hours :) thanks for knowing about it in the first place though! xD [12:26:01] this is really pissing me off [12:26:50] of course google cant index bugzilla [12:41:41] @replag [12:41:43] DaBPunkt: s1-rr-a-wd: 44s [-0.00 s/s]; s1-user-wd: 49s [-0.00 s/s]; s2-rr: 30m 27s [+0.12 s/s]; s2-user: 30m 27s [+0.12 s/s]; s2-user-c: error; s2-user-wd: 2h 4m 16s [-0.58 s/s]; s3-user: 15s [-0.00 s/s]; s3-user-wd: 44s [-0.00 s/s] [12:41:44] DaBPunkt: s4-user-wd: 43s [-0.00 s/s]; s5-user: 7h 36m 48s [+1.00 s/s]; s5-user-c: error; s6-user-wd: 45s [-0.00 s/s]; s7-user-wd: 44s [-0.00 s/s] [12:42:06] DaB. * [Toolserver-announce] Reboot of the linux-boxes today [14:06:34] Danny_B: I will reboot nightshade and yarrow tonight (see ML) [14:24:06] DaB. * [Toolserver-announce] Postmortem: General downtime yesterday [14:29:06] DaB. * Re: [Toolserver-l] Short downtime of s2 tomorrow [14:40:06] DaB. * Re: [Toolserver-announce] [Toolserver-l] 2. Try: sql-s5 (dewiki) will be read-only tomorrow [14:53:55] @replag [14:53:55] DaBPunkt: s1-rr-a-wd: 47s [-0.00 s/s]; s1-user-wd: 51s [+0.00 s/s]; s2-rr: 1h 1m 14s [+0.01 s/s]; s2-user: 1h 1m 14s [+0.01 s/s]; s2-user-c: error; s2-user-wd: 2h 7m 48s [-0.15 s/s]; s3-user: 22s [+0.00 s/s]; s3-user-wd: 51s [+0.00 s/s] [14:53:56] DaBPunkt: s4-user-wd: 48s [-0.00 s/s]; s5-user: 5h 18m 16s [-1.39 s/s]; s5-user-c: error; s6-user-wd: 51s [+0.00 s/s]; s7-user-wd: 51s [+0.00 s/s] [16:07:02] [[Special:Log/newusers]] create 10 * Vogone * (New user account) [16:33:03] @replag [16:33:04] liangent: s1-rr-a-wd: 48s [+0.00 s/s]; s1-user-wd: 49s [+0.00 s/s]; s2-rr: 2h 10m 31s [+0.77 s/s]; s2-user: 2h 10m 31s [+0.77 s/s]; s2-user-c: error; s2-user-wd: 2h 51m 32s [+0.48 s/s]; s3-user: 27s [-0.00 s/s]; s3-user-wd: 50s [+0.00 s/s] [16:33:05] liangent: s4-user-wd: 1m 10s [+0.00 s/s]; s5-user: 39m 4s [-3.14 s/s]; s5-user-c: error; s6-user-wd: 48s [+0.00 s/s]; s7-user-wd: 50s [+0.00 s/s] [17:14:43] DaBPunkt: Hi! Could it be that the HA failure knocked out the mailserver? When I "echo Test | mail -s Test timl" on yarrow, the message is neither forwarded to my set address, nor stored locally on any of the servers. [17:42:16] Sending mail from willow works. [17:46:51] Nightshade works as well, so it seems limited to yarrow. [17:49:06] And in fact, on yarrow "mailq" lists 206 queued mails. Are these automatically resent, or is admin intervention needed there? [17:50:17] addshore: https://bugzilla.wikimedia.org/show_bug.cgi?id=41345 [18:00:23] re [18:03:36] scfc_de: I will look for it [18:03:46] (I REALLY need to fix the aliasd-service…) [18:05:34] IIRC the aliasd functionality (preferring ~/.forward over LDAP) is /almost/ implemented in postfix, we would just need to set up some dummy files somewhere (sorry, forgot the details). So might be possible to get rid of aliasd. [18:06:01] What problems are in aliasd, BTW? [18:06:43] scfc_de: segmentation faults [18:07:54] I brought a book about c exspecially for the toolserver – but I find no time to read it… [18:08:23] scfc_de: but if you find a way to get rid of it I'm all ears :-) [18:11:01] Well, there's a ton of C expertise around if you need some advice :-). But it is not very nice to debug text manipulation in C. [18:12:20] From reading http://www.linuxquestions.org/questions/linux-server-73/postfix-ldap-and-forward-files-748756/ it seems that it should be enough to set up a (hourly/daily) cronjob that for each user who has a ~/.forward in his home directory adds a "$USER: $USER" to /etc/aliases. [18:19:28] scfc_de: I will give it a try next week. Thanks anyway [18:24:12] And yarrow's mail has arrived. Thanks! [18:24:58] cheers Betacommand [18:25:13] addshore: you can thank legoktm [18:25:24] :) [18:25:26] HAHA! will do :) [18:25:43] addshore: I couldnt find it [20:03:45] nightshade reboots in 2 minutes [20:04:13] if anyone has something open on nightshade he/she should close it NOW! [20:05:13] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10996 [20:05:13] wikidata replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 367.000000 [20:05:13] s5 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2495.000000 [20:05:13] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 119001 MB (19% inode=99%): [20:05:17] wikidata replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 50.000000 [20:05:18] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:05:19] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:05:19] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:05:19] APT on yarrow is WARNING: APT WARNING: 33 packages available for upgrade (0 critical updates). [20:05:19] wikidata replag on z-dat-s6-a is OK: QUERY OK: SELECT ts_rc_age() returned 51.000000 [20:05:19] wikidata replag on z-dat-s7-a is OK: QUERY OK: SELECT ts_rc_age() returned 51.000000 [20:05:19] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [20:07:53] i knew something was missing from this channel… :P [20:10:48] APT on z-dat-s2-b is WARNING: APT WARNING: 25 packages available for upgrade (0 critical updates). [20:11:19] aliasd on mayapple is CRITICAL: Connection refused [20:11:29] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12590.000000 [20:12:47] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on rosemary (146) [20:12:48] wikidata replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2220.000000 [20:12:58] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on rosemary (146) [20:13:08] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on rosemary (146) [20:13:08] MySQL on rosemary is CRITICAL: Cant connect to MySQL server on rosemary (146) [20:13:18] MySQL slave on rosemary is CRITICAL: Cant connect to MySQL server on rosemary (146) [20:13:18] Environment IPMI on rosemary is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:13:38] Load avg. on rosemary is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:13:48] SMTP on rosemary is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:13:58] RAID on rosemary is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:14:28] NTP on rosemary is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:15:38] SMTP on rosemary is OK: SMTP OK - 6.740 sec. response time [20:15:48] RAID on rosemary is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [20:16:07] Load avg. on rosemary is OK: OK - load average: 0.34, 0.53, 0.96 [20:16:19] NTP on rosemary is OK: NTP OK: Offset -0.011324 secs [20:16:58] Environment IPMI on rosemary is OK: ok: temperature ok fan ok voltage ok chassis ok [20:17:18] APT on yarrow is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:19:48] if anyone has something open on yarrow he/she should close it NOW! [20:21:28] Sun Grid Engine execd on nightshade is CRITICAL: CRITICAL: execd not communicating [20:24:08] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3636.000000 [20:25:41] nightshade and yarrow are back [20:31:28] NTP on yarrow is WARNING: NTP WARNING: Server has the LI_ALARM bit set, Offset -0.034122 secs [20:32:04] the user-store is missing at the moment, nosy looks for it [20:35:48] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3601.000000 [20:41:27] @replag [20:41:28] NTP on yarrow is OK: NTP OK: Offset -0.078738 secs [20:42:06] Merlissimo: s1-rr-a-wd: 1h 5m 45s [+1.00 s/s]; s1-user: error; s1-user-c: error; s1-user-wd: error; s2-rr: 2h 53m 2s [+1.00 s/s]; s2-user: 2h 53m 2s [+1.00 s/s]; s2-user-c: error; s2-user-wd: 3h 14m 29s [+0.71 s/s] [20:42:07] Merlissimo: s3-user-wd: 48s [+0.04 s/s]; s4-user-wd: 46s [-]; s5-rr-a: 1h 18m 4s [+1.00 s/s]; s5-user: error; s5-user-c: error; s6-user-wd: 1m 2s [+0.11 s/s]; s7-user-wd: 1m 1s [+0.10 s/s] [20:42:19] MySQL slave on rosemary is OK: Uptime: 1590 Threads: 13 Questions: 111 Slow queries: 0 Opens: 19 Flush tables: 1 Open tables: 12 Queries per second avg: 0.69 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [20:42:58] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2609.000000 [20:43:08] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2566.000000 [20:43:08] wikidata replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2606.000000 [20:43:08] MySQL on rosemary is OK: Uptime: 1642 Threads: 35 Questions: 1695 Slow queries: 64 Opens: 274 Flush tables: 1 Open tables: 257 Queries per second avg: 1.32 [20:47:48] APT on z-dat-s2-b is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:50:18] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2667 [20:57:28] Free Memory on turnera is CRITICAL: CRITICAL - 4.9% (410284 kB) free! [20:59:58] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1658.000000 [21:00:08] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3627.000000 [21:01:28] Free Memory on turnera is WARNING: WARNING - 5.5% (462956 kB) free! [21:02:18] MySQL slave on rosemary is OK: Uptime: 2790 Threads: 9 Questions: 1675360 Slow queries: 389 Opens: 465 Flush tables: 1 Open tables: 420 Queries per second avg: 600.487 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1777 [21:02:48] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1719.000000 [21:04:28] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 252704 MB (4% inode=67%): [21:04:58] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [21:05:08] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 118749 MB (19% inode=99%): [21:05:19] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:05:19] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:05:19] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:05:19] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [21:05:28] Free Memory on turnera is OK: OK - 7.9% (660216 kB) free. [21:05:38] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:05:38] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:05:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:05:48] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:11:17] aliasd on mayapple is CRITICAL: Connection refused [21:13:07] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13249.000000 [21:13:18] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:14:47] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12371 [21:19:06] DaB. * Re: [Toolserver-l] Reboot of the linux-boxes today [21:24:08] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7235.000000 [21:31:28] Free Memory on turnera is WARNING: WARNING - 5.1% (425804 kB) free! [21:35:08] wikidata replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3585.000000 [21:35:47] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4831.000000 [21:41:48] wikidata replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3401.000000 [21:45:08] wikidata replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1667.000000 [21:45:26] @replag [21:45:26] DaBPunkt: s1-rr-a-wd: 41m 6s [-0.39 s/s]; s1-user-wd: 26m 36s [+0.08 s/s]; s2-rr: 3h 33m 51s [+0.64 s/s]; s2-user: 3h 33m 51s [+0.64 s/s]; s2-user-c: error; s2-user-wd: 3h 48m 23s [+0.53 s/s]; s3-user: 44s [+0.01 s/s]; s3-user-wd: 58s [+0.00 s/s] [21:45:27] DaBPunkt: s4-user-wd: 54s [+0.00 s/s]; s5-rr-a: 2h 22m 3s [+1.00 s/s]; s5-user-c: error; s6-user-wd: 56s [-0.00 s/s]; s7-user-wd: 55s [-0.00 s/s] [21:49:47] wikidata replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 1617.000000 [21:59:28] Free Memory on turnera is CRITICAL: CRITICAL - 4.9% (407708 kB) free! [21:59:58] Environment IPMI on thyme is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [22:00:29] Free Memory on turnera is WARNING: WARNING - 5.7% (478836 kB) free! [22:01:19] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:01:28] Free Memory on turnera is CRITICAL: CRITICAL - 4.2% (353060 kB) free! [22:02:02] [[Main Page/lang]] ! 10https://wiki.toolserver.org/w/index.php?diff=7834&oldid=7624&rcid=21616 * Abshirdheere * (+37) () [22:04:28] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 252222 MB (4% inode=67%): [22:04:58] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [22:05:18] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:05:18] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [22:05:27] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:05:37] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:05:38] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:06:08] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 118424 MB (19% inode=99%): [22:06:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:06:48] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:11:18] aliasd on mayapple is CRITICAL: Connection refused [22:13:18] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:14:07] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9461.000000 [22:14:48] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7895 [22:23:29] Free Memory on turnera is OK: OK - 7.6% (640384 kB) free. [22:24:08] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10835.000000 [22:36:08] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [22:36:47] Environment IPMI on thyme is CRITICAL: Connection refused by host [22:37:48] Environment IPMI on thyme is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [22:43:47] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3518 [22:46:48] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [22:46:48] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [22:47:08] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [22:47:09] MySQL slave on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [22:47:28] Free Memory on turnera is WARNING: WARNING - 5.9% (497764 kB) free! [22:52:28] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (419456 kB) free! [22:53:48] MySQL slave on z-dat-s2-b is OK: Uptime: 6151 Threads: 8 Questions: 6371150 Slow queries: 139 Opens: 85131 Flush tables: 1 Open tables: 256 Queries per second avg: 1035.790 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1556 [22:54:49] wikidata replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 925.000000 [22:55:08] MySQL on thyme is OK: Uptime: 862 Threads: 8 Questions: 3241 Slow queries: 0 Opens: 54 Flush tables: 1 Open tables: 47 Queries per second avg: 3.759 [22:55:08] MySQL slave on thyme is OK: Uptime: 867 Threads: 11 Questions: 5265 Slow queries: 0 Opens: 58 Flush tables: 1 Open tables: 51 Queries per second avg: 6.72 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 873 [22:55:53] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 866.000000 [22:58:18] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:04:28] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 251759 MB (4% inode=67%): [23:04:58] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [23:05:19] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:05:19] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [23:05:38] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:05:38] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:06:08] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 118159 MB (19% inode=99%): [23:06:50] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:06:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:08:07] wikidata replag on z-dat-s2-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3397.000000 [23:10:21] @replag [23:10:22] DaBPunkt: s1-rr-a-wd: 31m 3s [-0.12 s/s]; s1-user-wd: 59s [-0.30 s/s]; s2-rr: 21s [-2.51 s/s]; s2-user: 21s [-2.51 s/s]; s2-user-c: error; s2-user-wd: 53m 56s [-2.05 s/s]; s3-user: 24s [-0.00 s/s]; s3-user-wd: 59s [+0.00 s/s] [23:10:23] DaBPunkt: s4-user-wd: 1m 2s [+0.00 s/s]; s5-rr-a: 3h 42m 37s [+0.95 s/s]; s5-user: 14m 45s [+0.04 s/s]; s5-user-c: error; s6-user-wd: 1m 3s [+0.00 s/s]; s7-user-wd: 1m 1s [+0.00 s/s] [23:11:18] aliasd on mayapple is CRITICAL: Connection refused [23:13:19] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:13:19] MySQL slave on cassia is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11920 [23:15:48] wikidata replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2184.000000 [23:16:53] @replag [23:16:53] DaBPunkt: s1-rr-a-wd: 37m 34s [+1.00 s/s]; s1-user-wd: 1m 8s [+0.02 s/s]; s2-user-c: error; s2-user-wd: 26m 52s [-4.15 s/s]; s3-user: 42s [+0.05 s/s]; s3-user-wd: 1m 14s [+0.04 s/s]; s4-user-wd: 1m 44s [+0.11 s/s]; s5-rr-a: 2h 24m 9s [-12.02 s/s] [23:16:54] DaBPunkt: s5-user: 16m 17s [+0.23 s/s]; s5-user-c: error; s6-user-wd: 1m 14s [+0.03 s/s]; s7-user-wd: 1m 7s [+0.02 s/s] [23:17:08] wikidata replag on z-dat-s2-b is OK: QUERY OK: SELECT ts_rc_age() returned 1566.000000 [23:20:47] wikidata replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 1526.000000 [23:23:08] s5 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3531.000000 [23:23:18] MySQL slave on cassia is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3220 [23:24:18] MySQL slave on cassia is OK: Uptime: 271282 Threads: 7 Questions: 505575510 Slow queries: 10745 Opens: 83074 Flush tables: 2 Open tables: 13939 Queries per second avg: 1863.652 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1729 [23:25:08] s5 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 787.000000 [23:35:19] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:35:59] @replag [23:36:00] DaBPunkt: s1-rr-a-wd: 59s [-1.91 s/s]; s1-user-wd: 54s [-0.01 s/s]; s2-user-c: error; s2-user-wd: 59s [-1.35 s/s]; s3-user: 60s [+0.02 s/s]; s3-user-wd: 59s [-0.01 s/s]; s4-user-wd: 7m 45s [+0.31 s/s]; s5-user: 31m 17s [+0.79 s/s] [23:36:00] DaBPunkt: s5-user-c: error; s6-user-wd: 58s [-0.01 s/s]; s7-user-wd: 54s [-0.01 s/s] [23:52:28] Free Memory on turnera is CRITICAL: CRITICAL - 2.0% (171748 kB) free! [23:57:05] I'd a weird trouble yesterday, on nightshade a subprocess of a process start with qcronsub -l arch=lx -l h_rt=INFINITY -l virtual_free=64M -notify $HOME/wsbot/wshocr [23:57:21] eat 14 Gb of memory before being killed [23:58:18] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.