[00:02:52] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 31113.000000 [00:06:11] toolserver.org HTTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:06:50] phe: I believe Danny_B is managing this. [00:07:11] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:07:20] phe: yes they do [00:07:25] what is the issue? [00:08:28] Danny_B, 5 Gb free on user-store and there is bunzip2 of a en dump in progress since 12 hours [00:09:20] user-store is sloooow, it take 50 seconds to read a 32 Ko file ;( [00:12:18] phe: Just want to confirm your observation: user-store is sloooow :-). [00:13:22] Danny_B: On which server is the bunzip2 running? [00:13:34] nightshade [00:13:35] nightshade [00:14:04] why people don't use directly compressed dump ? [00:14:29] because some tools don't work with compressed dumps [00:15:01] toolserver.org HTTP on ortelius is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.571 second response time [00:15:07] otoh, there is only the last one uncompressed [00:15:13] well, bzcat | broken-tool and read stding will do the trick :( [00:15:13] previous are compressed [00:16:47] Whatever it is, "ls -l /mnt/user-store/dumps/" just gives me the directories' names, not the details. Hmmm. [00:17:11] (But on local directories it works fine so LDAP is okay.) [00:19:04] well, the trouble is, bunzip2 will finish or fill user-store before ? [00:19:12] Might be useful to consider whether this bunzipping could be done directly on rosemary where the NFS volumes are mounted IIRC. [00:19:48] good idea [00:20:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:21:42] I think phe's question is more urgent, though :-). [00:22:23] it's not so fast, you have at least one hour left :) [00:23:16] i'll cleanup some older dumps [00:23:23] And what will it expand to and what it is now? (The file system is too damn slow to get some info.) [00:23:42] also, there will be probably some unused dumps [00:23:57] people never say they no longer need them [00:25:14] the new tool for better management of dumps is in progress though [00:25:32] Be careful about deleting them, though, I think some time ago it was considered as some kind of backup for WMF :-). I believe they now have other off-site backups. [00:25:45] Do you run it manually ATM or via cron? [00:26:14] perhaps the fs on rosemary is mounted with last access time on and you can know the oldest used dump [00:27:40] phe: i did not mean the age actually, i meant the given wiki [00:27:58] eg. somebody used some obscure lang dump year ago and no longer needs that [00:28:11] but since it has been used, the dump is still being updated [00:28:18] although no longer needed [00:30:48] beside that, using this dump is likely to slow down a lot user-store, with so little room on the disk it's likely to be highly fragmented [00:32:05] i actually wonder if anybody uses this one [00:32:18] if pages-articles isn't enough [00:33:39] I think adapting the existing tools to use the bzipped dumps should be something to look at. I think CPU isn't that expensive compared to shuffling gigabytes (terabytes?) of data over NFS :-). [00:33:53] also, cleanup of usr-store will take place soon, now when operators have been set, it's finally possible [00:34:59] scfc_de: as long as there are people willing to help to those who need uncompressed with the transition... [00:35:33] some users are not geeks, so they use simple stuff they copied from elsewhere [00:35:39] and just customized [00:36:51] It's Unix - whether to read from a file or a pipe isn't that different. If you know someone who needs help, give a shout. [00:36:58] erm [00:37:03] i cant find lo.wp's dump? [00:37:10] legoktm@nightshade:/mnt/user-store/dumps$ ls | grep lowiki [00:37:10] legoktm@nightshade:/mnt/user-store/dumps$ ls | grep lo [00:37:22] i havent read any scrollback btw [00:38:46] so um [00:38:48] boxofjuice: there are only dumps which have been used at least once. not all are copied [00:38:53] ok [00:38:58] ill just [00:39:03] ergh [00:39:04] so if you need it, download it to tmp [00:39:12] nah [00:39:14] and then it will be added to regular update [00:39:16] its small enough that i can use the API [00:39:20] or [00:39:22] ill just [00:39:25] run it from wmflabs [00:39:33] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:39:33] just read the readme files in dump dir ;-) [00:39:41] heh [00:39:46] i didnt even know there was a readme [00:40:34] a few of them [00:41:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:41:11] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [00:42:52] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1885 [00:42:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:44:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 4543 MB (0% inode=68%): [00:45:50] Danny_B: Would be nice once the dust has settled if you could post that to the mailing list and update https://wiki.toolserver.org/view/User-store. [00:45:53] Danny_B: you could download and unzip dump on tmp and them move to user-store only a least action. bulk readin/writing is fast on nfs [00:45:53] MySQL slave on z-dat-s2-b is OK: Uptime: 344985 Threads: 11 Questions: 566965867 Slow queries: 4839 Opens: 3265591 Flush tables: 1 Open tables: 256 Queries per second avg: 1643.450 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1692 [00:47:32] thanks for suggestions, i'll work on it [00:47:49] in the meantime, i removed the old dump [00:47:52] sge knows which server have enough tmp space [00:48:13] so 15G is there now [00:50:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:52:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 118988 MB (19% inode=99%): [00:59:18] so the enwiki just ended up [01:02:53] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32745.000000 [01:07:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:08:53] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2102 [01:20:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:24:14] how is it now? [01:24:34] should be faster [01:25:53] wikidata replag on z-dat-s2-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1969.000000 [01:39:29] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:41:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [01:42:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:42:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:44:11] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 98652 MB (1% inode=68%): [01:46:53] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3621 [01:48:13] Danny_B, a lot faster [01:50:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:51:53] wikidata replag on z-dat-s2-b is CRITICAL: (Service Check Timed Out) [01:52:52] wikidata replag on z-dat-s2-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2662.000000 [01:53:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 116951 MB (19% inode=99%): [02:02:53] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 34940.000000 [02:07:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:20:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:26:53] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3477 [02:29:01] wikidata replag on z-dat-s2-b is OK: QUERY OK: SELECT ts_rc_age() returned 1715.000000 [02:39:29] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:41:11] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [02:42:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:43:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:44:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 98029 MB (1% inode=68%): [02:50:53] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3624 [02:50:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:54:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 118596 MB (19% inode=99%): [03:02:53] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 37016.000000 [03:03:01] wikidata replag on z-dat-s2-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1928.000000 [03:07:02] wikidata replag on z-dat-s2-b is OK: QUERY OK: SELECT ts_rc_age() returned 1785.000000 [03:07:11] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:14:12] wikidata replag on z-dat-s2-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2002.000000 [03:21:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:39:29] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:41:11] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [03:41:29] wikidata replag on z-dat-s2-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2450.000000 [03:43:00] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:43:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:44:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 98100 MB (1% inode=68%): [03:50:53] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5956 [03:50:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:54:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 118419 MB (19% inode=99%): [04:02:53] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 31568.000000 [04:07:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:10:53] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3626.000000 [04:21:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:39:38] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:41:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [04:43:13] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:43:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:44:13] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 97964 MB (1% inode=68%): [04:50:54] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8855 [04:50:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:54:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 118270 MB (19% inode=99%): [04:56:53] wikidata replag on z-dat-s2-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3358.000000 [05:02:52] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 27181.000000 [05:11:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:16:43] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3365 [05:17:22] /tmp on willow is WARNING: DISK WARNING - free space: / 21312 MB (20% inode=98%): [05:18:23] / on willow is WARNING: DISK WARNING - free space: / 21253 MB (20% inode=98%): [05:18:43] MySQL slave on z-dat-s2-b is OK: Uptime: 361356 Threads: 6 Questions: 586735639 Slow queries: 5315 Opens: 3341951 Flush tables: 1 Open tables: 256 Queries per second avg: 1623.705 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1571 [05:21:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:25:03] wikidata replag on z-dat-s2-b is OK: QUERY OK: SELECT ts_rc_age() returned 1491.000000 [05:39:53] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:41:43] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [05:43:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:44:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:45:03] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 97688 MB (1% inode=68%): [05:51:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:55:43] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 118133 MB (19% inode=99%): [06:03:22] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25830.000000 [06:11:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:17:23] /tmp on willow is WARNING: DISK WARNING - free space: / 19694 MB (18% inode=98%): [06:18:23] / on willow is WARNING: DISK WARNING - free space: / 19692 MB (18% inode=98%): [06:21:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:39:53] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:41:43] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [06:43:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:44:23] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:45:03] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 97715 MB (1% inode=68%): [06:51:23] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:55:42] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 118008 MB (19% inode=99%): [07:03:23] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 27021.000000 [07:11:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:17:23] /tmp on willow is WARNING: DISK WARNING - free space: / 19567 MB (18% inode=98%): [07:18:23] / on willow is WARNING: DISK WARNING - free space: / 19566 MB (18% inode=98%): [07:21:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:39:53] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:41:42] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [07:43:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:44:23] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:45:02] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 97453 MB (1% inode=68%): [07:51:23] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:53:03] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 61924 MB (6% inode=99%): [07:55:43] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 117814 MB (19% inode=99%): [08:03:22] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 27372.000000 [08:11:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:17:22] /tmp on willow is WARNING: DISK WARNING - free space: / 19465 MB (18% inode=98%): [08:18:23] / on willow is WARNING: DISK WARNING - free space: / 19464 MB (18% inode=98%): [08:21:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:39:53] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:41:43] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [08:43:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:44:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:45:03] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 95104 MB (1% inode=68%): [08:51:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:55:43] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 117602 MB (19% inode=99%): [09:03:23] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28685.000000 [09:11:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:14:13] wikidata replag on z-dat-s2-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2077.000000 [09:17:23] /tmp on willow is WARNING: DISK WARNING - free space: / 19355 MB (18% inode=98%): [09:18:23] / on willow is WARNING: DISK WARNING - free space: / 19354 MB (18% inode=98%): [09:21:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:39:53] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:41:43] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [09:43:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:44:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:45:03] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 94753 MB (1% inode=68%): [09:49:37] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1977 [09:51:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:55:47] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 117374 MB (19% inode=99%): [10:03:28] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 29946.000000 [10:13:38] MySQL slave on z-dat-s2-b is OK: Uptime: 379059 Threads: 14 Questions: 599617020 Slow queries: 5758 Opens: 3486496 Flush tables: 1 Open tables: 256 Queries per second avg: 1581.856 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1642 [10:15:09] wikidata replag on z-dat-s2-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2507.000000 [10:17:28] /tmp on willow is WARNING: DISK WARNING - free space: / 19236 MB (18% inode=98%): [10:19:18] / on willow is WARNING: DISK WARNING - free space: / 19233 MB (18% inode=98%): [10:21:08] wikidata replag on z-dat-s2-b is OK: QUERY OK: SELECT ts_rc_age() returned 1431.000000 [10:21:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:40:48] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:41:48] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [10:42:08] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:43:47] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:44:08] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 57911 MB (5% inode=99%): [10:44:28] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:45:08] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 94180 MB (1% inode=68%): [10:49:09] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 60174 MB (6% inode=99%): [10:50:17] Free Memory on damiana is WARNING: WARNING - 5.1% (428532 kB) free! [10:51:18] Free Memory on damiana is CRITICAL: CRITICAL - 4.3% (357756 kB) free! [10:51:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:55:47] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 117132 MB (19% inode=99%): [10:56:18] Free Memory on damiana is WARNING: WARNING - 5.5% (463700 kB) free! [11:03:28] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 30844.000000 [11:17:18] Free Memory on damiana is CRITICAL: CRITICAL - 4.9% (413972 kB) free! [11:17:28] /tmp on willow is WARNING: DISK WARNING - free space: / 19144 MB (18% inode=98%): [11:19:17] Free Memory on damiana is WARNING: WARNING - 5.1% (424308 kB) free! [11:19:17] / on willow is WARNING: DISK WARNING - free space: / 19143 MB (18% inode=98%): [11:21:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:24:17] Free Memory on damiana is CRITICAL: CRITICAL - 4.8% (402500 kB) free! [11:40:48] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:41:47] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [11:43:48] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:44:28] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:45:08] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 93802 MB (1% inode=68%): [11:48:17] Free Memory on damiana is CRITICAL: CRITICAL - 2.8% (230448 kB) free! [11:50:48] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:51:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:51:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:55:50] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 116894 MB (19% inode=99%): [12:01:51] Free Memory on damiana is OK: OK - 83.0% (6956576 kB) free. [12:03:52] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 27980.000000 [12:11:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:17:51] /tmp on willow is WARNING: DISK WARNING - free space: / 19102 MB (18% inode=98%): [12:19:51] / on willow is WARNING: DISK WARNING - free space: / 19101 MB (18% inode=98%): [12:28:11] wikidata replag on z-dat-s2-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2099.000000 [12:41:27] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:41:50] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [12:43:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:44:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:45:10] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 93532 MB (1% inode=68%): [12:51:20] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:51:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:55:51] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 116662 MB (19% inode=99%): [13:04:52] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9133.000000 [13:09:11] wikidata replag on z-dat-s2-b is OK: QUERY OK: SELECT ts_rc_age() returned 1644.000000 [13:18:51] /tmp on willow is WARNING: DISK WARNING - free space: / 19056 MB (18% inode=98%): [13:20:51] / on willow is WARNING: DISK WARNING - free space: / 19054 MB (18% inode=98%): [13:23:51] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1967 [13:24:11] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1977.000000 [13:41:27] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:41:51] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:41:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [13:43:59] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:45:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 93161 MB (1% inode=68%): [13:45:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:51:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:51:50] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:55:51] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 116396 MB (19% inode=99%): [13:59:51] wikidata replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3539.000000 [14:15:51] wikidata replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1798.000000 [14:18:51] /tmp on willow is WARNING: DISK WARNING - free space: / 19012 MB (18% inode=98%): [14:19:18] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:19:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:20:51] / on willow is WARNING: DISK WARNING - free space: / 19010 MB (18% inode=98%): [14:21:18] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:21:50] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:22:28] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:23:50] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3149 [14:24:11] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3151.000000 [14:40:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:41:28] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:41:51] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:41:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [14:44:57] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:46:14] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 91942 MB (1% inode=68%): [14:52:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:55:51] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 116128 MB (19% inode=99%): [15:00:42] hello all [15:00:47] @replag [15:00:48] DaBPunkt: s1-rr-a-wd: 36s [-]; s1-user-wd: 45s [-]; s2-rr: 12s [-]; s2-user: 12s [-]; s2-user-c: error; s2-user-wd: 40s [-]; s3-user: 35s [-]; s3-user-wd: 35s [-] [15:00:49] DaBPunkt: s4-user-wd: 8m 10s [-]; s5-user: 51m 58s [-]; s5-user-c: error; s6-user-wd: 41s [-]; s7-user-wd: 40s [-] [15:00:58] lloks good [15:18:51] /tmp on willow is WARNING: DISK WARNING - free space: / 18971 MB (18% inode=98%): [15:20:51] / on willow is WARNING: DISK WARNING - free space: / 18969 MB (18% inode=98%): [15:23:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:23:51] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2813 [15:24:11] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2779.000000 [15:25:51] /tmp on willow is OK: DISK OK - free space: / 26986 MB (25% inode=99%): [15:25:51] aliasd on mayapple is CRITICAL: Connection refused [15:25:51] / on willow is OK: DISK OK - free space: / 26986 MB (25% inode=99%): [15:40:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:41:26] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:41:51] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:41:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [15:45:57] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:47:11] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 90891 MB (1% inode=68%): [15:53:50] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:55:50] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 115817 MB (19% inode=99%): [16:23:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:23:51] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2923 [16:24:11] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2937.000000 [16:25:50] aliasd on mayapple is CRITICAL: Connection refused [16:41:27] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:41:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:41:51] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:41:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [16:43:51] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3604 [16:44:12] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3615.000000 [16:45:58] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:47:11] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 87780 MB (1% inode=68%): [16:50:50] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2194 [16:53:50] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:55:50] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 115503 MB (19% inode=99%): [17:06:12] wikidata replag on z-dat-s2-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2126.000000 [17:10:17] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:10:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:15:50] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3619 [17:23:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:25:51] aliasd on mayapple is CRITICAL: Connection refused [17:34:12] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3644.000000 [17:41:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:41:54] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [17:42:26] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:43:51] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6018 [17:45:11] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6081.000000 [17:46:57] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:47:11] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 88678 MB (1% inode=68%): [17:53:50] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:56:51] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 115205 MB (18% inode=99%): [18:05:51] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:06:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:11:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:15:52] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6442 [18:25:54] aliasd on mayapple is CRITICAL: Connection refused [18:35:11] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4113.000000 [18:41:51] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:41:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [18:42:27] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:43:01] @replag [18:43:02] hk_kng: s1-rr-a-wd: 32s [-0.00 s/s]; s1-user: 2m 5s [-]; s1-user-wd: 34s [-0.00 s/s]; s2-rr: 1h 49m 48s [+0.49 s/s]; s2-user: 1h 49m 48s [+0.49 s/s]; s2-user-c: error; s2-user-wd: 1h 14m 14s [+0.33 s/s]; s3-user-wd: 31s [-0.00 s/s] [18:43:03] hk_kng: s4-user-wd: 6m 22s [-0.01 s/s]; s5-user: 2h 26m 26s [+0.43 s/s]; s5-user-c: error; s6-user-wd: 33s [-0.00 s/s]; s7-user-wd: 33s [-0.00 s/s] [18:43:52] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8811 [18:45:18] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8887.000000 [18:46:58] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:47:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 84236 MB (1% inode=68%): [18:54:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:56:51] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 114931 MB (18% inode=99%): [19:06:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:12:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:16:51] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7517 [19:25:52] aliasd on mayapple is CRITICAL: Connection refused [19:35:12] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6136.000000 [19:36:51] wikidata replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2084.000000 [19:41:50] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [19:41:50] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:42:27] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:44:51] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9759 [19:46:18] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9775.000000 [19:46:57] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:48:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 81056 MB (1% inode=68%): [19:54:50] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:56:51] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 114623 MB (18% inode=99%): [20:07:28] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:12:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:16:51] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9795 [20:18:06] DanielK_WMDE_: While looking at https://www.mediawiki.org/w/index.php?diff=656559: Didn't WMDE hire someone to build a (category) graph server? Is this live? Is there a status page for that project somewhere? [20:19:08] scfc_de: as far as I know, it's not live anywhere. The compontent that is not really ready is the live updating from the wiki database. [20:19:50] scfc_de: but yes, JohannesK was hired by WMDE for this, and is still working with it. Afaik, it'S used for data analysis internally [20:20:17] I have tried to plant the idea to integrate this into mediawiki at the foundation, but I'm not sure where I should fit it into their buisy schedule [20:20:24] perhaps write/post about it somewhere. [20:21:07] Plenty of places to do that. Either org's blog could certainly have a post. [20:21:09] What's the project's name? [20:21:51] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3636.000000 [20:25:56] scfc_de, Susan: graphserv is the backend. I once wrote a frontend called CatGraph. [20:26:08] DanielK_WMDE_: It would be nice if it could receive some more publicity :-). [20:26:13] The backend code is on github: https://github.com/jkroll20/graphcore and https://github.com/jkroll20/graphserv [20:26:17] scfc_de: yes, indeed [20:26:51] aliasd on mayapple is CRITICAL: Connection refused [20:27:08] client code is on the toolserver: [20:27:10] https://svn.toolserver.org/svnroot/daniel/duesenstuff/trunk/gpClient/ [20:27:18] https://svn.toolserver.org/svnroot/daniel/duesenstuff/trunk/CatGraph/ [20:35:17] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8023.000000 [20:41:51] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:41:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [20:43:27] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:44:50] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9789 [20:46:18] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9841.000000 [20:46:57] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:49:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 79831 MB (1% inode=68%): [20:54:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:56:52] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 114366 MB (18% inode=99%): [21:07:27] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:13:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:16:50] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5060 [21:21:50] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5895.000000 [21:26:50] aliasd on mayapple is CRITICAL: Connection refused [21:35:18] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8244.000000 [21:41:52] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [21:41:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:43:38] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:44:51] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7356 [21:46:18] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7259.000000 [21:47:13] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:49:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 77859 MB (1% inode=68%): [21:54:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:56:52] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 114056 MB (18% inode=99%): [22:07:28] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:10:19] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3434.000000 [22:10:52] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3234 [22:13:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:16:51] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8360 [22:17:20] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1507.000000 [22:17:52] MySQL slave on z-dat-s5-b is OK: Uptime: 111608 Threads: 5 Questions: 20254616 Slow queries: 1310 Opens: 59386 Flush tables: 1 Open tables: 254 Queries per second avg: 181.479 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1335 [22:21:50] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8318.000000 [22:26:51] aliasd on mayapple is CRITICAL: Connection refused [22:35:19] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10347.000000 [22:36:36] @replag [22:36:36] DaBPunkt: s1-rr-a-wd: 33s [+0.00 s/s]; s1-user: 1m 50s [-0.00 s/s]; s1-user-wd: 34s [-]; s2-rr: 2h 33m 23s [+0.19 s/s]; s2-user: 2h 33m 23s [+0.19 s/s]; s2-user-c: error; s2-user-wd: 2h 51m 0s [+0.41 s/s]; s3-user: 10s [-0.00 s/s] [22:36:37] DaBPunkt: s3-user-wd: 31s [-]; s4-user-wd: 2h 30m 19s [+0.62 s/s]; s5-user-c: error; s6-user-wd: 1m 51s [+0.01 s/s]; s7-user-wd: 28s [-0.00 s/s] [22:41:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:41:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [22:43:38] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:48:10] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:49:12] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 59220 MB (6% inode=99%): [22:49:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 76381 MB (1% inode=68%): [22:50:10] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 58456 MB (5% inode=99%): [22:52:11] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 61919 MB (6% inode=99%): [22:54:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:57:52] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 113740 MB (18% inode=99%): [23:05:34] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3608 [23:05:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 113709 MB (18% inode=99%): [23:05:40] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:05:40] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:05:40] aliasd on mayapple is CRITICAL: Connection refused [23:05:40] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [23:05:49] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 0.000000 [23:07:59] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:09:29] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3598 [23:24:29] MySQL slave on z-dat-s2-b is OK: Uptime: 426503 Threads: 8 Questions: 662970336 Slow queries: 6518 Opens: 3939889 Flush tables: 1 Open tables: 256 Queries per second avg: 1554.432 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1581