[00:16:28] nacht ts [00:23:12] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [00:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:46:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 52954 MB (8% inode=99%): [00:46:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [00:47:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:47:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:47:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13303.000000 [00:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 27019 [00:49:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 27002.000000 [00:50:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:51:22] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14336 [00:51:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:52:01] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:52:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1300600.000000 [00:59:52] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5841.000000 [01:02:32] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 48661 MB (4% inode=99%): [01:23:12] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [01:46:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 51890 MB (8% inode=99%): [01:46:13] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [01:46:13] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:47:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:47:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:47:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14533.000000 [01:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 26889 [01:49:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 26865.000000 [01:50:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:51:22] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12299 [01:51:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:52:02] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:52:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1301784.000000 [01:59:53] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3769.000000 [02:03:53] wikidata replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3596.000000 [02:23:12] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [02:31:52] wikidata replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1737.000000 [02:46:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 52836 MB (8% inode=99%): [02:46:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [02:47:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:47:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:48:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14379.000000 [02:49:01] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 25888 [02:49:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25896.000000 [02:50:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:51:22] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7250 [02:51:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:52:02] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:52:53] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1300985.000000 [03:16:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:23:12] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [03:42:22] MySQL slave on z-dat-s1-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3509 [03:46:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 51933 MB (8% inode=99%): [03:46:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [03:46:52] Load avg. on rosemary is WARNING: WARNING - load average: 14.13, 15.11, 13.31 [03:47:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:47:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:47:41] [[Special:Log/newusers]] create 10 * Lzhpdcde * (New user account) [03:48:53] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11577.000000 [03:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 25270 [03:49:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25249.000000 [03:50:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:51:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:52:03] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:52:53] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1301680.000000 [03:52:53] Load avg. on rosemary is OK: OK - load average: 12.99, 14.79, 13.84 [04:05:22] MySQL slave on z-dat-s1-b is OK: Uptime: 125154 Threads: 2 Questions: 98757462 Slow queries: 10 Opens: 1357 Flush tables: 2 Open tables: 124 Queries per second avg: 789.87 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1737 [04:16:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:23:12] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [04:46:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 51868 MB (8% inode=99%): [04:46:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [04:47:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:47:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:48:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8787.000000 [04:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 25333 [04:49:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25327.000000 [04:51:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:51:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:52:03] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:53:53] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1300714.000000 [05:16:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:23:12] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [05:46:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 51725 MB (8% inode=99%): [05:46:11] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [05:47:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:47:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:48:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4688.000000 [05:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 25068 [05:49:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25030.000000 [05:51:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:51:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:52:02] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:53:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1288646.000000 [06:02:52] wikidata replag on z-dat-s2-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3519.000000 [06:06:53] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3646.000000 [06:16:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:19:52] wikidata replag on z-dat-s2-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3581.000000 [06:23:12] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [06:46:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 51618 MB (8% inode=99%): [06:46:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [06:47:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:47:23] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:48:52] wikidata replag on z-dat-s2-b is OK: QUERY OK: SELECT ts_rc_age() returned 1727.000000 [06:49:01] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 24463 [06:50:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 24528.000000 [06:51:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:51:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:52:02] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:53:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1272064.000000 [07:06:52] Load avg. on nightshade is WARNING: WARNING - load average: 17.50, 16.62, 14.94 [07:11:52] Load avg. on nightshade is OK: OK - load average: 13.40, 15.00, 14.74 [07:16:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:23:12] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [07:29:53] Load avg. on nightshade is WARNING: WARNING - load average: 17.81, 17.31, 15.61 [07:46:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 51498 MB (8% inode=99%): [07:46:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [07:47:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:47:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:47:52] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 24.59, 23.36, 20.16 [07:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 26431 [07:49:52] Load avg. on nightshade is WARNING: WARNING - load average: 18.23, 21.48, 19.86 [07:50:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 26474.000000 [07:51:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:52:03] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:52:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:54:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1264083.000000 [07:58:53] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 25.27, 21.10, 20.15 [08:16:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:18:52] Load avg. on nightshade is WARNING: WARNING - load average: 16.98, 17.93, 18.57 [08:23:12] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [08:46:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 51341 MB (8% inode=99%): [08:46:11] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [08:47:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:47:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 26175 [08:50:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 26170.000000 [08:51:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:52:02] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:52:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:54:53] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1264561.000000 [09:01:53] Load avg. on nightshade is OK: OK - load average: 11.66, 12.24, 14.78 [09:16:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:23:12] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [09:39:02] fisheye.toolserver.org on web.amaranth is OK: HTTP OK: HTTP/1.1 200 OK - 274 bytes in 14.192 second response time [09:46:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 51146 MB (8% inode=99%): [09:46:11] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [09:47:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:47:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 26293 [09:50:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 26372.000000 [09:51:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:52:02] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:52:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:54:53] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1264312.000000 [10:16:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:21:53] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2007.000000 [10:26:02] wikidata replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1993.000000 [10:46:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 50961 MB (8% inode=99%): [10:46:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [10:47:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:47:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 27768 [10:50:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 27823.000000 [10:51:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:52:02] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:52:22] Load avg. on ptolemy is WARNING: WARNING - load average: 12.73, 15.19, 10.52 [10:52:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:53:22] Load avg. on ptolemy is OK: OK - load average: 9.83, 13.94, 10.37 [10:55:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1266544.000000 [11:16:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:21:52] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2358.000000 [11:26:02] wikidata replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2040.000000 [11:32:16] DixonD: You can force your job to be executed on Solaris only (where oursql is installed) by specifying "-l arch=sol". Or you can install oursql to your home directory with "python setup.py install --user" and then force Linux use with "-l arch=lx". I prefer the latter. [11:40:03] wikidata replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1799.000000 [11:42:52] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1624.000000 [11:46:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 50728 MB (8% inode=99%): [11:46:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [11:47:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:47:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 28994 [11:50:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28940.000000 [11:51:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:52:02] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:52:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:55:49] SGE lives correctly if in the wrapper script I put other bash commands? (like creating a folder and setting a couple variables)? [11:55:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1266492.000000 [11:56:53] fale: What do you mean? [11:57:02] scfc_de: https://github.com/Fale/ToolServer/blob/master/sge/wrapper/2spazi.sh [11:59:28] fale: I don't see why it shouldn't work, but haven't tried it. [11:59:45] scfc_de: Ok :) thanks :) I'm going to try [12:03:10] scfc_de: SGE does not return any error, but the job is not taken in the grid :( [12:03:38] ok, my fault... I putted some syntax error -.- [12:03:43] Do you have a log file in your $HOME? [12:05:13] scfc_de: it seems that the "=" is not supported [12:06:15] fale: Oh, I didn't actually check the syntax :-). Use 'NDATE = $(date +"%Y%m%d")' (only "$NDATE" when you want to use the value in NDATE). [12:06:40] scfc_de: I guessed that :D [12:14:56] Unable to run job: denied: "2spazi" is not a valid object name (cannot start with a digit). <-- SGE has weird ideas :D [12:16:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:16:33] scfc_de`: it does not work :D [12:16:54] scfc_de`: SGE ignores the variable declaration part [12:18:13] Hmmm. [12:18:48] Can you upload your current wrapper script again? [12:19:23] scfc_de: same url, just refresh it ;) [12:20:20] scfc_de: got it, still a space [12:20:47] no spaces around the = [12:21:25] shell scripting has some syntax quirks [12:21:31] Platonides: No, I guess it's the one after the "+" in date :-). [12:21:40] scfc_de: yes, that one :) [12:22:07] scfc_de: in fact now it works properly :) [12:22:16] fale: Which I told you already some time ago: "Use 'NDATE = $(date +"%Y%m%d")'" :-) [12:22:19] Platonides: yep, sad, I think :( [12:22:30] scfc_de: yeah, a copy-paste error :D [12:22:56] Hey, cheer up, it's bash! Try to write POSIX shell for a change :-). [12:23:54] scfc_de: ahah, never tryied [12:46:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 50540 MB (8% inode=99%): [12:46:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [12:47:02] wikidata replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2064.000000 [12:47:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:47:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 29786 [12:50:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 29877.000000 [12:51:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:52:02] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:53:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:55:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1264953.000000 [12:58:56] Hello all [13:02:32] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 48749 MB (4% inode=99%): [13:16:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:22:12] hi DaBPunkt :) [13:32:39] hello DaB [13:36:02] wikidata replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1716.000000 [13:36:35] I added the re-setuped enwiki-server as a -rr-server for testing [13:45:26] @replag [13:45:26] DaBPunkt: s1-user: 8h 44s [-0.01 s/s]; s1-user-wd: 3m 0s [-0.04 s/s]; s2-rr: 1m 14s [-0.01 s/s]; s2-user: 1m 14s [-0.01 s/s]; s2-user-c: error; s2-user-wd: 1m 13s [-0.30 s/s]; s3-user: 1m 32s [+0.00 s/s]; s5-user-c: 2w 15h 13m 4s [-0.58 s/s] [13:46:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 50365 MB (8% inode=99%): [13:46:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [13:47:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:47:23] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 28476 [13:50:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28580.000000 [13:51:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:52:02] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:53:50] @replag [13:53:50] DaBPunkt: s1-rr-a: error; s1-user: 7h 52m 32s [-]; s2-user-c: error; s3-user: 1m 37s [-]; s5-user-c: 2w 15h 15m 22s [-] [13:53:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:55:53] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1264377.000000 [14:05:13] @replag [14:05:14] DaBPunkt: s1-rr-a: error; s1-user: 7h 34m 32s [-1.58 s/s]; s2-user-c: error; s3-user: 1m 3s [-0.05 s/s]; s5-user-c: 2w 15h 12m 42s [-0.23 s/s] [14:16:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:38:22] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [14:38:23] MySQL slave on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [14:46:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 50169 MB (8% inode=99%): [14:46:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [14:47:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:47:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 24489 [14:50:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 24289.000000 [14:51:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:52:03] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:53:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:55:53] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1264157.000000 [15:03:52] Load avg. on nightshade is WARNING: WARNING - load average: 17.58, 16.04, 14.99 [15:16:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:38:22] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [15:38:22] MySQL slave on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [15:45:52] Load avg. on nightshade is OK: OK - load average: 12.52, 13.99, 14.81 [15:46:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 49974 MB (8% inode=99%): [15:46:13] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [15:47:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:47:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 21564 [15:50:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21373.000000 [15:51:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:52:02] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:53:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:55:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1265066.000000 [16:16:06] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:38:22] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [16:38:22] MySQL slave on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [16:40:52] Load avg. on nightshade is WARNING: WARNING - load average: 15.23, 15.58, 14.38 [16:42:53] Load avg. on nightshade is OK: OK - load average: 13.33, 14.92, 14.29 [16:46:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 49750 MB (8% inode=99%): [16:46:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [16:47:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:47:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:49:03] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 16345 [16:51:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:51:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16030.000000 [16:52:02] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:53:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:55:53] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1265414.000000 [17:16:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:27:06] Hi, could you help me with transition from "newtask -p batch" to qcronsub? My crontab lines used to look like this: 1 1 * * * newtask -p batch env PYTHONPATH=$HOME/wikt/pywikipedia python $HOME/wikt/moje/frequencyListUpdate.py >/dev/null [17:38:23] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [17:38:23] MySQL slave on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [17:46:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 49560 MB (8% inode=99%): [17:46:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [17:47:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:47:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14809 [17:51:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:52:02] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:52:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14596.000000 [17:54:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:55:53] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1265489.000000 [18:14:13] I have a toolserver question: when you add more coordinates to an article that has the "show all the coordinates in a GoogleMap," does it add the coords automatically, and if so, how long does it take for that to happen? [18:16:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:16:17] alkamid: Your cron*ie*tab entry should look something like "1 1 * * * qcronsub -l h_rt=H:MM:SS -l virtual_free=xM env PYTHONPATH=$HOME/wikt/pywikipedia python $HOME/wikt/moje/frequencyListUpdate.py" where "H:MM:SS" is the maximum duration you expect this job to run and "xM" is the amount of memory it will consume. Add "PATH=/opt/ts/gnu/bin:/sge/GE/bin/sol-amd64:/usr/bin:/bin" as the first line so that qcronsub is found by cronie. I [18:16:17] left out "> /dev/null" as it can hide error information. [18:17:21] purplebackpack89: Do you have an example article? [18:17:38] [[San Gabriel Pastoral Region]] [18:17:50] see the history for my edits that added coords [18:22:04] purplebackpack89: Unfortunately, the source of that CGI isn't readable by other Toolserver users. You have to ask Para him-/herself (apparently http://en.wikipedia.org/wiki/User:Para). [18:35:24] that guy hasn't edited since 2011? [18:35:50] and you're telling me if his tool's broken, nobody else except him can fix it? [18:38:22] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [18:38:22] MySQL slave on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [18:46:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 49253 MB (8% inode=99%): [18:46:13] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [18:47:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:47:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11462 [18:51:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:52:02] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:52:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11139.000000 [18:54:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:56:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1268800.000000 [19:06:02] wikidata replag on z-dat-s6-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2161.000000 [19:06:02] wikidata replag on z-dat-s7-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2162.000000 [19:06:02] wikidata replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2163.000000 [19:06:02] wikidata replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2162.000000 [19:06:42] wikidata replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2202.000000 [19:06:52] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2215.000000 [19:06:53] wikidata replag on z-dat-s2-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2214.000000 [19:16:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:30:02] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3601.000000 [19:30:02] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3602.000000 [19:30:02] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3602.000000 [19:30:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3603.000000 [19:30:41] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3642.000000 [19:30:53] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3655.000000 [19:30:53] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3654.000000 [19:33:13] scfc_de, I changed it according to your advice and now I get "Unable to read script file because of error: error opening env: No such file or directory" [19:38:07] scfc_de, in fact if I just put "python $HOME/wikt/moje/script.py" there, I get "error opening python: No such file or directory" [19:38:22] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [19:38:22] MySQL slave on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [19:39:06] alkamid: then there is a problem with your PATH [19:39:41] DaBPunkt, I put "PATH=/opt/ts/gnu/bin:/sge/GE/bin/sol-amd64:/usr/bin:/bin" on the first line of my cronietab, as scfc_de told me [19:40:40] mm, a little bit weird, but it should work [19:41:00] how is your login-name? [19:41:22] and where is your cronietab? [19:42:37] DaBPunkt, it's alkamid [19:43:16] as for the location of cronietab... I don't really know, I just edit it with "cronie -e" [19:43:43] alkamid: On which host do you use cronie? [19:43:47] alkamid: there is not "script.py"-file in /home/alkamid/wikt/moje/ [19:44:10] DaBPunkt, sorry, I didn't mean actual script [19:44:18] in my cronietab I used "testy.py" [19:44:26] scfc_de, willow [19:46:03] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 48982 MB (8% inode=99%): [19:46:11] @notify purplebackpack89 [19:46:11] I will notify you, when I see purplebackpack89 around here [19:46:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [19:47:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:47:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:48:04] alkamid: indepentend of the actual problem: The line miss a name for the task (that's the "-N"-parameter) [19:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9673 [19:49:54] DaBPunkt, I read that in this case the default will be my script's name, which is fine for me [19:50:05] or should I add it anyway? [19:51:00] alkamid: the task is now named "python". I doubt that this is helpful ;) [19:51:23] DaBPunkt, oh sorry, I thought it would be "testy.py" [19:51:26] BTW: when you look into /home/alkamid/python.e1596012 you see that the task was executed, but that a module is missing [19:51:29] I'll add names [19:51:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:52:03] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:52:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9642.000000 [19:54:10] DaBPunkt, then my second line does not work as I thought it would... it's "PYTHONPATH=$HOME/wikt/pywikipedia" [19:54:41] basically I need to include a path to python modules in my home folder [19:54:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:55:15] alkamid: sory, but I'm not a python-programmer. So I have no idea how the variable is called [19:55:55] DaBPunkt, that's allright, thanks for your help anyway [19:56:02] DaBPunkt: That's just an environment variable. [19:56:02] it worked with the old cron and newtask [19:56:08] alkamid: What second line? [19:56:11] now I need to figure out how to change it (: [19:56:39] scfc_de, you told me to put "PATH=/opt/ts/gnu/bin:/sge/GE/bin/sol-amd64:/usr/bin:/bin" as the first line in my cronietab [19:56:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1272402.000000 [19:56:54] so I naively thought I could put another line with PYTHONPATH [19:57:58] alkamid: That sets the environment for the call to qcronsub, but this environment isn't passed on to the actual python call. [19:58:18] scfc_de, oh... [19:58:52] alkamid: So you still need env or use another way (install beneath $HOME/.lib or whatever the default location is). [19:59:15] In my eyes the cleanest way is a simple wapper-script [19:59:41] alkamid: $HOME/.local/lib/python2.6/site-packages/ [19:59:49] scfc_de, now my line looks like this: 59 19 * * * qcronsub -N 'testcron' -b y -l h_rt=0:03:00 -l virtual_free=3M "env PYTHONPATH=$HOME/wikt/pywikipedia /usr/bin/python $HOME/wikt/moje/testy.py" [19:59:52] it makes alsothe SGE-variables more readable [20:00:14] but I get an error "Unable to run job: Script not executable: env." [20:00:50] alkamid: Are there quotes from env to test.py? Don't use them. [20:01:22] Instead "... virtual_free=3M env PYTHONPATH=..." [20:02:21] scfc_de, removed the quotes, the same behaviour [20:02:26] not executable [20:03:50] well, I know there are probably other ways - I just wanted things to stay more or less the same [20:04:08] but I can move the folder or search for another way to add pythonpath [20:04:22] alkamid: I just installed "*/3 * * * * qcronsub -l h_rt=0:10:00 -l virtual_free=10M env PYTHONPATH=$HOME/wikt/pywikipedia python" on willow, and it doesn't give me any error. [20:05:07] alkamid: What's your cronie line *exactly*? [20:05:55] scfc_de, http://pastebin.ca/2335340 [20:06:01] ignore the lines in the middle [20:06:49] alkamid: Takes a moment to set up a test ... [20:10:15] alkamid: Okay, I take back the "give me any error", my mail queue was too large :-). But we'll figure this out. Moment. [20:10:40] hehe, thanks a lot for helping me! [20:12:02] alkamid: Okay, try replacing "env" with "/usr/bin/env". *fingerscross* [20:13:02] if you need help by me, please send me a ping [20:13:38] DaBPunkt, thanks [20:14:28] scfc_de, it works! [20:14:36] thank you [20:14:57] I don't really understand why I have to write the full path [20:15:01] alkamid: Nice. "Unable to run job: Script not executable: env." has got to be one the most misleading error messages. [20:15:02] but if it works... [20:15:49] scfc_de, also I got another funny error "Unable to run job: Script length does not match declared length." [20:15:55] but then added "-b y" [20:16:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:16:15] also I don't know why but it helped [20:16:16] Probably q(cron)sub doesn't store the whole path to the specified program, and when SGE then executes the job, it can't find it anymore. [20:17:09] Yeah, "-b y" is another of those things that you would expect to work out of the box. (And which I have to admit didn't mention above :-).) [20:21:50] scfc_de, good that it's working now! You've been very helpful, thanks a lot [20:22:00] have a good evening/day, depending on where you are (: [20:22:02] bye-bye [20:30:03] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7201.000000 [20:30:03] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7202.000000 [20:30:03] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7202.000000 [20:30:42] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7243.000000 [20:30:52] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7256.000000 [20:30:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7255.000000 [20:31:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7263.000000 [20:33:53] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2060.000000 [20:38:22] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [20:38:23] MySQL slave on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [20:46:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 48652 MB (7% inode=99%): [20:46:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [20:47:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:47:23] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:49:03] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8870 [20:51:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:52:02] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:52:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8762.000000 [20:54:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:56:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1276002.000000 [20:58:52] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1604.000000 [21:16:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:22:52] / on ortelius is WARNING: DISK WARNING - free space: / 6245 MB (20% inode=92%): [21:30:03] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10801.000000 [21:30:03] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10802.000000 [21:30:03] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10802.000000 [21:30:42] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10843.000000 [21:30:52] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10856.000000 [21:30:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10855.000000 [21:31:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10863.000000 [21:36:05] Marlen Caemmerer * [Toolserver-announce] ts.wikimedia.org cleaned up [21:38:22] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [21:38:22] MySQL slave on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [21:46:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 48356 MB (7% inode=99%): [21:46:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [21:47:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:47:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8774 [21:51:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:52:03] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:52:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8798.000000 [21:54:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:56:53] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1279602.000000 [22:16:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:19:52] / on ortelius is OK: DISK OK - free space: / 12279 MB (40% inode=92%): [22:30:02] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14401.000000 [22:30:02] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14403.000000 [22:30:02] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14402.000000 [22:30:41] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14442.000000 [22:30:52] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14456.000000 [22:30:53] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14455.000000 [22:31:03] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14464.000000 [22:38:22] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [22:38:22] MySQL slave on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [22:41:22] MySQL on z-dat-s1-b is OK: Uptime: 13740 Threads: 2 Questions: 6963 Slow queries: 0 Opens: 53 Flush tables: 1 Open tables: 46 Queries per second avg: 0.506 [22:46:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 48137 MB (7% inode=99%): [22:46:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [22:47:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:47:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9563 [22:51:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:52:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9266.000000 [22:53:02] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:54:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:56:53] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1283202.000000 [23:16:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:30:02] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15199.000000 [23:30:02] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13064.000000 [23:30:02] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12948.000000 [23:30:43] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12903.000000 [23:30:52] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4309.000000 [23:30:53] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14887.000000 [23:31:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17304.000000 [23:33:32] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [23:33:52] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3347.000000 [23:37:53] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1663.000000 [23:38:22] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 29474 [23:46:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 47843 MB (7% inode=99%): [23:46:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [23:47:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:47:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9731 [23:51:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:52:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9524.000000 [23:53:03] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:54:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:56:54] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1284423.000000