[00:00:13] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 58513 MB (5% inode=99%): [00:05:22] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:05:54] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 45793.000000 [00:07:03] aliasd on mayapple is CRITICAL: Connection refused [00:07:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [00:07:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:08:13] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 243254 MB (4% inode=67%): [00:08:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:08:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 109662 MB (18% inode=99%): [00:09:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:10:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:13:53] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 13570 [00:14:13] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 59499 MB (6% inode=99%): [00:16:54] MySQL slave on z-dat-s6-a is OK: Uptime: 2375648 Threads: 6 Questions: 1000510250 Slow queries: 115873 Opens: 3930831 Flush tables: 1 Open tables: 3393 Queries per second avg: 421.152 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 313 [00:18:33] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15915.000000 [00:24:53] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9665 [00:26:53] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6040 [00:28:54] wikidata replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3229.000000 [00:36:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:41:53] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3645.000000 [01:05:22] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:05:54] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 48111.000000 [01:07:03] aliasd on mayapple is CRITICAL: Connection refused [01:07:04] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [01:08:13] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 244937 MB (4% inode=67%): [01:08:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:08:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:08:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 109420 MB (18% inode=99%): [01:09:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:10:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:16:53] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2175 [01:17:42] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2219.000000 [01:18:33] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17187.000000 [01:24:53] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10052 [01:36:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:41:53] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3613 [01:41:53] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5205.000000 [01:42:43] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3658.000000 [01:49:03] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 23.99, 26.65, 18.40 [01:50:03] Load avg. on ortelius is WARNING: WARNING - load average: 18.48, 24.72, 18.25 [02:04:03] Load avg. on ortelius is OK: OK - load average: 11.82, 13.30, 14.87 [02:05:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:05:53] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50766.000000 [02:07:03] aliasd on mayapple is CRITICAL: Connection refused [02:07:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [02:08:13] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 244657 MB (4% inode=67%): [02:08:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:08:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:08:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 107408 MB (17% inode=99%): [02:09:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:10:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:18:33] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 20569.000000 [02:24:54] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 13496 [02:27:03] Load avg. on ortelius is WARNING: WARNING - load average: 18.62, 18.22, 15.88 [02:36:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:41:03] Load avg. on ortelius is OK: OK - load average: 13.97, 14.49, 14.95 [02:41:13] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 58475 MB (5% inode=99%): [02:41:53] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7073 [02:41:53] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5023.000000 [02:42:43] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7117.000000 [02:43:13] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 58945 MB (6% inode=99%): [02:48:03] Load avg. on ortelius is WARNING: WARNING - load average: 20.64, 19.35, 17.23 [02:57:13] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 58507 MB (5% inode=99%): [03:00:03] Load avg. on ortelius is OK: OK - load average: 3.74, 10.95, 14.89 [03:01:54] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2217.000000 [03:01:54] MySQL slave on daphne is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2220 [03:05:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:05:53] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 53765.000000 [03:07:03] aliasd on mayapple is CRITICAL: Connection refused [03:07:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [03:08:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 244256 MB (4% inode=67%): [03:08:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:08:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:09:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:09:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 109101 MB (17% inode=99%): [03:10:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:18:13] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 58608 MB (6% inode=99%): [03:18:42] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 24091.000000 [03:19:13] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 58506 MB (5% inode=99%): [03:21:13] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 58621 MB (6% inode=99%): [03:24:53] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 17010 [03:25:54] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3657.000000 [03:25:54] MySQL slave on daphne is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3660 [03:36:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:41:54] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8107 [03:41:54] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6596.000000 [03:42:42] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7996.000000 [04:00:54] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3561 [04:01:13] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 57842 MB (5% inode=99%): [04:01:42] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3140.000000 [04:04:43] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1566.000000 [04:04:53] MySQL slave on z-dat-s5-b is OK: Uptime: 218835 Threads: 5 Questions: 43821040 Slow queries: 2921 Opens: 100613 Flush tables: 1 Open tables: 255 Queries per second avg: 200.246 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1461 [04:05:22] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:05:54] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 56836.000000 [04:07:03] aliasd on mayapple is CRITICAL: Connection refused [04:07:04] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [04:08:13] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 244163 MB (4% inode=67%): [04:08:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:08:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:09:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:09:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 108971 MB (17% inode=99%): [04:10:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:18:43] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 27279.000000 [04:24:54] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 20252 [04:25:53] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7258.000000 [04:25:54] MySQL slave on daphne is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7260 [04:36:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:42:54] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7690.000000 [05:05:33] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:05:53] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 58983.000000 [05:07:03] aliasd on mayapple is CRITICAL: Connection refused [05:07:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [05:08:13] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 244107 MB (4% inode=67%): [05:08:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:08:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:09:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:09:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 108831 MB (17% inode=99%): [05:11:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:18:43] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 30059.000000 [05:24:53] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 22787 [05:25:54] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10858.000000 [05:25:54] MySQL slave on daphne is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10861 [05:36:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:43:54] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7843.000000 [05:45:12] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 58969 MB (6% inode=99%): [05:54:54] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3111.000000 [05:54:54] MySQL slave on daphne is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3045 [05:56:53] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1749.000000 [05:56:53] MySQL slave on daphne is OK: Uptime: 5990502 Threads: 47 Questions: 1463368440 Slow queries: 205153 Opens: 276929 Flush tables: 1 Open tables: 1639 Queries per second avg: 244.281 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1699 [06:05:33] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:05:53] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 60641.000000 [06:07:03] aliasd on mayapple is CRITICAL: Connection refused [06:07:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [06:08:14] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 243992 MB (4% inode=67%): [06:08:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:08:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:09:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:09:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 108688 MB (17% inode=99%): [06:11:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:18:43] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 31564.000000 [06:24:53] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 25052 [06:36:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:43:54] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6679.000000 [06:44:13] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 58501 MB (5% inode=99%): [06:52:13] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 59193 MB (6% inode=99%): [07:05:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:05:53] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 62775.000000 [07:07:03] aliasd on mayapple is CRITICAL: Connection refused [07:07:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [07:08:13] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 243826 MB (4% inode=67%): [07:08:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:08:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:09:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:09:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 108530 MB (17% inode=99%): [07:11:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:18:43] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 34371.000000 [07:24:53] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 28461 [07:36:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:38:13] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 58321 MB (5% inode=99%): [07:40:13] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 58526 MB (6% inode=99%): [07:41:13] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 58202 MB (5% inode=99%): [07:43:53] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5533.000000 [07:47:43] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1856.000000 [07:47:53] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1854 [08:05:33] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:05:57] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 65364.000000 [08:07:03] aliasd on mayapple is CRITICAL: Connection refused [08:07:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [08:08:13] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 243594 MB (4% inode=67%): [08:08:53] Load avg. on nightshade is WARNING: WARNING - load average: 16.59, 17.76, 12.95 [08:08:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:08:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:09:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:09:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 108296 MB (17% inode=99%): [08:11:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:11:13] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 58583 MB (6% inode=99%): [08:18:43] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 37858.000000 [08:21:53] wikidata replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3533.000000 [08:24:53] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 32044 [08:25:13] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 58189 MB (5% inode=99%): [08:26:13] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 58532 MB (6% inode=99%): [08:31:13] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 58486 MB (5% inode=99%): [08:36:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:43:43] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3605.000000 [08:43:54] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3612 [08:57:13] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 59158 MB (6% inode=99%): [08:58:13] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 58177 MB (5% inode=99%): [09:04:54] Load avg. on nightshade is OK: OK - load average: 8.63, 12.50, 14.94 [09:05:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:05:53] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 68305.000000 [09:07:03] aliasd on mayapple is CRITICAL: Connection refused [09:07:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [09:08:13] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 242783 MB (4% inode=67%): [09:09:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:09:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 108108 MB (17% inode=99%): [09:09:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:09:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:11:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:12:53] Load avg. on nightshade is WARNING: WARNING - load average: 16.02, 15.09, 15.25 [09:18:43] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 38562.000000 [09:21:54] wikidata replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2198.000000 [09:24:53] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 33884 [09:29:53] wikidata replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1787.000000 [09:30:54] Load avg. on nightshade is OK: OK - load average: 8.51, 12.69, 14.81 [09:43:43] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5193.000000 [09:43:53] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5201 [10:05:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:05:54] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 69079.000000 [10:06:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:07:03] aliasd on mayapple is CRITICAL: Connection refused [10:07:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [10:08:13] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 242814 MB (4% inode=67%): [10:09:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:09:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 107856 MB (17% inode=99%): [10:09:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:09:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:11:04] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:18:43] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 40714.000000 [10:24:53] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 37128 [10:27:13] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 58816 MB (6% inode=99%): [10:43:43] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6897.000000 [10:43:53] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6902 [11:05:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:05:53] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 71105.000000 [11:06:13] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 58477 MB (5% inode=99%): [11:06:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:07:03] aliasd on mayapple is CRITICAL: Connection refused [11:07:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [11:08:13] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 241599 MB (4% inode=67%): [11:09:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:09:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 107604 MB (17% inode=99%): [11:09:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:09:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:11:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:12:03] Load avg. on ortelius is WARNING: WARNING - load average: 12.91, 15.96, 13.64 [11:14:03] Load avg. on ortelius is OK: OK - load average: 10.36, 14.18, 13.27 [11:14:13] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 58626 MB (6% inode=99%): [11:15:13] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 57662 MB (5% inode=99%): [11:18:43] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 44129.000000 [11:24:53] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 40689 [11:35:13] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 58694 MB (6% inode=99%): [11:43:43] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8673.000000 [11:43:53] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8685 [11:44:13] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 58497 MB (5% inode=99%): [11:47:03] Load avg. on ortelius is WARNING: WARNING - load average: 14.67, 15.29, 14.78 [11:48:13] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 58538 MB (6% inode=99%): [11:49:03] Load avg. on ortelius is OK: OK - load average: 13.78, 14.77, 14.65 [11:49:13] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 58491 MB (5% inode=99%): [12:00:03] Load avg. on ortelius is WARNING: WARNING - load average: 14.96, 15.52, 15.17 [12:05:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:05:54] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 73785.000000 [12:06:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:07:03] aliasd on mayapple is CRITICAL: Connection refused [12:07:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [12:08:13] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 241096 MB (4% inode=67%): [12:09:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:09:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 107354 MB (17% inode=99%): [12:10:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:10:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:11:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:11:04] Load avg. on ortelius is OK: OK - load average: 11.91, 14.12, 14.85 [12:18:43] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 47051.000000 [12:24:53] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 44131 [12:25:14] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 57824 MB (5% inode=99%): [12:43:43] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11975.000000 [12:43:53] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11987 [12:46:13] /sql on thyme is WARNING: DISK WARNING - free space: /sql 92147 MB (9% inode=99%): [13:05:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:05:54] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 76781.000000 [13:07:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:07:03] aliasd on mayapple is CRITICAL: Connection refused [13:07:04] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [13:08:13] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 240728 MB (4% inode=67%): [13:09:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:09:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 107051 MB (17% inode=99%): [13:10:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:10:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:11:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:18:43] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50522.000000 [13:24:53] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 47679 [13:43:43] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15199.000000 [13:43:53] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 15212 [13:57:53] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [14:05:33] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:05:54] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 79543.000000 [14:07:03] aliasd on mayapple is CRITICAL: Connection refused [14:07:04] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [14:08:13] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 240327 MB (4% inode=67%): [14:09:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:09:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 106740 MB (17% inode=99%): [14:10:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:10:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:11:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:16:31] @replag [14:16:31] Betacommand: s1-rr-a-wd: 53s [+0.00 s/s]; s1-user: 28m 35s [+0.02 s/s]; s1-user-wd: 22m 13s [+0.02 s/s]; s2-rr: 13h 44m 1s [+0.57 s/s]; s2-user: 13h 44m 1s [+0.57 s/s]; s2-user-c: error; s2-user-wd: 14h 22m 43s [+0.60 s/s]; s3-user: 19s [-] [14:16:32] Betacommand: s3-user-wd: 45s [+0.00 s/s]; s4-user-wd: 22h 14m 48s [+0.59 s/s]; s5-user: 4h 42m 42s [-]; s5-user-c: error; s6-user-wd: 46s [-0.00 s/s]; s7-user-wd: 46s [+0.00 s/s] [14:18:43] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 51795.000000 [14:22:13] /sql on thyme is OK: DISK OK - free space: /sql 119163 MB (12% inode=99%): [14:24:53] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2210.000000 [14:24:54] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 49595 [14:36:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:43:43] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18096.000000 [14:43:53] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 18107 [14:48:59] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3649.000000 [14:51:03] APT on mayapple is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [14:52:53] wikidata replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2024.000000 [14:57:53] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [15:05:33] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:05:54] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 82295.000000 [15:07:03] aliasd on mayapple is CRITICAL: Connection refused [15:07:04] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [15:08:13] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 239915 MB (4% inode=67%): [15:09:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:09:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 106449 MB (17% inode=99%): [15:10:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:10:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:11:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:18:43] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 52418.000000 [15:24:53] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 51080 [15:27:53] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3617.000000 [15:36:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:43:43] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 20557.000000 [15:43:53] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 20559 [15:45:53] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2131.000000 [15:48:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7249.000000 [15:51:03] APT on mayapple is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [15:57:53] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [16:05:43] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:05:54] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 85202.000000 [16:07:03] aliasd on mayapple is CRITICAL: Connection refused [16:07:04] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [16:08:13] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 239169 MB (4% inode=67%): [16:09:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:09:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 106108 MB (17% inode=99%): [16:10:57] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:10:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:11:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:13:53] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3651.000000 [16:18:42] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54987.000000 [16:24:54] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 54413 [16:27:53] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6917.000000 [16:36:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:43:48] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22505.000000 [16:43:53] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 22510 [16:48:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10850.000000 [16:51:02] APT on mayapple is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [16:57:53] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [16:58:03] APT on yarrow is CRITICAL: APT CRITICAL: 5 packages available for upgrade (5 critical updates). [17:05:43] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:05:53] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 87680.000000 [17:07:04] aliasd on mayapple is CRITICAL: Connection refused [17:07:04] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [17:08:13] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 238808 MB (4% inode=67%): [17:09:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:09:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 105847 MB (17% inode=99%): [17:10:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:10:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:11:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:13:53] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6725.000000 [17:18:43] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 57555.000000 [17:24:54] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 57727 [17:27:53] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10212.000000 [17:36:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:43:54] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25536.000000 [17:43:54] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 25540 [17:48:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14450.000000 [17:51:03] Load avg. on ortelius is WARNING: WARNING - load average: 16.62, 16.93, 13.57 [17:51:03] APT on mayapple is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [17:56:02] Load avg. on ortelius is OK: OK - load average: 10.07, 14.36, 13.49 [17:57:54] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [17:58:03] APT on yarrow is CRITICAL: APT CRITICAL: 5 packages available for upgrade (5 critical updates). [17:59:54] APT on nightshade is CRITICAL: APT CRITICAL: 5 packages available for upgrade (5 critical updates). [18:05:43] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:05:53] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 89805.000000 [18:07:03] aliasd on mayapple is CRITICAL: Connection refused [18:07:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [18:08:14] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 238103 MB (4% inode=67%): [18:09:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:09:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 105517 MB (17% inode=99%): [18:10:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:10:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:11:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:13:53] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10196.000000 [18:18:43] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 60384.000000 [18:19:36] [[Special:Log/newusers]] create 10 * Shsm * (New user account) [18:24:53] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 59410 [18:27:54] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13489.000000 [18:36:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:37:03] Load avg. on ortelius is WARNING: WARNING - load average: 19.19, 17.65, 14.93 [18:43:55] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28207.000000 [18:43:55] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 28211 [18:48:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18050.000000 [18:51:03] APT on mayapple is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [18:57:53] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [18:58:03] APT on yarrow is CRITICAL: APT CRITICAL: 5 packages available for upgrade (5 critical updates). [18:59:53] APT on nightshade is CRITICAL: APT CRITICAL: 5 packages available for upgrade (5 critical updates). [19:05:43] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:05:54] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 92336.000000 [19:07:03] aliasd on mayapple is CRITICAL: Connection refused [19:07:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [19:08:13] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 237836 MB (4% inode=67%): [19:09:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:09:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 105212 MB (17% inode=99%): [19:10:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:11:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:11:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:13:03] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 42.26, 27.18, 21.39 [19:13:53] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13642.000000 [19:18:43] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 62907.000000 [19:24:54] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 60968 [19:27:53] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16786.000000 [19:36:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:43:53] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 30543.000000 [19:43:54] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 30546 [19:44:19] why qcronsub doesn't working? :( [19:48:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21650.000000 [19:51:03] APT on mayapple is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [19:57:54] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [19:58:03] APT on yarrow is CRITICAL: APT CRITICAL: 5 packages available for upgrade (5 critical updates). [19:59:54] APT on nightshade is CRITICAL: APT CRITICAL: 5 packages available for upgrade (5 critical updates). [20:04:03] Load avg. on ortelius is WARNING: WARNING - load average: 17.10, 18.56, 19.84 [20:05:43] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:05:55] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 94918.000000 [20:07:03] aliasd on mayapple is CRITICAL: Connection refused [20:07:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [20:08:03] aliasd on mayapple is OK: TCP OK - 0.009 second response time on port 984 [500 Not found.] [20:08:13] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 237517 MB (4% inode=67%): [20:09:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:09:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 104982 MB (17% inode=99%): [20:10:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:11:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:11:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:13:45] hi Alchimista! :) [20:13:53] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16316.000000 [20:13:56] hey Gnumarcoo [20:18:42] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 64126.000000 [20:24:54] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 57980 [20:27:54] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18114.000000 [20:36:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:43:54] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 24317.000000 [20:43:54] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 24305 [20:48:54] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25250.000000 [20:51:13] APT on mayapple is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:54:53] APT on yucca is CRITICAL: APT CRITICAL: 5 packages available for upgrade (5 critical updates). [20:56:03] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 22.24, 21.21, 20.00 [20:57:54] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [20:58:03] APT on yarrow is CRITICAL: APT CRITICAL: 5 packages available for upgrade (5 critical updates). [20:58:07] Gnumarcoo: Has your qcronsub problem been solved? [20:58:23] scfc_de, nope [20:58:31] I'm trying to solve it... [20:58:35] What's wrong? [20:59:54] APT on nightshade is CRITICAL: APT CRITICAL: 5 packages available for upgrade (5 critical updates). [21:00:45] scfc_de: so, I'm trying to set a periodic task via qcronsub... but it does... nothing. [21:01:15] What do you do, where do you do it, how do you do it? :-) You have to be a bit more specific. [21:02:24] @hawthorn:~$ qcronsub -b y -wd /home/marcoo/pywikipediabot -l h_rt=0:15:00 -l virtual_free=100M "/home/marcoo/pywikipediabot/welcome.py -nlog -edit:0 -break -limit:250 -filter -random -savedata" [21:04:07] What does "qstat" immediately after that say? Is a job queued? [21:04:39] nothing: [21:04:46] Do you have log files in your home directory? (*.e1234 or something like that) [21:05:12] how can I see my log? [21:05:13] Free Memory on turnera is WARNING: WARNING - 6.2% (519064 kB) free! [21:05:33] For example with "less LOGFILE" [21:05:43] Could you try the command without quotes (")? [21:05:43] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:05:46] (excuse me, but I've very little knowledge of ts [21:05:53] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 96539.000000 [21:06:13] Free Memory on turnera is OK: OK - 7.4% (620536 kB) free. [21:06:22] scfc_de, without " it returns same result... [21:06:32] And qstat? [21:07:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [21:07:22] You're "marcoo" on TS? Then there's already a job running for you since 19:34Z. [21:07:34] scfc_de, qstat says that there is a job on queue [21:07:37] yeah [21:07:44] but it doesn't work. [21:07:45] qcronsub will not execute another instance of that until the first one finishes. [21:08:13] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 237104 MB (4% inode=67%): [21:08:21] Do you want to kill the first job? I think you can do that with "qdel job-ID". [21:08:29] ok [21:09:54] I've restart it... now I see if it works... [21:10:34] Hold on, the state for the job at 19:34Z was "Eqw" which seems to mean "Error", "queued" (?), "waiting". Merlissimo, under which circumstances does that occur? [21:10:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:10:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 104597 MB (17% inode=99%): [21:10:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:10:56] (now is qw) [21:11:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:11:19] Merlissimo: For reference: "1527024 1.00000 welcome.py marcoo Eqw 03/09/2013 19:34:26 1" [21:11:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:12:40] scfc_de, otherwise I'll edit job queue via cornie [21:12:43] *cronie [21:13:01] What do you mean by "edit job queue"? [21:13:54] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5016.000000 [21:14:00] edit crontab... [21:15:11] I don't quite understand what you mean. BTW, your job seems to have finished already. Was it successful? [21:15:31] no... [21:15:52] Do you have the job ID? [21:16:06] 1527564 [21:17:18] "qacct -j 1527564" says "exit_status 2". Do you have log files in your home directory for that job? [21:18:42] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 64216.000000 [21:19:53] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3577.000000 [21:22:31] scfc_de, no, i'm sorry... no logs [21:24:15] What does happen when you execute "/home/marcoo/pywikipediabot/welcome.py -nlog -edit:0 -break -limit:250 -filter -random -savedata" in /home/marcoo/pywikipediabot? [21:24:53] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1591.000000 [21:24:54] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 54018 [21:25:59] np [21:26:15] I'll solve it by myself.. tnx scfc_de [21:26:43] Okay. [21:27:58] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12677.000000 [21:36:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:43:54] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11292.000000 [21:43:54] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11269 [21:48:54] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28849.000000 [21:51:13] APT on mayapple is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [21:54:54] APT on yucca is CRITICAL: APT CRITICAL: 5 packages available for upgrade (5 critical updates). [21:56:14] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 23.03, 22.03, 23.04 [21:57:53] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [21:58:03] APT on yarrow is CRITICAL: APT CRITICAL: 5 packages available for upgrade (5 critical updates). [21:59:54] APT on nightshade is CRITICAL: APT CRITICAL: 5 packages available for upgrade (5 critical updates). [22:05:43] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:05:53] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 94540.000000 [22:07:13] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [22:08:13] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 236522 MB (4% inode=67%): [22:10:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:10:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 104310 MB (17% inode=99%): [22:10:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:11:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:11:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:12:53] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3150.000000 [22:12:54] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3134 [22:13:13] Free Memory on turnera is WARNING: WARNING - 6.6% (551712 kB) free! [22:14:13] Load avg. on ortelius is WARNING: WARNING - load average: 16.41, 17.85, 19.91 [22:18:42] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 63721.000000 [22:19:53] MySQL slave on z-dat-s5-b is OK: Uptime: 284536 Threads: 4 Questions: 53356434 Slow queries: 6737 Opens: 118810 Flush tables: 1 Open tables: 256 Queries per second avg: 187.520 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1795 [22:20:13] Free Memory on turnera is OK: OK - 7.4% (618600 kB) free. [22:20:54] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1406.000000 [22:22:13] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 24.34, 21.02, 20.41 [22:25:54] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 50248 [22:27:54] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9504.000000 [22:36:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:48:54] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32450.000000 [22:51:13] APT on mayapple is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [22:51:13] Free Memory on turnera is WARNING: WARNING - 6.0% (500796 kB) free! [22:54:53] APT on yucca is CRITICAL: APT CRITICAL: 5 packages available for upgrade (5 critical updates). [22:57:53] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [22:58:03] APT on yarrow is CRITICAL: APT CRITICAL: 5 packages available for upgrade (5 critical updates). [22:58:13] Free Memory on turnera is CRITICAL: CRITICAL - 4.6% (389388 kB) free! [22:59:53] APT on nightshade is CRITICAL: APT CRITICAL: 5 packages available for upgrade (5 critical updates). [23:03:38] Free Memory on turnera is OK: OK - 87.8% (7355880 kB) free. [23:06:57] Free Memory on turnera is OK: OK - 86.4% (7244056 kB) free. [23:07:58] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [23:08:57] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 236156 MB (4% inode=67%): [23:11:07] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:11:17] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:11:41] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 104047 MB (17% inode=99%): [23:11:47] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:12:27] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:19:07] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 64413.000000 [23:21:57] Load avg. on ortelius is WARNING: WARNING - load average: 17.93, 19.27, 19.93 [23:26:37] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 48540 [23:28:37] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6756.000000 [23:36:57] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:43:57] wikidata replag on z-dat-s6-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2059.000000 [23:47:37] wikidata replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3468.000000 [23:49:28] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36081.000000 [23:51:57] APT on mayapple is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [23:55:17] APT on yucca is CRITICAL: APT CRITICAL: 5 packages available for upgrade (5 critical updates). [23:57:38] wikidata replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1756.000000 [23:57:57] Load avg. on ortelius is OK: OK - load average: 9.75, 12.19, 14.88 [23:58:47] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [23:58:47] APT on yarrow is CRITICAL: APT CRITICAL: 5 packages available for upgrade (5 critical updates).