[00:01:33] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:02:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:02:33] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:06:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:06:05] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:06:05] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 465920 MB (8% inode=64%): [00:06:47] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13649.000000 [00:06:47] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13648.000000 [00:06:47] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16543.000000 [00:08:47] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 79742.000000 [00:08:47] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 103157 [00:08:47] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 56618 MB (9% inode=99%): [00:08:47] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 143072.000000 [00:08:47] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [00:08:53] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [00:15:47] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 103205.000000 [00:31:47] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:43:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:43:47] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:50:54] RAID on ptolemy is CRITICAL: ERROR - TOTAL: 3: FAILED: 0: DEGRADED: 1 [00:54:34] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22784.000000 [01:01:33] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:02:33] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:03:25] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:06:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:06:04] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:06:04] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 466110 MB (8% inode=64%): [01:06:47] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13911.000000 [01:06:47] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13909.000000 [01:06:47] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17928.000000 [01:08:47] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 79844.000000 [01:08:47] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 104106 [01:08:47] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 56421 MB (9% inode=99%): [01:08:47] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 144041.000000 [01:08:47] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [01:08:54] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [01:09:14] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 30836 MB (3% inode=99%): [01:15:47] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 104310.000000 [01:31:47] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:43:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:43:47] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:50:54] RAID on ptolemy is CRITICAL: ERROR - TOTAL: 3: FAILED: 0: DEGRADED: 1 [01:54:34] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22313.000000 [02:01:34] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:02:33] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:04:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:06:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:06:04] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:06:04] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 465943 MB (8% inode=64%): [02:06:47] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11750.000000 [02:06:47] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11748.000000 [02:06:47] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19554.000000 [02:08:54] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [02:09:47] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 106174 [02:09:47] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 79209.000000 [02:09:47] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 54327 MB (8% inode=99%): [02:09:48] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 144280.000000 [02:09:48] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [02:15:47] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 106438.000000 [02:43:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:43:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:50:53] RAID on ptolemy is CRITICAL: ERROR - TOTAL: 3: FAILED: 0: DEGRADED: 1 [02:53:47] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1802.000000 [02:54:34] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 23380.000000 [02:56:46] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1753.000000 [03:01:34] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:01:46] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:02:33] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:02:54] [[Special:Log/newusers]] create 10 * Fakegrass * (New user account) [03:04:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:06:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:06:05] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:06:05] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 465960 MB (8% inode=64%): [03:06:25] [[User talk:Fakegrass]] !N 10https://wiki.toolserver.org/w/index.php?oldid=7893&rcid=21730 * Fakegrass * (+1902) (Created page with "There is no question that in developed nations, synthetic turf has actually been widely used for several various functions. Nevertheless, in creating countries, fake yard stays a...") [03:07:46] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8401.000000 [03:07:46] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13781.000000 [03:07:46] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19318.000000 [03:08:55] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [03:09:54] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [03:10:41] [[Special:Log/delete]] delete 10 * MZMcBride * (deleted "[[02User talk:Fakegrass10]]": spam) [03:10:46] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 107351 [03:10:46] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 78850.000000 [03:10:46] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 56096 MB (9% inode=99%): [03:10:46] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 143692.000000 [03:10:53] [[Special:Log/block]] block 10 * MZMcBride * (blocked [[02User:Fakegrass10]] with an expiry time of infinite (account creation disabled): spam) [03:16:45] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 107312.000000 [03:43:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:43:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:50:54] RAID on ptolemy is CRITICAL: ERROR - TOTAL: 3: FAILED: 0: DEGRADED: 1 [03:54:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 23101.000000 [04:01:34] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:01:45] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:02:34] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:04:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:06:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:06:06] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:06:06] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 463388 MB (8% inode=64%): [04:08:45] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6293.000000 [04:08:46] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17442.000000 [04:08:47] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16281.000000 [04:09:54] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [04:09:54] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [04:10:46] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 108466 [04:10:46] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 55971 MB (9% inode=99%): [04:10:46] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 79621.000000 [04:10:46] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 143739.000000 [04:16:46] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 108557.000000 [04:30:59] [[Wiki server assignments]] ! 10https://wiki.toolserver.org/w/index.php?diff=7894&oldid=7885&rcid=21733 * 91.198.174.202 * (+4) (updated page) [04:43:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:43:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:50:54] RAID on ptolemy is CRITICAL: ERROR - TOTAL: 3: FAILED: 0: DEGRADED: 1 [04:55:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 20470.000000 [05:01:44] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:02:45] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:04:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:06:14] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 458996 MB (8% inode=64%): [05:07:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:07:05] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:09:45] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4340.000000 [05:09:45] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21102.000000 [05:09:45] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17547.000000 [05:09:54] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [05:09:54] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [05:10:55] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 144806.000000 [05:11:45] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 110398 [05:11:45] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 55847 MB (9% inode=99%): [05:11:45] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 80117.000000 [05:17:46] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 110627.000000 [05:31:55] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:34:45] wikidata replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3584.000000 [05:44:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:44:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:50:55] RAID on ptolemy is CRITICAL: ERROR - TOTAL: 3: FAILED: 0: DEGRADED: 1 [05:55:34] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17372.000000 [06:01:46] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:02:45] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:05:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:06:14] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 453170 MB (8% inode=64%): [06:07:14] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:08:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:09:45] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 24702.000000 [06:09:45] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21148.000000 [06:09:55] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [06:09:55] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [06:11:44] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 113147 [06:11:45] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 55717 MB (9% inode=99%): [06:11:45] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 80182.000000 [06:11:55] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 146785.000000 [06:12:15] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:12:45] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 42660 MB (8% inode=99%): [06:17:45] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 113508.000000 [06:34:45] wikidata replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1844.000000 [06:36:45] wikidata replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1771.000000 [06:44:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:44:56] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:50:54] RAID on ptolemy is CRITICAL: ERROR - TOTAL: 3: FAILED: 0: DEGRADED: 1 [06:55:34] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15481.000000 [07:01:44] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:02:45] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:02:45] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:05:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:06:14] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 448721 MB (8% inode=64%): [07:07:14] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:08:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:09:45] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28302.000000 [07:09:54] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 24752.000000 [07:09:54] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [07:09:54] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [07:11:45] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 116036 [07:11:45] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 55512 MB (9% inode=99%): [07:11:45] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 79857.000000 [07:12:55] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 148613.000000 [07:17:45] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 116397.000000 [07:44:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:44:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:51:05] RAID on ptolemy is CRITICAL: ERROR - TOTAL: 3: FAILED: 0: DEGRADED: 1 [07:56:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13645.000000 [07:56:34] Load avg. on nightshade is WARNING: WARNING - load average: 15.23, 17.51, 14.40 [08:01:44] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:02:45] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:02:45] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:05:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:06:14] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 448407 MB (8% inode=64%): [08:07:15] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:08:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:08:34] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 55.11, 33.35, 22.13 [08:09:45] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 26041.000000 [08:09:55] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 23303.000000 [08:09:55] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [08:09:55] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [08:11:45] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 118134 [08:11:45] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 55300 MB (9% inode=99%): [08:11:45] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 79318.000000 [08:13:34] Load avg. on nightshade is WARNING: WARNING - load average: 15.78, 21.62, 20.00 [08:13:54] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 150390.000000 [08:17:45] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 118317.000000 [08:24:12] [[Special:Log/newusers]] create 10 * فولف للمقاولات * (New user account) [08:45:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:45:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:47:33] Load avg. on nightshade is OK: OK - load average: 11.01, 13.05, 14.98 [08:51:05] RAID on ptolemy is CRITICAL: ERROR - TOTAL: 3: FAILED: 0: DEGRADED: 1 [08:56:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11670.000000 [09:01:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:02:45] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:02:45] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:06:14] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 446885 MB (8% inode=64%): [09:06:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:08:14] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:09:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:09:45] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 23977.000000 [09:10:53] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21367.000000 [09:10:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [09:10:53] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [09:11:45] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 55107 MB (9% inode=99%): [09:11:45] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 118816 [09:11:45] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 80957.000000 [09:14:55] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 151875.000000 [09:16:33] Load avg. on nightshade is WARNING: WARNING - load average: 16.27, 15.71, 14.94 [09:17:45] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 118747.000000 [09:19:46] [[Special:Log/newusers]] create 10 * Rob Kam * (New user account) [09:24:34] Load avg. on nightshade is OK: OK - load average: 10.81, 13.95, 14.74 [09:46:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:46:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:52:05] RAID on ptolemy is CRITICAL: ERROR - TOTAL: 3: FAILED: 0: DEGRADED: 1 [09:56:34] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9644.000000 [10:03:45] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:03:45] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:06:13] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 447717 MB (8% inode=64%): [10:06:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:09:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:09:14] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:09:44] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22082.000000 [10:10:54] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19132.000000 [10:10:55] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [10:10:55] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [10:11:45] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 120446 [10:11:46] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 54906 MB (8% inode=99%): [10:11:46] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 80498.000000 [10:14:58] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 153089.000000 [10:17:45] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 120584.000000 [10:31:45] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:46:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:46:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:52:04] RAID on ptolemy is CRITICAL: ERROR - TOTAL: 3: FAILED: 0: DEGRADED: 1 [10:56:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6327.000000 [11:03:45] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:03:45] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:05:04] RAID on ptolemy is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [11:06:14] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 447229 MB (8% inode=64%): [11:06:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:09:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:09:13] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:10:45] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19834.000000 [11:10:55] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17283.000000 [11:10:55] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [11:10:55] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [11:11:45] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 121966 [11:11:46] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 82240.000000 [11:12:14] /sql on ptolemy is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:14:54] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 155684.000000 [11:17:45] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 122044.000000 [11:31:45] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:37:44] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 91250 MB (15% inode=99%): [11:46:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:46:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:56:34] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3914.000000 [12:03:45] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:03:45] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:04:34] wikidata replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3583.000000 [12:06:14] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 446933 MB (8% inode=64%): [12:07:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:09:14] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:10:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:10:45] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17842.000000 [12:11:44] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 123255 [12:11:44] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 82368.000000 [12:11:54] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14521.000000 [12:11:54] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [12:11:54] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [12:14:54] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 156709.000000 [12:17:45] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 123187.000000 [12:31:55] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:32:08] @replag [12:32:09] hi there [12:32:18] mdupont: hay [12:32:28] where can i edit the crontab on willow? [12:32:36] i ahve a process going crazy [12:32:42] and it is getting slayed [12:32:47] i am trying to disable it [12:32:49] crontab -e ? [12:33:24] oh now it works [12:33:30] before it gave me an error [12:33:31] :( [12:33:45] of course that is the standard i used it before [12:34:27] thanks Betacommand [12:34:35] mdupont: np [12:38:46] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 91000 MB (14% inode=99%): [12:46:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:46:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:56:34] wikidata replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1649.000000 [13:03:45] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:03:45] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:06:14] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 445367 MB (8% inode=64%): [13:07:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:09:14] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 31022 MB (3% inode=99%): [13:10:13] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:11:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:11:43] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14515.000000 [13:12:03] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [13:12:44] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 123835 [13:12:44] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 84619.000000 [13:12:53] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11911.000000 [13:12:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [13:15:53] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 156642.000000 [13:18:43] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 123924.000000 [13:28:23] aliasd on yarrow is CRITICAL: Connection refused [13:39:44] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 90798 MB (14% inode=99%): [13:47:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:47:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:01:44] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:03:45] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:03:45] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:06:14] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 442598 MB (8% inode=64%): [14:08:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:11:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:11:12] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:11:44] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12245.000000 [14:12:03] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [14:12:53] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 86908.000000 [14:12:54] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9558.000000 [14:12:54] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [14:13:43] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 124800 [14:15:53] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 157917.000000 [14:18:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 124997.000000 [14:28:25] aliasd on yarrow is CRITICAL: Connection refused [14:39:45] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 90539 MB (14% inode=99%): [14:47:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:47:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:01:43] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:04:44] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:04:44] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:07:13] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 441828 MB (8% inode=64%): [15:09:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:11:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:11:14] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:11:44] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10307.000000 [15:12:05] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [15:12:54] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 88018.000000 [15:12:55] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8212.000000 [15:12:55] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [15:13:44] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 124700 [15:15:54] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 157493.000000 [15:17:33] wikidata replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1987.000000 [15:18:54] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 124881.000000 [15:24:23] @replag [15:27:34] Betacommand: s1-rr-a-wd: 1d 38m 55s [+0.54 s/s]; s1-user: 1d 10h 45m 11s [+0.14 s/s]; s1-user-wd: 1d 19h 44m 26s [+0.05 s/s]; s2-rr: 28m 14s [+0.14 s/s]; s2-user: 28m 14s [+0.14 s/s]; s2-user-c: error; s2-user-wd: 1d 38m 55s [+0.54 s/s]; s3-user-wd: 2h 53m 52s [-0.18 s/s] [15:27:35] Betacommand: s4-user-wd: 39m 52s [-]; s5-rr-a-wd: 3m 27s [-]; s6-user-wd: 2h 9m 32s [-0.54 s/s]; s7-user-wd: 2h 48m 46s [-0.62 s/s] [15:28:24] aliasd on yarrow is CRITICAL: Connection refused [15:36:23] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2136 [15:39:44] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 90315 MB (14% inode=99%): [15:47:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:47:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:58:22] hello all [16:00:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3628.000000 [16:01:45] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:03:28] DaBPunkt: Da? [16:03:42] was gibts? [16:04:24] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3641 [16:04:24] auf sql-s3-rr eiert eine sehr invollständige centralauth Kopie rum [16:04:32] mysql> SELECT MAX( lu_attached_timestamp ) FROM centralauth_p.localuser; [16:04:36] 20101205144919 [16:05:19] ich werde den view entfernen. Das wird noch aus der zeit sein als centralauth auf s3 war. [16:05:40] Ja, bitte... das ist unnötig verwirrend [16:05:44] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:05:44] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:05:45] (wenn jemand nix vorhat kann er die wmf mal bitten solche altbestände zu droppen…) [16:06:06] Werde Chris drauf ansprechen, wenn ich dran denke [16:07:13] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 441120 MB (8% inode=64%): [16:08:14] DaBPunkt: any eta for lag issues on s1? [16:09:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:10:30] Betacommand: sql-s1-rr is lag-free since some days. sql-s1-user is stagnating AFAIS. I will look if it is swapping and announce a reboot if needed [16:11:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:11:12] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:11:44] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9089.000000 [16:12:54] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7068.000000 [16:12:54] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 91201.000000 [16:12:54] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [16:13:04] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [16:14:43] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 123989 [16:15:57] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 157880.000000 [16:18:54] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 123976.000000 [16:28:24] aliasd on yarrow is CRITICAL: Connection refused [16:28:28] @replag [16:31:41] Betacommand: s1-rr-a-wd: 1d 1h 36m 51s [+0.90 s/s]; s1-user: 1d 10h 31m 17s [-0.22 s/s]; s1-user-c: 4m 45s [-]; s1-user-wd: 1d 19h 58m 20s [+0.22 s/s]; s2-rr: 1h 20m 46s [+0.82 s/s]; s2-user: 1h 20m 46s [+0.82 s/s]; s2-user-c: error; s2-user-wd: 1d 1h 36m 51s [+0.90 s/s] [16:31:42] Betacommand: s3-user: 12s [-0.00 s/s]; s3-user-wd: 2h 29m 19s [-0.38 s/s]; s4-user-wd: 1h 13m 42s [+0.53 s/s]; s5-rr-a-wd: 10m 58s [+0.12 s/s]; s6-user-wd: 1h 57m 45s [-0.18 s/s]; s7-user-wd: 2h 24m 31s [-0.38 s/s] [16:39:45] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 90110 MB (14% inode=99%): [16:48:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:48:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:00:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5191.000000 [17:01:44] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:01:50] sorry, there was a mistake in tsbot-config about s1-rr-a-wd [17:01:55] @replag [17:04:24] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5413 [17:05:44] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:05:45] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:08:13] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 440434 MB (8% inode=64%): [17:08:50] DaBPunkt, how do I kill a task? [17:09:04] I need to know now. [17:09:10] Cyberpower678: a process or a SGE-job? [17:09:26] It's from a crontab in willow. [17:09:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:09:44] Cyberpower678: a process or a SGE-job? [17:10:00] Process I think. [17:10:23] do you have the pid? [17:10:26] if not: [17:10:39] ps -ef|grep *insertNameHere* [17:11:14] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:11:40] the pid is the first number in every line [17:11:44] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8736.000000 [17:11:49] kill *pid* [17:11:53] kills it [17:12:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:12:08] if it doesn't stop, try kill -9 *pid* [17:12:14] thanks. [17:12:28] Wow I have 5 instances. [17:12:46] be careful, one of them is your shell ;) [17:13:03] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [17:13:24] DaBPunkt, which one? [17:13:45] the one with "bash" [17:13:53] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7227.000000 [17:13:53] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 93712.000000 [17:13:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [17:13:55] which login-name do you have? [17:14:09] cyberpower678 [17:14:43] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 124009 [17:15:29] Cyberpower678: 22437 is your shell. 21466 is your ssh-instanz and 22433 and 15629 also. 22665 is a bot. [17:15:56] 21490 and 15635 are sftp-transport-clients [17:16:03] Hello [17:16:07] 22672 is another bot [17:16:14] What happens if i kill them? [17:16:14] Are there any tool devs around? [17:16:18] I have a little problem [17:16:21] hello Qcoder00 [17:16:30] * DaBPunkt has to leave for dinner [17:16:33] bye [17:16:47] Is there a tool on toolserver to find image captions? [17:16:53] Cyberpower678: if you kill them your sftp session will get closed [17:16:53] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 159857.000000 [17:17:02] Qcoder00: probably not, since that involves page text [17:17:25] legoktm, that's not so bad. I can easily reopen that. [17:17:36] $kill pid then :) [17:17:40] It was that adding image descriptions at en would be aided by being able to pull descriptions from image captioning [17:18:10] legoktm, will that irreversibly kick me from toolserver. [17:18:11] ? [17:18:12] This could either be a script, but a Toolserver tool might work better as it could be generalised to any project [17:18:25] Cyberpower678: not irreversibly [17:18:34] it might kick you though [17:18:55] will logging back in fix it or does DaB. have to restart something. [17:19:07] oh youll be able to just log back in [17:19:26] Cyberbot II is running rampant. [17:19:42] i know [17:19:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 123874.000000 [17:19:56] it keeps popping up on my watchlist :P [17:22:52] It'll popup even more in just a moment. I am having it wipe the pp-pc1/2 templates and retagging them to clean up a mess Cyberbot II created when Wikipedia failed. [17:23:23] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3523 [17:23:54] yes. it's killed [17:24:11] :) [17:26:25] bbl in a bit [17:26:29] dont break anything :P [17:28:24] aliasd on yarrow is CRITICAL: Connection refused [17:36:23] MySQL slave on z-dat-s2-b is OK: Uptime: 411194 Threads: 12 Questions: 650076000 Slow queries: 6525 Opens: 3745174 Flush tables: 1 Open tables: 256 Queries per second avg: 1580.947 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1697 [17:40:43] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 89906 MB (14% inode=99%): [17:48:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:48:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:57:25] legoktm: next time you can tell him about pkill -u USERNAME [17:57:36] ok [17:57:56] legoktm: that kills everything under your account [18:00:07] legoktm: Ive found that useful if you just need to wipe the slate and start over [18:00:13] heh [18:00:25] i suppose [18:00:34] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6156.000000 [18:00:52] hello [18:00:56] maintenance tonight [18:01:39] legoktm: Ive got half a dozen IRC Bots that sometimes just need mass restarted [18:01:59] * legoktm has 4 :P [18:02:53] 4 dozen? [18:03:15] i'll start with willow [18:03:23] will be rebooted in some minutes [18:06:43] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:06:43] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:08:14] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 440019 MB (8% inode=64%): [18:08:50] will reboot willow now [18:09:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:10:55] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1146 MB (9% inode=99%): [18:11:14] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:11:52] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9036.000000 [18:11:52] Sun Grid Engine execd on willow is CRITICAL: Connection refused by host [18:12:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:12:45] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 43240 MB (8% inode=99%): [18:12:54] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 1309 MB (11% inode=99%): [18:13:05] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [18:13:55] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1096 MB (9% inode=99%): [18:13:55] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 93865.000000 [18:13:55] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7528.000000 [18:13:55] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [18:14:45] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 120659 [18:16:24] willow is back [18:17:03] install volume not working, going to fix this [18:17:53] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 160481.000000 [18:19:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 120753.000000 [18:26:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:28:53] nosy: wrong /mnt/user-store/ mounted to willow [18:29:23] aliasd on yarrow is CRITICAL: Connection refused [18:31:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:32:46] Danny_B: actually its not mounted because of ipv6 reverse zone is not ready yet [18:34:47] weird, i see /mnt/user-store/sk/... [18:40:44] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 89652 MB (14% inode=99%): [18:41:37] user-store is back [18:42:01] Danny_B: probably sk just creates dirs that are not there automatically [18:44:50] hemlock will be next [18:45:07] nothing should be badly disturbed [18:46:54] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 1489 MB (12% inode=99%): [18:48:39] nosy: please make sure munin works in the end [18:48:52] ok [18:48:55] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1203 MB (10% inode=99%): [18:48:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:49:25] * DaBPunkt will look at ortelius [18:50:16] dschwen: Your vips-files are filling the tmp-folder AGAIN! [18:50:51] some are older than 1 week [18:52:54] /tmp on ortelius is OK: DISK OK - free space: /tmp 4360 MB (37% inode=99%): [18:54:44] wikidata replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1882.000000 [19:00:34] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7832.000000 [19:07:45] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:07:45] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:09:14] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 438217 MB (8% inode=64%): [19:09:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:11:54] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9498.000000 [19:12:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:12:13] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:13:05] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [19:14:54] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8035.000000 [19:14:54] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 93173.000000 [19:15:43] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 120874 [19:17:54] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 161036.000000 [19:19:55] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 120681.000000 [19:26:23] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [19:26:33] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [19:26:45] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [19:26:45] / on thyme is UNKNOWN: NRPE: Unable to read output [19:26:45] RAID on thyme is UNKNOWN: NRPE: Unable to read output [19:26:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:27:04] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [19:27:04] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [19:27:04] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [19:27:14] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [19:29:23] aliasd on yarrow is CRITICAL: Connection refused [19:37:18] My interface isn't loading. What's wrong? [19:39:09] Hello? [19:40:45] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 89420 MB (14% inode=99%): [19:41:10] tsnag, ? [19:48:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:51:11] Cyberpower678: maintenance is in progress [19:51:15] and tsnag is a bot [19:51:37] Oh and oh. [19:52:08] Is that why the PC task keeps stalling? [19:53:14] possibly [19:53:25] cyberpower678@willow:~$ ps -ef|grep PCbot.php [19:53:26] cyberpow 25245 25202 0 19:51:01 ? 0:00 /bin/sh -c cd /home/cyberpower678/bots;/usr/bin/php PCbot.php >$HOME/CyberbotII [19:53:26] cyberpow 25246 25245 0 19:51:01 ? 0:00 /usr/bin/php PCbot.php [19:53:26] cyberpow 25753 20409 0 19:53:06 pts/2 0:00 grep PCbot.php [19:53:38] Which is my bot and which should I not touch? [19:54:15] legoktm, ^ [19:54:30] um [19:54:44] wikidata replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2313.000000 [19:54:44] all 3 are your bot [19:55:27] DaBPunkt, said there are untouchables that shouldn't get deleted like the grep one. [19:58:54] well [19:58:58] i dont know what your bot does [19:59:01] or how it works [19:59:56] It keeps randomly stalling in the middle of a POST command and apparently pid keeps disappearing. [20:00:39] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9945.000000 [20:00:57] Cyberpower678: which host? [20:01:07] you do a post to a php on willow? [20:01:08] Willow [20:01:14] or via web servers? [20:01:31] willow just changed ips [20:01:43] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:01:47] nosy, meaning? [20:02:07] might be dns somewhere is still holding old ips or something [20:02:30] And that's causing PIDs to disappear? [20:03:07] i dont know [20:03:19] pids disappear because programs exit [20:03:22] i would say... [20:03:30] not sure if this helps [20:06:29] Looks like it's stable again. [20:06:43] * Cyberpower678 scratches his head and wonders. [20:06:51] ill break some more here now :D [20:06:57] * legoktm scratches Cyberpower678's head and wonders. [20:07:15] legoktm, a little more to the left. [20:07:33] daphne would be next to reboot that means s4-user will go away for some time [20:07:45] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:07:45] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:07:53] so ill prepare now [20:07:55] :P [20:09:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:10:02] nosy, stop breaking everything. [20:10:06] :p [20:10:13] Cyberpower678: nope [20:10:13] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 437914 MB (8% inode=64%): [20:10:18] its maintenance time :D [20:10:24] time to break things [20:10:38] my favorite job [20:10:56] Usually maintenance means improving and keeping it running. [20:11:13] But I may have the wrong definition. [20:11:49] you seem to not be an admin ;) [20:12:04] nosy, huh? [20:12:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:12:07] i will reboot daphne now [20:12:13] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:12:54] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9917.000000 [20:13:01] here breaking means definitelly keeping it up since our ip space will be gone in 2 weeks [20:13:07] so we need to renumber [20:14:54] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8584.000000 [20:14:54] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 92832.000000 [20:15:43] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 120555 [20:16:36] nosy, i don't get it. What do you mean by ip space will be gone? [20:17:05] we have to give our network addresses back and we already have new ones [20:17:16] so ips have to be changed wherever we use the external ones [20:17:22] nosy, back to who? [20:17:27] ripe [20:17:31] sorry maintenance now [20:17:54] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 162796.000000 [20:19:55] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 120158.000000 [20:26:25] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [20:26:33] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [20:26:45] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [20:26:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:27:04] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [20:27:05] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [20:27:05] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [20:27:14] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [20:27:44] / on thyme is UNKNOWN: NRPE: Unable to read output [20:27:44] RAID on thyme is UNKNOWN: NRPE: Unable to read output [20:29:13] [[Special:Log/newusers]] create 10 * Miguel2706 * (New user account) [20:29:23] aliasd on yarrow is CRITICAL: Connection refused [20:32:43] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3640.000000 [20:41:44] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 89191 MB (14% inode=99%): [20:46:17] ok s4-user is working for me [20:48:43] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2185.000000 [20:48:43] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2186.000000 [20:50:00] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:51:00] nosy, you created an artificial sentient intelligence. [20:51:11] It calls itself XANA. [20:51:19] What did you do? [20:51:21] :p [20:51:23] wtf [20:51:30] you said [20:51:36] I created an artificial sentient intelligence. [20:51:41] that must be it [20:52:12] and it calls itself XANA. [20:52:26] must be true [20:52:38] did you meet already? [20:52:54] * Cyberpower678 is curious if nosy knows what I'm referring to. [20:53:02] nope [20:53:08] Thought so. [20:53:13] Code Lyoko [20:53:55] XANA is a multiagent computer program initially designed to counter a military project but has gained self awareness and wants to enslave all of mankind. [20:54:27] nosy, ^ [20:54:37] hehe...sounds like an admin...true [20:54:54] ok...ill renumber clematis now [20:55:11] nosy, stop fixing things. [20:55:33] you wish, i wish but thats my job [20:55:46] do you get paid? [20:56:28] Cyberpower678: She's WMDE staff, so yes [20:57:08] OK then. Carry on. And don't create a demonic artificial intelligence. [20:57:39] :p [21:01:43] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:07:45] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:07:45] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:09:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:09:40] i did not create toolserver i just maintain it [21:10:13] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 437686 MB (8% inode=64%): [21:11:33] ok hawthorn is next [21:12:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:12:13] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:12:45] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3625.000000 [21:12:45] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3626.000000 [21:12:55] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13516.000000 [21:14:48] it reboot [21:14:51] it reboots [21:14:55] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 96432.000000 [21:14:55] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12185.000000 [21:15:02] so lets see if mail still works afterwards [21:15:44] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 118303 [21:17:55] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 166397.000000 [21:20:54] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 118049.000000 [21:21:08] ok...mail still working... [21:23:14] NTP on hemlock is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [21:23:44] MySQL slave on daphne is CRITICAL: Access denied for user tsnagios7643@185.15.59.197 (using password: YES) [21:23:44] MySQL on daphne is CRITICAL: Access denied for user tsnagios7643@185.15.59.197 (using password: YES) [21:23:54] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [21:24:03] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [21:24:24] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user tsnagios7643@185.15.59.197 (using password: YES) [21:24:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user tsnagios7643@185.15.59.197 (using password: YES) [21:25:45] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3423.000000 [21:25:45] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3251.000000 [21:26:25] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [21:26:33] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [21:26:44] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [21:26:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:26:59] ok, i think maintenance is over for today [21:27:04] *wave* [21:27:05] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [21:27:05] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [21:27:05] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [21:27:13] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [21:27:44] / on thyme is UNKNOWN: NRPE: Unable to read output [21:27:44] RAID on thyme is UNKNOWN: NRPE: Unable to read output [21:29:23] aliasd on yarrow is CRITICAL: Connection refused [21:30:45] s4 replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1395.000000 [21:32:44] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1629.000000 [21:32:44] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6498.000000 [21:41:46] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 88925 MB (14% inode=99%): [21:49:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:56:06] RAID on daphne is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [22:01:43] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:04:31] DaBPunkt, this is due to a bug in VIPS that has been fixed. I will add a ticket to have an uptodate version of VIPS installed. Will clean the tmp files [22:06:15] dschwen: "find /tmp/ -maxdepth 1 -user dschwen -name "vips*" -atime +1" should do [22:06:28] no [22:06:29] dschwen: "find /tmp/ -maxdepth 1 -user dschwen -name "vips*" -atime +1 -delete" should do [22:06:48] the tmp files are just a symptom [22:06:58] yes, of course [22:08:44] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:08:44] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:10:13] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 437313 MB (8% inode=64%): [22:10:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:12:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:13:13] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:13:53] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14414.000000 [22:14:55] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 98454.000000 [22:14:55] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13083.000000 [22:15:44] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 116344 [22:18:54] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 166404.000000 [22:21:54] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 116385.000000 [22:23:44] MySQL slave on daphne is CRITICAL: Access denied for user tsnagios7643@185.15.59.197 (using password: YES) [22:23:44] MySQL on daphne is CRITICAL: Access denied for user tsnagios7643@185.15.59.197 (using password: YES) [22:24:13] NTP on hemlock is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [22:24:23] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user tsnagios7643@185.15.59.197 (using password: YES) [22:24:32] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user tsnagios7643@185.15.59.197 (using password: YES) [22:24:54] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [22:26:24] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [22:26:33] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [22:26:44] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [22:27:06] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [22:27:06] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [22:27:06] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [22:27:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:28:12] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [22:28:44] / on thyme is UNKNOWN: NRPE: Unable to read output [22:28:44] RAID on thyme is UNKNOWN: NRPE: Unable to read output [22:29:23] aliasd on yarrow is CRITICAL: Connection refused [22:33:44] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6687.000000 [22:41:46] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 88690 MB (14% inode=99%): [22:49:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:01:44] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:08:45] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:08:46] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:10:13] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 437301 MB (8% inode=64%): [23:10:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:12:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:13:13] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:13:54] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14879.000000 [23:14:55] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13485.000000 [23:14:55] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 99076.000000 [23:16:17] @replag [23:16:44] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 113852 [23:18:55] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 167087.000000 [23:20:44] nacht ts [23:21:27] nacht DaB [23:22:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 113665.000000 [23:23:43] MySQL slave on daphne is CRITICAL: Access denied for user tsnagios7643@185.15.59.197 (using password: YES) [23:23:43] MySQL on daphne is CRITICAL: Access denied for user tsnagios7643@185.15.59.197 (using password: YES) [23:24:13] NTP on hemlock is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [23:24:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user tsnagios7643@185.15.59.197 (using password: YES) [23:24:55] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [23:25:24] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user tsnagios7643@185.15.59.197 (using password: YES) [23:26:33] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [23:26:44] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [23:27:23] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [23:27:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:28:03] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [23:28:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [23:28:03] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [23:28:13] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [23:29:46] / on thyme is UNKNOWN: NRPE: Unable to read output [23:29:46] RAID on thyme is UNKNOWN: NRPE: Unable to read output [23:30:24] aliasd on yarrow is CRITICAL: Connection refused [23:33:44] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6825.000000 [23:42:45] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 88449 MB (14% inode=99%): [23:51:00] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output