[00:22:32] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=1.395020/1.25, alarm hl:np_load_long=1.069336/1.75, alarm hl:mem_free=1513.000000M/300M [00:23:30] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [00:23:41] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:27:32] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=1.570312/1.25, alarm hl:np_load_long=1.188476/1.75, alarm hl:mem_free=1303.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=1.570312/1.50, alarm hl:np_load_long=1.188476/2.00, alarm hl:mem_free=1303.000000M/250M [00:30:10] SSH on nightshade.mgmt is CRITICAL: Server answer: [00:33:02] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.049805/1.00, alarm hl:np_load_long=0.736328/1.50, alarm hl:mem_free=11448.000000M/300M [00:33:11] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 55750 MB (5% inode=99%): [00:33:31] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:34:03] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [00:53:11] Load avg. on nightshade is WARNING: WARNING - load average: 14.55, 16.77, 14.21 [00:55:11] Load avg. on nightshade is OK: OK - load average: 10.90, 14.46, 13.65 [01:00:52] I've got a request [01:01:32] Sp33dyphil: what? [01:02:12] Load avg. on nightshade is WARNING: WARNING - load average: 13.79, 15.74, 14.45 [01:02:31] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=1.452637/1.25, alarm hl:np_load_long=1.144043/1.75, alarm hl:mem_free=661.000000M/300M [01:02:38] Betacommand: This is not important, but how long would it take to create a function that warns people not to enter a page, like a popup? [01:03:11] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [01:03:14] Sp33dyphil: what do you mean> [01:04:31] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [01:12:31] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=1.414551/1.25, alarm hl:np_load_long=1.234863/1.75, alarm hl:mem_free=264.000000M/300M [01:23:41] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:27:01] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.083984/1.00, alarm hl:np_load_long=1.171875/1.50, alarm hl:mem_free=10379.000000M/300M [01:29:01] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [01:30:11] SSH on nightshade.mgmt is CRITICAL: Server answer: [01:32:11] Load avg. on nightshade is WARNING: WARNING - load average: 24.96, 19.03, 15.18 [01:33:12] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 55632 MB (5% inode=99%): [01:33:32] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:35:12] Load avg. on nightshade is OK: OK - load average: 8.61, 13.96, 13.84 [01:43:01] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=2.073242/1.00, alarm hl:np_load_long=1.212891/1.50, alarm hl:mem_free=10194.000000M/300M: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=2.073242/1.10, alarm hl:np_load_long=1.212891/1.75, alarm hl:mem_free=10194.000000M/300M [01:49:10] Load avg. on nightshade is WARNING: WARNING - load average: 17.54, 16.20, 14.56 [02:03:11] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [02:23:51] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:30:11] SSH on nightshade.mgmt is CRITICAL: Server answer: [02:34:12] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 55509 MB (5% inode=99%): [02:34:32] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:37:59] nacht ts [02:41:11] Load avg. on nightshade is WARNING: WARNING - load average: 16.48, 14.43, 13.88 [02:42:13] Load avg. on nightshade is OK: OK - load average: 10.09, 13.02, 13.42 [02:46:11] Load avg. on nightshade is WARNING: WARNING - load average: 16.04, 15.30, 14.29 [03:03:13] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [03:23:52] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:28:21] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:28:31] SMTP on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:29:01] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [03:29:21] SMTP on hyacinth is OK: SMTP OK - 0.126 sec. response time [03:30:12] SSH on nightshade.mgmt is CRITICAL: Server answer: [03:34:31] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:35:12] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 56378 MB (5% inode=99%): [03:57:32] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=1.511230/1.25, alarm hl:np_load_long=1.113281/1.75, alarm hl:mem_free=1949.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=1.511230/1.50, alarm hl:np_load_long=1.113281/2.00, alarm hl:mem_free=1949.000000M/250M [03:58:32] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [04:02:31] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=1.592774/1.25, alarm hl:np_load_long=1.245605/1.75, alarm hl:mem_free=1999.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=1.592774/1.50, alarm hl:np_load_long=1.245605/2.00, alarm hl:mem_free=1999.000000M/250M [04:03:41] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=1.382812/1.25, alarm hl:np_load_long=1.563965/1.75, alarm hl:mem_free=896.000000M/300M [04:09:42] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [04:23:00] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:24:51] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:30:21] SSH on nightshade.mgmt is CRITICAL: Server answer: [04:33:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [04:35:22] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 56264 MB (5% inode=99%): [04:35:31] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:37:42] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [04:47:31] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=1.588379/1.25, alarm hl:np_load_long=1.227539/1.75, alarm hl:mem_free=1720.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=1.588379/1.50, alarm hl:np_load_long=1.227539/2.00, alarm hl:mem_free=1720.000000M/250M [04:49:31] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [05:02:31] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=1.536133/1.25, alarm hl:np_load_long=1.249024/1.75, alarm hl:mem_free=1906.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=1.536133/1.50, alarm hl:np_load_long=1.249024/2.00, alarm hl:mem_free=1906.000000M/250M [05:25:03] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:28:31] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=1.404785/1.25, alarm hl:np_load_long=1.128418/1.75, alarm hl:mem_free=1682.000000M/300M [05:30:22] SSH on nightshade.mgmt is CRITICAL: Server answer: [05:33:11] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [05:33:32] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [05:33:41] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=1.657715/1.25, alarm hl:np_load_long=1.093750/1.75, alarm hl:mem_free=1110.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=1.657715/1.50, alarm hl:np_load_long=1.093750/2.00, alarm hl:mem_free=1110.000000M/250M [05:35:22] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 56012 MB (5% inode=99%): [05:35:31] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:35:40] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [05:42:42] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=1.276855/1.25, alarm hl:np_load_long=1.220703/1.75, alarm hl:mem_free=1873.000000M/300M [05:54:41] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [05:57:41] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=1.979492/1.25, alarm hl:np_load_long=1.465820/1.75, alarm hl:mem_free=2083.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=1.979492/1.50, alarm hl:np_load_long=1.465820/2.00, alarm hl:mem_free=2083.000000M/250M [06:02:22] Load avg. on nightshade is WARNING: WARNING - load average: 22.66, 19.17, 14.76 [06:07:23] Load avg. on nightshade is OK: OK - load average: 9.25, 13.78, 13.69 [06:07:42] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [06:10:42] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=1.649414/1.25, alarm hl:np_load_long=1.177734/1.75, alarm hl:mem_free=1204.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=1.649414/1.50, alarm hl:np_load_long=1.177734/2.00, alarm hl:mem_free=1204.000000M/250M [06:12:43] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [06:22:42] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=1.267090/1.25, alarm hl:np_load_long=1.208984/1.75, alarm hl:mem_free=1113.000000M/300M [06:25:02] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:31:23] SSH on nightshade.mgmt is CRITICAL: Server answer: [06:33:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [06:34:01] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.115234/1.00, alarm hl:np_load_long=0.714844/1.50, alarm hl:mem_free=12252.000000M/300M: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.115234/1.10, alarm hl:np_load_long=0.714844/1.75, alarm hl:mem_free=12252.000000M/300M [06:35:02] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [06:35:42] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:36:22] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 56046 MB (5% inode=99%): [07:02:42] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=1.536133/1.25, alarm hl:np_load_long=1.188965/1.75, alarm hl:mem_free=972.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=1.536133/1.50, alarm hl:np_load_long=1.188965/2.00, alarm hl:mem_free=972.000000M/250M [07:04:42] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [07:12:42] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=1.316406/1.25, alarm hl:np_load_long=1.210938/1.75, alarm hl:mem_free=544.000000M/300M [07:25:02] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:32:32] SSH on nightshade.mgmt is CRITICAL: Server answer: [07:33:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [07:35:42] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:36:22] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 55926 MB (5% inode=99%): [07:54:01] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.693359/1.00, alarm hl:np_load_long=0.791015/1.50, alarm hl:mem_free=12115.000000M/300M: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.693359/1.10, alarm hl:np_load_long=0.791015/1.75, alarm hl:mem_free=12115.000000M/300M [07:55:02] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [08:02:42] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=1.308594/1.25, alarm hl:np_load_long=1.129395/1.75, alarm hl:mem_free=879.000000M/300M [08:03:42] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [08:04:31] /aux0 on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 50585 MB (5% inode=99%): [08:12:42] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=1.444336/1.25, alarm hl:np_load_long=1.149902/1.75, alarm hl:mem_free=578.000000M/300M [08:25:12] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:32:31] SSH on nightshade.mgmt is CRITICAL: Server answer: [08:33:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [08:35:42] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:36:23] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 55814 MB (5% inode=99%): [09:25:13] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:26:22] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 38.96, 20.30, 12.29 [09:26:41] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=4.845703/1.25, alarm hl:np_load_long=1.516601/1.75, alarm hl:mem_free=1687.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=4.845703/1.50, alarm hl:np_load_long=1.516601/2.00, alarm hl:mem_free=1687.000000M/250M [09:32:32] SSH on nightshade.mgmt is CRITICAL: Server answer: [09:33:14] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [09:34:23] Load avg. on nightshade is WARNING: WARNING - load average: 10.41, 22.66, 18.42 [09:35:41] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:37:22] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 55707 MB (5% inode=99%): [09:38:11] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.506836/1.00, alarm hl:np_load_long=0.642578/1.50, alarm hl:mem_free=12019.000000M/300M: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.506836/1.10, alarm hl:np_load_long=0.642578/1.75, alarm hl:mem_free=12019.000000M/300M [09:38:41] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=1.810547/1.25, alarm hl:np_load_long=1.228027/1.75, alarm hl:mem_free=910.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=1.810547/1.50, alarm hl:np_load_long=1.228027/2.00, alarm hl:mem_free=910.000000M/250M [09:39:13] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [09:39:23] Load avg. on nightshade is OK: OK - load average: 6.58, 12.29, 15.00 [09:42:41] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [09:43:41] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [09:45:41] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=5.094727/1.25, alarm hl:np_load_long=2.569824/1.75, alarm hl:mem_free=1824.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=5.094727/1.50, alarm hl:np_load_long=2.569824/2.00, alarm hl:mem_free=1824.000000M/250M [09:49:26] nosy: are you competent to explain the rules? [09:55:42] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [09:56:42] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=2.330078/1.25, alarm hl:np_load_long=1.493652/1.75, alarm hl:mem_free=651.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=2.330078/1.50, alarm hl:np_load_long=1.493652/2.00, alarm hl:mem_free=651.000000M/250M [10:07:25] Giftpflanze: dont know [10:07:32] what do you mean? [10:07:59] which rules do you mean? [10:17:41] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=0.583984/1.25, alarm hl:np_load_long=1.906250/1.75, alarm hl:mem_free=2806.000000M/300M [10:19:42] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [10:22:41] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=6.991699/1.25, alarm hl:np_load_long=2.781738/1.75, alarm hl:mem_free=2295.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=6.991699/1.50, alarm hl:np_load_long=2.781738/2.00, alarm hl:mem_free=2295.000000M/250M [10:25:12] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:32:32] SSH on nightshade.mgmt is CRITICAL: Server answer: [10:35:42] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:36:42] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [10:37:22] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 55581 MB (5% inode=99%): [10:43:42] Load avg. on willow is WARNING: WARNING - load average: 16.38, 14.70, 13.15 [10:44:41] Load avg. on willow is OK: OK - load average: 10.57, 13.39, 12.79 [11:03:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [11:26:11] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:32:32] SSH on nightshade.mgmt is CRITICAL: Server answer: [11:35:42] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:38:22] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 55439 MB (5% inode=99%): [11:44:11] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.092774/1.00, alarm hl:np_load_long=0.643555/1.50, alarm hl:mem_free=12091.000000M/300M [11:45:11] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [11:56:11] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=4.741211/1.00, alarm hl:np_load_long=1.208984/1.50, alarm hl:mem_free=12190.000000M/300M: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=4.741211/1.10, alarm hl:np_load_long=1.208984/1.75, alarm hl:mem_free=12190.000000M/300M [11:56:32] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 58.68, 29.24, 15.16 [11:56:42] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=7.423828/1.25, alarm hl:np_load_long=1.876953/1.75, alarm hl:mem_free=2295.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=7.423828/1.50, alarm hl:np_load_long=1.876953/2.00, alarm hl:mem_free=2295.000000M/250M [11:57:41] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=1.479980/1.25, alarm hl:np_load_long=1.074707/1.75, alarm hl:mem_free=998.000000M/300M [11:58:01] Load avg. on ortelius is WARNING: WARNING - load average: 18.27, 12.07, 6.56 [12:01:02] Load avg. on ortelius is OK: OK - load average: 14.83, 13.57, 8.16 [12:03:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [12:04:41] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [12:12:42] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=1.644043/1.25, alarm hl:np_load_long=1.243164/1.75, alarm hl:mem_free=661.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=1.644043/1.50, alarm hl:np_load_long=1.243164/2.00, alarm hl:mem_free=661.000000M/250M [12:15:01] Load avg. on ortelius is WARNING: WARNING - load average: 17.47, 12.39, 9.16 [12:19:31] Load avg. on nightshade is WARNING: WARNING - load average: 5.49, 10.27, 19.97 [12:21:02] Load avg. on ortelius is OK: OK - load average: 5.53, 13.45, 11.22 [12:22:42] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [12:25:33] Sun Grid Engine execd on wolfsbane is WARNING: short@wolfsbane exceedes load threshold: alarm hl:np_load_short=1.222168/1.00, alarm hl:np_load_long=0.880859/1.50, alarm hl:mem_free=2873.000000M/300M: all.q@wolfsbane exceedes load threshold: alarm hl:np_load_short=1.222168/1.10, alarm hl:np_load_long=0.880859/1.75, alarm hl:mem_free=2873.000000M/300M [12:26:22] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:26:33] Sun Grid Engine execd on wolfsbane is OK: short@wolfsbane OK: all.q@wolfsbane OK [12:27:32] Load avg. on nightshade is OK: OK - load average: 7.49, 7.57, 14.55 [12:28:41] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [12:30:32] Sun Grid Engine execd on wolfsbane is WARNING: short@wolfsbane exceedes load threshold: alarm hl:np_load_short=1.001465/1.00, alarm hl:np_load_long=0.899414/1.50, alarm hl:mem_free=2328.000000M/300M [12:32:33] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 36.78, 22.03, 18.41 [12:32:33] SSH on nightshade.mgmt is CRITICAL: Server answer: [12:32:41] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=4.530273/1.25, alarm hl:np_load_long=2.289062/1.75, alarm hl:mem_free=1864.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=4.530273/1.50, alarm hl:np_load_long=2.289062/2.00, alarm hl:mem_free=1864.000000M/250M [12:34:02] Load avg. on ortelius is WARNING: WARNING - load average: 16.10, 10.77, 9.09 [12:35:43] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:38:32] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 48394 MB (4% inode=99%): [12:40:01] Load avg. on ortelius is OK: OK - load average: 10.00, 13.59, 11.21 [12:42:32] Load avg. on nightshade is WARNING: WARNING - load average: 6.21, 17.98, 19.80 [12:49:33] Load avg. on nightshade is OK: OK - load average: 4.67, 9.04, 14.81 [12:51:41] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [12:54:01] Load avg. on ortelius is WARNING: WARNING - load average: 21.28, 13.43, 10.17 [12:56:02] Load avg. on ortelius is OK: OK - load average: 14.36, 13.29, 10.51 [12:56:22] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=3.508789/1.00, alarm hl:np_load_long=2.629883/1.50, alarm hl:mem_free=11022.000000M/300M: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=3.508789/1.10, alarm hl:np_load_long=2.629883/1.75, alarm hl:mem_free=11022.000000M/300M [12:57:42] Load avg. on willow is WARNING: WARNING - load average: 15.31, 14.91, 12.82 [12:58:41] Load avg. on willow is OK: OK - load average: 13.98, 14.60, 12.84 [13:02:41] Load avg. on willow is WARNING: WARNING - load average: 13.23, 15.58, 13.77 [13:03:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [13:11:41] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=1.785156/1.25, alarm hl:np_load_long=1.636230/1.75, alarm hl:mem_free=203.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=1.785156/1.50, alarm hl:np_load_long=1.636230/2.00, alarm hl:mem_free=203.000000M/250M [13:12:12] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [13:26:22] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:27:41] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [13:32:42] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=1.388184/1.25, alarm hl:np_load_long=1.534180/1.75, alarm hl:mem_free=923.000000M/300M [13:33:33] SSH on nightshade.mgmt is CRITICAL: Server answer: [13:34:43] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [13:36:52] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:39:32] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 22921 MB (2% inode=99%): [13:40:53] nosy: the rights for dumps are quite weird [13:41:02] i can't change to subdirs [13:42:40] danny_b|backup good luck with dumps! :) [13:44:09] thx ;-) [13:44:42] Danny_B|backup: who owns the dirs? [13:45:15] ufortunately it came a bit later than i expected and now it seems i won't have about two weeks so much time to work on it as intensive as i wanted to [13:45:57] (e.g. did a root untar, and thus change the uids to the ones in the tar archive) [13:45:59] nosy (or anybody): do mmp have .forward available as well? [13:48:10] I would expect so - mmps are just uses [13:48:12] users* [13:52:05] ah, i set rwxrw---- originally, so that's why i didn't have access, now it works [13:52:42] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=1.502441/1.25, alarm hl:np_load_long=1.263184/1.75, alarm hl:mem_free=902.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=1.502441/1.50, alarm hl:np_load_long=1.263184/2.00, alarm hl:mem_free=902.000000M/250M [14:03:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [14:16:21] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=2.295898/1.00, alarm hl:np_load_long=0.828125/1.50, alarm hl:mem_free=11677.000000M/300M: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=2.295898/1.10, alarm hl:np_load_long=0.828125/1.75, alarm hl:mem_free=11677.000000M/300M [14:18:22] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [14:19:36] Danny_B|backup: I tested mailing to the pywikipedia mmp, but .forward doesn't seem to work... [14:19:51] :-/ [14:19:59] nor do I get an error mail back... to keep things interesting [14:27:21] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:34:32] SSH on nightshade.mgmt is CRITICAL: Server answer: [14:34:50] Danny_B|backup: or.. apparently it does, as the other maintainer *did* get the e-mail [14:34:53] :| [14:35:11] and what was the address used? [14:35:41] I sent an email to pywikipedia@toolserver.org; the .forward lists valhallasw@toolserver.org, multichill@toolserver.org [14:35:58] oh [14:36:05] it doesn't [14:36:51] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:36:57] Danny_B|backup: see /home/projects/p/y/w/pywikipedia/.bashrc [14:37:24] anyway - .forward does work [14:37:30] it was an error on my part [14:38:10] will check when i'm back, thx for proofing [14:38:12] afk [14:40:31] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 55987 MB (5% inode=99%): [14:50:05] do any of the toolserver admins know how to restart the IRC bot, ACCBot? It runs on the toolserver, I imagine under the ~acc account [14:53:20] acc::6009:overlordq,stwalkerster,cobi,sql [14:53:23] ask any of those people [15:03:13] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [15:18:21] valhallasw: only cobi and stwalkerster are active and both are AFK (and have been for about a day) but thanks [15:27:21] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:34:43] SSH on nightshade.mgmt is CRITICAL: Server answer: [15:36:52] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:40:33] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 55848 MB (5% inode=99%): [16:12:58] [[Special:Log/newusers]] create 10 * Dermcesbomebliodi * (New user account) [16:13:30] [[User:Dermcesbomebliodi]] !N 10https://wiki.toolserver.org/w/index.php?oldid=6547&rcid=8611 * Dermcesbomebliodi * (+503) (Created page with ". A Joel Perez Carrero La mujer pirata December 5, 2011 at 7:27pm ยท 2 Jim Henriquez Al hacer ese truco, te dan la misma exp y monedas que cuando las completas sin truco? De no ...") [16:27:21] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:30:30] zz [16:33:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [16:34:44] SSH on nightshade.mgmt is CRITICAL: Server answer: [16:37:02] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:41:32] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 55642 MB (5% inode=99%): [17:27:32] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:35:42] SSH on nightshade.mgmt is CRITICAL: Server answer: [17:37:02] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:41:33] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 55488 MB (5% inode=99%): [17:54:41] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 55.48, 26.58, 14.48 [17:55:02] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=7.365723/1.25, alarm hl:np_load_long=1.803223/1.75, alarm hl:mem_free=1622.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=7.365723/1.50, alarm hl:np_load_long=1.803223/2.00, alarm hl:mem_free=1622.000000M/250M [17:55:32] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=3.621094/1.00, alarm hl:np_load_long=1.366211/1.50, alarm hl:mem_free=11478.000000M/300M: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=3.621094/1.10, alarm hl:np_load_long=1.366211/1.75, alarm hl:mem_free=11478.000000M/300M [17:55:42] Load avg. on nightshade is WARNING: WARNING - load average: 29.48, 24.38, 14.47 [17:58:33] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [17:58:59] i wanted to run "python interwiki.py -auto -transcludes:portale" and i just received a mail from the slayerd, which killed the process. but, since it is cpu consuming, how to run interwikis on ts? [17:59:03] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [18:00:42] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 37.95, 22.95, 15.84 [18:01:32] nickanc: afaik slayerd reacts to memory consumption, not cpu [18:01:32] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.035156/1.00, alarm hl:np_load_long=1.253906/1.50, alarm hl:mem_free=10908.000000M/300M [18:02:02] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=2.854492/1.25, alarm hl:np_load_long=1.956055/1.75, alarm hl:mem_free=1371.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=2.854492/1.50, alarm hl:np_load_long=1.956055/2.00, alarm hl:mem_free=1371.000000M/250M [18:02:38] what was the exact reason slayerd killed your process? [18:02:45] true valhallasw. it reacted to memory consumption. i was wondering where was my head when i wrote the previous message. [18:03:01] One or more of your processes on the host willow [18:03:01] were exceeding the configured memory limit, which is 1000 megabytes. [18:03:01] I have killed enough of your processes to bring your usage back to the [18:03:01] threshold limit, which is 750 megabytes. [18:03:11] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [18:03:16] python (pid 26930), using 982 megabyte(s). [18:03:17] command: python interwiki.py -auto -transcludes:portale [18:04:32] strange. maybe the transcludes pagegenerator is broken [18:05:05] well, [[it:template:Portale]] is one of the most used tmps on it.wiki [18:06:53] yes, but it shouldn't load all items at once [18:06:59] try running it again, with -v [18:07:12] (and save that to a log file!) [18:07:34] maybe there is something clearly wrong (i.e. retrieval of hundreds of pages at once) [18:10:09] ok. i have the old log, if you want ot check it. [18:10:19] with -v? [18:10:19] * nickanc always saves logs [18:10:21] no [18:10:47] (-v shows the used api queries, which is immensely useful) [18:12:17] it is working valhallasw with -v now [18:12:45] yes, until it's killed again [18:13:12] * nickanc is waiting for slayerd [18:13:28] when it is killed again, can i mail you the log? [18:14:12] I cannot guarantee I have time to look at it [18:27:32] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:35:43] SSH on nightshade.mgmt is CRITICAL: Server answer: [18:38:01] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:41:31] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 55321 MB (5% inode=99%): [18:54:09] valhallasw it ha just been killed [19:01:42] the log is at http://toolserver.org/~nickanc/interwiki.log [19:03:11] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [19:03:44] hm [19:04:08] it doesn't use the api to retrieve page batches, I forgot... [19:16:02] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=1.698730/1.25, alarm hl:np_load_long=1.881836/1.75, alarm hl:mem_free=1373.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=1.698730/1.50, alarm hl:np_load_long=1.881836/2.00, alarm hl:mem_free=1373.000000M/250M [19:20:02] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [19:23:02] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=1.946777/1.25, alarm hl:np_load_long=1.874512/1.75, alarm hl:mem_free=1464.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=1.946777/1.50, alarm hl:np_load_long=1.874512/2.00, alarm hl:mem_free=1464.000000M/250M [19:28:31] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:35:42] SSH on nightshade.mgmt is CRITICAL: Server answer: [19:37:02] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [19:38:02] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:41:42] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 56154 MB (5% inode=99%): [19:43:57] nosy: question was: (how) does checking for linkrot interfer with "impact other networks" from the rules? [19:53:20] 3(created) [MNT-1173] Add the index on the hstore columns on osm_mapnik database again, that were forgotten during the re-import; Maintenance: ptolemy; Minor work <10https://jira.toolserver.org/browse/MNT-1173> (Kai Krueger) [20:00:02] Sun Grid Engine execd on nightshade is WARNING: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=1.232422/1.50, alarm hl:np_load_long=1.751465/1.75, alarm hl:mem_free=1766.000000M/250M [20:03:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [20:04:42] /aux0 on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 49901 MB (5% inode=99%): [20:17:01] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [20:22:03] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=2.211914/1.25, alarm hl:np_load_long=1.923340/2.00, alarm hl:mem_free=1566.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=2.211914/1.50, alarm hl:np_load_long=1.923340/1.75, alarm hl:mem_free=1566.000000M/250M [20:28:31] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:32:12] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [20:36:42] SSH on nightshade.mgmt is CRITICAL: Server answer: [20:38:11] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:42:26] Is something overloaded at Nightshade? Download of files is realy slow (30 KB/s, I get 10 times that speed on my laptop) [20:42:42] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 55963 MB (5% inode=99%): [20:47:54] [[Special:Log/newusers]] create 10 * Gabby8228 * (New user account) [20:57:12] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=2.057617/1.25, alarm hl:np_load_long=1.924316/2.00, alarm hl:mem_free=1668.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=2.057617/1.50, alarm hl:np_load_long=1.924316/1.75, alarm hl:mem_free=1668.000000M/250M [20:59:30] hi, external links on a page, are they put in an extra table, or can you only do a text search for them? [21:00:56] Akoopal: externallinks table. see mediawiki table structure [21:01:05] (somewhere on mediawiki.org :p) [21:01:12] ok :-) [21:05:22] Tanvir: Why are you running over a dozen interwiki bots? [21:06:54] Wtf, we have over 75 interwiki.py running at Nightshade at this very moment [21:12:12] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [21:15:11] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=1.704590/1.50, alarm hl:np_load_long=1.929199/2.00, alarm hl:mem_free=1747.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=1.704590/1.50, alarm hl:np_load_long=1.929199/1.75, alarm hl:mem_free=1747.000000M/250M [21:17:02] multichill: 69 on willow + 41 on nightshade = 110 interwiki.py bots [21:21:11] Merlissimo: Lol, beria likes to welcome people [21:24:02] Maarten Dammers * [Toolserver-l] interwiki.py [21:28:32] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:29:18] what! 110 interwiki bots!??? [21:29:44] Multichill, no, only one. [21:29:45] * aude is eager for an extension or wikidata to handle interwiki links [21:30:03] everyone comes looking after the mail [21:30:19] multichill: most cpu waste is currently done by johang because he run multiple processes which creates a temp file on user store. He writes many many bytes there instead of using the default tempdir which would need less nfs traffic to hemlock [21:31:23] The /tmp is only 512MB [21:32:14] Merlissimo: What location would you suggest? [21:32:30] sge is setting temp differently depending on queue and space. if you use -l tmp_free=100M you'll get 100M tmp space for sure [21:32:46] Aude, 110 interwiki bots are too many that I agree. [21:33:13] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [21:33:26] multichill: alswas TMPDIR (als mktemp would use by default) [21:33:37] Maybe we can think about creating something to discourage new interwiki bots.. dunno if it's ethical. [21:33:47] But we have enough, yes. [21:34:10] Tanvir: At the moment you don't have one bot running unless it spawns a lot of new processes [21:34:16] Tanvir: i think the big problem is, that we have 5-6 bots that are all watching the big wikis rc [21:34:36] Mhm.. that can be Merlissimo. [21:35:30] Multichill, I am running the same number of process that I used to run a year ago. [21:36:17] multichill: can you write a new generator that reads from a web page? perhaps we can make a global rc, which a bot can request and all entries once deliverd are not send to a request of another bot [21:36:36] New generator? Just use -file: [21:36:52] SSH on nightshade.mgmt is CRITICAL: Server answer: [21:37:00] i am not using py very much because i have my own framework [21:37:02] Uh, anyone know if something is wrong with the mono installation on the ts? [21:38:12] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:38:20] Firebolt: http://lists.wikimedia.org/pipermail/toolserver-l/2012-January/004646.html [21:38:23] Tanvir: But that is not one bot, right? [21:39:01] Multichill, you mean mine? I have only one inter-wiki bot, User:WikitanvirBot [21:39:06] ty jeremyb [21:39:25] Tanvir: I'm talking about the number of processes/instances [21:39:36] Not about the amount of user accounts [21:39:47] Yes, I am running several process. [21:42:31] Merlissimo: afaik interwiki.py does not actually watch the rc feed [21:42:31] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.160156/1.00, alarm hl:np_load_long=1.028320/1.50, alarm hl:mem_free=13589.000000M/300M: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.160156/1.10, alarm hl:np_load_long=1.028320/1.75, alarm hl:mem_free=13589.000000M/300M [21:42:42] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 55814 MB (5% inode=99%): [21:43:12] (although I wouldn't know why that would be useful, either) [21:44:20] valhallasw: Tanvir is running a generator which request the last 100 entries vom rc again and again. if there is not edit on a wiki he run on the same 100 pages again and again [21:44:35] wow. that is pretty useless [21:44:42] 100 entries? o.O [21:44:51] Where did you get that Merlissimo? [21:44:53] (i don't know the exact value) [21:45:32] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [21:45:49] Merlissimo, okay then. :) [21:46:40] My bot patrol pages on big wikis only, and bnwiki (my home wiki). [21:47:44] reza is running -recentchanges:200 / beria -recentchanges:200 and tanvir isn't using sge [21:48:09] Merlissimo, sge? [21:48:19] sun grid engine [21:48:57] that why i cannot see you full command, because its cut after some characters on process list [21:50:45] -recentchanges:200 ? o.O [21:51:46] i think it should be much more coordinated so that one rc-entry isn't processed twice (same bot and other bots) [21:52:59] Merlissimo: yes, /and/ we should stop wikitext-scaping and use the database, /and/ we should have a central database instead of the mess we have now [21:53:23] luckily, there is a mediawiki extension that does just that... still not installed, unfortunately [21:53:26] Valhallasw, *nods* [21:55:12] valhallasw: 2/3 are not so important for ts [21:55:19] ? [21:57:51] 2) there is no bandwhich bottleneck not wmf 3) tanvor would be out of work [21:58:21] 2) bandwidth is not the problem. memory is [21:58:31] besides, it decreases the speed at which things can be done [21:58:42] 3) is important for the ts, as it would clear out a lot of resources [21:58:53] Willow is damn slow now.. [21:59:10] thus: I disagree. [21:59:12] strongly [21:59:23] (thats the reason why i have my own interwiki bot, which is using api only) [22:15:12] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=0.488770/1.50, alarm hl:np_load_long=2.010742/2.00, alarm hl:mem_free=2450.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=0.488770/1.50, alarm hl:np_load_long=2.010742/1.75, alarm hl:mem_free=2450.000000M/250M [22:19:11] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [22:22:12] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=2.335938/1.50, alarm hl:np_load_long=2.018555/2.00, alarm hl:mem_free=2392.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=2.335938/1.50, alarm hl:np_load_long=2.018555/1.75, alarm hl:mem_free=2392.000000M/250M [22:26:12] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [22:28:42] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:32:13] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=1.756348/1.50, alarm hl:np_load_long=1.535645/2.00, alarm hl:mem_free=2444.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=1.756348/1.50, alarm hl:np_load_long=1.535645/1.75, alarm hl:mem_free=2444.000000M/250M [22:33:11] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [22:36:32] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.042969/1.00, alarm hl:np_load_long=1.145508/1.50, alarm hl:mem_free=13976.000000M/300M [22:37:31] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [22:37:52] SSH on nightshade.mgmt is CRITICAL: Server answer: [22:38:12] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:43:42] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 55616 MB (5% inode=99%): [22:51:31] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.385742/1.00, alarm hl:np_load_long=1.155274/1.50, alarm hl:mem_free=13860.000000M/300M: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.385742/1.10, alarm hl:np_load_long=1.155274/1.75, alarm hl:mem_free=13860.000000M/300M [23:03:12] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=2.023926/1.50, alarm hl:np_load_long=2.636230/2.00, alarm hl:mem_free=2222.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=2.023926/1.50, alarm hl:np_load_long=2.636230/1.75, alarm hl:mem_free=2222.000000M/250M [23:03:12] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=1.850586/1.50, alarm hl:np_load_long=1.632324/2.00, alarm hl:mem_free=2358.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=1.850586/1.50, alarm hl:np_load_long=1.632324/1.75, alarm hl:mem_free=2358.000000M/250M [23:03:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [23:04:13] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [23:07:12] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=1.634766/1.50, alarm hl:np_load_long=1.620605/2.00, alarm hl:mem_free=2326.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=1.634766/1.50, alarm hl:np_load_long=1.620605/1.75, alarm hl:mem_free=2326.000000M/250M [23:29:42] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:33:01] Nickanc Wikipedia * Re: [Toolserver-l] interwiki.py [23:37:51] SSH on nightshade.mgmt is CRITICAL: Server answer: [23:38:12] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:43:42] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 55499 MB (5% inode=99%): [23:53:12] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=1.514649/1.50, alarm hl:np_load_long=1.494629/2.00, alarm hl:mem_free=2267.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=1.514649/1.50, alarm hl:np_load_long=1.494629/1.75, alarm hl:mem_free=2267.000000M/250M [23:54:21] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [23:57:06] hm. I have no clue where my crontab is running -_- [23:57:15] oh, there it is