[00:08:00] <awight>	 !log deployment-prep Push ORES git-lfs to look at stuff
[00:08:03] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL
[00:20:47] <zhuyifei1999_>	 !log toolsbeta deleted all instances I just created except k8s master because of chicken-and-egg problem
[00:20:48] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[00:46:19] <awight>	 !log deployment-prep roll back ORES beta to master
[00:46:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL
[02:06:39] <ragesoss>	 andrewbogott: that's the server that runs Programs & Events Dashboard, which continuosly tracks contribution stats for several hundred ongoing education programs and other editing and outreach events.
[02:07:05] <ragesoss>	 is the CPU causing problems, or you're wondreing what it's doing?
[02:08:07] <ragesoss>	 It pretty much has a continuous queue of programs to pull in new stats for and update them, so more or less the current pattern of CPU usage is expected indefinitely.
[14:41:00] <revi>	 any WMCS staff available?
[14:44:15] <revi>	 (topic: toolforge abuse)
[14:55:30] <zhuyifei1999_>	 revi: not “staff” here, but if the abuse is ongoing and urgently needs killing, I can help with that
[14:55:38] <revi>	 I don't think it's ongoing
[14:55:42] <revi>	 Alaa: knows better
[14:56:33] <Alaa>	 Thanks revi
[14:57:31] <revi>	 arwiki CU did some checkuser block and other community member is complaining about their toolforge bot being blocked in arwiki
[14:57:46] <revi>	 so we thought toolforge is somehow being abused :P
[14:58:26] <Reedy>	 lol
[14:58:52] <revi>	 whatever... I need to be on meeting in 2 minutes so see you in an hour or so
[14:58:52] <zhuyifei1999_>	 oh, well, can you ask the CUs to see who’s running from a cloud IP address?
[14:59:11] <revi>	 Alaa: ^ you should do it :P
[14:59:12] <Alaa>	 zhuyifei I'm with you :D
[14:59:32] * revi is afk now
[14:59:42] <Alaa>	 "who’s running from a cloud IP address"?
[15:00:25] <zhuyifei1999_>	 I mean, which blocked user has the IP that belongs to toolforge
[15:01:21] <Alaa>	 one of those 
[15:01:23] <Alaa>	 https://ar.wikipedia.org/wiki/%D8%AA%D8%B5%D9%86%D9%8A%D9%81:%D8%B5%D8%A7%D9%86%D8%B9%D9%88_%D8%AF%D9%85%D9%89_%D8%AC%D9%88%D8%A7%D8%B1%D8%A8/Jamesbebo
[15:01:27] <Alaa>	 user:Ramykhaled
[15:01:36] <Alaa>	 user:Hero9000
[15:01:47] <Alaa>	 those 2 especially 
[15:02:54] <zhuyifei1999_>	 I don’t think we have logs on networking here on toolforge, and even if we do it wouldn’t be feasible for us to grep the amount of data just to search for some possibly-rare network accesses
[15:03:37] <Alaa>	 user:Ramykhaled, caught with abusefilter by 52,000 hits on 15 min
[15:03:48] <Alaa>	 user:Hero9000 made 16,000 edits on 5 min 
[15:04:32] <Alaa>	 Hero9000 do 16,000 in ar.wiki/3000 on Meta
[15:04:43] <Alaa>	 in less than 20 min :D
[15:05:14] <Alaa>	 Vandalism-only accounts 
[15:07:13] * zhuyifei1999_ will grepping for these three usernames on toolforge login accounts (not network data) shows nothing
[15:07:31] <zhuyifei1999_>	 um, scratch that '/me will'
[15:08:00] <Alaa>	 umm
[15:09:11] <zhuyifei1999_>	 I see three autoblocks in https://ar.wikipedia.org/wiki/%D8%AE%D8%A7%D8%B5:%D9%82%D8%A7%D8%A6%D9%85%D8%A9_%D8%A7%D9%84%D9%85%D9%86%D8%B9?wpTarget=&wpOptions%5B%5D=userblocks&wpOptions%5B%5D=rangeblocks&limit=50&wpFormIdentifier=blocklist
[15:09:37] <zhuyifei1999_>	 all belong to different accounts
[15:10:21] <Alaa>	 Hero9000 = Ramykhaled = Ramykhaled90
[15:10:39] <Alaa>	 this one #226664 related to another sock
[15:12:39] <zhuyifei1999_>	 I mean, if many toolforge bots are being blocked on arwiki, wouldn't the list have many autoblocks?
[15:12:58] <zhuyifei1999_>	 uh, maybe not
[15:13:25] <Alaa>	 Umm I don't think so! 
[15:13:50] <zhuyifei1999_>	 which toolforge bot is concerned? I'll see if that IP of the host it is running on is autoblocked
[15:14:01] <Alaa>	 user:ASammourBot 
[15:14:05] <zhuyifei1999_>	 k
[15:16:18] <zhuyifei1999_>	 username 'controll' has a declared name of 'ASammour', and belongs to two tools, 'tools.sammour' & 'tools.rchv', neither have ongoing jobs
[15:16:38] <zhuyifei1999_>	 I might have to read their log files... :(
[15:17:56] <Alaa>	 he said that he try today to run his bot on Toolforge, then he faced that his bot fall under the block and not run
[15:18:26] <Alaa>	 I'm trying to translate what he said :D 
[15:20:15] <zhuyifei1999_>	 can you ask him where the bot is running from? if directly, ask where he ssh'ed into. if on grid, ask for the output of qstat which the bot is running
[15:20:45] <zhuyifei1999_>	 though I still think CUs are better for finding who's been using a toolforge IP
[15:21:39] <zhuyifei1999_>	 fwiw, the logs of 'tools.sammour' is flooding. it'll take me a long time to find an error message
[15:21:48] <Alaa>	 O.o all this
[15:22:17] <zhuyifei1999_>	 *which the bot is running => when the bot is running
[15:22:43] <Alaa>	 The user is inactive now, so we should wait
[15:23:02] <Alaa>	 I'll told him to send an email to you, okay?
[15:26:30] <zhuyifei1999_>	 um, can he email tool-labs-standards-committee@lists.wikimedia.org and CC zhuyifei1999@gmail.com?
[15:37:10] <Alaa>	 zhuyifei1999 ofc
[15:39:40] <zhuyifei1999_>	 k thanks
[15:51:04] <subbu>	 can someone help me with some vagrant and nfs complaints i am seeing on a labs vm?
[15:55:34] <revi>	 and hmmmm from the information I have, there's no WMF IPs involved
[15:56:09] <revi>	 so yeah they should mail the team directly
[15:56:53] <subbu>	 andrewbogott ?
[15:57:15] <revi>	 meanwhile I thought 52000 edits in 5 minutes should trigger ratelimit
[15:57:19] <andrewbogott>	 subbu: I can help in a few minutes… what's the instance and project?
[15:57:34] <subbu>	 mw-base.wikitextexp.eqiad.wmflabs
[15:59:59] <subbu>	 https://gist.githubusercontent.com/subbuss/2e2b1fb80c49339da031dab0975ba67d/raw/25064d793060d2713b86b8f22ea9b126d99cf3b4/gistfile1.txt
[16:00:54] <subbu>	 but, on a different vm (mw-expt.wikitextexp.eqiad.wmflabs) which is identical as far as i know, it moves past that just fine.
[16:09:28] <andrewbogott>	 subbu: ah, sorry, that's deep in the workings of Vagrant which I don't know much about.
[16:09:52] <andrewbogott>	 It's probably worth just tearing down any existing Vagrant things and then rebooting the host to see if that gets it unstuck
[16:09:58] <paladox>	 subbu try mwvagrant reload
[16:10:03] <paladox>	 sometimes it does that
[16:10:11] <paladox>	 so you have to keep reloading until it work
[16:10:59] <subbu>	 paladox, i see .. i tried reloading once and it didn't work.. but, if that is the trick, i will try a couple more times :)
[16:11:07] <paladox>	 ok :)
[16:12:28] <subbu>	 paladox, ha .. it took me 4 tries .. but that did it!
[16:12:41] <paladox>	 :)
[19:34:51] <ASammour>	 Hello all. In arabic wikipedia there was 3 attempts of vandalism, and the latest one was yesterday. Fortunately the abuse filters catch all the edits (52,000 hits)
[19:36:44] <Hauskatze>	 well, 52.000 hits are very concerning
[19:39:21] <ASammour>	 The checkusers in arwiki has done many checkuser for the accounts responsible for three attempts.
[19:40:56] <ASammour>	 All these attempts has done with highly rate of edit (more than 5000 edits in 1 minute)
[19:41:51] <ASammour>	 I wonder if there is a way to check the toolforge tools, to find if there any suspicious actions happened in yesterday's log?
[19:59:34] <Reedy>	 Were they actually from toolforge tools?
[20:00:42] <chicocvenancio>	 ASammour: why check toolforge logs?
[20:01:58] * chicocvenancio reading related(?) scrollback
[20:04:23] <ASammour>	 I'm not sure, but the high rate of edits make me wonder if they use toolforge.
[20:06:34] <chicocvenancio>	 there was previous discussion here saying the same
[20:06:46] <chicocvenancio>	 Alaa: is this handled?
[20:07:42] <Alaa>	 chicocvenancio: No, because the bot operator was inactive, but 
[20:07:50] <Reedy>	 You can do high rate of edits anywhere
[20:07:58] <Alaa>	 but now he's active ASammour
[20:08:52] <chicocvenancio>	 Reedy: from the scrollback, an ip was blocked and an user failed to edit from toolforge, they're assuming the IP was WMCS' and that it was abused 
[20:08:52] <Alaa>	 We are here now
[20:08:52] <Alaa>	 https://phabricator.wikimedia.org/T192668
[20:09:12] <Reedy>	 This is easy to work out
[20:09:13] <Alaa>	 Can any user use open proxy with toolforge?
[20:10:01] <chicocvenancio>	 Alaa: that is considered abuse
[20:11:04] <Reedy>	 it's an ipv6 address that one use edited from
[20:11:41] <chicocvenancio>	 Reedy: is it WMCS'?
[20:11:49] <Reedy>	 The first one isn't
[20:12:04] <Reedy>	 it's registered to uk2.net
[20:12:39] <Reedy>	 Another uk2
[20:14:59] <chicocvenancio>	 sounds like it was not toolforge abuse
[20:16:49] <Reedy>	 Few in the US.. Some from egypt
[20:16:57] <chicocvenancio>	 ASammour, Alaa: what about tailoring new abusefilters for these edits?
[20:17:29] <Alaa>	 today, the user caught with 52,000 hits with ar.wiki abusefilters :D
[20:17:39] <Alaa>	 52,000 hits in around 15 min 
[20:18:29] <chicocvenancio>	 sounds like it is very automated
[20:18:56] <Reedy>	 chicocvenancio: It would look like none of the edits came from wmcs
[20:19:08] <chicocvenancio>	 wgRateLimits is a good idea, but I'd consider Abusefilter the first line and best defense
[20:19:12] <chicocvenancio>	 Reedy: Great!
[20:19:28] <Reedy>	 Considering the user agent of one of the users is FF 59 (granted, it can be faked)
[20:20:10] <Alaa>	 the problem here that 
[20:20:52] <Alaa>	 But when the approximate count of hits in last 24 hours will be at least 5 % from count of all edits that was done in last 24 hours, the filter is turned off automatically as a security mesaure.
[20:21:24] <chicocvenancio>	 ha
[20:21:29] <zhuyifei1999_>	 block them ;)
[20:21:38] <Alaa>	 block whom?
[20:21:50] <chicocvenancio>	 the user hitting the filter
[20:21:59] <zhuyifei1999_>	 yeah
[20:22:24] <chicocvenancio>	 in an emergency, head to #wikimedia-stewards and ask for the block there (if no local sysops can take care of it)
[20:23:01] <Alaa>	 I'm a steward :D 
[20:23:35] <zhuyifei1999_>	 if they are not running the edits from wikimedia cloud, there's nothing we can do about it unfortunately :(
[20:23:54] <Alaa>	 Yeah, I notice that :/
[20:25:21] * chicocvenancio laughs about attempting to teach steward stuff to a steward
[20:26:17] <Alaa>	 :D :D I'm a new one elected before 2 months :D 
[20:27:11] <chicocvenancio>	 that does not improve my blunder. I actually saw your name on the election 2 months ago...