[08:26:49] <bennofs>	 is there some way to debug processes running on the toolforge grid?
[13:57:39] <Sagan>	 does somebody know if there was a problem with memcached or similar? We had a problem in dewiki with an abusefilter that should have matched, but didn't
[13:58:00] <Sagan>	 and it's kind of annoying, because the purpose is to enforce arbcom actions
[13:59:43] <James_F>	 Sagan: Nothing obvious comes to mind; could you file a Phabricator task?
[14:01:24] <Sagan>	 ok, will do then
[14:06:40] <jeblad>	 Seems like an old account of mine didn't get merged properly when we moved to global accounts. Anyone still doing cleanup of such problems?
[14:09:55] <Sagan>	 jeblad, maybe the stewards can help you with that?
[14:12:09] <jeblad>	 Perhaps I should just link them to the same account. No email, so I can't reset the password.
[14:12:45] <jeblad>	 Perhaps that is why the merge failed.
[14:32:52] <Daimona>	 Hello, could someone with shell access help me analyze some cache keys, please? (Or redirect me to the right place to ask)
[14:36:45] <bawolff>	 If nobody responds, you could try #wikimedia-operations
[14:36:54] <bawolff>	 I'd offer to help but i don't have shell access anymore
[14:37:13] <Reedy>	 Daimona: How many are we talking about?
[14:37:15] <Daimona>	 Sure, thanks. I wanted to try here because it's not serious
[14:37:21] <Daimona>	 Reedy: In theory, one
[14:37:31] <Daimona>	 But maybe we don't even need to check the key
[14:37:38] <Daimona>	 Debug logs could be sufficient
[14:39:46] <bawolff>	 To certain extent though, memcache keys are best effort, maybe the throttling key referenced in that bug just happened to be evicted from cache
[14:40:54] <Daimona>	 That's what I was thinking
[14:41:34] <Daimona>	 That said, I think I've rebuilt what the key should look like
[14:41:41] <Daimona>	 But maybe it's better if we move to PM
[14:42:14] <bawolff>	 Its not like this channel is so busy that you're interferring with other conversations ;)
[14:42:36] <Daimona>	 It was just a precaution in case this is considered somewhat private :)
[14:42:50] <Daimona>	 But well, it should already be pretty obvious from the task
[14:43:24] <bawolff>	 They key wouldn't have an IP address in it, and the abusefilter that uses it is public, so I don't think it could be considred private data
[14:43:41] <Reedy>	 I mean, it's easy enough to sanity check it before re-posting it
[14:43:48] <Daimona>	 Sure
[14:43:49] <Reedy>	 Unless it's many lines lon
[14:43:51] <Reedy>	 g
[14:44:03] <Daimona>	 So, in theory, it should be enough to grep debug logs for: dewiki:abusefilter:throttle:290:page:Wikipedia Diskussion:Schiedsgericht/Anfragen/Diskussionskomplex im Umfeld "Anetta Kahane":user:Brainswiffer
[14:44:10] <Daimona>	 I'm uncertain about the quotes etc.
[14:44:39] <Daimona>	 See https://phabricator.wikimedia.org/T210709#4811680 for sample messages
[14:45:22] <bawolff>	 Are spaces even allowed in memcached keys?
[14:45:56] <Daimona>	 That's another good question
[14:46:18] <Reedy>	 I'd also presume not
[14:46:39] <Reedy>	 Passing the components to wfMemcKey() might be a saner way to find out
[14:46:41] <Daimona>	 Actually, I think we should call Title::getDBKey() instead of getPrefixedTitle
[14:46:41] <bawolff>	 Most of the rest of MW hashes page titles before putting them in memcached keys
[14:46:56] <Daimona>	 Or hashing, yes
[14:47:51] <Daimona>	 Apparently, spaces are automatically replaced with underscores
[14:48:00] <Daimona>	 Quotes seem to be allowed
[14:48:36] <bawolff>	 I think it does hash things
[14:48:49] <bawolff>	 line 842 of AbuseFilterRunner.php
[14:49:02] <bawolff>	 $identifier = sha1( implode( ':', $identifiers ) );
[14:49:47] <Daimona>	 Eurgh
[14:50:28] <Daimona>	 Indeed...
[14:50:32] <bawolff>	 So that would make it be like dewiki:abusefilter:throttle:290:page,user:<whatever the sha1 is>
[14:50:58] <Daimona>	 Exactly
[14:51:08] <Daimona>	 But well, maybe we don't even need the sha1
[14:51:52] <Daimona>	 Grepping until "user:" may be enough
[14:52:34] <Daimona>	 Although, actually, that would be "user,page"
[14:53:44] <bawolff>	 right
[14:54:44] <Daimona>	 Now that I think about it, what could've happened is:
[14:55:10] <Daimona>	 -The key got created yesterday, and it reached 2 by the end of the day
[14:55:30] <Daimona>	 - Then it got to, say, 4 today
[14:55:44] <Daimona>	 - Then it expired, and the new key started off from 0
[14:57:29] <Daimona>	 Or maybe not, today at 06:58 the filter disallowed an edit, so by that time the key was at 5
[14:57:57] <Daimona>	 But there've been 7 edits since then, in less than 24 hours
[14:59:57] <bawolff>	 So the hash would be sha1( "404212:Wikipedia Diskussion:Schiedsgericht/Anfragen/Diskussionskomplexim Umfeld \"Anetta Kahane\"" )
[15:00:38] <bawolff>	 which is ce12284db0992734fd982f4a3ddd1f5f2b5c5408
[15:00:58] <bawolff>	 so dewiki:abusefilter:throttle:290:user,page:ce12284db0992734fd982f4a3ddd1f5f2b5c5408
[15:01:08] <bawolff>	 I think
[15:01:42] <Daimona>	 seems correct
[15:03:06] <Daimona>	 Rn my main guess is that the key was evicted for whatever reason
[15:05:49] <bawolff>	 So Abusefilter has debug level logging at mwlog1001. I don't have access to that anymore (only logstash which only does "info" level logging), but that debug logging should be enough to verify what happened with this key
[15:07:26] <bawolff>	 Reedy: Any chance you could paste on to the bug, what the results of grep ce12284db0992734fd982f4a3ddd1f5f2b5c5408 AbuseFilter.log is (on mwlog)
[15:07:38] <Reedy>	 I can have a look, yeah
[15:07:44] <Daimona>	 Agreed - and I also don't have access to mwlog. Poking the key should be unnecessary unless something ill is happening
[15:08:00] <Reedy>	 Obviously, stating the obvious... Something isn't guaranteed to be in MW before its expiry time
[15:08:12] <bawolff>	 That kind of assumes I managed to reconstruct that hash right, not sure how confident I feel about that
[15:09:53] <bawolff>	 Maybe this is a sign i should try and get my shell access back :)
[15:11:34] <Reedy>	 0 results... But like you say, dunno if that means the hash is right ;p
[15:12:33] <Daimona>	 This means we're doing it right
[15:12:56] <Daimona>	 It would've been too easy otherwise
[15:13:11] <Daimona>	 What happens if you grep "dewiki:abusefilter:throttle:290:user,page" instead?
[15:30:19] <Reedy>	 reedy@mwlog1001:/srv/mw-log$ grep "dewiki:abusefilter:throttle:290" memcached.log 
[15:30:19] <Reedy>	 reedy@mwlog1001:/srv/mw-log$ 
[15:57:17] <Daimona>	 Uhm something's missing?
[16:16:29] <Reedy>	 reedy@mwlog1001:/srv/mw-log$ zgrep "dewiki:abusefilter:throttle:290" archive/memcached-keys.log-201912*
[16:16:29] <Reedy>	 reedy@mwlog1001:/srv/mw-log$ 
[16:23:49] <Daimona>	 I think it should be in mwlog (?)
[16:56:32] <Reedy>	 Daimona: ?
[16:56:36] <Reedy>	 The files are there on disk
[16:57:03] <Reedy>	 reedy@mwlog1001:/srv/mw-log$ ls -al memcached.log 
[16:57:03] <Reedy>	 -rw-r--r-- 1 udp2log udp2log 6179236 Dec 17 16:56 memcached.log
[17:26:45] <Daimona>	 Reedy: And is there any line like https://phabricator.wikimedia.org/T210709#4811680 ?
[17:28:16] <Reedy>	 Not that specific one, but yes
[17:28:52] <Reedy>	 reedy@mwlog1001:/srv/mw-log$ grep -c ":abusefilter:" memcached.log 
[17:28:52] <Reedy>	 529
[17:29:17] <Reedy>	 lots of "ITEM TOO BIG"
[17:35:53] <Daimona>	 Erm...
[17:36:16] <Daimona>	 Could you please paste them somewhere?
[17:40:41] <Reedy>	 they're mostly check-stash from what i can see
[17:40:56] <Reedy>	 that's almost all that's in memcached.log for AF
[19:33:25] <Nemo_bis>	 I keep getting error 503s...
[21:42:10] <jeblad>	 Huh, login.wikimedia.org hangs?
[21:44:00] <jeblad>	 Got a timeout on the auth key but was still logged in? 
[21:54:08] <Reedy>	 A few people complaining about some potential EU issues
[21:54:25] <jeblad>	 It is BREXIT!!
[21:54:34] <cdanis>	 indeed, was just looking at that.  looks like ATS on cp3050 was freaking out, not sure why yet.
[21:54:37] * jeblad off to get his coffee
[21:54:38] <cdanis>	 I've depooled the host.
[21:56:08] <jeblad>	 Note that I got logged in even if the auth key had timed out. That could be an idication of a defunc 2fa system.