[08:25:30] <don-vip>	 I have requested a memory increase for my tool: https://phabricator.wikimedia.org/T230284#6087891
[09:54:34] <Nemo_bis>	 "ERROR 2013 (HY000): Lost connection to MySQL server during query"
[09:54:48] <Nemo_bis>	 Did the timeouts/querykillers become more aggressive on the replicas
[09:56:51] <Nemo_bis>	 Seems to be a couple minutes now
[10:31:57] <arturo>	 Nemo_bis: perhaps, I don't know. I suggest you send an email to the cloud@ mailing list to talk with folks
[10:33:25] <arturo>	 don-vip: we will discuss the request in the next team meeting
[10:33:45] <don-vip>	 arturo: thanks! :)
[12:24:55] <Nemo_bis>	 arturo: ok thanks, done https://lists.wikimedia.org/pipermail/cloud/2020-April/001073.html
[12:38:50] <arturo>	 thanks!
[12:39:46] <arturo>	 Nemo_bis: I believe you have a reply now
[13:38:57] <Examknow>	 !help
[13:38:57] <wm-bot>	 If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-kanban
[13:39:04] <Examknow>	 PAWS has 504 error
[13:39:09] <arturo>	 ouch
[13:39:38] <arturo>	 Examknow: which URL?
[13:40:06] <arturo>	 the first 2 URLs I tried just work
[13:40:30] <Examknow>	 oh NVM it works now :D
[13:40:40] <Examknow>	 sorry temporary issue
[13:41:56] <arturo>	 heh
[13:42:06] <arturo>	 fair
[15:23:35] <AmandaNP>	 Maybe asking in here will help. Anyone know about recent changes or issues with OAuth?
[15:23:50] <AmandaNP>	 I'm getting E004 for the first time ever
[15:32:59] <bd808>	 AmandaNP: hmmm... nothing I have heard of recently becoming widespread. https://www.mediawiki.org/wiki/Help:OAuth/Errors#E004
[15:33:28] <AmandaNP>	 I don't get why it would stop now though, like how many years into working...
[15:33:43] * AmandaNP starts to pull out dev tools
[15:42:32] <AmandaNP>	 I don't even know where to start looking to diagnose this..
[16:38:00] <AmandaNP>	 bd808: I see the WMF released 1.35wmf30 today...I am assuming it's something with https://www.mediawiki.org/wiki/MediaWiki_1.35/wmf.30#OAuth
[16:38:07] <AmandaNP>	 stwalkerster: ^
[16:38:26] <Reedy>	 AmandaNP: That not on all wikis though
[16:38:47] <Reedy>	 you can see the list on https://tools.wmflabs.org/versions
[16:39:20] <AmandaNP>	 True, but oauth uses mediawikiwiki
[16:39:25] <AmandaNP>	 Which is group 1
[16:39:31] <AmandaNP>	 err 0
[16:39:35] <bd808>	 click the group to expand
[16:40:19] <AmandaNP>	 ya mediawikiwiki is definitely on 1.35wmf30
[16:40:21] <bd808>	 but generally, 0 is testwiki and mediawiki.org; 1 is everything except wikipedias, and 2 is wikipedias
[16:43:14] <Reedy>	 The error logs for OAuth don't look to have changed for mediawiki.org since the train
[16:44:31] <Reedy>	 huge spice overnight UTC
[16:45:12] <Reedy>	 I just did https://test.wikipedia.org/w/index.php?title=User_talk%3AReedy&type=revision&diff=427937&oldid=180259
[16:45:21] <Reedy>	 Using https://tools.wmflabs.org/oauth-hello-world/enduser.php
[16:45:37] <AmandaNP>	 Oh yes, OAuth is working for other applications, I know that much
[16:45:53] <Reedy>	 I was just confirming it wasn't completely broken
[16:45:57] <Reedy>	 Cause that's a slightly different issue
[16:45:58] <AmandaNP>	 fair
[16:46:19] <Reedy>	 I moved a load of code around, and if it was fully broken, it was going to be likely my fault
[16:46:29] <Reedy>	 I'm not saying that your breakage isn't my fault, obviously
[16:48:10] <AmandaNP>	 I'm trying to find the url that is set in the code to test another wiki, but someone else coded this and it's not making it easy
[16:51:42] <AntiComposite>	 https://github.com/UTRS/utrs/blob/136d7dbb23c76602ccf5e8f60a9cafef36ef4c65/public_html/login.php#L60 ?
[16:51:46] <Reedy>	 Hmm
[16:51:51] <Reedy>	 I wonder if it's upset the caching somehow
[16:53:29] <Reedy>	 As it can return an instance of MWOAuthToken
[16:56:50] <Reedy>	 But that's only cached for ~10 minutes
[16:57:59] <Reedy>	 13:03 liw@deploy1001: rebuilt and synchronized wikiversions files: group0 wikis to 1.35.0-wmf.30
[16:58:06] <Reedy>	 So nearly 5 hours ago
[17:05:33] <AmandaNP>	 Reedy: so I just changed the URL to enwiki to login, and it works fine
[17:05:53] <AmandaNP>	 AntiComposite: ya sorry already found it
[17:06:52] <AmandaNP>	 Do you wish for me to open a Phab ticket for this?
[17:07:31] <bd808>	 AmandaNP: so the bug you are seeing is that OAuth authorization fails against mediawiki.org but succeeds against enwiki using the same client code and grant?
[17:08:46] <bd808>	 if so that does sound likely to be somehow related to today's train
[17:08:56] <AmandaNP>	 I get E004 on using mediawiki.org, but works fine on enwiki. All I changed was https://github.com/UTRS/utrs/commit/5adf8916b304f7d38efa866609c32aecad07f46c
[17:09:12] <bd808>	 *nod*
[17:09:16] <Reedy>	 		return "OAUTH:$wgMWOAuthCentralWiki:" . implode( ':', $args );
[17:09:29] <Reedy>	 bd808: As above, it feels cache-y relating to the changing of NS of the cache object
[17:09:35] <bd808>	 We should start a Phabricator task for this if there is not one yet 
[17:09:37] <Reedy>	 So might be worth the back compat alias
[17:10:17] <bd808>	 Reedy: hmmm could be yeah
[17:10:40] <Reedy>	 It's the right sort of area
[17:10:42] <Reedy>	 Considering...
[17:10:53] <Reedy>	 			if ( $token === null || !( $returnToken instanceof MWOAuthToken ) ) {
[17:10:53] <Reedy>	 				throw new MWOAuthException( 'mwoauthdatastore-request-token-not-found', [
[17:11:01] <Reedy>	 And then the rest of the code ends up linking E004
[17:11:14] <Reedy>	 So when is MWOAuthToken not an MWOAuthToken..
[17:17:06] <Reedy>	 https://gerrit.wikimedia.org/r/593008
[17:18:50] <bd808>	 Reedy: I think it would be good ot have a bug to link to that. And maybe a comment about how it should be safe to remove after... some time?
[17:18:58] <Reedy>	 Yeah
[17:19:06] <Reedy>	 CI is gonna shout at me anyway :P
[17:19:42] <bd808>	 What's the cache TTL for those?
[17:20:34] <Reedy>	 10 minutes from what I can see
[17:20:54] <Reedy>	 		$this->cache->set( $cacheKey, $token, 600 ); // 10 more minutes. Kindof arbitray.
[17:21:04] <Reedy>	 But the cache key is
[17:21:05] <Reedy>	 		return "OAUTH:$wgMWOAuthCentralWiki:" . implode( ':', $args );
[17:21:09] <Reedy>	 So shared-y/globally
[17:21:39] <bd808>	 oh... so if its being hit from Group 0 wikis and Group 2 wikis... battle
[17:21:47] <bd808>	 yuck
[17:22:10] <Reedy>	 Yeah. Not 100% sure, but it seems very likely
[17:22:24] <Reedy>	 Especially as we're serialising in some way the object to stick it into cache
[17:23:04] <bd808>	 I think the object is basically a named tuple with 2 slots if I remember right from N years ago
[17:25:52] <AmandaNP>	 sorry, got caught up with a local CU
[17:26:38] <AmandaNP>	 do you want me to test a group 2 wiki?
[17:27:05] <bd808>	 AmandaNP: you did when you switched to enwiki
[17:27:18] <AmandaNP>	 oh I thought they were group 1 for some reason
[17:27:32] <bd808>	 I think Reedy is on the track of this and it is related to today's train hitting mediawiki.org
[17:27:49] <bd808>	 its a yucky cached object bug
[17:28:19] <AmandaNP>	 ok, do we want a phab for this still? And should I be the one to file?
[17:28:48] <bd808>	 yes, please. Unless you already made a bug Reedy?
[17:28:55] <Reedy>	 Nope, was poking for a fix
[17:29:05] <AmandaNP>	 kk, i'll go make this
[17:29:21] <Reedy>	 thanks
[17:32:23] <Reedy>	 I think that PS should work... But phan will no doubt bitch
[17:33:29] <Reedy>	 Oh, nope, it's fine because subclasses etc
[17:33:52] <Reedy>	 I'll amend the commit summary to add more detail when we have the bug number
[17:34:15] <AmandaNP>	 https://phabricator.wikimedia.org/T251306
[17:36:02] <AmandaNP>	 heh Jdforrester
[17:36:46] <Reedy>	 bd808: https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/OAuth/+/593008/
[17:43:41] <bd808>	 Reedy: so pretty much only needed as a backport in theory I guess?
[17:43:48] <Reedy>	 In theory
[17:43:52] <Reedy>	 James_F just merged to master :D
[17:44:05] <bd808>	 :) that works too
[17:44:05] <Reedy>	 Which isn't probably a bad thing, incase we need to carry it forward
[17:44:19] <James_F>	 Yeah, caches being what they are.
[17:44:22] <Reedy>	 But if we see it's fixed in prod on friday/over the weekend, we can revert it out
[17:44:27] * James_F nods.
[17:44:39] <Reedy>	 And we've been bitten by not merging to master before...
[17:44:46] <bd808>	 thanks for jumping on that Reedy. and thanks for being the canary AmandaNP 
[17:44:53] <Reedy>	 :)
[17:44:55] <AmandaNP>	 np
[17:44:58] <Reedy>	 I try and fix what I break... usually
[17:45:06] <Reedy>	 AmandaNP: Mind testing the fix when it's deployed?
[17:45:15] <AmandaNP>	 sure, just let me know when
[17:45:22] <Reedy>	 Probably in ~10 mins
[17:45:22] <AmandaNP>	 UTRS is still marked as down anyway
[17:45:25] <Reedy>	 See how long jerkins takes
[17:48:27] <James_F>	 OAuth has a short CI pipe.
[17:51:35] <Reedy>	 Just over 6 minutes :P
[17:51:45] <Reedy>	 AmandaNP: Feel free to try now
[17:54:58] <AmandaNP>	 Ok, I need a few more mins
[17:55:17] <Reedy>	 Sure, no rush
[18:02:46] <AmandaNP>	 Reedy: yes, working now
[18:02:59] <Reedy>	 Sweet. Thanks for testing and reporting
[18:03:21] <James_F>	 Reedy: Should we Resolve?
[18:03:28] <Reedy>	 I just did :)
[18:04:31] <James_F>	 Ta.
[22:58:04] <bstorm_>	 !log tools rebuilding docker-registry.tools.wmflabs.org/maintain-kubeusers:beta T247455
[22:58:07] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[22:58:08] <stashbot>	 T247455: Dumps not accessible from container pods - https://phabricator.wikimedia.org/T247455