[00:13:38] 14:00:48 why did people think my proposal to have a "check for life" for a tool every 6 months was a bad idea? <-- I think people are still scarred from how poorly it went on the Toolserver, which didn't encourage multi-maintainer tools or publishing source code. So when a tool hoster/maintainer vanished, it it was dead, for good. [00:15:00] [telegram] Not sure that is incredibly different today. [00:16:06] [telegram] Things don’t *have* to die, but I haven’t seen a lot of folks trying to save old, abandoned tools. [00:19:07] I think the main thing is that (in theory) all the code is freely licensed, and Toolforge roots are empowered to give it to people who are at least interested in reviving stuff. Previously even if someone was interested in reviving a tool, Toolserver roots couldn't necessarily do that [00:20:33] (or maybe they could and chose not to :shrug:) [00:33:15] !log admin shutting down cloudservices2002-dev to see if we can live without it. This is in anticipation or rebuilding it entirely for T253780 [00:33:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [00:33:18] T253780: Upgrade cloudservices nodes to Debian Buster - https://phabricator.wikimedia.org/T253780 [07:48:22] Hi. [07:48:44] I am trying to use the urlShortener API on toolforge. [07:49:09] I am getting this error "urlshortener-blocked". [07:49:21] What must I do? [08:03:44] Do I need to specify something in the header? [08:30:55] acagastya: I'm not familiar with that piece of software. Perhaps send an email to the mailing list cloud@ ? [08:31:54] Thanks, arturo. I will wait till EOD -- if I get no reply, I will put it on the mailing list. [08:35:34] 👍 [08:42:56] I have noted, arturo, my timezone is prolly a bad timezone to be in. Most of the devs are sleeping while I would be coding. [09:05:23] acagastya: that error message occurs when the ip/user is blocked. are you logged in? [09:11:26] Majavah: that is used by a tool on toolforge. [09:12:21] Not unless there is a block-first condition for toolforge tools. [09:14:45] I wouldn't be surprised if anon editing from wmcs was blocked [09:17:07] What does it have to do with wmcs? [09:17:43] Or anon editing. [09:18:07] The bot is not trying to edit -- it is merely querying the API. [09:19:54] Is it trying to create a short url on get details of existing one? [09:21:07] Let's consider enwp's page Fiji. [09:22:17] It sends a post request to https://en.wikipedia.org/wiki/India -- and ideally it should (if the shortened URL does not exist; create a shortened url, and then) return the shortened url. [09:22:58] I can run the tool on my laptop -- and it works fine. But when I run it on toolforge -- that time it does not work. [09:23:30] acagastya: would https://meta.wikimedia.org/wiki/Special:UrlShortener work ? [09:23:37] that's the "w.wiki" shortener [09:23:38] Yes, and creating a short url requires the account/ip not be blocked [09:23:56] It does not require an account. [09:24:06] And yes, I am talking about w.wiki indeed. [09:24:33] It has only one API endpoint, as far as I know. [09:24:48] So is it blocked for the toolforge tools? [09:24:56] If so, how do I request access? [09:25:17] i just tested the link above and was able to create a short url [09:25:20] Just log in? My guess would be that toolforge has anon-only block [09:25:22] without logging in [09:25:34] and i did not see it related to toolforge [09:25:47] What do you mean I should log in? [09:26:18] In the exact same way you can authenticate any other API request [09:26:47] The bot is trying to query the endpoint where it is hosted. [09:27:12] When I run locally, I don't have to authenticate it -- it is not tied to any OAuth login. [09:27:56] I'll be at home in 10 mins, I'll explain better then [09:31:13] The API most certainly does not require one to login. [09:32:30] If I were to use Tor, w.wiki is blocked because the IP is blocked because it is open proxy. [09:33:17] But if I host it locally, or run the query on a ReST client such as insomnia -- it isn't blocked. [09:33:36] The docs say nothing about any token required for this. [09:34:10] And if the IP of toolforge server is blocked by default, is there a way to request to be whitelisted? [09:43:13] based on the source code, that error message can't occur unless the server thinks that the account/ip is blocked, right? https://github.com/wikimedia/mediawiki-extensions-UrlShortener/blob/master/includes/UrlShortenerUtils.php#L49 [09:44:34] Yes. [09:45:13] yes, and you create them normally without logging in from your own laptop, as that does not require logging in, as your home IP is not blocked [09:45:25] but you can't do that from toolforge, with the exact same code [09:45:32] correct? [09:45:43] So, toolforge IP must be blocked, right? [09:45:59] that's the only possible explanation I get [09:46:09] let me check if I can create them from toolforge when I've logged in [09:46:46] If toolforge is blocked, is there a way to request whitelist? [09:48:09] acagastya: most likely that way is to create a ticket in phabricator and ask there [09:49:13] Is that the only option? Just asking before I file a ticket. [09:53:33] the easiest way is still probably just to login with any account [09:54:14] How would I do that? [09:54:23] The API does not request any token. [09:54:27] It is an IRC bot. [09:55:42] if you're using a mediawiki api framework like pywikibot, they have it built-in. if not, you send an api call to action=login [09:55:45] see https://www.mediawiki.org/wiki/API:Login#Method_1._login [09:57:23] It is just an IRC bot, written in NodeJS, sends a POST request to the API. No it does not use a framework. Just a fetch request. [09:59:36] unfortunately this is annoyingly complicated for a simple operation like that :( [09:59:45] the first step is that you send a request to https://meta.wikimedia.org/w/api.php?action=query&meta=tokens&type=login&format=json [10:00:28] you get that token into a variable [10:00:57] Even if I login -- I don't see the API docs mentioning any param to supply a token. [10:04:35] I think that by default it just adds a login token to current session, but it's been a while since I last did anything without a library of some sort [10:04:51] but I think that by default keeps your account logged in by a session [10:05:46] If I knew the param to supply the token -- I would just copy the token and put it in params. [10:06:48] I think there should be a mechanism to whitelist (upon request) tools from toolforge. [10:07:07] !log tools.zppixbot-test tweaking stuff for testing [10:07:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot-test/SAL [10:07:14] in an ideal world, yes, but how do you know which request comes from which tool? [10:11:43] I don't know -- maybe someone who knows more about how this extension works could look into it. [10:12:53] mmmm [10:13:08] What do you suspect, Majavah, which parameter and what content should I add to the request? [10:13:21] the IP range of toolforge, and domains, are well-known [10:13:49] is anyone having issues with git. Git pulling from github's now just hanging? [10:14:09] * RhinosF1 has noticed Notifico and Travis are slow so it might be a github issue [10:14:19] RhinosF1: tried few hours ago and it worked. [10:14:33] Reception123: anything can happen in hours [10:14:37] acagastya: [10:15:16] acagastya: I suggest you open a phab task about the URL shortener thing [10:15:16] Well -- assuming someone tells me which param I need to add -- I would have to git clone again. [10:15:22] arturo: I'm not seeing any blocks in Special:BlockList for WMCS/toolforge but curl 'https://en.wikipedia.org/w/api.php?action=query&meta=userinfo&uiprop=blockinfo&format=json' suggests that they're blocked by... user without empty username [10:15:44] acagastya: no idea about the param name, sorry [10:16:17] Majavah: could you rephrase "they're blocked by... user without empty username"? [10:17:00] acagastya: raw json better? {"blockid":null,"blockedby":"","blockedbyid":0,"blockreason":"{{Private IP range softblock|172.16.3.190}} "} [10:17:21] "systemblocktype":"wgSoftBlockRanges" [10:18:08] aha, https://www.mediawiki.org/wiki/Manual:$wgSoftBlockRanges "IP ranges that should be considered soft-blocked (anon-only, account creation allowed). The intent is to use this to prevent anonymous edits from shared resources such as Wikimedia Cloud VPS. " [10:19:05] Okay -- that means a token might save the day -- I just need to figure out which one. [10:20:10] !log tools.zppixbot Known inconsistency in git - unable to pull changes [10:20:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL [10:21:00] !log tools.zppixbot Known inconsistency in git - unable to pull changes (NOW RESOLVED) [10:21:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL [10:21:52] acagastya: https://www.mediawiki.org/wiki/API:Edit#Example has a JS code example for editing pages, in theory you should just be able to change the last step from that code to make it work [10:29:28] filed T253844 for those blocks not being visible anywhere on the UI [10:29:29] T253844: Show blocks created by $wgSoftBlockRanges when searching for a block for a given IP - https://phabricator.wikimedia.org/T253844 [10:30:36] !log toolsbeta livehacking puppetmaster to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/599139 [10:30:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:31:00] !log toolsbeta livehacking puppetmaster and toolsbeta-proxy-1 to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/599139 (T253816) [10:31:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:31:02] T253816: *.toolforge.org hostnames unexpectly treated as tools.wmflabs.org when the URL's path starts with a match to a grid engine webservice - https://phabricator.wikimedia.org/T253816 [10:37:44] thanks Majavah [10:43:05] Majavah: Looking into. [10:44:58] So if I got it right -- appening &token=csrfToken should suffice, right? [10:45:27] Now, I had created an account for my bot ages ago -- and it does not have a csrf token. [10:45:58] I am also trying to understand BotPasswords. [10:47:08] acagastya: so bot passwords are just a way of allowing your bot account authenticate without adding the real password somewhere. you can create those at Special:BotPasswords on-wiki [10:47:35] Using my account, or the bot account? [10:47:45] your bot account [10:48:04] Okay. [10:48:25] And that IP address textbox -- I don't know what to do with that. [10:50:31] easiest way is to leave it as default [10:52:43] Okay -- when I try to access csrf token with bot account, I see a string. [10:52:59] Should I just use it, or should I still go for bot passwords? [10:53:07] I think that the CSRF token is the wrong thing here [10:53:41] all examples I could find store the authentication data in HTTP session storage [10:54:27] Would I have to log in every time the bot is restarted, right? [10:54:37] yes [11:05:28] Majavah: what I don't understand is why did I need bot password? [11:06:12] acagastya: you need that to log in to MediaWiki, so you can use the url shortener api without being blocked [11:06:18] It is a four step process -- but when requesting a csrf token -- we don't make use of any data we fetched from the first two. [11:07:10] For posting the request, we need just the csrf token, right? Not the bot password. [11:07:13] actually step 3 is unneeded now, but step 2 sets a cookie to the HTTP session object [11:07:23] you need the cookie from step 2 [11:08:10] If the step 3 is unneeded, wait? Don't we need csrf token? [11:08:28] no, I think that CSRF token is not needed here [11:09:05] A bot can have a serverside cookie? [11:09:17] I don't know -- /me goes to try. [11:09:41] yeah, that's why every single example creates some sort of cookie jar [11:12:10] In that case ... Is that `jar: true` important? [11:13:15] yes, that's the part making everything work [11:27:23] !log tools merging change to front-proxy: https://gerrit.wikimedia.org/r/c/operations/puppet/+/599139 (T253816) [11:27:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:27:27] T253816: *.toolforge.org hostnames unexpectly treated as tools.wmflabs.org when the URL's path starts with a match to a grid engine webservice - https://phabricator.wikimedia.org/T253816 [11:27:30] !log toolsbeta cleanup livehackings [11:27:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [11:28:50] ahoyhoy - I have some questions about puppet setup in deployment-prep. Is here the right place to ask or should I ask in #wikimedia-releng? [11:29:33] hnowlan: try the different channels I would say :-) [11:31:03] cool :) [11:41:38] Majavah: I get a 301 when I try to run the second step. [11:41:59] Is there a way I can check I am logged in or not? [11:42:30] acagastya: what exact url you're trying to call? [11:43:03] api.php?action=query&meta=userinfo&format=json shows the current user [11:43:25] POST req to https://mediawiki.org/w/api.php with these params: [11:43:44] action, lgname, lgpassword, lgtoken format [11:45:36] where do you get redirected? 301 is a redirect [11:48:01] I tried to check if it logged me in or not: I get this. [11:48:10] {"batchcomplete":"","query":{"userinfo":{"id":0,"name":,"anon":""}}} [11:50:05] I don't know where is it redirected -- does not give me a target. [11:50:59] thar result means that you arent logged in [11:51:11] the target is in "Location" header [11:54:38] At the end of step two, csrf is empty. [11:54:51] Now I need to figure out how to read the header from requests. [11:55:22] Great -- I have now tripped too many login attempts. [12:04:10] * zhuyifei1999_ thinks it's probably gonna be easier if you use a bot library [12:05:01] Is there a bot library for Node? [12:05:09] yes [12:05:29] let me find it [12:08:44] Wow. Request is deprecated. [12:08:53] acagastya: note that request != requests [12:09:46] I don't think the code has any mention of requests. [12:10:25] ohh somehow read that code wrong :/ [12:10:37] zhuyifei1999_: are you looking for https://www.npmjs.com/package/mwbot [12:11:09] https://www.mediawiki.org/wiki/API:Client_code#JavaScript [12:11:22] I was looking for individual bots [12:12:33] Step 2 is failing -- incorrect password. [12:13:34] Incorrect username/password. [12:14:16] Tell me, username is the username of my bot user account on MediaWiki, or is it the name I entered when creating bot passwords. [12:14:32] Though I don't know why they should be any different -- they have the same name. [12:14:33] username is "Account name@Bot password name" [12:15:15] Ah. [12:15:17] Wait. [12:16:35] It still managed to fail. [12:16:57] Password is the BotPassword -- the autogenerated one, right? [12:17:05] yes [12:17:47] Which I would have to hard code. [12:17:50] It fails. [12:18:54] Should it be botname@password? [12:19:26] That too does not work. [12:19:37] either username as "bot name@bot password name" and "password" OR "bot name" and "bot name" and "bot password name@bot password value" [12:19:59] If I have an account User:Foo, and in the BotPassword, I type "bar" and the password is baz... [12:20:15] Is the username Foo@bar and password baz@bar? [12:20:38] No, in that case username would be "Foo@bar" and password "baz" [12:21:14] That didn't work. [12:22:53] At this point in time, I should wonder if the example is correct? [12:25:33] Holy molly. [12:25:42] It is User:Foo's password. [12:25:56] Is this how is should be, Majavah? [12:26:16] wait what? [12:27:20] Talk about expectation subversion. [12:29:33] It logs me in, says this endpoint is deprecated and... [13:05:13] Well, Majavah, I authenticated -- and sent the code, and yet I get blocked message. [13:13:38] I tried enough -- and it is futile. [13:16:49] Thank you for your help. [13:19:16] getting blocked after login is unexpected [13:19:35] it should be anon block right? [13:20:38] Yeah. [13:21:15] I have spent hours behind this -- now I ought to step back and look at other things. [13:36:46] !log admin rebuilding cloudservices2002-dev with Buster [13:36:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:13:20] Maybe, zhuyifei1999_, a simpler way would be allowing tools to bypass the block for w.wiki -- would save a great deal. [14:14:17] My authentication code -- which despite being what was in that example didn't work -- it was longer than the tool's code itself. [14:14:31] use a library [14:15:28] Would a library be using the same code flow for authentication? [14:15:51] If so, the results won't be different -- if not, then the example code does not work. [14:17:07] usually a library gets updated more frequently and gets tested than examples [14:17:40] actually, which example are you following? I'll see what's wrong with it [14:18:44] https://www.mediawiki.org/wiki/API:Edit#Example [14:19:17] If there is an official library, or a reliable library, I would use it to get this done once and for all. [14:22:03] there has never been an 'official' library, all of them are non-wmf-maintained [14:22:27] but https://www.mediawiki.org/wiki/API:Client_code#JavaScript should be reliable [14:22:59] nodemw is used by a commons admin bot, fwiw [14:34:43] Thanks for the link, zhuyifei1999_. I will try it out. [14:34:56] k [14:49:59] !log tools cleanup /etc/apt/sources.list.d/ directory in all tools-k8s-* VMs [14:50:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [14:59:54] acagastya: I see a lot of backscroll and may have missed something, but I think the answer to your first question is that yes, anon editing is blocked from Toolforge (and all of Cloud VPS). [15:00:04] this is by design [15:04:38] Thanks for clarifying that, bd808. [15:04:50] bd808: (fortunately you are here) I am trying to login using this mwbot now -- and I am getting "You have made too many recent login attempts. Please wait 5 minutes before trying again." Is it possible for me to request temporary exempt? [15:05:03] I have a static IP. [15:06:30] acagastya: not easy to grant an exception, no. What you tripped is a protection against brute force password guessing attacks. [15:06:57] make yourself a beverage and walk around a bit. 5 minutes will pass quickly :) [15:07:32] Wasn't trying to brute force -- just trying to understand the API who does not have the best of the documentation. [15:08:06] sure, but the algorithm doesn't understand your intent, just what happens on the wiki [15:08:40] Yes, but can a temporary pass be granted? If not, I have no other option to wait. [15:08:53] acagastya: you have to wait [15:09:21] Ah. That is the highlight of the day. [15:09:23] Okay. [15:09:48] !log tools upgrading tools-k8s-control-1 to 1.16.10 (T246122) [15:09:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:09:52] T246122: Upgrade the Toolforge Kubernetes cluster to v1.16 - https://phabricator.wikimedia.org/T246122 [15:17:39] !log tools upgrading tools-k8s-control-2 to 1.16.10 (T246122) [15:17:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:17:46] T246122: Upgrade the Toolforge Kubernetes cluster to v1.16 - https://phabricator.wikimedia.org/T246122 [15:41:15] !log tools upgrading tools-k8s-control-3 to 1.16.10 (T246122) [15:41:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:41:18] T246122: Upgrade the Toolforge Kubernetes cluster to v1.16 - https://phabricator.wikimedia.org/T246122 [15:43:38] Finally it works! [15:44:04] I would try to look at the API, and see if I can avoid using the library or not. [15:44:27] But in any case, to see if it will work on toolforge, can I request temp soft block for my IP? [15:53:12] acagastya: no, we do not have the ability to change login failure rate detection based on ip. Not saying that this is 100% technically impossible, but it is not something we can just click a button for. [15:54:01] Okay -- I get it. [15:58:39] !log tools upgrading tools-k8s-worker-[1..10] to 1.16.10 (T246122) [15:58:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:58:42] T246122: Upgrade the Toolforge Kubernetes cluster to v1.16 - https://phabricator.wikimedia.org/T246122 [16:01:28] !log tools kubectl upgraded to 1.16.10 on all bastions T246122 [16:01:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:08:57] Everything works, bd808, zhuyifei1999_, Majavah. [17:09:00] Thank you all. [17:54:22] !log tools upgraded tools-k8s-worker-[11..15] and starting on -21-29 now T246122 [17:54:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:54:26] T246122: Upgrade the Toolforge Kubernetes cluster to v1.16 - https://phabricator.wikimedia.org/T246122 [18:08:07] (np) [19:33:15] upon careful inspection -- even after logging in successfully -- it is blocked. [19:33:59] Do I need to supply a token? [19:35:23] can you show the error you get acagastya ? [19:36:22] "it is blocked" -- what is "it" [19:36:38] http://dpaste.com/0CW12NM [19:37:35] sounds like you need to add a Content-Type HTTP header acagastya :) [19:37:44] oh hang on [19:37:55] ` "info": "Blocked users can't make short URLs.",` [19:37:57] this is interesting [19:38:00] you're able to edit right? [19:38:11] Yes -- I am able to. [19:38:23] error message is from this code -- https://github.com/wikimedia/mediawiki-extensions-UrlShortener/blob/cca182dd178362af9bad30833e8981b5a2cd4c59/includes/UrlShortenerUtils.php#L48-L50 [19:38:41] Yes, indeed. [19:38:42] so the next question I guess is what authed user [19:38:52] User:enwnbot [19:39:30] It is not blocked -- I used the w.wiki logged in and it works fine. [19:39:44] which wiki are you running this against? [19:39:48] https://meta.wikimedia.org/wiki/Special:CentralAuth/Enwnbot [19:39:57] The code runs locally. But when I run on toolforge -- it doesn't. [19:40:05] not attached many places, probably not autoconfirmed anywhere [19:40:23] I am using the meta for querying -- w.wiki requires that. [19:40:31] you don't need autoconfirmed for this though do you? [19:41:22] No, as I said -- I can go to w.wiki, logged in and make new ones. [19:42:40] as your account but not as your bot account? [19:43:22] Do you mean bot with bot pass? [19:43:43] (I meant, I would log in using User:enwnbot and access w.wiki and it would work) [19:44:06] ... so you can do it via the UI but not the API? [19:45:21] No, I can't -- when that is running on toolforge. [19:45:22] using eval.php on metawiki, User::newfromname('Enwnbot')->isBlocked() and ->isBlockedGlobally() both return false [19:45:26] It runs locally. [19:46:02] acagastya: are you sure that you are passing the session cookies on after your bot logs in? [19:46:11] That's really the only thing that makes sense to me here [19:47:16] acagastya: add "&assert=user" to your api calls and you can make sure that you are logged in. You will get an explict error when you are not [19:47:57] or "&assertUser=Enwnbot" to be explict about which user [19:48:51] ooh that's a good idea. forgot we could do that now [19:49:15] Assuming I have stripped everything sensitive from this JSON -- this is the details that are saved. [19:49:48] https://gist.github.com/acagastya/94f8a4a98cfc03d16d06ad99b6c86dda [19:50:39] that's the auth call, yes. But then you have to continue to use that session for subsequent calls. [19:51:58] your "works locally, fails in toolforge" sounds like it would be caused by auth actually not working and your local IP bieng allowed to edit as anon [19:53:11] bd808: this library that I am using passes the session for further calls. [19:53:30] It is npm.im/mwbot [19:53:53] prove it to me by adding "&assert=user" to the api call you make to create the short link [19:57:00] I did that. And the error message I get is the same as the dpaste link I shared above. [20:01:16] Ah, wait. [20:01:28] acagastya: how about trying a call to "action=query&meta=userinfo&uiprop=rights"? Just trying to get you to a place where ther in info to help debug [20:01:36] *where there is [20:10:45] bd808: just asking -- do I need to do that on toolforge -- or (if things are going wrong) I can try on my laptop? [20:12:31] acagastya: try from wherever, but ultimately you are saying the problem is in Toolforge [20:12:53] so it is either your code, or a weird bug I have never heard of before [20:13:52] ~30% of all edits across the Wikimedia project wikis are made from code running in Toolforge or elsewhere in Cloud VPS. If there was a wide spread problem with authed API actions we would know really soon. [20:27:54] That is quite significant! [20:44:38] it is a pretty cool stat -- https://wmcs-edits.wmflabs.org/#wmcs-edits [20:47:25] wait, so across all wikis, ~1/3 edits come from toolforge or other CloudVPS systems? that's wild [21:02:50] the number of edits to wikidata and the dependence of wikidata editing on data import tools skews that number upwards, but yeah. The bots and other editing tools that y'all make have a huge impact on the projects. :) [21:03:56] hello! susana_ and I working on the toolforge migration for plagiabot (the test account for copypatrol). It looks like we can't get the .lighttpd.conf right. This page is just a 302 to nowhere https://plagiabot.toolforge.org/ [21:04:25] the previously working rewrite rule was `url.rewrite-if-not-file += ( "(.*)" => "/plagiabot/index.php/$0" )`, we tried that and similar variations [21:04:55] https://plagiabot.toolforge.org/index.php does work, however (though it's looking in the wrong place for the assets) [21:05:09] any ideas? [21:05:17] musikanimal: drop the "/plagiabot" prefix [21:06:28] oh we have, I was just saying what worked before in case the slashes and stuff mattered [21:06:29] musikanimal: the bash tool's router rewrite is `url.rewrite-if-not-file += ( "(.*)" => "/index.php/$0" )` [21:06:34] !log tools upgrading tools-k8s-worker-[30-60] to kubernetes 1.16.10 T246122 [21:06:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [21:06:38] T246122: Upgrade the Toolforge Kubernetes cluster to v1.16 - https://phabricator.wikimedia.org/T246122 [21:07:37] right now we have `url.rewrite-if-not-file += ( "(.*)" => "/index.php/$0" )`, still no dice :( [21:09:27] musikanimal: hmm... something funky [21:09:45] the root url is emitting redirects to nowhere [21:10:24] literally sending a "location:" header with no destination [21:12:00] yeah it's weird [21:12:23] I feel like tools is running very slowly. I'm waiting for almost 5 minutes just for a composer install to install 3 dependencies. [21:12:25] I tried using the gridengine too [21:12:27] From cache [21:12:52] I just ran composer install on plagiabot and it went pretty fast, and not from cache [21:13:28] but I've definitely had that happen to me before, usually for a tool account I haven't touched in a while [21:14:12] It sits at Loading composer repositories with package information for a bit [21:14:25] Seems like the network might be bottlenecked right now. [21:14:34] Cyberpower678: from a bastion or a kuberntes pod? [21:14:43] That and someone has been running a python application on login. [21:14:54] bd808: in English please? [21:14:55] :p [21:15:15] Cyberpower678: where are you running this code? [21:15:37] From login stretch [21:15:50] if bastion and Kubernetes are not english, you may not have been paying attention around here for the last 3 years :) [21:16:14] lol. No, I likely just mix things up when I don't refer to them in a while. [21:16:40] I suppose wherever login-stretch.tools.wmflabs.org lives? [21:16:42] My guess is NFS contention from that shared bastion. Let me look [21:17:20] Cyberpower678: update your ssh config to "login.toolforge.org" for better future proofing :) [21:17:22] Someone is also running a very long running Python application on that host as well. :p [21:17:41] Cool. [21:19:22] Added ssh entry. [21:19:45] !log tools Killed 7 python processes run by user 'mattho69' on login.toolforge.org [21:19:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [21:23:22] !log tools.lziad Admins killed `node dist/index.js` process running on login.toolforge.org bastion; Use a Kubernetes container or the Job Grid please [21:23:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lziad/SAL [21:30:31] musikanimal: https://plagiabot.toolforge.org/en works. I think there is something weird in the app code that when it is trying to redirect to the language code [21:30:55] maybe it expects the toolname prefix in the route? [21:31:04] ohh, yeah probably [21:31:13] I shall debug! [21:41:03] * Cyberpower678 admonishes musikanimal for plagiarizing iabot in plgiabot's name. [21:41:14] =P [21:41:14] :p [21:53:48] Hey, I'm getting access denied errors when trying to connect to the db replicas from a Toolforge bastion connection - is this a known issue? [21:55:31] Connecting to ToolsDB still seems to be working fine. [21:57:26] Naypta, wfm, can you give an example of a thing failing? [21:59:34] just running sql enwiki gives me access denied [21:59:50] same for sql nowiki or sql eswiki [21:59:53] but sql tools works fine [22:00:56] interesting [22:01:20] it was working yesterday for sure bc I was running queries against it [22:01:51] yeah confirmed your user can't authenticate [22:02:14] well that's always great news :p [22:02:31] very interesting that it was working yesterday [22:02:36] any idea why that's happened? *stares blankly* [22:03:03] krenair@tools-sgebastion-07:~$ sudo sudo -iu naypta sql enwiki [22:03:03] ERROR 1045 (28000): Access denied for user 'u24474'@'10.64.37.27' (using password: YES) [22:03:27] this should Just Work, AFAIK... it works under my user... [22:03:34] yeah, exactly [22:03:43] it's really strange that it would only be the wiki databases and not tools too [22:04:18] Naypta, Krenair: there was a new server pooled there today. It may have some missign auth records [22:04:21] the only thing i can think is that someone got annoyed at the queries i was running yesterday and blocked my access to the dbs, but I've not been messaged to tell me that, and I don't think the queries were that taxing, plus they were running against analytics [22:04:28] ahh that might be it [22:04:31] ah [22:04:58] do you have any idea how long it'll take for the auth records to propagate? [22:05:23] Naypta: we probably need to rebuild them. Could you open a phab task please? [22:05:34] sure thing, on the cloud services kanban? [22:05:45] sure [22:05:49] (y) will do! [22:08:09] https://phabricator.wikimedia.org/T253930 In the inbox! [22:18:27] Naypta: are you trying as your own user, or as a tool? And if tool, which tool? [22:18:45] My own user - I can try as a tool account too if that's helpful [22:18:58] Tool account not working either [22:19:31] Naypta: *nod*just trying to narrow down the time that grants are missing starting from [22:19:45] so your user would be "older" than any tool you made [22:19:55] only very slightly aha but yeah [22:24:37] Naypta: you should be able to use `sql --cluster=web ` to get in. That server will have a lower limit on how long a query can run for, but it has your user GRANTs active [22:25:05]