[00:01:57] sDrewth: its something we need to fix on the gird for folk who are using pywikibot and python2. There is a task somewhere... [00:02:17] T248376 [00:02:18] T248376: Install python3-requests from stretch-backports for compatibility with pywikibot - https://phabricator.wikimedia.org/T248376 [00:03:30] it also happens on python3 [00:04:01] yeah, the task has both. I just remembered it as part of the py2->py3 deprecation [00:04:35] bd808, is it then okay to run the low resource tasks occasionally from command line, as they don't get the errors [00:04:43] you can use a venv to install a newer version of requests for yourself and still use the shared pywikibot files or install it seperately [00:05:13] AntiComposite, I am not a real hacker [00:05:15] if you need to run something like that interactively, use `webservice --backend=kubernetes python3.7 shell` [00:05:16] sDrewth: it should just be a warning right? [00:05:43] bd808, archivebot.py fails in the grid [00:05:50] ugh [00:05:59] there is a case where it's an error [00:06:00] well, it fails for me [00:06:10] * AntiComposite goes and looks at the task again [00:06:31] the shared pywikibot install is both awesome and horrible ;) [00:06:41] at enWS we don't archive many pages automatically for length, but we do have about 10+ pages that we like to tidy [00:07:08] it is awesome, as I like just being a user [00:07:38] I keep dreaming of a future where we have a Docker container to make pywikibot on the Kubernetes grid really simple. I should stop dreaming and do some work to make that happen [00:09:15] AntiComposite, for me with archivebot.py it seems main issue is "ImportError: No module named pathlib2" [00:09:30] ah well there's your problem [00:09:38] yes [00:09:58] problems I have, solutions are my goal [00:10:02] that's the unstable right? I thought that got fixed in stable [00:10:22] * sDrewth does their best of best clueless faces [00:10:30] bd808, only thing unstable here is me [00:11:09] as I said, it runs fine from command line, but not in the grid [00:11:17] sDrewth: so if it seems to work for you from a bastion, I would say to unblock you today you should use the dev bastion (dev.toolforge.org) and run it there. [00:11:34] I'd suggest trying to run it from a webservice shell too [00:11:52] unless that doesn't have shared pywikibot. [00:11:55] * AntiComposite can't remember [00:11:58] the webservice shell requires venv [00:12:41] which is why we need a special container to make it easier on the Kubernetes cluster [00:14:02] just to double-check, are you using `python`, `python2`, or `python3`? [00:16:10] bd808, get same error when run from dev.toolgorge grid [00:16:49] sDrewth: yes. the grid will be the same from any bastion. That bastion is just a better palce to run things locally than login.toolforge.org [00:17:05] aka login-tools, aka login.tools.wmflabs.org [00:17:16] E_TOMMANYNAMES [00:17:18] oh, you mean run command line from bastion dev [00:17:21] yeah [00:17:26] I misunderstood [00:17:40] no worries. I was probably not very clear [00:17:46] okay, I will cron it there for once a week [00:18:22] don't want to push friendships [00:18:24] that won't work [00:18:36] local crons are very very bad [00:18:39] okay [00:18:50] if you need to run it now, run it there interactively [00:19:00] I will remember (maybe) to do it weekly [00:19:15] I'll get the pacakge bug fixed tomorrow [00:19:38] I thought it was just a warning and not a full brakage [00:20:31] !log tools Docker rebuild failed in toolforge-python2-sssd-base: "zlib1g-dev : Depends: zlib1g (= 1:1.2.8.dfsg-2+b1) but 1:1.2.8.dfsg-2+deb8u1 is to be installed" [00:20:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [00:20:53] * bd808 shakes fist at the ancient jessie docker containers [00:21:29] ok, I need to go do IRL things. See y'all online [00:24:38] thanks. I have added to the ticket [02:16:05] bd808: not sure if you want that as a level 2 or 3 header, but added a note here regarding our earlier conversation about email - https://wikitech.wikimedia.org/w/index.php?title=News%2FToolforge.org&type=revision&diff=1862759&oldid=1862755 [02:17:17] DSquirrelGM: I'm honestly not sure that needs to be said, but ok [02:17:40] the migration is pretty clearly about *webservice* I thought [02:21:23] for a user coming here later after this goes into effect, it's likely going to be a question as to why the email and site name are going to be different [02:22:33] that's my point of view at least [03:04:27] bd808: is that all it would take? a dockerized pywikibot setup? [03:05:14] hare: maybe? [03:06:20] my pywikibot fu is pretty weak, but I think that we could make a container that has the core and requirements installed in it that should be relatively easy to then use to launch jobs on the Kubernetes cluster [03:07:20] one question I have about that (and need to talk to the #pywikibot folks about) is how often we would need to rebuild that container to have a reasonably stable but also well patched version [03:08:09] we don't have a CI/CD layer for building the containers automatically right now, so that would probably be a step we would want too [03:47:27] The container should be rebuilt when there is a pwb release. 4 times this year and 21 times in the previous 3 years. [04:06:34] JJMC89[m]: that seems pretty manageable :) [04:18:59] !log tools python3 build.py --image-prefix toolforge --tag latest --no-cache --push --single jessie-sssd [04:19:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [04:29:46] !log Running rebuild_all for Docker images to pick up toollabs-webservice v0.66 [try #2] (T154504, T234617) [04:29:47] bd808: Unknown project "Running" [04:29:47] T234617: Toolforge. introduce new domain toolforge.org - https://phabricator.wikimedia.org/T234617 [04:29:47] T154504: Make webservice backend default to kubernetes - https://phabricator.wikimedia.org/T154504 [04:29:55] !log tools Running rebuild_all for Docker images to pick up toollabs-webservice v0.66 [try #2] (T154504, T234617) [04:29:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [05:14:07] hare: I wrote some things down -- T249787 [05:14:07] T249787: Create Docker image for Toolforge that is purpose built to run pywikibot scripts - https://phabricator.wikimedia.org/T249787 [05:18:26] bd808: re pwb releases per year. I used the calendar years. So it would be more like 7/year, not counting the ones this year. [05:19:21] https://pypi.org/project/pywikibot/#history [05:19:24] JJMC89[m]: ok. Even if we need to rebuild once a month that's not bad. I was more worried the answer would be "nightly" [05:20:11] master on toolforge updates nightly, but stable is only when the team does a release (manual). [05:21:22] *nod* [05:47:52] Hi people, quick question. There is a way to increase add more RAM to a could instance or should I just create a new instance and install everything again? [07:29:03] I have never heard of a way to add RAM to an existing instance [08:25:40] dsaez: https://wikitech.wikimedia.org/wiki/Help:Cloud_VPS_Instances#Increase_quotas_for_projects ? [08:26:28] thanks andre_ this seems to be for increasing HDD, anyhow, I've just created a new instance and copied everything there, it was not difficult :) [08:27:03] dsaez: second sentence says "Quotas refer to one or more of CPU, RAM, [...]". If that is wrong it should be corrected. :) [08:27:27] ooh, you are right, i would have a look [08:38:09] yup [09:34:34] !log meet create project with ladsgroup and kevinpayravi as projectadmins (T249159) with 1 floating IP [09:34:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Meet/SAL [09:34:36] T249159: Request creation of "meet" VPS project - https://phabricator.wikimedia.org/T249159 [11:18:23] !log tools bump nproc limit in bastions https://gerrit.wikimedia.org/r/c/operations/puppet/+/587715 (T219070) [11:18:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:18:27] T219070: Failure of "kubectl get pods" for the editgroups project - https://phabricator.wikimedia.org/T219070 [11:25:20] !log tools.sge-status `tools.sge-status@tools-sgebastion-07:~$ webservice restart --canonical` [11:25:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.sge-status/SAL [11:27:07] !log tools.bd808-test2 `tools.bd808-test2@tools-sgebastion-07:~$ webservice restart --canonical` [11:27:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bd808-test2/SAL [11:34:12] !log tools.bd808-test2 first attempt to register in the proxy failed with: `toolforge.webservice.proxy.ProxyException: Port registration failed!` A retry succeeded [11:34:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bd808-test2/SAL [11:36:04] !log tools.gtirloni-sandbox `tools.gtirloni-sandbox@tools-sgebastion-07:~$ webservice restart --canonical` [11:36:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.gtirloni-sandbox/SAL [11:40:36] Lucas_WMDE: you around? [12:14:57] any webservice developer around with a tool running in kubernetes want to do some early testing for toolforge.org? [12:15:03] Urbanecm: ? [12:19:39] arturo: what would be my role? :-) [12:19:41] (hi) [12:32:27] arturo: am now [12:32:43] though I probably shouldn’t be testing my private tools on working time ^^ [12:33:14] would the test be “restart with --canonical and see what happens”? :) [12:38:24] arturo: actually, I have a WMDE tool that needs a minor redeploy anyways [12:38:31] but which isn’t often used as far as I’m aware [12:38:35] basically the perfect guinea pig [12:50:43] !log tools.wdmm updated to python3.7 (including venv rebuild) [12:50:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wdmm/SAL [12:50:57] Lucas_WMDE: How.. convienient ;P [12:50:57] ^ that’s the guinea pig tool, just did a python update in preparation [12:51:06] (it was still on 3.5) [12:52:02] Lucas_WMDE: yes, the canonical thing [12:52:07] though I’ll be unavailable for 2¼ hours starting in 8 minutes :/ [12:52:13] okay, we can try the canonical thing in that time I think [12:52:35] Urbanecm: you have any tool in the k8s cluster? [12:53:25] arturo: yes, if starting through `webservice --backend=kubernetes something` means that. [12:53:58] Urbanecm: yeah exactly. Would you like to be an early adopter of the new domain? [12:54:03] Sure! [12:54:19] if so try webservice restart --canonical [12:54:33] Note many of my tools are tied to OAuth; I can change the callback URL, but I'd need tools.wmflabs.org/toolname to redirect to toolname.toolforge.org. Would that do so? [12:54:55] hmmm, I had the tool loaded with the old scheme and tried to submit a form after a start with --canonical [12:55:01] the POST seems to have been redirected with 301 [12:55:05] yes, that's exactly the thing the canonicsl swtich does [12:55:12] shouldn’t it be a temporary redirect though? [12:55:18] not a permanent one? [12:55:24] arturo: great. Let's try it then 🙂 [12:55:40] Lucas_WMDE: mmmm let me double check [12:55:45] also, apparently the next request my browser did was a *GET* to the new URL, which resulted in 405 Method Not Allowed, presumably from my tool [12:55:55] I don’t remember which redirect is the one that preserves methods and which doesn’t [12:56:05] presumably 301 was interpreted by Firefox as “turn POST to GET” [12:56:17] anyways, let me test the tool properly first [12:56:36] seems to work when I load it from the new URL directly \o/ [12:57:16] so the new domain works on its own, but the redirect isn’t perfect yet, I think [12:57:31] a GET to the old URL (https://tools.wmflabs.org/wdmm/) also returns 301 Moved Permanently [12:57:53] !log tools.wdmm restarted with --canonical, see https://w.wiki/MS$ [12:57:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wdmm/SAL [12:59:01] !log tools.wdmm deployed 410c0d5db3 (azbwiki override) with webservice restart [12:59:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wdmm/SAL [12:59:20] ^ a plain `webservice restart` seems to preserve the --canonical flag, which is what I expected, good [12:59:58] Lucas_WMDE: a restart without --canonical should clear the redirect [13:00:01] I have to go to meetings now but can test a bit more later [13:00:03] uh [13:00:05] hm [13:00:11] the old domain should still be reachable though [13:00:17] I'm only talking about the redirect [13:01:15] the POST to GET thing is a thing to explore [13:01:38] right now t.w.o/wdmm redirects and wdmm.t.o doesn’t [13:01:46] and that was after a `webservice restart` without --canonical [13:01:55] mmm [13:02:05] (testing with curl because I don’t trust my browser after those permanent redirects) [13:02:09] the behavior might be different in the grid and in k8s [13:02:19] sorry, really off now [13:02:24] thanks Lucas_WMDE ! [13:03:28] https://wikinity.toolforge.org/, login seems to work, however, maps from maps.wikimedia.org don't seem to load (they get HTTP 429) [13:03:30] see https://wikinity.toolforge.org/s/1568 as an example [13:04:04] Urbanecm: because some ACL or something in the maps server? [13:04:24] sounds plausible [13:04:47] I'm trying to get a debugging copy of this tool back up, so I can make sure it worked from the old wmflabs.org domain [13:05:09] 429 means "too many requests" [13:05:13] that's weird, isn't it? [13:05:41] I think the maps team has whitelist for requests from trusted domains, to stop external users from overloading it [13:05:47] I'm not sure if wmflabs.org was there or not... [13:06:36] but anyway is my own browser doing the requests to maps, not the tool in the backend, right? [13:06:51] yes [13:07:20] Lucas_WMDE: for the record, yes, the annotation in the ingress object is permanent redirect: https://tools.wmflabs.org/k8s-status/namespaces/tool-wdmm/ingresses/wdmm-legacy/ [13:07:46] yes, it checks the referrer [13:08:02] it's implemented at the cache level: if it's not a wikimedia referrer, 1 request for an uncached tile is too many [13:08:15] AntiComposite: that makes sense [13:08:22] then we need to whitelist toolforge.org [13:08:27] arturo: okay, so wmflabs.org works, toolforge.org not [13:08:38] will create a phab task about this [13:08:41] thanks [13:08:44] who should I ping about this? [13:11:33] https://wikitech.wikimedia.org/wiki/Incident_documentation/20200204-maps is the incident, https://phabricator.wikimedia.org/T244278 is the task [13:13:08] arturo: I guess #maps and #operations would do [13:16:06] I just opened T249815 [13:16:09] T249815: maps: whitelist/reduce ratelimit from requests with toolforge.org referrer - https://phabricator.wikimedia.org/T249815 [13:16:41] thanks [13:21:59] also sent a patch: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/587731 [13:22:31] that means two of us :D [13:23:11] arturo: abandoned mine :) [13:23:12] I like yours more :-P [13:24:08] we went from 0 patches, to 2 patches, now back to 0 :p [13:24:14] LOL [13:24:19] Urbanecm: go ahead with yours! [13:24:36] done :D [13:26:00] arturo: hopefully someone merges that soon, otherwise, I'd need to revert that temporarily 🙂 [13:26:15] let me ping some folks on IRC [13:26:22] thanks! [13:26:49] arturo: do you want me to test with some other tools, or do you want to have just one and monitor what happens? [13:26:50] Urbanecm: anyway, feel free to restart your webservice without the canonical switch [13:26:56] that should wipe the redirection [13:27:28] wait, not in the k8s cluster [13:27:33] so you will need a full stop & start [13:27:55] ack [13:28:01] Urbanecm: will ping tomorrow for another tool. With the 2 we tested today I have couple of things to explore [13:28:21] okay, noted. Thanks for your work on this! [13:28:30] (or, feel free to do at your own pace and open phab tasks anyway!) [13:31:37] brb [14:06:43] hi [14:10:10] I could not add the bot files to toolforge, What is the solution؟ [14:14:00] Clayology, could you be a bit more specific about what you were trying to do and what stopped you? [14:30:20] I was trying to put bot python files in my tool on [14:39:19] !help [14:39:19] Clayology: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [14:46:41] !help [14:46:41] Clayology: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [14:47:06] Clayology: Are you receiving any error messages? [14:48:17] we'll need to know more about the steps you're using to add files and what exactly is not working for you [15:13:15] !log tools Rebuilding all stretch and buster Docker images. Jessie is broken at the moment due to package version mismatches [15:13:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:52:36] !log admin cleaned up a bunch of leaked .eqiad.wmflabs dns records [16:52:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [17:25:01] !log tools.wikibugs Updated channels.yaml to: 4f74eadc0b86a0fc7c0ed502651518446b6c6a60 More Product Infrastructure repos [17:25:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL [18:04:27] !log tools.wikibugs Updated channels.yaml to: f3ca868d48da550c5882dd07f3008e78e6093ce3 Drop the #wikimedia-interactive channel, no longer used [18:04:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL [18:29:25] !log tools.wikibugs Updated channels.yaml to: 10c446c88bbebfe5153b16f489ab8d60e81d8c96 Partially revert "Drop the #wikimedia-interactive channel, no longer used" [18:29:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL [19:57:17] !log admin upgrading eqiad1 designate to rocky [19:57:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL