[00:04:40] !log tools Shut off tools-puppetmaster-01 - to be deleted in one week T245365 [00:04:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [00:04:43] T245365: Replace tools-puppetmaster-01 (jessie) with a buster puppetmaster - https://phabricator.wikimedia.org/T245365 [00:26:59] guess that depends on whether the bot's using its own account or the user's, ningu [00:27:12] DSquirrelGM: the user's [00:27:22] it's fine if the user has to authorize ia-upload separately [00:27:34] then yes they'd probably have to [00:27:38] but I dunno how that would actually work, code-wise [00:28:06] I dunno if I can redirect to ia-upload, possibly have them click allow, then redirect back to my tool ... or whatever [00:29:26] DSquirrelGM: alternatively, I guess yeah, I could create an account for the bot and it would import via ia-upload on the bot's behalf, then create wikisource pages on the user's behalf [00:29:30] I dunno [00:30:15] the wikisource pages probably have to be the user's, not bot's, but all this is up for discussion I guess [01:19:04] ningu: I would say talk to tpt about what your tool would need to work with theirs. My guess is that the tow of you would need to work out an API and auth mechanism. https://wikitech.wikimedia.org/wiki/Tool:IA_Upload has some good contact info links. [01:19:16] ah ok [01:19:18] yeah, can do [01:19:34] but it sounds possible in general? [01:21:11] potentially. the tricky bit will be figuring out how to hand off the user I imagine. Both tools are using OAuth, but with different grants so there is no intrinsic way to handle that. [01:21:53] what about the version where there's a bot user on whose behalf ia-upload is run? [01:22:10] basically my tool would be registered with ia-upload as the bot, and it would only act on the user's behalf to edit pages [01:22:38] I don't know what best/right ways are to do this [01:24:08] I have never used ia-upload, but looking at the web ui it seems to require OAuth. Automating OAuth from your backend to that tool would not be simple. And prossibly impossible if you are still planning on your tool being primarily an client side js app [01:24:27] *possibly (typing is hard today I guess) [01:24:34] yeah sorry, this would be a back-end tool that is something else [01:24:40] if the upload tool is storing access tokens in cookies (which is actually a sane thing to do now with the new domain name structure!), it should be pretty simple [01:24:42] the plan is a work in progress :) [01:25:00] the front-end would be the page-edit tool [01:25:10] tool 1 = importer, tool 2 = editor [01:25:23] that is still going to have the issue that both our OAuth 1.0a and 2.0 flows are browser based for issuing the grants [01:25:31] yeah I see [01:25:32] which is the ticky bit [01:25:36] *tricky [01:25:55] another way is to require the user to go and import and come back to my tool [01:26:08] If ia-uload was an OAuth provider rather than a OAuth relying party it might be easier [01:26:09] if ia-upload can redirect back to my tool at the end [01:26:24] just redirect the user to a dedicated endpoint on ia-upload wich does the upload (possibly with authorization) and redirects back [01:26:42] it will be invisible to the user when authorization is not needed [01:26:52] tgr: right, that seems best -- it's transparent which is great -- I just want to make it easy to use and not have the user get lost [01:27:21] although if your tool is frontend only then security-wise that might be problematic [01:27:43] I don't think I need to modify what ia-upload does in terms of actual upload process. just want to hand off to it, then continue [01:27:49] since any third party will be able to pretend to be your tool [01:28:14] tgr: true but it will be hosted on wmflabs.org [01:28:23] so if we can restrict it that way, maybe? [01:28:41] anyway, the import tool doesn't have to be front-end [01:28:56] it's much easier if the edit one can be, though [01:29:00] ningu: that's not much of a restriction :) Cloud VPS has a lot of users [01:29:01] I guess you can do some horrible hack to prove that the user has authorized you [01:29:20] like putting data in a custom user preference key [01:30:07] an HMAC signed assertion would work using a shared secret for the signing assuming that ningu's tool has a backend to securely manage the secret and signing [01:30:49] There are lots of ways to do two party authn/z communication [01:31:04] hence my suggestion to talk with Tpt [01:31:11] yeah, will do [01:31:16] once/if this grant is approved haha [01:34:39] it doesn't have a backend though [01:34:58] tgr: nah this is a separate tool for import [01:35:06] the other one is for editing and is front-end [01:35:16] this one hasn't been written yet, can be whatever we want [01:35:51] the front-end react app is already pretty well developed. this would just be a wrapper of sorts around ia-upload which would proceed and make new wikisource pages after the import [01:35:58] following a sensible template [01:36:24] it can be back-end in nodejs or python or whatever [01:36:27] in that case yeah, there are plenty of options [01:37:10] my immediate goal is to make sure what I'm proposing in the grant actually makes sense [01:37:31] which grant is that? [01:37:32] I'll have to talk to tpt to actually do it, as you said [01:37:39] https://meta.wikimedia.org/wiki/Grants:Project/PanLex/Integration_of_palm-leaf_transcription_platform_with_Wikisource [01:37:51] not submitted yet [01:38:00] I think the budget is too high actually :P realizing it isn't as much work [02:09:30] Regarding cookies and other keys, etc., is there a way to force a tool to only operate on the new isolated domain? [02:09:52] or have it redirect to itself on the new domain [02:11:47] if it's not invoked from that domain to start with [02:14:13] DSquirrelGM: great question. We have not setup a simple way to do this redirect yet, but yes it is possible via manual changes to your tools' -legacy ingress object [02:14:39] See it in action by visiting https://tools.wmflabs.org/openstack-browser/project/tools [02:15:01] which now hangs for me... :/ [02:15:38] fixed itself apparently. [02:16:16] going to go hunting around in the php docs to see if I can find something useful [02:16:26] DSquirrelGM: Basically what you need to do is similar to what I wrote up at https://wikitech.wikimedia.org/wiki/User:BryanDavis/Kubernetes#Make_a_tool_redirect_to_another_tool_WITHOUT_running_a_webservice [02:17:01] I will write up exactly the legacy ingress change on that page too [02:20:36] well I'm planning on having my tools be web-based anyway [02:27:52] !log tools.admin Hard stop/start cycle to pick up lates ingress config from webservice [02:27:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.admin/SAL [02:38:06] DSquirrelGM: https://wikitech.wikimedia.org/wiki/User:BryanDavis/Kubernetes#Redirect_requests_for_tools.wmflabs.org/my-tool_to_my-tool.toolforge.org [02:38:24] It is a relatively small change. [02:53:38] hi, i like dongs, and i like them a lot [02:53:39] hi, i like dongs, and i like them a lot [02:53:40] hi, i like dongs, and i like them a lot [02:53:49] hi, i like dongs, and i like them a lot [02:53:50] hi, i like dongs, and i like them a lot [02:53:51] hi, i like dongs, and i like them a lot [02:55:04] hi, i like dongs, and i like them a lot [02:55:05] hi, i like dongs, and i like them a lot [02:55:06] hi, i like dongs, and i like them a lot [02:55:20] hi, i like dongs, and i like them a lot [02:55:21] hi, i like dongs, and i like them a lot [02:55:22] hi, i like dongs, and i like them a lot [02:55:38] hi, i like dongs, and i like them a lot [02:55:39] hi, i like dongs, and i like them a lot [02:55:40] hi, i like dongs, and i like them a lot [02:55:40] hi, i like dongs, and i like them a lot [02:55:41] hi, i like dongs, and i like them a lot [02:55:42] hi, i like dongs, and i like them a lot [02:56:23] hi, i like dongs, and i like them a lot [02:56:25] hi, i like dongs, and i like them a lot [02:56:25] hi, i like dongs, and i like them a lot [02:56:29] hi, i like dongs, and i like them a lot [02:56:30] hi, i like dongs, and i like them a lot [02:56:30] hi, i like dongs, and i like them a lot [02:56:31] hi, i like dongs, and i like them a lot [02:56:31] hi, i like dongs, and i like them a lot [02:56:32] hi, i like dongs, and i like them a lot [02:56:52] hi, i like dongs, and i like them a lot [02:56:53] hi, i like dongs, and i like them a lot [02:56:54] hi, i like dongs, and i like them a lot [02:56:58] !ops [02:56:58] Zppix: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [02:57:49] hi, i like dongs, and i like them a lot [02:57:50] hi, i like dongs, and i like them a lot [02:57:51] hi, i like dongs, and i like them a lot [02:57:54] hi, i like dongs, and i like them a lot [02:57:54] hi, i like dongs, and i like them a lot [02:57:54] hi, i like dongs, and i like them a lot [02:57:55] hi, i like dongs, and i like them a lot [02:57:56] hi, i like dongs, and i like them a lot [02:57:57] hi, i like dongs, and i like them a lot [02:58:04] hi, i like dongs, and i like them a lot [02:58:05] hi, i like dongs, and i like them a lot [02:58:06] hi, i like dongs, and i like them a lot [02:58:41] hi, i like dongs, and i like them a lot [02:58:42] hi, i like dongs, and i like them a lot [02:58:43] hi, i like dongs, and i like them a lot [02:59:44] oh goody. the irc trolls are out tonight [03:00:01] bd808: i dont know why you think that [03:00:40] hi, i like dongs, and i like them a lot [03:00:41] hi, i like dongs, and i like them a lot [03:00:42] hi, i like dongs, and i like them a lot [03:00:43] hi, i like dongs, and i like them a lot [03:00:44] hi, i like dongs, and i like them a lot [03:00:44] hi, i like dongs, and i like them a lot [03:00:55] bd808: do /mode +q ~a [03:01:04] or /mode #wikimedia-cloud +r [03:01:05] Or just +r [03:01:08] ^ [03:01:13] they'll still joinpart spam [03:01:15] I hate locking this channel :/ [03:01:25] temporarily [03:01:31] thats why i suggested ~a [03:01:52] But as Anti said, they'll still join-spam-part [03:02:14] Ty [03:02:46] usually lasts no more than ~30 min [03:03:05] * bd808 makes a note to self to undo tomorrow [03:17:18] This works well enough for my purposes: [03:22:52] honestly, having this channel be identified users only makes a certain amount of sense [03:23:57] Requiring IRC auth to ask for help in Toolforge is not a nice idea. That's like making enwiki require named accounts to edit IMO [03:24:22] bd808: it seems to have stopped [03:24:27] maybe [03:26:18] this channel's not usually a target for this kind of stuff, that's the weird part [03:28:13] the jerks find us once in a while, but we do seem to get missed most of the time [03:31:35] I will try taking the restriction off in my daytime tomorrow [03:31:55] I'd rather not let them back in with no admins around to take action [03:39:39] just curious what time zone you're in [03:48:09] DSquirrelGM: utc-7 [03:49:22] so 2 hours west of here [03:51:31] bd808: so its 8:51 there? [03:58:19] my irc client is in UTC so it's 03:58 ;) [11:29:13] !log tools.openstack-browser restarting webservice as the page is mostly not responding (in both domains) [11:29:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.openstack-browser/SAL [13:33:43] !log admin [codfw1dev] disable puppet in cloudnet servers to hack neutron.conf for tests related to T245606 [13:33:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [13:33:48] T245606: CloudVPS: enable BGP in the neutron transport network - https://phabricator.wikimedia.org/T245606 [13:35:43] !log admin [codfw1dev] disable puppet in cloudcontrol servers to hack neutron.conf for tests related to T245606 [13:35:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:49:52] !log tools moving tools-k8s-worker-19 and tools-k8s-worker-18 to cloudvirt1022 (as part of draining 1014) [14:49:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:33:46] !log admin migrating all VMs on cloudvirt1014 to cloudvirt1022 [15:33:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [19:22:04] !log admin updating designate pool config for https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/572213/ [19:22:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [21:11:25] bd808: hmm, we should update node/npm on Toolforge maybe [21:11:47] node is at 8.11 and npm at 5.8 [21:15:22] hauskatze: we track what Debian or WMF are using. Toolforge Kubernetes has nodejs v10.15.2. The grid engine & bastions are going to have the older stuff until they have an operating system upgrade. [21:15:54] bd808: ah, makes sense [21:16:11] bd808: another q; can we now run cron jobs via k8s right? [21:16:19] instead of crontab -e; etc. [21:16:20] ? [21:17:03] Yes, the 2020 Kubernetes cluster has support for cronjobs objects. No nice tools for using it yet, but some info at https://wikitech.wikimedia.org/wiki/Help:Toolforge/Kubernetes#Kubernetes_cronjobs [21:18:08] I might try to migrate my bot jobs maybe [21:18:18] they're one-time python scripts [21:18:27] some run daily, other montly [21:18:31] *monthly [21:18:37] I'll surely take a look [21:21:10] *nod* I have one cronjob setup, but it has not actually hit its run date yet. I'm trying it out for a pywikibot monthly job that archives my talkpage on wikitech [21:22:24] not a bad idea [21:22:35] bd808: omg why didn't you asked me? I run a dayly archivebot.py script [21:22:47] *daily [21:22:52] damn English :-) [21:22:59] he probably wants the practice [21:23:17] hauskatze: where is that documented!? :) [21:23:26] or not :P [21:23:43] bd808: wikitech:User:MABot ? [21:23:56] although I think I don't run archivebot for wikitech and just redirects [21:24:05] easy enough to setup though if you want it [21:24:29] My marker page is https://wikitech.wikimedia.org/wiki/User:BryanDavis/archivebot. I was going to make sure it actually works before telling folks on Wikitech about it [21:26:12] bd808: https://github.com/MarcoAurelioWM/MABot-jobs/blob/master/archivebot.sh [21:26:39] hauskatze: I would be happy to have your bot take over and offer archivebot.py runs for everyone who may want it on Wikitech [21:27:22] We don't have a lot of super busy talk pages, but there are probably a few more than mine that could use it [21:28:08] (I may or may not have something to do with you needing it) [21:29:00] bd808: I could create a daily cronjob. Since archivebot.py relies on a configurable, per-page on-wiki template to tell the bot what and when it has to archive stuff... [21:29:28] daily is probably overkill [21:32:27] I'll set up something and let you know :) [21:32:55] hauskatze: yup. I have only set mine up for a monthly run so far, but it is using all the normal archivebot template stuff so if your bot took over I would just have to edit my talk page data to point to your marker template [21:33:21] I could use yours for now bd808 [21:33:40] so I can conduct some tests before "getting into the market" so to speak ;) [21:34:44] Sure. I think it would make sense to create the marker page under your bot's User namespace though. That would make it easy to document there too for folks to find and use themselves [21:36:48] Yup, okay :D [21:40:21] I'm fighting with some npm thing but I'll craft something soonish [21:40:24] probably today [21:42:40] hauskatze: there is no deadline ;) [21:47:54] bd808: I'm sure plently of people in the graveyard would disagree :-) [21:59:10] does anyone happen to know why ia-upload seems to use djvu format exclusively? [21:59:43] ningu historical reasons [22:01:04] ok [22:03:18] on a related topic (only related if I explain a few things though haha) is there any rule about whether a tool on wmflabs.org can/should sent client-side js requests to other domains? [22:03:29] send* [22:04:02] You mean loading external resources? [22:04:09] so I am thinking of having a tool that would send api requests to fetch info directly from the internet archive [22:04:31] in this case it would be an image, so yeah basically an external resource, but it wouldn't be in , would be dynamically loaded in js and written to canvas [22:05:02] the static img's I need would be from wikimedia commons [22:05:05] (probably) [22:05:38] I don't remember, to be honest [22:05:46] ok [22:05:47] Loading resources from outside of WMF servers without user action is frowned on (and will generate CSP warnings) [22:06:10] with user consent it is ok, but will still generate a CSP warning [22:06:31] bd808: yeah so the issue is, (1) wmf doesn't provide an official IIIF endpoint, (2) the existing unofficial endpoint at zoomviewer doesn't support djvu [22:06:49] I can ask for user consent, no problem [22:07:01] you could potentially proxy requests through, stripping any unnecessary info in the process? [22:07:09] Krenair: yes that too [22:07:27] would not be a problem for me as long as it's ok with wmf [22:07:45] unnecessary info meaning user-identifiable info? [22:07:53] yeah so like not passing along their IP/UA etc. [22:07:56] sure [22:08:00] IP, user-agent, cookies, etc [22:08:02] or other browser fingerprinting details [22:08:21] let me see if nodejs's http-proxy-middleware has a good flag for that [22:08:29] basically we hope that tools in Toolforge and Cloud VPS are as privacy respecting as enwiki [22:08:46] ok, good to know that's the main concern, because that means I can address it :) [22:10:12] ningu: https://wikitech.wikimedia.org/wiki/Wikitech:Cloud_Services_Terms_of_use and https://wikitech.wikimedia.org/wiki/Help:Toolforge/Terms_and_conditions are the written policies [22:11:13] actually, using an actual proxy library is overkill [22:11:26] oh and https://wikitech.wikimedia.org/wiki/Help:Toolforge/Rules which is linked from [[Help:Toolforge/Terms and conditions]] [22:11:29] I'll just take the relevant part of the request and make a separate request and pipe it back [22:11:36] Imho your better asking consent for something that may or not happen then not ask and something happen without consent. [22:13:28] Zppix: I dunno, if it retrieves higher-res images on the backend from IA where the request originates only from the server with nothing else sent along, that doesn't seem like such a big deal [22:14:11] Like i said thats just my 2 cents [22:15:16] ningu, thanks for asking, it made me refresh my mind (and thanks bd808 for the details responses) [22:15:40] It is very reasonable for a tool to take user input, go get some data from the internet at large, and then show the results to a user. That is a very common use case. [22:15:46] the "right" solution is official IIIF on WMF but that doesn't exist yet (there is a task for it though) [22:16:14] ningu: don't hold your breath ;) [22:16:37] well, the even righter solution is to serve high-res images from the start and not enhance the lower-res ones, but that requires all users to have high enough speed connections, which is even less under my control, so... [22:16:41] there are tasks asking for thousands of things [22:17:13] most users will be in bali where I know the connection speed varies a lot [22:17:21] in the big city with a good data plan it's fine [22:17:31] bd808: I'm about to test the archiver bot [22:17:40] hauskatze: so fast [22:17:51] fast is my middle name :) [22:17:59] haus-schnell-katze [22:18:04] Jawhol [22:18:07] lol [22:18:31] of course cron complained that the file was not executable [22:18:33] fixed [22:18:33] not "fast katze" which would be "almost cat" [22:18:44] Hausgepard (upgraded from cat to cheetah) [22:18:50] hahaha [22:19:21] mutante: maybe in _your_ house, I'd be a bit reluctant [22:20:05] lol [22:21:17] well bd808 it seems to work [22:22:27] and now if I can cron it usin K8s I can die a happy man [22:22:39] hauskatze: yeah, looks like it did what I would have expected [22:22:42] ningu: hahaa [22:32:52] bd808: re k8s crons, do kubectl create the file for me or should I? [22:34:16] hauskatze: you will need to write a YAML doc and then apply it to the cluster using kubectl. [22:34:56] hauskatze: here is what mine looks like -- https://wikitech.wikimedia.org/wiki/User:BryanDavis/Kubernetes#Run_a_cronjob [22:36:05] looks very complex [22:36:20] definitely more complex than just regular crontab -e [22:37:42] yes. the "cron" part is really just the schedule line. The rest is configuring the job by telling it what container to use and what runtime args to pass to the container [22:37:52] oh, kubernetes say `kubectl run` does create the file for you [22:39:05] hauskatze: yes, but you are unlikely to get everything setup that way to run pywikibot [22:39:12] oof [22:39:16] :( [22:39:27] the `kubectl run` method is using cli args to do all the work of the yaml file [22:40:00] can't it help me to build the skeleton at least? [22:40:17] Most of the YAML needed is boilerplate, but there are places you need to stick in the right directories and things [22:42:45] hauskatze: Just make a file starting from the YAML I posted at https://wikitech.wikimedia.org/wiki/User:BryanDavis/Kubernetes#Attach_to_a_running_pod and changing any "bd808-pywikibot" to the name of your tool [22:43:06] bha. wrong link -- https://wikitech.wikimedia.org/wiki/User:BryanDavis/Kubernetes#Run_a_cronjob [22:43:37] I'll try to do that tomorrow [22:43:45] it's late [22:45:20] !log cloudinfra Swapped 185.15.56.64 floating IP (backing puppetmaster.cloudinfra.wmflabs.org) over to cloud-puppetmaster-03 from cloud-puppetmaster-01 [22:45:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Cloudinfra/SAL [23:36:17] !log tools.quickcategories deployed 1f063050d9 (many code cleanups including some minor bugfixes) [23:36:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.quickcategories/SAL