[07:17:03] !log deployment-prep enable suggested values paramter in TemplateData and VisualEditor (T271825) [07:17:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [07:17:09] T271825: Add suggested values to TemplateData and the related input in VE (ENUM) - https://phabricator.wikimedia.org/T271825 [07:19:17] !log deployment-prep enable changes to the descriptions in the VE transclusion dialog (T273425) [07:19:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [07:19:22] T273425: Move descriptions below field label - https://phabricator.wikimedia.org/T273425 [10:18:16] !log tools seting the retention on the tools-prometheus VMs to 250GB (they have 276GB total, leaving some space for online data operations if needed) (T279990) [10:18:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:18:23] T279990: [tools] prometheus out of space - https://phabricator.wikimedia.org/T279990 [10:18:33] Hi! I am running @wikilinksbot on Telegram, and I just updated a package that it relies on. However, the new version of that package needs Python 3.6, while the python3 in toolforge is 3.5.3. Is there a way to fix that? [10:20:51] @jhsoby: the Kubernetes cluster has 3.7 (would require creating k8s yaml files, there are a few examples on wikitech but still can be fairly difficult), or wait some time until we have a Debian Buster grid running for 3.7 [10:22:52] Majavah: Thanks. Iˈve never dealt with anything Kubernetes before. Whatˈs the best place to start looking into it? https://wikitech.wikimedia.org/wiki/Help:Toolforge/Kubernetes I presume? [10:27:00] !log wikilearv create project (T279694) [10:27:06] arturo: Unknown project "wikilearv" [10:27:06] T279694: Request creation of wikilearn VPS project - https://phabricator.wikimedia.org/T279694 [10:27:10] !log wikilearn create project (T279694) [10:27:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilearn/SAL [10:29:48] @jhsoby: Yes. There's also a framework/wrapper for non-webservices in kubernetes to simplify things in development, but that will still take a while. Depending on your timeframe waiting for the Buster grid or raw Kubernetes files are your best option [10:30:51] !log wikisource quota bump, CPU from 8 to 11, RAM from 16GB to 24GB (T280217) [10:30:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikisource/SAL [10:30:54] T280217: Request increased quota for wikisource Cloud VPS project - https://phabricator.wikimedia.org/T280217 [10:35:06] Majavah: Ok, thanks. I think I will just downgrade the package i was using and keep on with the same setup i already have, and look into kubernetes later – seems a tad too complicated to delve into now [13:23:28] Did something happen with cron emails? Im not getting them any longer [13:41:54] https://words.yuvi.in/post/kbatch/ is something I wanted to build while at wikimedia, and now hope to build elsewhere :) Majavah is the wrapper for non-webservices in kubernetes something similar? Do you know where it's being developed? [13:58:09] yuvipanda: https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/EnhancementProposals/Toolforge_jobs [14:01:31] ah, very interesting! Quite a different architecture than what I had in mind - I think kbatch is trying to be much closer to how webservice is designed [14:04:38] particularly focusing on making sure there's no REST API component [14:05:31] arturo: that's a pretty good document! Thanks for writing it up :) [14:06:45] thanks :-) [14:07:03] at the end of the day, the goal is the same, quoting your doc: [14:07:11] ""This is a small UX improvement over kubectl, basically."" [15:07:09] @jhsoby: running a bot process on Kubernetes is not extremely complex. There are notes at https://wikitech.wikimedia.org/wiki/Help:Toolforge/Kubernetes#Kubernetes_continuous_jobs about how I run stashbot which would be similar to your python telegram bot. [15:07:28] I would be happy to answer questions if you decide to try it. [15:22:19] bd808: Great to know! I might ask you sooner or later, when I have the time to dedicate to it 😊 [19:01:01] !log toolsbeta updated the maintain-kubeusers:beta image to https://gerrit.wikimedia.org/r/c/labs/tools/maintain-kubeusers/+/680244 [19:01:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [20:13:05] !log admin reboot cloudservices1004 [20:13:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [20:21:34] !log admin reboot cloudservices1003 [20:21:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [20:51:09] Hi Wikimedia Cloud — I'm an intern at WMF trying to run a Golang-based web app through Toolforge. I'm trying to install Go on my Toolforge instance, but unfortunately I don't have the permissions to make those file changes. Any advice? Should I just try to get approved to use VPS? Another service? [20:56:48] * Reedy wonders if he's going to come back (he's on my team) [20:57:22] Reedy: you can them to compile on their laptop and just deploy the binary to their tool [20:57:36] I can compile on their laptop? :D [20:58:29] sure! Or better yet, make a blubberfile and a pipelinelib config to compile and then copy the binary out for deoploying on toolforge :) [21:00:11] b.storm made a "bare" toolforge runtime container recently for folks who are deploying rust and golang things [21:00:46] but until we get build packs and a CI/CD for that, compiling go, rust, c++, etc on toolforge is an adventure in slow pain [21:03:30] heh, isp change [21:03:38] welcome back Hal :) [21:03:49] (I copy-pasta'd to him on Slack) [21:06:16] T277749 `docker-registry.tools.wmflabs.org/toolforge-buster-standalone:latest` is the new bare image. [21:06:17] T277749: [Toolforge] Generic webservice not working on Kubernetes - https://phabricator.wikimedia.org/T277749 [21:08:13] Thanks so much for helping out! I'll take a look at the toolforge bare image [21:11:58] bd808: I just noticed I haven't gotten any mail from my Toolforge cronjobs since the 13th. Sending stuff from the command line didn't come through either. [21:12:13] well that's "fun" [21:12:52] let me see if I can find the mail server.... [21:15:43] `exim -bp` says there is nothing enqued which seems fishy [21:20:14] anomie: I just tried sending you a mail directly from the mail server. I got my cc copy basically immediately. Can you look for it? Subject "Testing from mail server" From: root@tools.wmflabs.org [21:20:40] bd808: It came through [21:21:02] is that the address you expect to get tool mail at? [21:21:21] Yes [21:23:22] ok. getting somewhere. I tried to send another from a bastion and it is stuck in queue on the bastion. Now to figure out why [21:25:00] we have out of date config! Probably in hiera somewhere... [21:25:28] the bastion is trying to send to tools-mail-02.tools.eqiad.wmflabs as the smarthost. But it should be tools-mail-03.tools.eqiad1.wikimedia.cloud [21:27:05] !log tools Update hiera `profile::toolforge::active_mail_relay: tools-mail-03.tools.eqiad.wmflabs`. was -2 which is decommed. [21:27:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [21:30:06] anomie: It will take a bit (~30 minutes?) for that hiera fix to hit all the grid nodes. Fingers crossed that when it does your stuck mails will start to get sent out. [21:30:37] bd808: ok. I'll try to remember to check it again later [21:30:55] /me braces for root mail impact [21:31:02] /me [21:31:04] yeah... [21:31:23] @Alex Telegram does funny stuff for /me :) [21:31:28] Telegram's handling of /me is weird [21:31:31] Yeah [21:49:57] !log tools tagged the latest maintain-kubeusers and deployed to toolforge (with kustomize changes to rbac) after testing in toolsbeta T280300 [21:50:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [21:50:01] T280300: Validate that maintain-kubeusers works with k8s 1.18 - https://phabricator.wikimedia.org/T280300 [21:52:58] !log tools Update hiera `profile::toolforge::active_mail_relay: tools-mail-03.tools.eqiad1.wikimedia.cloud`. Was using wrong domain name in prior update. [21:53:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [21:59:06] htriedman: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#Other_/_generic_web_servers might help you [21:59:42] we don't have a lot of detailed docs on working with the edge case "generic" containers [22:00:32] basically it you use the golang, jdk, or ruby containers you have to also make some shell script to "launch" your process inside the container when it starts up [22:01:39] htriedman: folks who hang out here a lot (like legoktm) may be able to give you tips on what they do for these kinds of webservices that they run. [22:01:47] ohai [22:02:37] https://gerrit.wikimedia.org/r/plugins/gitiles/labs/tools/shorturls/+/refs/heads/master/service.template and https://gerrit.wikimedia.org/r/plugins/gitiles/labs/tools/shorturls/+/refs/heads/master/run-webserver.sh is what I recommend [22:03:37] the service.template file allows you to just run `webservice start` and not pass in the same args every time [22:05:51] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#Webservice_templates [22:19:08] !log tools `clush -w @exec -b "sudo exiqgrep -z -i | xargs sudo exim -Mt"` [22:19:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [22:20:08] !log tools `clush -w @all -b "sudo exiqgrep -z -i | xargs sudo exim -Mt"` [22:20:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [23:18:43] !log paws deploying updated paws-hub image for https://github.com/toolforge/paws/pull/63 [23:18:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL