[02:07:47] revi: im confused too i see 0 alerts in any tech chans im in [02:08:30] my poor stewardbot had a job queued for a while so /-p [03:58:18] revi, Zppix, can you tell me more about what you think is broken? [03:58:48] looks like time fixed it [03:58:59] but at that moment I couldn't get my continuous job running :-p [03:59:34] ok, so not so much 'the cloud' being broken as just slow grid scheduling? [03:59:43] Was it a grid job or k8s? [04:04:28] Grid [04:09:32] so "it's just the grid being the grid" is a reasonable explanation [05:37:28] !log admin downtimed cloudnet1004 for another week T271058 [05:37:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [05:37:33] T271058: cloudnet1004/cloudnet1003: network hiccups because broadcom driver/firmware problem - https://phabricator.wikimedia.org/T271058 [09:12:01] I'm not too sure what I'm doing wrong, but I can't get python libraries loaded to the kubernetes instance [09:12:20] Trying to finish putting the WikiAuthBot discord bot up [09:14:03] I am assuming that is the location to run it. [09:14:19] Reqs Python3.7+ to run in it's current form [09:16:25] Hi, I am trying to run a script in grid which tried to allocate 25MB but gives out a memory error. This works fine locally or in PAWS though. Is there any way to get memoru access in toolforge when running in grid? [09:16:50] *to get more memory access [09:19:31] IVORK: how did you install them? [09:20:02] Tried just pip3 from bastion [09:20:25] but running commands into kubectl doesn't get the libraries, nor recognise pip [09:20:40] If easier, happy to screenshare on the enwiki discord. [09:21:27] you need to install them in an interactive k8s pod, the bastion and kubernetes cluster use a different Debian and python version [09:22:03] Any hints as to how? I swear I've been through every Toolforge help page [09:22:08] you can use `webservice --backend=kubernetes python3.X shell` [09:23:11] what guide were you using to deploy it? maybe it should be updated [09:23:44] tanny411: did you already see https://wikitech.wikimedia.org/wiki/Help:Toolforge/Grid#Allocating_additional_memory? [09:24:05] Just straight from Help:Toolforge/Kubernetes [09:24:06] pip isn't loaded natively in the interactive shell [09:24:09] since the default is over the amount you're requesting I'd just use the default [09:24:36] IVORK: are you familiar with python virtual environments? [09:24:58] Not overly [09:25:30] Only to the extent of following the guides [09:26:24] in short, in the kubernetes interactive shell you create one using python3 -m venv venv (that'll create one in ./venv, that is the "venv" directory that will be created in your shell's current directory) [09:26:56] then you activate it using `source $PATH/venv/bin/activate`, you'll again need to adjust the path [09:27:31] that will allow you to use pip, python etc tools, but instead of installing dependencies globally they will be specific to that venv [09:28:05] then you just either always activate the venv or use `$PATH/venv/bin/python` instead of `python` to use it [09:28:39] Is this a keepalive state if I were to exit, now there's no kubectl lol [09:29:20] Have individually had pip3/kubectl & python3.7, but yet to all at once [09:29:54] sorry, not exactly sure what you're asking [09:31:56] Now in webservice shell & venv, but kubernetes not natively installed, which I had assumed was the only permitted way to run instances like this. [09:32:17] Keep alive state as in there's a way to exit the console while leaving the process running [09:35:10] yeah, looks like kubectl isn't available inside the kubernetes container itself [09:36:02] can't you just open two ssh connections and use kubectl on the bastion on the another one? [09:49:29] Majavah:++ [09:49:37] ? [09:53:48] Majavah: sorry :), that was an expression of gratitude [09:53:58] ah, thanks! [10:00:56] Majavah: Thanks! [11:28:37] can someone with permissions remove this wikitech spam page? https://wikitech.wikimedia.org/wiki/User:Experiencenissanleaf [11:31:21] done [11:31:55] ty [11:38:43] !log admin [codfw1dev] drop `coudinfra` project (we are using `cloudinfra-codfw1dev` there) [11:38:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [11:42:50] !log admin [codfw1dev] drop `tools` project, create `tools-codfw1dev` [11:42:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [11:45:05] !log admin [codfw1dev] create instance `tools-codfw1dev-bastion-1` in `tools-codfw1dev` to test stuff related to T272397 [11:45:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [11:45:08] T272397: cloud: drop NAT exception for dumps NFS - https://phabricator.wikimedia.org/T272397 [11:59:01] !log admin [codfw1dev] create instance `tools-codfw1dev-bastion-2` (stretch) in `tools-codfw1dev` to test stuff related to T272397 [11:59:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [11:59:05] T272397: cloud: drop NAT exception for dumps NFS - https://phabricator.wikimedia.org/T272397 [12:01:05] !log admin [codfw1dev] drop instance `tools-codfw1dev-bastion-1` in `tools-codfw1dev` (was buster, cannot use it yet) [12:01:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [15:22:03] !log testlabs deleted bstorm-(haproxy|toolforge|nfs)-test [15:22:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Testlabs/SAL [17:39:05] bstorm: for when you have time: https://gerrit.wikimedia.org/r/c/operations/puppet/+/663289/ [18:26:21] * bd808 waves to nikkinikk [18:27:06] bd808: did my question post? I got an ambigous error message after i posted, but i see it myself....haha [18:27:52] nikkinikk: I only saw a join notice for you and now words until this question about the question :) [18:28:03] *no words (typing is hard) [18:28:17] hahaha [18:28:30] ok repostin! [18:28:36] I have a question about deploying to a given toolforge instance. Ive been getting latest versions of code just by doing git pulls inside the instance itself, is there an example of an existing project that does deploys to toolforge through CI? is that possible at all? [18:29:23] there is not currently a deployment tool for toolhub, but some folks have made their own adhoc solutions [18:29:59] this is actually a project that phamhi is doing some work towards for everyone, but that project is in really early stages [18:31:10] nikkinikk: this is one adhoc solution that a toolhub community member came up with -- https://wikitech.wikimedia.org/wiki/Help:Toolforge/Auto-update_a_tool_from_GitHub [18:31:27] its pretty sketchy though :) [18:32:36] nikkinikk: legoktm may have some good tips for building something useful for yourself. I know he has done some work maintaining tools that have reasonably nice deployment tooling [18:33:09] and he was working on the project that phamhi is working on now before he joined the serviceops team [18:34:17] bd808: hahaha "sketchy". ill take a look at those docs/ask legoktm thanks! i dont think it needs anything too advanced, as the tool is only a proof of concept. but i was just wondering/for future projects :) [18:35:06] nikkinikk: I have a few things that effectively just git pull on a timer [18:35:44] for wikibugs we have a HTTP endpoint that shells out to git pull and CI makes a curl request to it [18:38:39] legoktm: cool maybe i could take a look at wikibugs then - i was thinking it could be a jenkins job? Not sure how much effort that is to actually set up though! [18:39:17] I think it's pretty simple, see https://gerrit.wikimedia.org/g/integration/config/+/fefa31ae96af4e4e75169a7ec284b51b4e91b7de/jjb/labs.yaml#16 [18:39:24] jenkins would have to have ssh keys to a toolforge account and membership for that user in your project. It's possible. [18:40:08] or some "secured" automation thing to talk to inside the tool would work too. That [18:40:34] That's basically the idea of the github hook workflow [18:40:37] oh there's a task for that somewhere... [18:41:17] https://phabricator.wikimedia.org/T157893#6269991 [18:44:37] legoktm: we should fix that :) [18:45:09] I'm looking for jenkins-bot@ right now, it's not configured in exim so it's probably in GSuite [18:45:14] I was working with Jeena on releng about something similar, on automating npm publishes through jenkins and she was starting to look at holding npm tokens in jenkins for doing publishes. sounds somewhat similar/related [18:46:28] legoktm: we can fix the email address with LDAP + wikitech direct access if someone can figure out where is should really go [18:47:07] nikkinikk: yes, same kind of idea. [18:47:41] I just emailed jenkins-bot@, let's wait a day to see if I get a response and if not just do it ourselves [18:48:09] Hmm, can we not add jenkins-deploy to tools using Horizon? I don't think that requires action on the users' part [18:50:05] there's a bit more than adding via horizon that Striker takes care of, but I could trick that into happening without the account password I think [18:51:22] it just has to be enough for Striker to let others add the jenkins-deploy user to their tool, right? [18:51:58] the account would need full toolforge maintainer setup so that it could `become` a tool [18:52:31] so, it really should just go through Striker is what you're saying [18:52:40] yes :) [18:53:14] we could make a brand new account for this too. just seemed like reusing would be a tiny bit easier [18:55:06] that was hashar's suggestion [18:55:09] too [18:55:21] legoktm: I think really all we need is to pick a new email address that actually goes to releng folks [18:56:21] https://lists.wikimedia.org/mailman/listinfo/releng ? [18:57:03] legoktm: yeah, that's a closed list right? [18:57:22] Just wouldn't want password reset emails going public :) [18:58:40] yes [19:10:13] Hi all, silly questions: if I want to download a PAWS notebook example as-is, for example this one: https://public.paws.wmcloud.org/11901271/Lab%201%20-%20Revision%20Histories.ipynb where I can find it? If I download that it is not the actual notebook (even if the extension is .ipynb), but an HTML version of the notebook [19:11:58] CristianCantoro: https://public.paws.wmcloud.org/11901271/Lab%201%20-%20Revision%20Histories.ipynb?format=raw [19:12:39] `?format=raw` will give you the actual notebook [19:13:46] CristianCantoro: see https://wikitech.wikimedia.org/wiki/PAWS/Getting_started_with_PAWS#Fork the Fork section, notice what chico mentioned with the url param at the end [19:16:16] chicocvenancio, joakino: thank you, both! So simple, and yet... I jumped directly to the examples and then I was a little bit lost [19:17:09] it is a bit tricky to be honest, we have T264603 to track improving the experience at some point [19:17:10] T264603: Make forking notebooks in PAWS easier - https://phabricator.wikimedia.org/T264603 [19:18:14] CristianCantoro: you're welcome. A nicer interface for paws public and forking is something we've needed forever. Might be a nice project for a future hackathon, if those ever happen again. [19:20:26] I've added a small note in the example page: https://wikitech.wikimedia.org/w/index.php?title=PAWS%2FPAWS_examples_and_recipes&type=revision&diff=1898387&oldid=1896311 [19:20:45] mostly because I expected to find the word "download" somewhere there [20:38:36] bd808: discussing in -releng right now [21:25:58] !log tools.discordwiki reinstate maintainer zoranzoki21 T274344 [21:26:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.discordwiki/SAL [21:26:01] T274344: Restore access to the tool discordwiki on Toolforge - https://phabricator.wikimedia.org/T274344 [22:05:19] !log tools.lexeme-forms deployed 8e718af67e (JS fix) [22:05:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL [22:25:50] !log tools.lexeme-forms deployed 81166d5c17 (reduce T230833 workaround / "und" language codes) [22:25:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL