[00:13:44] I put "backoffLimit: 0" in the jobTemplate spec and it works, the script failed and the job didn't create a new pod :) [00:15:11] danilo: ah! Nice. "set .spec.backoffLimit to specify the number of retries before considering a Job as failed" [00:15:14] grr, people are running bots on login.toolforge.org [00:15:33] legoktm: you have root there. Kill them [00:15:46] (with fire, preferably) [00:17:52] people are always running stuff on the bastions [00:18:16] doing... [00:18:21] and they're not even using a tool account -.- [00:18:37] is that the mattho69 account? [00:19:17] yes [00:19:49] I'll just email them [00:20:19] cool. I was going to suggest a message on their talk page [00:20:41] oh, that works too [00:20:48] https://wikitech.wikimedia.org/wiki/User_talk:Mattho69 [00:22:36] https://wikitech.wikimedia.org/wiki/User_talk:Mattho69#Don%27t_run_scripts_on_login.toolforge.org [00:25:18] I created a script that put me in a shell inside a k8s pod, I use that when I am testing or run something heavy, maybe it is a good idea encourage other users do that instead of use the bastion [00:26:01] sometimes we need to interact with the script and grid engine does not allow that [00:26:11] well there's the dev.toolforge.org server for that [00:26:13] webservice --backend=kubernetes python3.7 shell [00:26:35] legoktm: dev is only slightly less disruptive than login [00:26:57] and if you're writing python scripts, you might as well use 3.7 and not 3.5 [00:27:02] k8s or grid is best pretty much always [00:27:27] it's been fine for me for the past 2ish weeks, but then maybe I'm the problem? :| [00:27:43] I would love to have `ssh login.toolforge.org` just drop you into k8s pod [00:27:51] we can use that webservice command for things not related to webservice? [00:27:59] but that would freak everyone out [00:27:59] yup [00:28:08] danilo: for the shell action yes [00:28:08] hm, now I think nfs is lagging :| [00:29:54] legoktm: iotop looks pretty boring on login [00:30:50] I'm in a webservice python3.7 shell, trying to run `webservice-python-bootstrap --fresh` and it's hanging at the `rm -rf /data/project/checker/www/python/venv` step [00:31:01] (on the "checker" tool) [00:31:50] hmmm... that could be NFS. That's not on the quota for login, but instead its quota limited for which ever exec node you ended up on [00:32:07] as soon as I complained in here, it started working again [00:32:09] thanks IRC? [00:34:35] bd808: thanks for taking a look :) [00:34:55] np [00:35:13] * bd808 returns to his regularly scheduled not working on Saturday ;) [06:55:54] ! [14:40:26] bd808, andrewbogott: lmk when one of you are around [14:40:57] It's a weekend, so probably not for 24 hours [14:41:11] that's fine tbh [14:41:45] * bd808 is here in some sense [14:41:56] bd808: pming [14:42:18] You really didn't need to ping him again, did you? [14:45:43] Reedy: not really, I have 10000 more important things to think about than whether I really had to ping someone to say I PM'd [14:47:10] that's a lot of important things [14:47:50] It's all good Reedy :) [14:51:44] Reedy: it is [21:17:10] !log tools.zppixbot-test reset & updated status.py config T255922 [21:17:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot-test/SAL [21:18:36] !log tools.zppixbot reset & updated status.py config T255922 [21:18:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL [21:24:45] !log tools.zppixbot reset & updated mhphab config T255922 [21:24:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL [21:24:51] !log tools.zppixbot-test reset & updated mhphab config T255922 [21:24:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot-test/SAL