[03:28:59] tools-sgebastion-08 is really slow right now because I'm trying to compile some rust code [03:37:21] I wonder if it's the resource limits that are making rustc slow [03:37:40] each process isn't able to take more than 5% CPU according to top [06:53:07] summary of my experience with rust today: https://phabricator.wikimedia.org/T194953#6183849 [09:52:32] !log tools.zppixbot-test run dbclean.py for nick_id 28 & 34 - SUCESS [09:52:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot-test/SAL [09:52:35] Reception123: ^ [09:52:54] ack [10:06:12] !log tools.zppixbot run dbclean.py using {P11346} for T254154 START [10:06:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL [10:06:15] T254154: Establish data retention guidelines - https://phabricator.wikimedia.org/T254154 [10:06:16] Reception123: ^ [10:08:05] !log tools.zppixbot run dbclean.py using {P11346} for T254154 END [10:08:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL [10:17:52] !log tools.zppixbot cleanup 129 133 171 293 301 (orphan nickname & id, not in above list) [10:17:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL [11:00:39] !log tools renewed TLS cert for prometheus to contact toolforge k8s (T250874) following docs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes/Certificates#external_API_access [11:00:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:00:44] T250874: Refresh external certs for the toolforge k8s cluster after the upgrade - https://phabricator.wikimedia.org/T250874 [11:39:10] !log tools renewed TLS cert for k8s metrics-server (T250874) following docs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes/Certificates#internal_API_access [11:39:59] arturo: stashbot isn’t here [11:40:32] mmm and why is that [11:40:41] Looks like it’s been join/quitting [11:42:12] root@tools-k8s-control-3:~# kubectl get pods -n tool-stashbot [11:42:12] NAME READY STATUS RESTARTS AGE [11:42:12] stashbot-85cc8c6c55-vgn67 1/1 Running 0 4d18h [11:42:12] stashbot.bot-5bbf56f756-vvq9r 0/1 CrashLoopBackOff 6 4d13h [11:43:05] apparently it has issues connecting to elasticsearch [12:23:56] !log tools renewed TLS cert for k8s metrics-server (T250874) following docs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes/Certificates#internal_API_access [12:24:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:24:00] T250874: Refresh external certs for the toolforge k8s cluster after the upgrade - https://phabricator.wikimedia.org/T250874 [14:19:04] !log tools.zppixbot-test upgrading python to 3.7 [14:19:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot-test/SAL [14:52:10] !log tools.zppixbot-test reverted to python3.5 [14:52:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot-test/SAL [16:24:48] Can someone confirm there is no python 3.8 image on docker for toolforge? [16:27:21] RhinosF1: according to https://docker-registry.tools.wmflabs.org/v2/_catalog , not today [16:27:45] arturo: any plan? [16:27:56] * RhinosF1 is going to try and do py3.7 soon [16:28:04] I don't think so, but may or may not happen "soon" [16:29:55] That’s fine. If I read correctly, py3.7 goes until 2023. Should give us an extra 2.5 years to move to 3.8? [16:44:04] hopefull we'll have dropped py2 by then. [17:44:14] I guess a Python 3.8 image would have to wait for Debian Bullseye (2021-ish) at least? [17:51:08] yeah, or a strong need to pull it in from an alternate apt repo [17:51:46] we really need to start deprecating old language versions before we pile on a bunch more new ones :) [17:59:31] good point ^^ [18:11:46] !log tools.stewardbots Stopping webservice [18:11:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [18:16:58] !log tools.stewardbots webservice --backend=kubernetes --canonical php7.2 start [18:17:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [20:40:36] !log tools.zppixbot Add MacFan4000 to maintainers [20:40:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL [20:40:49] !log tools.zppixbot-test Add MacFan4000 to maintainers [20:40:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot-test/SAL [20:46:25] !log tools.zppixbot add me to admins and reboot [20:46:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL [20:57:13] does anyone know how I can get our kuberentes pod to update the image to python3.7? [20:58:08] changing the image in the yaml file indicates it's using everything inside the venv setup [20:58:24] and I have no clue how to get it working on python3.7 [20:59:11] !log tools.zppixbot add Examknow to maintainers [20:59:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL [20:59:15] !log tools.zppixbot-tes add Examknow to maintainers [20:59:15] Zppix: Unknown project "tools.zppixbot-tes" [20:59:17] !log tools.zppixbot-test add Examknow to maintainers [20:59:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot-test/SAL [20:59:23] RhinosF1: you have to re-deploy to make changes in the on disk yaml take effect in the cluster [20:59:55] we talked about this a bit a few days ago. There is no magic connection between the yaml on disk and the running cluster [21:00:01] bd808: I did that and it's using a python installation in our venv for python3.5, it's that venv I can't upgrade [21:00:17] you have to build a new venv [21:00:27] the venv is on NFS and mounted into the container [21:00:49] so you have to use a python3.7 shell, create a python3.7 venv using that [21:01:14] create a py3.7 shell? [21:01:27] webservice python3.7 shell [21:02:00] that's simple [21:03:14] !log tools.zppixbot-test add me to admins and reboot [21:03:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot-test/SAL [21:04:22] !log tools.zppixbot-test updating to python 3.7 - DO NOT RESTART [21:04:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot-test/SAL [21:14:13] * RhinosF1 sees this as promissing [21:20:57] !log tools.zppixbot-test Switch to 3.7 DONE - Safe to reboot [21:20:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot-test/SAL [21:40:53] `webservice-python-bootstrap --fresh` is your friend [21:45:56] legoktm: what? [21:46:12] * RhinosF1 still has the main instance to do if test survives a day [21:48:04] RhinosF1: if you set everything up using the normal file paths, then webservice-python-bootstrap --fresh will automatically create a new virtualenv for you. [21:48:15] from the webservice shell, run `webservice-python-bootstrap --help` [21:48:53] legoktm: -bash: webservice-python-bootstrap: command not found [21:49:56] tools.newusers@interactive:~$ webservice-python-bootstrap --help [21:49:56] usage: webservice-python-bootstrap [-h] [--fresh] [21:49:58] I used webservice python3.7 shell && python3 -m venv zppixbottest37 which seems to have worked fine [21:50:03] ah [21:50:44] and then just changed then yaml & .sh file to point to 3.7 [21:50:53] uh, okay [21:51:26] legoktm: is there anything "wrong" with that? [21:59:46] legoktm: they are running a bot with a custom deployment, so the nice tool you made for python webservices doesn't directly help :) [22:00:05] oh gotcha [22:00:16] makes more sense now [22:00:32] bd808: our bot and nice havent been heard in the same sentence [22:01:13] * bd808 pets zppixbot and says RhinosF1 didn't mean that [22:01:47] * RhinosF1 has spent a lot of this global crisis trying to undo 2.5 years of core hacks and crazy code [22:02:20] * RhinosF1 so far hasnt found much more reason for hacking and not upstreaming than error generation and making his life hard [22:03:23] Or using the proper way of creating new commands