[07:23:05] bd808: Hey, I read your email, I answer here to avoid flooding this mail-list. I discovered the issue with the version on Tuesday, and also fought that my venv is not consistent. So Yesterday I reinstalled everything from scratch. And I can confirm, before sending the email, that the venv was with Python 3.7 and the image was with 3.7, and it [07:23:06] crashed. Actually, with venv 3.7 and image 3.5, it works (no idea why). I use presently both at 3.5 and it works. The test is easy to reproduce, it is only a "hello world" in the deployment file. [10:28:30] !log admin [codfw1dev] reimaging labtestvirt2003 (cloudgw) T261724 [10:28:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:28:35] T261724: cloudgw: evaluate / validate setup in codfw1dev - https://phabricator.wikimedia.org/T261724 [13:36:40] 200~Grund: IRC error on #wikimedia-cloud: err_bannedfromchan [15:16:48] can I change the directory the logs of lighttpd are stored in (from $HOME to $HOME/logs)? [15:17:00] I tried server.errorlog = "{home}/logs/error.log" in the lighttpd config but then lighttpd doesn't start… [15:17:34] In general it seems, that most settings in the local lighttpd.conf results in 503 – Is there any way to see the logs of lighttpd's start? [15:18:17] The wiki only says "Sometimes merge fails if an option is already set in the default configuration" [15:55:23] Nudin_: unfortunately with the version of lighttpd that we are using you cannot change a setting that is set in the global config by overriding it in your $HOME/.lighttpd.conf. [15:56:12] I've thought a bit about how we might work around this by making the processing that happens inside of the `webservice` command a bit more complicated, but I haven't actually tried to write code for it yet. [15:58:26] Nudin_: if you are running your webservice on the Kubernetes cluster, the `kubectl logs ` command should show you the log messages that happen when the container first starts. There are some tips for seeing log events in Kubernetes at https://kubectl.docs.kubernetes.io/pages/container_debugging/container_logs.html that might help. [16:03:51] !log admin [codfw1dev] briefly live-hacked python3-neutron source code in all 3 cloudcontrol2xxx-dev servers to workaround /31 network definition issue (T263622) [16:03:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [16:03:55] T263622: codfw: more vlans setup changes in the cloudgw PoC - https://phabricator.wikimedia.org/T263622 [16:17:09] !log admin [codfw1dev] `root@cloudcontrol2001-dev:~# openstack subnet create --network wan-transport-codfw --gateway 185.15.57.8 --no-dhcp --subnet-range 185.15.57.8/31 cloud-gw-transport-codfw` (with a hack -- see task) (T263622) [16:17:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [16:17:13] T263622: codfw: more vlans setup changes in the cloudgw PoC - https://phabricator.wikimedia.org/T263622 [17:07:46] !log tools rebuilding docker images with locales-all T263339 [17:07:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:07:51] T263339: Add tr_TR.utf8 locale to ToolForge - https://phabricator.wikimedia.org/T263339 [19:22:52] (not sure if my previous message went through, since I had gotten disconnected from the channel) I'm getting a 502 error accessing one of my VMs: https://wikifarm.wmflabs.org/cpt. Does this have to do with the .wikimedia.cloud transition, or is something else going on? Thanks! [19:22:52] I can ssh into the VM, so it is up. [19:36:13] CindyCicaleseWMF It's unlikely to have to do with the .cloud renaming. It's somewhat possible that something else is happening with the proxy layer though. I'll look in a moment. [19:36:28] what project is that in? [19:36:35] pluggableauth [19:36:39] thank you! [19:38:40] CindyCicaleseWMF: did you try a local curl to see if the web server is working properly on that host? [19:39:05] I did not. I will try that now. [19:39:38] I would guess that you just need to restart the web service there but it could be other things :) [19:46:36] Ah, I figured it out. Thanks for the confirmation that it didn't have to do with the rename, andrewbogott! [19:46:44] cool! [20:14:17] !log tools.lexeme-forms deployed fd8c692798 (fix a crash; debug code still in place) [20:14:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL [23:26:56] @bd808 thx for the answer. Should we update the wikitech-wikipage that claims that the configs are merged? It sounds like overwriting should be possible in most cases. [23:29:24] Nudin_: what happens right now is that the $HOME/.lighttpd.conf is appended to the default settings. That used to work to override most things, but then a change was made in newer versions of lighttpd that made some settings immutable once they are given in the config. [23:29:37] I thought we updated the docs about that... [23:30:50] heh. it is kind of hidden in the help page I was thinking of -- https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web/Lighttpd#Header,_mimetype,_character_encoding,_error_handler