[07:38:13] Good morning, I have an issue with my static files, most probably still an understanding mistake from my side. I use Django. In my uwsgi.ini file I have check-static = /data/project/cycling-init-bot/www/python/src/staticfiles, in this folder (I checked the address with cd, it is correct), I have some files. In my settings.py I have STATIC_ROOT = [07:38:13] os.path.join(BASE_DIR, 'staticfiles') and STATIC_URL = '/static/'. I expected to get the files from staticfiles at the following url: https://cycling-init-bot.toolforge.org/static/ But obviously there is nothing, so says the browser. I made collectstatic, but it just fills the folder staticfiles. Where is my understanding mistake? I also know [07:38:14] about https://tools-static.wmflabs.org/cycling-init-bot.toolforge but I understood that it is more for "constants", like css files. [10:56:50] !log paws aborrero@paws-acme-chief-01:~$ sudo systemctl restart acme-chief.service [10:56:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [10:58:28] !log paws aborrero@paws-acme-chief-01:~$ sudo systemctl restart uwsgi-acme-chief.service [10:58:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [10:59:08] !log paws last 2 commands should help puppet agent in the paws project, previously it had issues fetching acme-chief certs because an API update [10:59:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [11:46:43] Hey, just a question, do you know why it's not possible to use arbitrary Dockerfiles in toolforge k8s? even if we use WMF docker registry only. [11:48:45] Amir1: you would then need to upload the resulting docker img to our toolforge registry, right? We don't support bring-your-own-container yet, even if it is based on the WMF docker registry [11:49:17] * arturo relocating [11:49:19] Thanks! [13:13:34] >_> [13:13:36] <_< [13:28:16] * andrewbogott waves at yuvipanda [13:30:50] Hello, I have an issue with my static files, most probably still an understanding mistake from my side. I use Django. In my uwsgi.ini file I have check-static = /data/project/cycling-init-bot/www/python/src/staticfiles, in this folder (I checked the address with cd, it is correct), I have some files. In my settings.py I have STATIC_ROOT = [13:30:51] os.path.join(BASE_DIR, 'staticfiles') and STATIC_URL = '/static/'. I expected to get the files from staticfiles at the following url: https://cycling-init-bot.toolforge.org/static/ But obviously there is nothing, so says the browser. I made collectstatic, but it just fills the folder staticfiles. Where is my understanding mistake? I also know [13:30:51] about https://tools-static.wmflabs.org/cycling-init-bot.toolforge but I understood that it is more for "constants", like css files. [13:41:51] Psemdel: I suggest you send an email to cloud@l.w.m [13:42:19] why is it such a complex question? [13:51:26] !log admin rebooting cloudvirt1012 [13:51:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:05:47] !log admin reimaging 1014 over and over in an attempt to get partman right [14:05:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:41:08] Amir1: beyond the registry issue, today Toolforge's Kubernetes cluster is still recreating an environment similar to the gird engine environment where home directories and other things are mounted from our NFS servers and things in the containers run as the tool user. This requires us to put specific config into the container image that connects the container to the LDAP directory. [14:42:59] We can technically run arbitrary containers on the Kubernetes cluster, but doing that will break a lot of long held assumptions about how Toolforge works. [14:47:40] !log admin rebooting cloudvirt1012, chasing config weirdness in the linuxbridge agent [14:47:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [15:29:02] Hi cloud folks. Is there any mechanism/procedure for getting additional disk space added to a VM ? [15:30:25] dancy: https://wikitech.wikimedia.org/wiki/Help:Adding_Disk_Space_to_Cloud_VPS_instances [15:30:51] Thanks! I'll read through it. [15:37:34] dancy: I didn’t realize this was for deployment-prep… the situation there might be more complicated, I’m not sure [15:38:00] OK. I guess we'll find out! [15:38:21] Anybody for my problem above? [15:40:33] !log admin downgrade linux kernel from linux-image-4.19.0-11-amd64 to linux-image-4.19.0-10-amd64 on cloudvirt1012 [15:40:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [15:47:44] Psemdel: have you tried printing `os.path.join(BASE_DIR, 'staticfiles')` to be sure it is what you want? [15:48:26] how can I print it? [15:48:39] I mean local I make print() and I see it in my terminal [15:48:41] but here? [15:49:17] i'm not sure, how are you running the app? [15:49:39] python manage.py ... [15:49:48] webservice... [15:50:15] i don't know much about django, i was looking at https://uwsgi-docs.readthedocs.io/en/latest/StaticFiles.html and https://docs.djangoproject.com/en/3.1/howto/static-files/ [15:50:46] ok, I tried python manage.py migrate, and it is printed [15:50:53] thank you [15:50:55] is it the path it should be? [15:51:32] this has the deployment config option https://docs.djangoproject.com/en/3.1/howto/static-files/#deployment , which i think you were already doing [15:51:51] now that path printed should match the path on the uwsgi.ini configuration [15:54:56] in my uwsgi is data/project [15:55:02] and printed is mnt/nfs/labstore-secondary-tools-project/ [15:55:42] I thought that about the "root" of my project was data/project [15:55:47] is it not the case? [15:56:03] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web [15:56:08] for instance there [15:56:56] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web/Python [15:56:57] here also [15:57:56] why are you using BASE_DIR? why don't you try setting the path manually like in the docs: STATIC_ROOT = "/var/www/example.com/static/" [15:58:18] oh i see it is in the docs [15:59:03] the /mnt/nfs/labstore-secondary-tools-project/ and /data/project paths are the same on disk. /data/project is a symlink to /mnt/nfs/labstore-secondary-tools-project [16:00:18] then what you are missing is "/www/python/src/static" after BASE_DIR to point to the folder [16:01:22] In the docs on wikitech, BASE_DIR should be == $HOME/www/python/src [16:02:10] Psemdel: can you confirm what the value of BASE_DIR is when you print it? see bd808 above ☝️ [16:02:24] I am looking [16:03:03] BASE_DIR is [16:03:13] mnt/nfs/labstore-secondary-tools-project/cycling-init-bot/www/python/src [16:03:17] with / in front [16:03:26] wait I know what makes the difference [16:04:10] no still the smae [16:04:32] I had a os.abspath [16:06:48] I tried to replace the data/project in uwsgi.ini with mnt... [16:06:58] but uwsgi is not happy with it [16:07:00] https://paste.toolforge.org/view/1d4918e0 [16:10:16] I reverted my changes, here is the log, without error [16:10:17] https://paste.toolforge.org/view/452a7671 [16:10:33] but still nothing at https://cycling-init-bot.toolforge.org/static/ [16:10:40] Psemdel: ok. I'm poking around inside your tool. I think what may have happened is that your config has become a mix of paths that work on the bastions (/mnt/nfs/labstore-secondary-tools-project/...) but not inside a Kubernetes container (/data/project/...). [16:11:08] ok [16:11:12] where is my config? [16:11:13] i occasionally get http 502 errors on my tool. is there anything i can do to avoid that? (it's probably not a problem in my code since it happens even when serving static html pages.) [16:11:31] On the bastions, /data/project is a symlink to /mnt/nfs/labstore-secondary-tools-project, but inside the Kubernetes containers the tool's $HOME is mounted directly under /data/project [16:12:06] should I delete everything and start again from scratch? [16:12:23] MatmaRex: are you seeing that when restarting things, or generally when you think the webservice is running "normally"? [16:12:52] Psemdel: just change any path you have hardcoded from /mnt/nfs/labstore-secondary-tools-project/... to /data/project/... [16:12:56] when it should be running normally [16:13:43] Yes I did it already [16:13:45] MatmaRex: I think we would have to catch it in the act to try and figure out what is keeping the frontend proxy from getting to your tool's backend webservice. [16:13:46] it was just a try [16:14:31] Psemdel: it looks like your container started properly this time [16:14:44] yes, there is no error [16:14:49] bd808: alright, thanks. it has never lasted more than ~30 seconds, so that might be tough, but that also means it's not a big problem. [16:15:12] (i'm just using the `webservice` command) [16:15:47] it is just that I (and my website) can't find the static, which should be at an url under /static, if I understood well, and here I see nothing [16:18:41] <[1997kB]> hey, anybody use dbeaver for connecting to tools dbs? [16:20:24] <[1997kB]> I am trying to set up dbeaver but receiving an error "Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost." [16:30:04] Psemdel: I don't see STATIC_ROOT defined in your settings.py. Is that intentional? https://docs.djangoproject.com/en/3.1/howto/static-files/deployment/ [16:31:16] CREATE DATABASE disquaire; [16:31:22] no sorry [16:31:58] STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles') [16:32:03] it is there [16:32:15] staticfiles exist in my order [16:32:19] folder [16:32:58] (venv) tools.cycling-init-bot@tools-sgebastion-07:~/www/python/src/staticfiles$ diradmin main.679c0136ff07233c8dc1.js polyfills-es5.6e5c6b8b085276707109.js runtime.c51bd5b1c616d9ffddc1.jsassets polyfills.8a6242a766db0bcb54d3.js rest_framework styles.81653438fc75fec3ab63.css [16:33:31] I expect those file to be at the url https://cycling-init-bot.toolforge.org/static/ [16:33:41] as my STATIC_URL is '/static/' [16:42:27] Psemdel: I fixed it! https://cycling-init-bot.toolforge.org/static/main.679c0136ff07233c8dc1.js [16:43:02] Psemdel: the change I made was in your uwsgi.ini file -- static-map = /static=/data/project/cycling-init-bot/www/python/src/staticfiles [16:44:00] [1997kB], can you provide more details on how you configured things? [16:44:15] I think that with the check-static setting the uwsgi service tries to look for the path static/... inside your named static directory. With the static-map setting I told it to strip off the static/ part [16:52:59] !log metricsinfra silence all the deployment-prep alerts for another 30 days [16:53:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Metricsinfra/SAL [16:58:52] bd808: many thanks, what a surprise, I was just eating and now it works [16:58:59] it's like magic! [16:59:21] :) glad to help [17:00:04] I updated the doc on wikitech too [17:11:47] Yes, I am always thinking about updating doc, but honestly most of the time the doc is correct, it is just necessary to read with more attention [17:21:29] If you had not solved the problem, I would have renamed my staticfiles folder in static, for convention [17:21:32] I just tried [17:21:41] and it also solves the problem [17:21:51] so the folder must be static [17:21:56] or use static map [17:22:15] this staticfiles came from Heroku [17:42:11] bd808: if you want to debug, https://dtcheck.toolforge.org/ is returning 502 right now [17:42:28] (the problem i mentioned a few hours ago) [17:42:40] and it is loading for me... :( [17:42:42] annnd now it loads [17:43:05] let me se if I can see anything in the event logs though MatmaRex [17:43:50] pod has been running for 21 hours, so not that [17:43:55] i just got a 502 again though [17:45:49] MatmaRex: the proxy server is logging messages like this -- 2020/09/29 17:43:46 [error] 13890#13890: *177423556 upstream prematurely closed connection while reading response header from upstream, client: , server: , request: "GET /dtstats.html HTTP/2.0", upstream: "http://172.16.0.99:30000/dtstats.html", host: "dtcheck.toolforge.org", referrer: "https://dtcheck.toolforge.org/" [17:46:16] I think that is saying that the tool is sometimes not returning a clean response [17:46:43] the tool just serves static html files though. they are updated in a cron job (which runs hourly, so not right now) [17:47:15] i've just used `webservice start` (and `... restart`) [17:47:40] i don't think i even configured anything. i made a .lighttpd.conf file though by copy-pasting some things from wikitech [17:49:46] MatmaRex: I'm wondering if maybe there is contention for NFS reads from the Kubernetes node your webservice is scheduled on. That could cause strange interruptions [18:14:09] MatmaRex: I've been doing a lot of reloads of your tool's webpage and no further 502 errors. We could open a task to dig into this more, but my current guess is either NFS contention or some transient issues with the Toolforge ingress. There have been intermittent issues as we continue to move virtual machines into the new Ceph storage backend this week and last week. [18:14:43] bd808: thanks, it's not critical, i was just worrying if i did something wrong [18:15:11] :) fair enough. For what it is worth your setup looks good [18:15:20] thanks. you're the best [23:48:33] !log clouddb-services restarting postgresql on clouddb1003 to clear the number of connections [23:48:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Clouddb-services/SAL