[11:20:39] !log admin T202636 distributing default routes using classless-static-route for all VMs in main/labtest (dnsmasq/nova-network) [11:20:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [11:20:43] T202636: Allow routing between eqiad and eqiad1 regions - https://phabricator.wikimedia.org/T202636 [11:22:10] !log bastion rebooting bastion-restricted-01 to test new setup from T202636 [11:22:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Bastion/SAL [11:44:00] (03CR) 10Lokal Profil: [C: 04-1] "Readme is the cause of the -1. the others are thoughts/suggestions." (033 comments) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/459769 (owner: 10Jean-Frédéric) [11:58:07] (03CR) 10Lokal Profil: Include empty sources in Monuments database statistics (031 comment) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/459783 (https://phabricator.wikimedia.org/T203349) (owner: 10Jean-Frédéric) [11:58:59] (03CR) 10Jean-Frédéric: Add Bash script to run arbitrary Erfgoed Python script (033 comments) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/459769 (owner: 10Jean-Frédéric) [12:00:52] (03CR) 10Jean-Frédéric: Include empty sources in Monuments database statistics (031 comment) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/459783 (https://phabricator.wikimedia.org/T203349) (owner: 10Jean-Frédéric) [14:00:36] Technical Advice IRC meeting starting in 60 minutes in channel #wikimedia-tech, hosts: @chiborg & @amir1 - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [14:50:35] Technical Advice IRC meeting starting in 10 minutes in channel #wikimedia-tech, hosts: @chiborg & @amir1 - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [14:53:51] (03PS1) 10Jean-Frédéric: Default date to today in daily uploads [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/460026 [14:53:53] (03PS1) 10Jean-Frédéric: Retrieve date and category from URL parameters [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/460027 (https://phabricator.wikimedia.org/T144724) [14:55:48] (03CR) 10jerkins-bot: [V: 04-1] Retrieve date and category from URL parameters [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/460027 (https://phabricator.wikimedia.org/T144724) (owner: 10Jean-Frédéric) [14:56:34] (03CR) 10jerkins-bot: [V: 04-1] Default date to today in daily uploads [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/460026 (owner: 10Jean-Frédéric) [15:04:20] (03PS2) 10Jean-Frédéric: Default date to today in daily uploads [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/460026 [15:04:22] (03PS2) 10Jean-Frédéric: Retrieve date and category from URL parameters in daily uploads [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/460027 (https://phabricator.wikimedia.org/T144724) [15:08:15] (03CR) 10jerkins-bot: [V: 04-1] Retrieve date and category from URL parameters in daily uploads [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/460027 (https://phabricator.wikimedia.org/T144724) (owner: 10Jean-Frédéric) [15:11:13] (03PS3) 10Thiemo Kreuz (WMDE): Default date to today in daily uploads [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/460026 (owner: 10Jean-Frédéric) [15:12:04] (03CR) 10Thiemo Kreuz (WMDE): [C: 031] "According to https://www.mediawiki.org/wiki/Manual:Coding_conventions/JavaScript slice() should be preferred over substr() and substring()" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/460026 (owner: 10Jean-Frédéric) [16:07:06] hi guys [16:07:39] i'm trying to configure flask in the server wcdo.eqiad.wmflabs for the project i'm working on [16:08:03] does anyone know about flask + nginx? i'm following this tutorial but I'm stuck with the ports [16:08:06] https://www.digitalocean.com/community/tutorials/how-to-serve-flask-applications-with-uswgi-and-nginx-on-ubuntu-18-04 [16:11:21] I've got flask+uwsgi+nginx working fine in labs [16:11:37] bare in mind you won't be running on top of bionic in labs [16:11:42] it'll be stretch or jessie, maybe trusty [16:12:19] sorry, but i have no idea what bionic, stretch, jessie or trusty are :S [16:12:20] yeah looks like wcdo.wcdo.eqiad.wmflabs is stretch [16:12:31] currently i have nginx just to share some datasets [16:12:37] bionic = ubuntu 18.04, trusty = ubuntu 14.04 [16:12:43] but i'd like to set up flask to serve a web [16:12:55] jessie = debian 8, stretch = debian 9 [16:13:04] ah, ok [16:13:41] so that tutorial is technically for a different distribution/version than what you're running, but it's ubuntu vs. debian so often the same [16:13:51] what should I do? i'm following the tutorial [16:13:55] i already have nginx [16:13:58] well let's see [16:14:03] pointing at a particular folder [16:14:04] which part are you stuck on? [16:14:57] well, it seems i can't access :5000 [16:16:00] !log bastion force-rebooted again bastion-restricted-01 due to some additional tests with DHCP [16:16:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Bastion/SAL [16:17:10] marmiquel, well, ok [16:17:21] how familiar are you with linux server administration? [16:18:03] very very few. chicocvenancio did set up the nginx and is helping me but i guess he's on holidays [16:18:20] ok [16:18:32] so you're trying to connect to localhost:5000 but it's failing? [16:19:05] yup, using the ip [16:19:09] which IP? [16:19:59] 127.0.0.1? [16:20:03] 10.68.23.230? [16:20:31] looks like you don't have a public floating IP so that rules that particular gotcha out [16:20:41] 208.80.155.156 [16:20:47] oh [16:20:50] yeah no that won't work [16:20:53] for several reasons [16:21:29] that's the IP of a proxy which is set up to pass wcdo.wmflabs.org through to http://10.68.23.230:80 [16:22:25] ok, so i should use the other one [16:22:33] depends for what purpose really [16:23:44] marmiquel, so you want uwsgi running on port 5000 [16:23:51] first I suggest: sudo lsof -i :5000 [16:23:51] i want to serve pages with d3 viz [16:25:09] alright well I don't know what that is but I can probably help you with some basics [16:25:21] d3 vizualizations (javascript) [16:25:30] ok [16:25:31] they would be triggered or integrated in flask apps [16:25:40] so [16:25:44] that would retrieve data from sqlite3 db [16:25:51] what does the command say? [16:27:17] WSGI app 0 (mountpoint='') ready in 1 seconds on interpreter 0x55d5c96ab440 pid: 23700 (default app) [16:27:17] *** uWSGI is running in multiple interpreter mode *** [16:27:17] spawned uWSGI worker 1 (and the only) (pid: 23700, cores: 1) [16:27:26] this is working. but i can't access the ip: [16:27:29] http://10.68.23.230:5000/ [16:27:37] how are you trying to access it? [16:27:38] curl? [16:28:03] you're not sticking that into your local web browser are you? [16:28:27] because that won't work without some configuration that most people won't have by default [16:29:34] ahh, i didn't know [16:29:41] it doesn't work on a local browser, nope [16:29:42] well it is a 10. address [16:29:44] private IP range [16:29:48] you won't be on that private network [16:29:48] so [16:30:02] `sudo lsof -i :5000` should show your uwsgi process [16:30:25] `curl http://10.68.23.230:5000/` *from within the instance* should work [16:30:45] can you confirm? [16:31:15] 1sec [16:31:45] works [16:31:49] ok good [16:32:58] are you able to continue with the tutorial from here marmiquel? [16:34:35] i'll continue, if i can't and u don't mind i'll ping u. still, i don't know when i'll be able to see it in my local browser... anway, i continue :) [16:34:43] ok [16:35:30] you'll be able to see it in your local browser once you've got nginx running on port 80, proxying through to uwsgi locally - your web browser will need to go to https://wcdo.wmflabs.org/ instead of the public IP of the proxy, and no custom ports [16:35:47] it looks like you already have nginx on port 80 [16:37:22] cool [16:37:28] i'll tell you [16:59:05] I have to go but I'll be back later and there are probably others around who can help [17:13:36] I will let you know when I see Framawiki around here [17:13:36] @notify Framawiki [17:17:10] Krenair: gone? [17:17:32] anyone knows about nginx and flask? [17:17:39] i could complete the tutorial: https://www.digitalocean.com/community/tutorials/how-to-serve-flask-applications-with-uswgi-and-nginx-on-ubuntu-18-04 [17:18:04] but i'd like to keep the functionality of displaying a folder content with nginx [17:18:23] which is the previous use i gave to nginx (I showed a folder with datasets) [17:19:07] now after setting up flask and a specific app, it seems it conflicts with the previous nginx config and it does not show the folder content [17:19:40] marmiquel, back for a bit [17:20:36] marmiquel, well exact nginx config is up to you, you could make it only proxy to uwsgi with a certain path [17:21:11] so instead of "location / {" you do "location /d3/ {" [17:21:32] https://pastebin.com/eY4tBSR6 [17:21:46] (03PS19) 10Zhuyifei1999: Update dependencies [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/428140 (https://phabricator.wikimedia.org/T192731) (owner: 10Framawiki) [17:21:53] the first two lines were there before, and they were pointing at the folder i was sharing using nginx [17:22:02] (03PS16) 10Zhuyifei1999: Port to Python3 [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/440007 (https://phabricator.wikimedia.org/T192698) (owner: 10Framawiki) [17:22:03] the second two lines are the ones to enable flask [17:22:27] (03PS17) 10Zhuyifei1999: Port to Python3 [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/440007 (https://phabricator.wikimedia.org/T192698) (owner: 10Framawiki) [17:22:27] i'd like to set up configs for each of both uses [17:24:13] okay [17:24:21] so end the location block after autoindex on; [17:24:26] create a new location block [17:24:31] location /d3/ { [17:24:34] followed by the uwsgi stuff [17:24:36] i see, just before opening the { [17:24:55] that way you get your directory listing at / and your uwsgi stuff if you type go to /d3/ [17:25:11] you could make an empty directory at /srv/wcdo/site/d3/ to make it appear in the listing [17:25:29] do you understand what I mean? [17:27:23] yes [17:29:13] cool, it works [17:29:25] cool [17:29:26] thank you so much Krenair [17:29:29] np [17:29:33] you've been very helpful :) [17:49:23] thanks for having a good technical discussion here Krenair and marmiquel! I love seeing people working things out in our shared spaces [17:50:38] :))) i got really happy i could move forward this point. yup. [17:50:57] :) [18:27:44] Krenair: are you around? [18:45:31] !help hi, we will perform a big quarry maintenance (https://phabricator.wikimedia.org/T202588) that needs instance recreation. But we are already using 7 of 8 available instance slots. Is it possible to increase a bit this limit to allow us to have legacy, dev and new servers in parallel the time of migration ? [18:45:31] framawiki: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [18:47:42] framawiki: that's fine but usually we process quota increases through phab. I'm afk so don't have a link handy but ping me again in a couple hours if you can't find where to where it. [18:48:31] framawiki: https://phabricator.wikimedia.org/project/view/2880/ [18:50:07] i'll look, thanks [18:58:43] framawiki: let's just delete the dev ones [18:59:30] I'm outside and don't have my 2fa keys with me, so feel free to do that :P [19:06:12] (03CR) 10Zhuyifei1999: [C: 032] Update dependencies [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/428140 (https://phabricator.wikimedia.org/T192731) (owner: 10Framawiki) [19:06:35] (03Merged) 10jenkins-bot: Update dependencies [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/428140 (https://phabricator.wikimedia.org/T192731) (owner: 10Framawiki) [19:06:51] (03CR) 10Zhuyifei1999: [C: 032] Port to Python3 [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/440007 (https://phabricator.wikimedia.org/T192698) (owner: 10Framawiki) [19:07:09] (03Merged) 10jenkins-bot: Port to Python3 [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/440007 (https://phabricator.wikimedia.org/T192698) (owner: 10Framawiki) [19:16:23] if marmiquel comes back tell him to check his meta user talk [19:18:54] hey Krenair [19:19:12] i [19:19:13] hi [19:19:30] i wanted to ask you how you reload the app once you change the .py [19:19:49] i tried sth and i messed up the nginx. i restored back to the old files [19:19:55] but it still gone [19:20:20] 502 [19:20:28] uh well I haven't tried your tutorial [19:20:38] wikimedia puppet uwsgi will set up a uwsgi-{appname} service [19:20:47] so I'd do e.g. sudo service uwsgi-certcentral restart [19:21:00] which would restart uwsgi for the certcentral API [19:21:43] marmiquel, is the 502 coming from the proxy or your own nginx? [19:21:51] bare in mind that the proxy also runs nginx just to confuse things :) [19:22:00] must be mine [19:22:06] http://wcdo.wmflabs.org [19:22:25] yes [19:22:28] ok [19:22:30] first things first [19:22:46] http://tools.wmflabs.org/openstack-browser/project/wcdo shows the proxy points to http://10.68.23.230:80 [19:22:47] so [19:22:50] on your instance [19:22:52] sudo lsof -i :80 [19:22:58] that will show you what process is running on port 80 if anything [19:23:00] what does it say? [19:23:15] nginx [19:23:21] many processes [19:23:49] okay [19:23:52] so if you do [19:24:11] curl -H "Host: wcdo.wmflabs.org" http://10.68.23.230:80 [19:24:12] from your instance [19:24:15] what happens? [19:24:30] same message i see [19:24:33] on my browser [19:24:41] the html code [19:25:01] can you paste your nginx config? [19:25:54] https://pastebin.com/9BFH9qHG [19:26:14] i haven't changed any line of it. i did change one line in th uwsgi.ini i created before in order to tell reload on pythong changes... [19:26:17] that was a mistake [19:26:32] and i did call that ini, and that's when it broke [19:26:45] i just restored the .ini without that line [19:26:55] and tried to restart nginx several times without success [19:28:00] Okay so the first thing I notice [19:28:02] https://wcdo.wmflabs.org/datasets/ works fine [19:28:14] meaning something is up with uwsgi_pass unix:/srv/wcdo/wcdo_app.sock; [19:28:16] yes [19:28:25] ls -lh /srv/wcdo/wcdo_app.sock [19:28:29] ^ what does that show? [19:28:49] No such file or directory [19:28:56] wait [19:29:15] (btw, this nginx config has changed a lot since I saw it :)) [19:29:35] no such file, nop [19:29:45] i thought this .sock was sth made up by the app, not a real file [19:29:54] marmiquel, yeah so your problem is you're telling nginx to proxy to a socket that doesn't exist [19:29:55] well [19:29:56] in the tutorial it did not suggest to create this file [19:29:58] it'll be a socket file [19:30:00] special type of file [19:30:20] can you paste your uwsgi config? [19:30:52] looking for it [19:31:25] basically you can configure uwsgi and nginx to talk to each other over a port or a socket file like this [19:31:57] https://pastebin.com/ppTZbEtw [19:32:03] this is exactly the same recommended by the tutorial [19:32:05] uwsgi will set up the file (if you tell it to) and you can tell nginx to talk to it that way [19:32:14] i wanted to add that unfortunate line... [19:32:16] hmm [19:32:27] marmiquel, try changing that socket line to be the full path [19:32:35] socket = /srv/wcdo/wcdo_app.sock; [19:32:42] then restarting the uwsgi service [19:32:48] then see if the file appears [19:34:33] sudo service uwsgi reload i guess [19:35:31] well [19:35:40] I don't know if your service name is actually `uwsgi` [19:35:45] nope [19:35:46] also I would restart rather than reload [19:35:54] i'm looking for the way to restart [19:37:46] does `ls /etc/systemd/system/uwsgi*` give you anything? [19:38:33] i see there's a wcdo_app.service there [19:38:37] in system [19:38:47] but not a uwsgi* [19:39:13] ok [19:39:27] out of curiosity, pastebin that wcdo_app.service file? [19:40:02] https://pastebin.com/1KEpGWwg [19:40:08] this was suggested by the tutorial [19:40:43] alright this is what you want [19:40:47] sudo service wcdo_app restart [19:40:59] yes, i'm looking for the way to restart uwsgi though [19:41:12] yeah that should do the trick [19:41:13] sudo service wcdo_app restart [19:41:34] bad gateway still [19:41:52] does /srv/wcdo/wcdo_app.sock exist? [19:42:08] uhm [19:42:14] i see sth [19:42:19] srw-rw---- 1 marcmiquel www-data 0 Sep 12 19:41 wcdo_app.sock; [19:44:30] perhaps i should delete it [19:44:40] in case it is confusing the uwsgi [19:45:34] !log quarry created new quarry database and user in quarry-db-01.quarry.eqiad.wmflabs T202588 [19:45:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [19:45:39] T202588: Do the big Quarry migration - https://phabricator.wikimedia.org/T202588 [19:46:05] Krenair: what do you think? [19:47:47] I can't do anything right now my internet connection is being useless [19:48:06] we only just got the file to exist, don't delete it [19:50:31] ok [19:50:43] restarting the app service or stopping cause no effect [19:51:19] I would try to check what processes are currently talking on that socket [19:51:50] problem is I've forgotten how and I literally cannot even browse to unix.stackexchange.com right now [19:52:03] despite having an OK ping to IRC [19:52:38] i see : S [19:53:33] * framawiki follows Krenair instructions to debug quarry 502 error [19:55:58] perhaps i did start a second uwsgi process [19:56:04] and they are in conflict somehow [19:57:02] marmiquel, ps auxwf | grep uwsgi [19:57:54] you should have one grep process and one uwsgi process [19:58:02] is that correct? [19:58:10] https://pastebin.com/8nySvcq1 [20:00:13] ok [20:00:20] too many procsses? [20:00:32] I think that's ok [20:00:42] sudo lsof -U | grep wcdo [20:01:01] you should get back an entry like this: [20:01:04] uwsgi 10269 www-data 3u unix 0xffff9ccea6ae8400 0t0 4454408 /run/uwsgi/certcentral.sock type=STREAM [20:01:11] https://pastebin.com/H3DGXtMt [20:01:16] several [20:01:38] not certcentral.sock though [20:01:47] yes [20:01:52] certcentral.sock is one for my particular problem [20:01:56] sorry [20:01:58] my particular program [20:02:04] !log quarry stopped celery-quarry-worker on quarry-runner-0{1,2} T202588 [20:02:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [20:02:08] T202588: Do the big Quarry migration - https://phabricator.wikimedia.org/T202588 [20:02:10] you'll have one mentioning wcdo [20:02:19] https://pastebin.com/H3DGXtMt [20:02:32] so [20:02:35] we've established that [20:02:38] uwsgi is running [20:02:43] and listening on the socket [20:02:49] aren't there too many? [20:03:06] 6 lines appear [20:03:16] after the lsof [20:03:26] I'm not sure [20:03:33] I think uwsgi does do some multi-process thing [20:03:51] ok [20:03:54] !log quarry set quarry-main-01 mariadb read-only T202588 [20:03:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [20:04:02] oh hey, have you checked your nginx error log? [20:04:08] nope [20:05:06] it'll probably be at something like /var/log/nginx/error.log [20:05:08] https://pastebin.com/PyfeaMnL [20:05:13] yes, it was that. i pasted it here [20:06:07] the .sock gives trouble [20:06:11] okay so why is it still saying "unix:/srv/wcdo/wcdo_app.sock failed (2: No such file or directory)" if /srv/wcdo/wcdo_app.sock exists? [20:06:28] i guess because the wcdo_app.sock is empty [20:06:34] no [20:06:41] srw-rw---- 1 marcmiquel www-data 0 Sep 12 19:50 wcdo_app.sock; [20:06:50] it exists but it is 0 [20:07:18] it's a socket [20:07:54] I would expect the size to be 0 [20:08:07] what I don't get is why nginx thinks it doesn't exist [20:08:52] however, this socket was created automatically [20:09:20] yes [20:09:58] maybe nginx was upset about it not existing when nginx started, and hasn't noticed it does exist now, caching the failure? try `sudo service nginx restart` ? [20:10:50] nothing [20:11:32] sigh [20:11:47] anything new in /var/log/nginx/error.log or is it the same old? [20:12:33] the same [20:15:24] ok sanity check time [20:15:24] sudo -u www-data ls -lh /srv/wcdo/wcdo_app.sock [20:16:00] uhm, it asks me for a password [20:16:06] weird [20:16:16] since i log in the server i never need a password for a sudo [20:16:33] and i login with the public-private keys [20:16:45] so now i don't know what password [20:17:14] uh [20:17:14] try [20:17:14] sudo sudo -u www-data ls -lh /srv/wcdo/wcdo_app.sock [20:17:16] that way you become root first then try to sudo to www-data :) [20:17:58] ls: cannot access '/srv/wcdo/wcdo_app.sock': No such file or directory [20:19:27] ok [20:19:28] sudo ls -lh /srv/wcdo/wcdo_app.sock [20:20:26] same [20:23:48] wtf [20:23:48] I thought it existed earlier [20:24:06] ok [20:24:25] let's re-check some stuff [20:24:28] it's weird [20:24:32] because when i do ls -l [20:24:37] it appears on the directory list [20:24:38] is uwsgi running? ps auxwf | grep uwsgi [20:25:01] so, when i do "sudo ls -lh /srv/wcdo/wcdo_app.sock" it says it doesnt exist [20:25:12] but when i do the ls -l it appears as: [20:25:17] srw-rw---- 1 marcmiquel www-data 0 Sep 12 20:12 wcdo_app.sock; [20:25:38] wait [20:25:40] hang on [20:25:44] the semicolon at the end [20:26:03] https://pastebin.com/zxFvsRvD [20:26:08] ohhhh [20:26:18] the semicolon is ... because i did put it in the.ini file [20:26:25] when u suggested me to introduce the full path [20:26:38] you know where earlier I said [20:26:38] 20<Krenair>30 socket = /srv/wcdo/wcdo_app.sock; [20:26:42] yeah that was bad [20:26:52] the semicolon at the end shouldn't be there [20:27:11] !log quarry backed up old db to /data/project/dump-2018-09-12.sql and restoring to new server T202588 [20:27:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [20:27:16] T202588: Do the big Quarry migration - https://phabricator.wikimedia.org/T202588 [20:27:20] try removing it and restarting uwsgi [20:27:33] sorry about this [20:27:36] cross fingers [20:29:13] restarting sudo systemctl restart wcdo_app [20:29:25] we are progressing somewhere [20:29:28] now the error is different [20:29:37] Internal Server Error [20:29:40] http://wcdo.wmflabs.org/ [20:29:53] ok [20:30:05] so [20:30:22] sudo service wcdo_app status [20:30:53] as there's likely an error inside the uwsgi process [20:31:08] https://pastebin.com/QG3W6KCr [20:33:08] yeah so at this point it's between uwsgi and your code [20:33:51] do i need to restart the wcdo_app every time i change sth in the code? [20:34:18] you might need to reload or something, not sure [20:34:32] I always just restart it yeah [20:35:18] can you pastebin your code? [20:35:23] yes [20:35:33] hey [20:35:34] it works now [20:35:42] i changed the .py to the one i was using before [20:35:49] and restarted [20:35:51] and now it works [20:35:59] ok [20:36:07] i need to check the other wcdo_app.py i wanted to use [20:36:09] which error had [20:36:15] so,... [20:36:20] in a way there were two errors stacked [20:36:25] one for the 502 [20:36:31] and one for the internal error [20:36:36] yes [20:37:05] this kind of thing happens with systems that proxy requests around a lot [20:38:46] :) [20:38:50] well, thanks again! [20:38:55] np [20:38:59] it's been an annoying problem, hopefully solved. :) [20:39:06] happily* [20:41:03] !log quarry switched quarry.wmflabs.org proxy to new quarry-web-01.quarry.eqiad.wmflabs [20:41:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [20:41:25] did you get stuff sorted framawiki? [20:42:30] Krenair: yes, and we're almost done with our migration. [20:42:34] !log quarry unset read-only on new database T202588 [20:42:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [20:42:37] T202588: Do the big Quarry migration - https://phabricator.wikimedia.org/T202588 [20:43:33] oh good [20:45:48] !log quarry set read-only again on new database due to a bug in worker code [20:45:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [20:53:37]