[14:05:36] !log openstack T212302 creating openstack-puppetmaster-01 and cloudvps-upgrade-test VM instances [14:05:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Openstack/SAL [14:05:39] T212302: CloudVPS: upgrade: jessie -> stretch & mitaka -> newton - https://phabricator.wikimedia.org/T212302 [15:39:49] !log ores staging ores-wmflabs-deploy:7a3bdf [15:39:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL [16:06:55] !log ores ran "sudo service uwsgi-ores restart" on ores-staging-01 [16:06:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL [18:13:51] !log ores staging ores-wmflabs-deploy:f3dbd2 [18:13:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL [18:17:03] !log ores deploying ores-wmflabs-deploy:f3dbd2 [18:17:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL [18:21:59] Do we really only have two worker nodes these days? [18:22:18] and we still have three web nodes lol [18:25:07] ores.wmflabs.org is up to date and looking good. [18:28:22] awight, when you get back, https://gerrit.wikimedia.org/r/#/c/mediawiki/services/ores/deploy/+/481210 is ready for you. [18:28:27] I think that is ready to go to beta. [18:29:05] ^ articlequality is still broken [18:29:15] Is it? Seemed to work fine for me. [18:29:44] for the labs repo maybe--because the submodule is hosted on github [18:29:58] Na. the pull worked fine for prod (from gerrit) [18:30:02] try cloning the articlequality repo and please lmk what happens [18:30:08] OK will do. [18:30:12] I get an empty dir [18:30:25] which is confirmed by gerrit views of the repo [18:30:51] Oh. our config has us pulling our submodule from https://gerrit.wikimedia.org/r/scoring/ores/wikiclass [18:31:04] 'o' [18:32:07] Sure enough, it seems our glwiki doesn't get pulled down. [18:32:09] hmm [18:32:40] last commit is "Merge pull request #63 from wiki-ai/fawiki_wp10" [18:33:11] last commit on github is "Updates for revscoring 2.3.0 (#71)" [18:33:13] So that's not right. [18:33:22] So it is broken after all! [18:34:24] Here's what I get when running submodule update -i with either the master branch or your deployment patch: [18:34:27] Cloning into '/Users/awight/work/ores-prod-deploy/submodules/articlequality'... [18:34:30] error: Server does not allow request for unadvertised object 73e7aa93845023d58032d2cc9cc350aa2ded6131 [18:34:33] Fetched in submodule path 'submodules/articlequality', but it did not contain 73e7aa93845023d58032d2cc9cc350aa2ded6131. Direct fetching of that commit failed. [18:34:41] then investigating further, my local repo is empty [18:53:15] bd808: should all flask tools use http://werkzeug.pocoo.org/docs/0.14/contrib/fixers/#werkzeug.contrib.fixers.ProxyFix ? [19:41:49] legoktm: I'm not sure honestly. I threw it in the blog post after a quick google search for X-Forwarded-Proto things. I haven't actually tried using it myself. [19:55:31] lemme try it then [20:08:45] bd808: I have it running on https://tools.wmflabs.org/ldap/user/legoktm but I don't see any visible difference [20:09:02] app.wsgi_app = ProxyFix(app.wsgi_app) [20:09:20] is what I added, per http://flask.pocoo.org/docs/1.0/quickstart/#hooking-in-wsgi-middlewares [20:12:31] looking at the view-source of the generated pages it seems that all the css and links are relative rather than absolute, so I don't think it would make any difference in that app. Really the only need it to sniff for X-Forwarded-Proto if you are generating full URLs including a protocol that link back to your own app [20:20:56] flask by default does relative links if you use url_for AIUI [20:24:01] http://flask.pocoo.org/docs/1.0/quickstart/#url-building "The generated paths are always absolute, avoiding unexpected behavior of relative paths in browsers." I think they mean absolute in the sense of using the full URL path, but no hostname/protocol [21:22:49] heh i spent like five minutes trying to figure out why i couldn't reach past the bastion hosts until i realized i hadn't set up the proxy ssh config on that machine [21:37:38] !log tools Truncated /data/project/.system/accounting after archiving ~30 days of history [21:37:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [22:08:32] is bast4001 broken? [22:08:36] fork failed: Resource temporarily unavailable / ssh_exchange_identification: Connection closed by remote host [22:10:19] brion: yeah, it was replaced by bast4002 a while ago [22:11:24] aha thx [22:14:04] bd808: odd, bast4001 asks me for a password, it doesn't seem to know my key. lemme double-check my config tho [22:14:55] yeah that was my config [22:14:58] ok i'm all set :D [22:15:00] thanks! [22:15:09] s/4001/4002 [22:16:21] np. I tripped over the same thing sometime last month. I think the reality was that the SRE folks thought that everyone had switched to 4002 long ago, but some of us had not [22:17:11] might be nice to have stable names like bast-eqiad or bast-ulsfo ... but then when they change you have the "ssh key changed" warnings that are so annoying :D [22:35:48] brion: bd808 https://people.wikimedia.org/~dzahn/bastion.sh.txt :p [22:36:55] mutante: :) I think you threw that at me last month too [22:39:11] `sed -i -e "/ProxyCommand/ s/${IS_BASTION}/${SHOULD_BASTION}/g" $SSH_CONFIG` -- old school! The cool kids are using ProxyJump these days [22:39:24] just some hacky attempt because i felt bad we keep breaking it :) i didn't mean for it to be offical..hence only on people.wm.org :P [22:39:36] oh @ ProxyJump [22:40:43] I think ProxyJump is to ProxyCommand as tmux is to screen ;) [22:41:10] mostly if you know how to use the older setup you probably keep using it [22:41:32] * bd808 uses tmux but with screen key bindings [22:41:55] TIL [22:42:18] yea, it seems it is to make it simpler but no technical difference [22:42:22] thanks [22:42:41] good to know [22:45:31] yeah, it does the same things but is harder to use improperly [22:54:30] mutante: nice, i'll bookmark that :) [22:54:43] ;) [22:57:10] ah tmux is great. i only recently started moving to it from screen and it's like .... so much betters [23:00:26] it really is :) [23:01:00] it is also possible to nest tmux in tmux in a much more sane way than nesting screen