[07:55:37] !log git - shutting down instance gerrit-test7, backups created and uploaded to deploy1002 in devtools, disassociating floating IP (T236569) [07:55:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Git/SAL [07:55:40] T236569: "git" Cloud VPS project jessie deprecation - https://phabricator.wikimedia.org/T236569 [07:59:14] !log git - shutting down instance puppet-paladox, backups created and uploaded to deploy-1002 in devtools (T236569) [07:59:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Git/SAL [08:04:42] !log devtools - broken puppet again from prod changes. this time: deploy-1002 - []' is not applicable to an Undef Value. mediawiki/mcrouter_wancache.pp, line: 19 [08:04:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL [09:05:58] !log devtools - puppet fixed on deploy-1002 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/594900 [09:06:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL [09:06:39] !log devtools - avoiding the need for a second role for deployment_servers in cloud with https://gerrit.wikimedia.org/r/c/operations/puppet/+/594903 [09:06:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL [09:07:01] !log devtools - puppetmaster-1001 - Permission denied @ rb_sysopen - /var/lib/puppet/volatile/GeoIP/.geoipupdate.lock [09:07:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL [09:09:18] mutante: thanks for doing !log of your activities :-) [09:17:15] arturo: :) yep! i would forget immediately and this way i can share later with others [09:17:57] now trying to figure out how to fix permissions on local puppetmaster with the GeoIP stuff [09:24:09] !log devtools - cloud puppetmasters still affected by https://phabricator.wikimedia.org/T83447#5807825 [09:24:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL [11:12:33] !log toolsbeta livehack toolsbeta-puppetmaster-03 with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/594925 and https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/594926 (T251297 and T250866) [11:12:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [11:12:37] T251297: Refactor the toolforge::k8s::kubeadm* modules - https://phabricator.wikimedia.org/T251297 [11:12:37] T250866: Stage packages for upstream kubeadm v1.16.9 to use in Toolforge - https://phabricator.wikimedia.org/T250866 [12:04:34] !log deployment-prep - puppet broken on deployment_servers - fix deployed in https://gerrit.wikimedia.org/r/c/operations/puppet/+/594932 [12:04:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [12:07:58] !log deployment-prep - puppet still broken on deployment_servers due to unrelated pre-existing issues, also no alerts about it in shinken [12:08:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [12:34:46] !log deployment-prep removing role::labs::lvm::srv from deployment servers since this is now included in role:deployment_server and should neve have been a role in the first place [12:34:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [12:36:42] !log toolsbeta cleanup livehacks in toolsbeta-puppetmaster-03 [12:36:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [14:46:30] !log git deleted instances gerrit-test7 and puppet-paladox T236569 [14:46:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Git/SAL [14:46:34] T236569: "git" Cloud VPS project jessie deprecation - https://phabricator.wikimedia.org/T236569 [16:33:00] !log toolsbeta livehack toolsbeta-puppetmaster-03 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/594945 (T251297 and T250866) [16:33:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:33:04] T251297: Refactor the toolforge::k8s::kubeadm* modules - https://phabricator.wikimedia.org/T251297 [16:33:04] T250866: Stage packages for upstream kubeadm v1.16.9 to use in Toolforge - https://phabricator.wikimedia.org/T250866 [18:57:05] !log tools pushing new toollabs-webservice package v0.69 to the tools repos [18:57:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:03:34] !log tools toollabs-webservice 0.69 now pushed to the Toolforge bastions [19:03:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [21:09:45] hello! I'm trying to migrate the `musikanimal` account to toolforge.org. All this tool should do is redirect to other tools via .lighttpd.conf. After restarting with --canonical, the webservice keeps crashing/restarting. I tried the gridengine instead of k8s and it immediately dies. Any ideas? [21:17:12] musikanimal, guess I can take a look at it in k8s [21:17:33] thanks [21:18:05] musikanimal, the tool is just called `musikanimal`? [21:18:13] yep [21:18:50] krenair@tools-k8s-control-1:~$ sudo -i kubectl -n tool-musikanimal logs -l toolforge=tool [21:18:51] 2020-05-07 21:16:25: (configfile.c.1289) source: /var/run/lighttpd/musikanimal line: 619 pos: 9 parser failed somehow near here: /.* [21:19:00] right now it's throwing a 503 https://musikanimal.toolforge.org/, but the old URL seems to be redirecting properly https://tools.wmflabs.org/musikanimal [21:19:21] bah [21:19:45] I was missing a comma! [21:21:05] :D [21:21:09] classic [21:21:57] btw you can get logs for your containers in this way too [21:22:02] okay that was it! thank you [21:22:04] but like [21:22:12] yeah I should have checked the logs. Silly me [21:22:22] from tools-sgebastion-*, and without sudo or needing to specify a namespace. just need to be logged in as the tool [21:29:41] musikanimal: you might be interested in https://wikitech.wikimedia.org/wiki/User:BryanDavis/Kubernetes#Make_a_tool_redirect_to_another_tool_WITHOUT_running_a_webservice [21:30:21] musikanimal: that lets you tell the k8s ingress layer how to do all the work and avoid running a lighttpd pod just to serve redirects [21:30:21] oh that is interesting. I'll look into it. Thanks! [21:31:07] I would be glad to help you figure out how to do fancier things with that too. Anything that is jsut redirecting should be safe to do as a bare ingress [21:32:03] * bd808 should update that example with toolforge.org changes [21:39:47] yeah there are 10 or so different rewrite rules. I'm not certain how to do it with ingress [21:41:26] musikanimal: I might take a peek at some point. You can actually do pretty much full nginx vhost config in an ingress so I have a hunch it is possible [21:43:47] https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#server-snippet -- that lets you do almost anything [21:51:32] !log tools rebuilding the docker images for Toolforge k8s [21:51:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [22:09:26] could this consumer get approved? sorry for all the older versions, just trial and error. I think this one is configured correctly https://meta.wikimedia.org/wiki/Special:OAuthListConsumers/view/fc76f181e00ce5e798e17db4426f3147 [22:11:03] done [22:13:20] ugh that didn't work either [22:20:34] musikanimal: what kind of error are you getting? My eyeball doesn't see a difference in the grant requests you have been sending in. [22:22:12] it's an application layer error, just click on Login https://global-search.toolforge.org/ [22:22:37] the application config and the OAuth consumer seem identical to https://meta.wikimedia.org/wiki/Special:OAuthListConsumers/view/6dee02302d82c220e56321ec90f48c0b which IS working [22:22:48] https://global-search-test.toolforge.org/ [22:24:04] It's as if the "Allow consumer to specify a callback" option isn't set, but it is [22:25:23] the url I end up with looks pretty wonky... let me watch the network flow my browser sees [22:26:31] It should be a little wonky; see the URL https://global-search-test.toolforge.org/ gives you [22:26:46] Well, you can't see it I guess because it works and sends you to Meta [22:27:21] *nod* I see the 302. yeah looks pretty much the same [22:27:57] I guess let me try creating yet another consumer [22:28:04] musikanimal: rubber duck debugging here. Have you double checked the tokens you are using? [22:28:18] like are you using the old token from the new url? [22:28:29] * bd808 makes mistakes like that [22:29:14] it's possible I put in the wrong secret, which unfortunately I did record anywhere else than .env [22:29:58] *didn't [22:30:16] musikanimal: I can check your secret... if I remember where I hid the notes on how to do that are [22:30:39] awesome, thank you [22:30:46] https://www.mediawiki.org/wiki/User:BDavis_(WMF)/Notes/Recover_OAuth_secret [22:31:09] I feel like it would show a different error if that were the case, though [22:31:56] it might, I honestly can't remember [22:37:00] musikanimal: ok, looking in the db tells me that 2 of the new grants have the callback flag and one does not. When consumer key are you using? [22:37:11] *which [22:37:16] fc76f181e00ce5e798e17db4426f3147 [22:37:37] ok, the latest one. That has the oarc_callback_is_prefix flag set [22:37:40] I also reset the secret so I'm sure that's right [22:38:51] what's the consumer key for the test grant that works? I can compare at the db level and see if I spot weirdness [22:39:23] 6dee02302d82c220e56321ec90f48c0b [22:41:13] * bd808 squints at query output [22:41:47] I'm not seeing a difference :( [22:43:21] yeah, I'll try a new consumer. Thanks for checking though! [22:45:27] I guess I can approve consumers myself with the steward rights! I don't know that I'm allowed to though [22:45:55] oh no! did the world make you a steward! :) [22:46:26] congratulations :) [22:46:39] I don't mind hitting the button for you [22:46:48] hehe thanks [22:55:42] musikanimal: weirdly I'm computing a different secret key than the one you jsut put in .env [22:55:51] but maybe I'm computing it wrong [22:56:11] hmm [22:56:49] I'll PM you what I computed. Worst case its still broken, best case we found a bug