[01:35:16] !log tools.labstore-profiling Migrated to Stretch job grid [01:35:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.labstore-profiling/SAL [01:35:28] !log tools.labstore-profiling Shutdown webservice [01:35:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.labstore-profiling/SAL [03:14:11] !log tools.toolserverdb Disabled Stretch grid cron job "push-data" because $HOME/push-data executable does not exist (T212664) [03:14:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.toolserverdb/SAL [03:14:14] T212664: toolserverdb: /data/project/toolserverdb/push-data: not found - https://phabricator.wikimedia.org/T212664 [14:05:33] !log tools rebooted tools-docker-registry-{03,04}, tools-puppetmaster-02 and tools-puppetdb-01 (load avg >45, not accessible) [14:05:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [14:37:12] !log wikidata-dev wikidata-shex installed UniversalLanguageSelector, cldr, Babel [14:37:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidata-dev/SAL [19:07:19] !log tools umounted /mnt/nfs/dumps-labstore1006.wikimedia.org for T217473 [19:07:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:07:26] T217473: labstore1006 spontaneous reboot - https://phabricator.wikimedia.org/T217473 [19:37:28] !log admin umounted /mnt/nfs/dumps-labstore1006.wikimedia.org across all VPS projects for T217473 [19:37:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [19:37:31] T217473: labstore1006 spontaneous reboot - https://phabricator.wikimedia.org/T217473 [20:25:36] Anyone know if tools user 'Wahrani' is here under a different name? [20:30:53] What's the current situation with virtualenvs and python3 under k8s? Are those Stretch-based images (i.e., can I create the venv from the stretch bastion)? Or do I somehow need to create it from the k8s container (and if so, how? can I exec in even if I have not started a webservice?) [20:37:00] !log tools.multichill killed find_paint because it was filling up /tmp on its exec node [20:37:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.multichill/SAL [20:45:48] valhallasw`cloud: k8s containers normally are jessie-based afaik and you can get in by `webservice shell` (but this you should know i guess) [20:46:36] annika: webservice shell actually doesn't seem documented -- https://wikitech.wikimedia.org/wiki/Help:Toolforge/Kubernetes#Shell mentions kubectl exec [20:46:57] ah, but https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#python_(Python3_+_Kubernetes) does [20:47:01] i admit that i never look into the docs [20:47:13] too many pages! [20:47:30] "webservice shell is only supported for kubernetes backend" [20:47:31] uh. [20:47:39] oh, webservice --backend=kubernetes python shell [20:47:40] right. [20:48:24] oh, yeah, sorry, i didn't test for no running k8s webservice [21:48:29] I asked about this earlier and got some advice about it being a bad idea, but I need to circle back to it... [21:48:29] I want to be able to SSH in to the Wikimedia Cloud server that runs Programs & Events Dashboard as a dedicated `deploy` user, so that multiple people can have deploy access. We tried setting it up so that either I or Wes could deploy by logging in with our own accounts, but we discovered some major hurdles with that approach, because the Ruby on Rails deployment tool that we use (Capistrano) is designed around the assumption of a single user [21:48:29] doing all the deployments. [21:48:29] Any ideas? [21:55:45] Like, we could create a Wikimedia developer account `ProgramsAndEventsDashboard` or similar, and then add each of the Dashboard team's public keys to it, and only use that account for the purpose of deploying to the Dashboard server. It's not clear to me whether that's an allowed way to use a Wikitech account, though. [21:59:32] ragesoss, gotta be a rule against that somewhere... [21:59:55] yeah, I would have guessed so, just haven't spotted it yet. [21:59:59] ragesoss, just make a single user and allow everyone necessary to sudo to it? [22:00:18] ragesoss: I think the most practical solution would be to have some sort of a service user added locally (although I'm not 100% sure in which uid range), and then sudo to it. That's close to what toolforge does for tool users. [22:00:31] Krenair: that isn't very straightforward, if possible at all, with Capistrano. [22:00:43] (the sudo access to a different user, that is) [22:00:47] why not? [22:00:55] ssh in, sudo to the right user, run the tool [22:02:13] You run Capistrano from the remote computer, with a configuration that says which user to log in as, and then it executes deployment actions — most of which are standard and provided by Capistrano or its plugins — as that user. [22:02:57] that doesn't sound like a supported method of access into labs [22:03:44] but you should find you are able to run the tool from inside [22:03:59] service user + adding the key to /etc/ssh/userkeys/ ? [22:04:06] that seems to be used for clush on tools [22:04:10] yes [22:04:29] cumin in deployment-prep too [22:04:36] so then you ssh to bastion as your own user, and use proxycommand to ssh as the deployuser to the actual labs hist [22:04:36] also mwdeploy in deployment-prep as that's how prod works [22:04:50] s/hist/instance [22:05:05] that could be the same instance in practice [22:05:23] cool. that sounds like it might work. [22:06:53] We'd only tried /home//.ssh/authorized_keys, but gave up quickly on that to try and see if we could make multi-user-deployment work. [22:07:31] (and the answer was, eventually, with enough chmod and chown, but not in a way that will be practical to keep doing)