[13:54:58] Hi everybody o/ I am new to WMCS. I have a maintenance script that I want to run every 12 hours, so I want to setup a cron job. But I don't know how to do that on WMCS. Is there a tutorial somewhere or is there anyone who can help me out? :) [14:02:32] I think in WMCS that’s pretty much left up to you [14:02:46] install cron or some other cron-daemon (I don’t know if Debian installs one by default) [14:02:50] or use systemd timers instead [14:03:13] should be similar to non-WMCS Linux/Debian systems in any case [14:04:02] Okay, I will see if it works :) [14:04:04] Thx [14:04:17] good luck :) [14:04:23] Thank you :) [14:05:03] dutchtom: take a look at https://wikitech.wikimedia.org/wiki/Help:Toolforge/Grid#Scheduling_jobs_at_regular_intervals_with_cron [14:05:12] Ah great thanks [14:06:09] phamhi: isn’t that for Toolforge, not Cloud VPS? [14:06:25] Yeah sorry I use Toolforge :P [14:06:27] (though I’m not sure which one DutchTom was asking about, I assumed Cloud VPS = separate VM) [14:06:29] ah ok :D [14:06:33] in that case that’s the better link indeed [14:06:38] Thanks haha [14:06:49] Sorry, I was a bit unclear [14:58:57] !log toolsbeta deleting all toolsbeta-test-* VMs (master, worker, etcd, lb) T236074 [14:59:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [14:59:00] T236074: Toolforge: rebuild the new k8s toolsbeta deployment and write final docs - https://phabricator.wikimedia.org/T236074 [15:07:07] !log toolsbeta create 3 VMs toolsbeta-test-k8s-etcd-{1,2,3} T236074 [15:07:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:07:10] T236074: Toolforge: rebuild the new k8s toolsbeta deployment and write final docs - https://phabricator.wikimedia.org/T236074 [15:13:26] !log toolsbeta refresh config in prefix puppet `toolsbeta-test-k8s-etcd` to account for new servers T236074 [15:13:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:13:29] T236074: Toolforge: rebuild the new k8s toolsbeta deployment and write final docs - https://phabricator.wikimedia.org/T236074 [17:32:31] !log tools Rebuilding all jessie and stretch docker images to pick up toollabs-webservice 0.46 [17:32:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:40:50] Krenair, arlo and i both got this error while running scap deploy on deployment-deploy01.deployment-prep [19:40:51] 18:02:54 ['/usr/bin/scap', 'deploy-local', '-v', '--repo', 'parsoid/deploy', '-g', 'default', 'fetch', '--refresh-config'] on deployment-parsoid09.deployment-prep.eqiad.wmflabs returned [255]: sign_and_send_pubkey: signing failed: agent refused operation [19:41:33] on releng, i was told that this could be related to some deployment keys work recently and that you might possibly know more. [19:52:31] subbu, hi [19:52:43] root@deployment-deploy01:~# SSH_AUTH_SOCK=/var/run/keyholder/agent.sock ssh deploy-service@deployment-parsoid09 [19:52:45] this seems to work [19:52:53] yet when you change agent.sock to proxy.sock it refuses [19:56:25] subbu, keyholder-proxy is logging that it refuses this. which user on the deployment box do we expect to be able to use deploy-service credentials? [19:56:53] all parsoid-devs [19:56:58] Krenair: SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh -l deploy-service deployment-parsoid09.deployment-prep.eqiad.wmflabs gets denied access [19:57:00] o.O [19:57:31] subbu, that does not appear to be a user? [19:57:37] so the keyholder is wrong [19:58:02] oh sorry .. i was not giving you groups / users. [19:58:12] mobrovac, works if you change proxy to agent, so yeah seems to be a keyholder auth thing [19:58:17] question is which user we expect this to work from [19:58:57] Krenair: all in group wikidev [19:59:30] interesting, we have that set up for mwdeploy [19:59:44] root@deployment-deploy01:~# grep wikidev /etc/keyholder-auth.d/* [19:59:44] root@deployment-deploy01:~# [19:59:46] ugh [19:59:51] /etc/keyholder-auth.d/mwdeploy.yml:wikidev: [mwdeploy] [19:59:58] ^ only result [20:00:04] which suggests this box has not been configured to permit that mobrovac [20:00:49] doesn't mean we couldn't, but... I do wonder how this was working before, if it was? [20:01:20] Krenair: deploy-service.yml is the correct one [20:01:41] yeah: [20:01:43] root@deployment-deploy01:~# cat /etc/keyholder-auth.d/deploy_service.yml [20:01:43] --- [20:01:43] deploy-service: [deploy_service] [20:01:43] ops: [deploy_service] [20:01:43] root@deployment-deploy01:~# [20:02:15] ops group exists in LDAP, deploy-service does not [20:02:26] deploy-service is not in /etc/group either [20:02:30] ah ok Krenair, there is a problem on deploy01, usrs that are able to deploy ar not in the deploy-service group any more [20:02:36] yeah [20:04:03] Is it possible we were manually curating a group and its membership on this box? [20:04:33] puppet should make the local deploy-service user (I think... let me find the puppet code) [20:05:31] there must have been some change somewhere recently [20:05:45] because now deploy-service can';t log into any of the boxes it used to [20:06:07] I think the problem is deploy-service doesn't exist? [20:06:13] At least not on the deployment server. [20:06:32] Clearly the user exists on the target machine (deployment-parsoid09) or we wouldn't be able to SSH to it. [20:06:34] the "scap::target" define makes the $deploy_user [20:07:32] this is defined in hiera [20:07:36] let me look around [20:11:17] it seems like the deploy-service group def from admin is not being honoured [20:14:05] from admin as in the admin module? [20:14:12] that module is not used in labs [20:15:29] yeah from there [20:16:02] well somehow the def on deploy01 and now it's not [20:16:16] this means that all services' deploys in beta are broken [20:18:13] where do the users/grous defs come from? [20:21:12] * mobrovac needs to slip away [20:21:50] mobrovac: problems with deploying on beta? [20:25:13] hauskater, services, specifically, not mw [20:25:49] for some reason the group people were in to access the key needed to deploy services has vanished [20:25:56] was it puppetised? not sure [20:28:16] Krenair: keyholder again? [20:28:33] I don't remember any services group on keylogger.d [20:29:02] I'm finishing some work stuff, then I can take a look and learn [20:29:28] I think the Hiera settings y'all will want to look at are documented at https://wikitech.wikimedia.org/wiki/User:BryanDavis/Scap3_in_a_Cloud_VPS_project#Adding_a_scap3_project [20:30:01] profile::keyholder::server::agents was the key that controlled all the access the last time I setup a scap3 server [20:30:54] bd808: but there was a change in profile::keyholder that made "unencrypted" keys to be refused [20:31:28] so Krenair and myself had to set dummy passwords for the certs there for scap to work again (for which Alex thinks it's just a distraction from the real problem) [20:31:56] * hauskater back to boring legal theories [20:32:15] so first of all.... no, I didn't just set dummy passwords [20:32:33] I ditched the compromised keys and used the ones we generated inside the project [20:32:44] (and theoretically have not shared outside) [20:32:56] secondly, I don't think this is the same thing [20:33:16] It sounded to me like trusted_groups was not set as expected/hoped [20:33:45] I'm pretty sure that's the Hiera bit that configures who can read/write proxy.sock [20:33:45] AFAIK keyholder will refuse unencrypted keys by default, we only ran into this because we unexpectedly deployed the compromised keys that weren't encrypted like our ones [20:40:54] bd808, trusted_groups stuff gets written to /etc/keyholder-auth.d/ files? [20:43:36] Krenair: yes. The .yml files in /etc/keyholder-auth.d are mappings of authorized user groups to ... deploy accounts? I think that's the RHS of the mapping. LHS is definitely local user group [20:44:15] yeah I think it's system groups -> keys that can be used from keyholder [20:44:17] oh, I bet the RHS is a list of ssh keys actually [20:44:23] yes [20:44:30] which is sort of the accounts, but really the keys because that's all keyholder deals with [20:44:35] sure [20:45:24] on my striker-deploy04 server, the deploy_service.yml has mappings for the wikidev and ops groups [20:46:17] which somehow come into existence from the hiera stuff I linked above [20:49:42] okay so we could just add wikidev [20:49:48] that would solve most problems for us probably? [20:51:25] woah [20:51:29] ssh: connect to host gerrit.wikimedia.org port 29418: Connection refused [20:51:32] this is the maintenance window? [20:51:34] ok [20:52:37] yup [20:52:42] tea time [21:27:19] Krenair: gerrit's back :) [21:27:27] ty [22:04:11] Krenair mobrovac bd808 hauskater thanks for looking into this. [22:04:52] I think I'll just cherry-pick the patch on deployment-puppetmaster and run puppet [22:05:06] you're the expert :) [22:05:18] but beta is a bit of a mess imho [22:06:00] hm the last two puppet.git cherry-picks are my WIP stuff [22:06:04] it's partly my fault [22:06:07] :) [22:09:05] oh and naturally [22:09:08] service keyholder-proxy restart [22:09:26] subbu, anyway `SSH_AUTH_SOCK=/var/run/keyholder/proxy.sock ssh deploy-service@deployment-parsoid09` and `SSH_AUTH_SOCK=/var/run/keyholder/proxy.sock ssh deploy-service@deployment-mediawiki-parsoid10` work from my own user now [22:09:40] great. will ask arlo to retry. [22:33:57] subbu / krenair still unable to deploy parsoid to beta? [22:35:35] hauskater, are you asking me? or telling me? a few mins back, arlo said he will retry, but i haven't heard anything from him yet. [22:36:13] subbu: asking, but you already answered :) [23:10:31] Krenair arlo says he deployed to the beta cluster. [23:10:41] cool