[05:42:47] What's with replag? [08:31:11] is there a way to delete a file from cloud-vagrant without SSH-ing in? [08:31:54] it ate up all the space (some of the node services have crappy logging settings) and now it can't open a shell [08:56:48] tgr is that a virtual machine? [09:15:01] Hi, JFYI I've merged a patch to fix an issue with keyholder's proxy on buster (Py3.7+). As a result, due to its current puppetization, that triggers a refresh of the agent too, requiring to re-arm keyholder (sudo keyholder arm). [09:21:51] volans: where would we need that? [09:22:51] arturo: anywhere that uses the keyholder::agent define, that unfortunately I don't think is searchable in the puppetclass browser [09:23:07] cumin masters, scap deploy comes to mind [09:23:36] ok [09:24:39] arturo: or we can run a cloud-wide check for something like: [09:24:40] ps fax | grep '/usr/local/bin/ssh-agent-prox[y]' [09:25:08] or its presence on disk [09:28:09] yup [10:13:52] !log cloudinfra re-arm keyholder in cloud-cumin-01 [10:13:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Cloudinfra/SAL [10:15:02] arturo: I'm also merging a patch to remove the notify of the agent when the proxy is changed, AFAIK we can restart the proxy as much as we want without the agent being restarted [10:15:12] and notifying only the proxy instead [10:15:24] 👍 [10:16:01] volans: BTW this is the list of affected servers [10:16:04] https://www.irccloud.com/pastebin/BfVEYe90/ [10:17:16] I can take care of af-puppetmaster02.automation-framework.eqiad.wmflabs [10:17:22] ok [10:18:08] most of the projects I don't have any involvement with them. I'm not sure if we should simply leave a message in the SAL or actually use my root powers to re-arm keyholder myself [10:18:27] you most likely don't have the key [10:18:37] s/key/passphrase/ [10:18:39] oh, and that, indeed [10:21:29] in this case, for example, this is confusing [10:21:34] aborrero@tools-acme-chief-02:~$ sudo keyholder arm [10:21:34] Enter passphrase for /etc/keyholder.d/authdns_acmechief: [10:21:56] but the one stored in pwstore is not the expected password [10:22:02] ofc not, that's the prod one [10:22:18] so the thing is I don't know where is this pass stored hehe [10:22:35] everyone has it stored in their own way, I don't think there is a standard [10:22:42] as cloud doesn't support real secrets :D [10:23:03] we have several secrets in pwstore [10:23:18] for example `cumin-openstack-master-key-passphrase` [10:24:15] I have the one in the af-puppetmaster, and I've already re-armed it [10:24:34] but if you're referring to those in deployment-prep I don't have those [10:24:41] hashar probably [10:24:52] btw jbond-buster.puppet.eqiad.wmflabs doesn't have keyholder [10:24:57] not sure why it matches your search [10:25:04] ah yeah my bad, ssh connect :D [10:25:27] hehe [10:26:50] ok I found the keyholder pass for tols-acme-chief-02 [10:27:19] !log tools re-arm keyholder in tools-acme-chief-02 (password in labs/private.git @ tools-puppetmaster-01) [10:27:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:28:06] !log tools re-arm keyholder in tools-acme-chief-01 (password in labs/private.git @ tools-puppetmaster-01) [10:28:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:29:26] !log cloudinfra re-arm keyholder in cloud-cumin-02 (password in pwstore) [10:29:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Cloudinfra/SAL [10:30:46] sorry for the mess, with the latest patch merged from now on only the proxy will be restarted on a proxy change [10:30:49] not requiring the re-arm [10:30:56] it's ok volans [10:31:02] I honestly don't know why it was that way before [10:31:31] also, thanks for taking care of this corner of the infra volans :-P [10:31:43] :) [11:45:05] !log tools created 3 new VMs `tools-k8s-worker-[3,4,5]` (T239403) [11:45:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:45:09] T239403: toolforge: new k8s: scale up a bit the cluster before final tests and initial migrations - https://phabricator.wikimedia.org/T239403 [13:09:39] !log tools.wikibase-databridge-storybook npm run build-storybook && tar -C storybook-static -c . | ssh toolforge "sudo -i -u tools.wikibase-databridge-storybook sh -c 'rm -rf www/static/*; tar -C www/static/ -x'" # deploy locally built storybook (T237513) [13:09:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibase-databridge-storybook/SAL [15:54:35] !log design deleting VM 've-font' on Prateek's instructions [15:54:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Design/SAL [15:55:08] !log design deleting design-research-methods on Jonathan's instructions [15:55:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Design/SAL [16:02:31] !log visualeditor deleting dumpgrepper and togetherjs instances on Bartosz's instructions. [16:02:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Visualeditor/SAL [20:18:55] arturo: yes, vagrant running on a Cloud VPS box that has role::mediawiki::vagrant [20:19:03] so an lxc-based VM [20:19:27] not super important, I can just destroy and redo the vagrant box [20:19:54] but seems simple enough, on virtualbox I'd know how to do it, but I don't know much about lxc [20:47:32] The replication lag time on s1 with 31 Hours and s8 with ~14 hours is known? [22:01:06] Krenair: Thanks for the report. Old code :-( [22:03:11] Wurgl, yeah I figured. thanks for fixing it [22:03:20] by the way I was wondering what your itwiki block was about [22:03:39] ? [22:03:46] it? [22:03:54] https://meta.wikimedia.org/wiki/Special:CentralAuth/Wurgl [22:04:00] some other country [22:04:00] ltwiki, sorry [22:04:13] lithuanian wikipedia [22:04:34] I visitied the page and the red bell ringed [22:05:25] So I asked, if they could consider to greet people when the create the account in their country or at the first entry [22:05:34] The guy did not like that [22:05:59] so blocked you for 'Nekonstruktyvi veikla'? xD [22:06:26] not constructive... something [22:06:45] We had different ideas about greeting people. I did not accept his opinion and he not mine [22:09:38] https://lt.wikipedia.org/w/index.php?title=Naudotojo_aptarimas:Homo_ergaster&diff=prev&oldid=5012022#Why_do_I_get_this_invitation? [22:10:31] However, I do not care. I do not even not care :-) [22:12:15] "if you don't understand Lithuanian you are not much welcome here" jeez [22:12:31] better keep that in mind next time I'm running tourbot [22:16:15] I missed some reference/data in german wikipedia and there was an article in lt.wiki. So I looked, if there is some information. Google translate helps reading it … and the global login created an automagic account … [23:39:15] !log tools.wd-image-positions deployed f07ce73792 (fixes for somevalue/novalue and user script) [23:39:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-image-positions/SAL