[05:17:57] s.a [09:47:59] !log admin draining coludvirt1013 for reboot (T280641) [09:48:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:48:03] T280641: ceph: Upgrade to latest Nautilus/Octopus to fix CVE-2021-20288 - https://phabricator.wikimedia.org/T280641 [10:37:41] !log admin draining coludvirt1016 for reboot (T280641) [10:37:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:37:44] T280641: ceph: Upgrade to latest Nautilus/Octopus to fix CVE-2021-20288 - https://phabricator.wikimedia.org/T280641 [11:16:54] !log admin draining and rebooting coludvirt1017, last one today (T280641) [11:16:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [11:16:57] T280641: ceph: Upgrade to latest Nautilus/Octopus to fix CVE-2021-20288 - https://phabricator.wikimedia.org/T280641 [13:05:46] * chicocvenancio gently points cloud people to https://toolsadmin.wikimedia.org/tools/membership/status/1055 very active wikipedia and wikidata volunteer from Brazil [13:09:51] the backlog in toolforge membership requests seems to be quite high overall, I don't unfortunately have access to process those yet [13:10:22] * dcaro looking [13:30:29] just did a full review, thanks for the notice! [13:31:35] there are some pending for feedback from almost a month ago, I guess those could be closed too [13:32:14] there's a couple with a bit less than a month, I'll give them a few more days [13:32:22] (closed the ones of >30 days) [14:12:20] o/ I got a `Puppet failure on diff-privacy-beam-test.wmf-research-tools.eqiad1.wikimedia.cloud` email this morning (https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Cloud_VPS_alert_Puppet_failure_on). Went and ran `sudo run-puppet-agent` to see if that fixed it and got the message below. I suspect this is because I had an older instance with the exact same name and a config file is out-of-date somewhere as a result [14:12:20] (the first instance would not rebuild for some reason when I needed to so I deleted it and created anew w/ same name). Any help fixing it would be appreciated (or happy to file a phab ticket if this is more complicated). I did try the `sudo find /var/lib/puppet/ssl ...` command on the instance but that didn't fix the issue. [14:12:26] https://www.irccloud.com/pastebin/AxpMeath/ [14:13:58] isaacj: I think you forgot to sudo [14:14:15] wait no, it's there xd [14:14:55] dcaro: yeah, and for more context, i haven't changed any of the puppet config etc. so i think this is just an automated process but i wasn't even aware it was happening until i got the email :) [14:15:52] do you mind if I ssh and check around? [14:16:19] dcaro go for it -- many thanks. in the worse case where it has to be rebuilt, it's not the end of hte world just a hassle [14:19:53] did the cleanup on the server side too, and now it's running ok [14:20:02] isaacj: ^can you verify? [14:21:11] dcaro: yep, thanks! is this something i could have easily fixed in case it happens again in the future? [14:21:37] isaacj: if you have ssh to the puppetmaster yes, just cleaning up the cert (not sure if you do though) [14:21:54] otherwise you need to get someone with access to it, not sure why it got borked though [14:24:19] ahh...that sounds like something i should avoid mucking around. well i'll save this conversation so should it happen again i can at least provide better guidance. i haven't ever gotten this error in the year or two of working with Cloud VPS which is why I assume it was my creation of an instance with the exact same name as an older instance but no idea really [14:26:04] yep, that might create an issue as the certs are hostname-identified, so two hosts with the same name would have the same 'cert name', and complain [14:38:42] :thumbs up: i'll avoid that pattern then in the future (or at least give more time for configs to be cleared). thanks again! [14:56:37] In theory old Puppet client certs are removed when an instance is deleted, but in practice leaks are possible. [16:26:24] Hey, a toolforge root around please for a PM? [16:27:51] Urbanecm, sure