[06:57:18] <taavi>	 morning
[07:22:48] <godog>	 greetings
[07:27:51] <dcaro>	 morning
[07:29:47] <godog>	 getting ready for the tools nfs switch in 30m 
[07:32:10] <godog>	 tempting to not reboot the currently stuck nfs workers
[07:37:05] <dcaro>	 oh, sorry, I started rebooting them xd
[07:37:16] <dcaro>	 I can stop though
[07:37:34] <godog>	 dcaro: no all good, thank you!
[07:37:35] <dcaro>	 (kinda like before getting coffee I run the cookbook every morning)
[07:37:41] <godog>	 lol
[07:37:59] <godog>	 actually more like lolsob
[07:39:25] <dcaro>	 🤞 for the upgrade
[07:39:32] <dcaro>	 (to fix it so I don't do it anymore xd)
[07:40:24] <godog>	 heheh yeah 🤞 indeed
[07:43:01] <volans|off>	 🤞
[07:43:18] * volans hides for the duration of the upgrade :-P
[07:43:59] <godog>	 hahah I noticed users are at least quite lenient with toolforge/us
[08:01:21] <godog>	 ok starting
[08:06:54] <godog>	 we're off to a not so great start https://phabricator.wikimedia.org/P83774
[08:07:51] <taavi>	 that seems to be due to this: https://phabricator.wikimedia.org/P83775
[08:08:47] <godog>	 thank you
[08:09:14] <taavi>	 godog: for some reason, the roles are applied on the `tools-nfs-` prefix level, not on the VM directly, and the cookbook doesn't seem to be able to handle that
[08:10:04] <godog>	 mmhh ok I'm tempted to move the puppet settings from tools-nfs- prefix onto to tools-nfs-2
[08:10:07] <godog>	 and try again
[08:10:13] <taavi>	 godog: tryr now? I applied 'role::wmcs::nfs::standalone' to the nfs-3 VM directly
[08:10:28] <godog>	 ok trying again
[08:10:59] <godog>	 yeah that's better, thank you taavi !
[08:11:10] <godog>	 making a note to fix this later
[08:12:38] <godog>	 ok partial success
[08:13:17] <taavi>	 hm?
[08:15:07] <godog>	 wmcs-prepare-cinder-volume failed, I'm finishing manually
[08:17:28] <godog>	 waiting for dns to propagate
[08:18:23] <taavi>	 [for later] why did that fail?
[08:21:43] <godog>	 indeed I'll document that too
[08:21:54] <godog>	 ok flip is done, I'll start rebooting nfs workers
[08:22:43] <godog>	 taavi: would you minding checking on non-nfs-workers ?
[08:22:52] <taavi>	 will do
[08:23:23] <godog>	 thx
[08:25:38] <volans>	 from the alert seems that toolforge is unhappy
[08:25:56] <dcaro>	 yep, until we reboot the nfs workers it will not be happy I think
[08:26:16] <godog>	 yeah I see a couple of pages re: 5xx
[08:27:01] <godog>	 so I started wmcs.toolforge.k8s.reboot on a bunch of nfs workers, currently waiting for phase drain on the first one
[08:27:05] <dcaro>	 let me know when the first nfs worker is rebooted, we can check if that one comes healthy
[08:27:13] <dcaro>	 *come up
[08:27:21] <godog>	 ok, it'll be tools-k8s-worker-nfs-1:
[08:27:55] <dcaro>	 ack
[08:28:03] <godog>	 still in wait_drain though
[08:28:14] <dcaro>	 we can split the workers if you want
[08:28:18] <taavi>	 godog: we're maxing out the haproxy connection limits with all the nfs-enabled tools so even non-nfs web services are not doing great, but otherwise the non-nfs workers are doing just fine
[08:28:25] <taavi>	 I will next look at the non-k8s nfs clients
[08:28:32] <godog>	 taavi: thank you!
[08:28:38] <godog>	 dcaro: sure
[08:29:25] <godog>	 I did these https://phabricator.wikimedia.org/P83778
[08:29:47] <dcaro>	 damn sorting xd
[08:29:54] <godog>	 mmhh except that the command line with two spaces doesn't work
[08:29:59] <godog>	 anyways
[08:30:27] <godog>	 ok my bad there's some garbage in there, the cookbook has exited anyways so we can coordinate dcaro 
[08:31:38] <godog>	 dcaro: nfs-1 is back up
[08:31:39] <dcaro>	 np
[08:32:52] <godog>	 dcaro: recommendations/suggestions on how we can split the workers ?
[08:33:25] <dcaro>	 lets get the list of remaining workers and split it in two
[08:33:32] <godog>	 ok
[08:33:40] <dcaro>	 all workers https://etherpad.wikimedia.org/p/2025-10-workers-reboot
[08:33:53] <dcaro>	 (sorted the same way the cookbook did I think)
[08:34:28] <godog>	 ok I'll keep going with the first half
[08:35:33] <dcaro>	 ack, I think we missed 22 for some reason
[08:36:16] <godog>	 yes that's my bad copy/pasta
[08:36:53] <taavi>	 it seems like the nfs server on clouddumps1001 is also having some sort of an issue
[08:37:14] <taavi>	 https://phabricator.wikimedia.org/P83780
[08:37:31] <dcaro>	 that's the client thing again?
[08:37:32] <godog>	 ah yes I know what's going on, clouddumps1001 needs kicking of stuck nfs clients
[08:37:36] <godog>	 indeed
[08:38:25] <godog>	 this https://phabricator.wikimedia.org/T404833#11189123
[08:38:26] <taavi>	 it needs what?
[08:38:31] * taavi looks
[08:39:06] <dcaro>	   458  echo expire> /proc/fs/nfsd/clients/1398/ctl 
[08:39:13] <dcaro>	 something similar to that, but with the right client id
[08:39:39] <taavi>	 that did it, thanks
[08:39:50] <dcaro>	 ack
[08:40:40] <godog>	 thoughts on just rebooting nfs workers as opposed to waiting for draining? I suspect we'll be here all day otherwise
[08:41:12] <taavi>	 +1
[08:41:39] <godog>	 if you are curious on what clients reconnected: on tools-nfs-3 grep ^name: /proc/fs/nfsd/clients/*/info
[08:41:57] <dcaro>	 +1
[08:42:21] <godog>	 ok! what cookbook is recommended for that ?
[08:42:57] <taavi>	 wmcs.vps.instance.force_reboot can hard reboot individual vms
[08:43:49] <godog>	 ack thanks, will it attempt soft reboot first too ? either acpi or ssh I suppose ?
[08:44:09] <taavi>	 that one does not. it just issues the openstack hard reboot command
[08:44:44] <godog>	 ok thank you, I'll test with a gentler 'reboot'
[08:47:06] <dcaro>	 https://gerrit.wikimedia.org/r/c/cloud/wmcs-cookbooks/+/1195629
[08:48:02] <dcaro>	 it seems to be failing to soft-reboot though :/, a bit quick
[08:48:27] <taavi>	 most often with stuck nfs issues you need a full hard reboot
[08:48:29] <dcaro>	 just timing ot
[08:48:31] <dcaro>	 yep
[08:48:37] <dcaro>	 it has 30s timeout
[08:49:15] <godog>	 yeah indeed
[08:49:24] <godog>	 hard reboot it is
[08:52:14] <godog>	 yeah I think also hard reboot -> will need clouddumps1001 action
[09:07:17] <dcaro>	 78 did take a long time to come back up :/
[09:09:50] <godog>	 ack
[09:09:53] <godog>	 availability is recovering
[09:10:49] <dcaro>	 🎉
[09:11:29] <godog>	 or not, spoke too soon
[09:11:32] <dcaro>	 81 is taking it's time to reboot too
[09:11:57] <dcaro>	 hahahaha, I can see myself jumping on every spike up and down
[09:12:18] <taavi>	 is someone looking at those stuck clouddumps1001 clients?
[09:12:24] <godog>	 taavi: I am yes
[09:12:30] <godog>	 fixing them as we go
[09:12:43] <taavi>	 thanks
[09:12:51] <godog>	 """fixing"""
[09:17:54] <dcaro>	 last nfs worker just rebooted
[09:18:20] <godog>	 \o/
[09:18:29] <dcaro>	 haproxy availability seems at similar levels than before
[09:19:05] <dcaro>	 memory usage has spiked though it seems
[09:19:10] <godog>	 can confirm, I kicked all 'courtesy' clients from clouddumps1001
[09:19:32] <dcaro>	 let's see if the memory usage stabilizes as before https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview?orgId=1&from=now-1h&to=now&timezone=utc&var-cluster_datasource=P8433460076D33992&var-cluster=tools
[09:19:50] <taavi>	 dcaro: I would expect memory usage to go down as k8s processes all the disruption with nodes suddenly going down and pods failing health checks
[09:20:00] <taavi>	 (i.e. that matches with what we've seen before)
[09:20:00] <dcaro>	 yep
[09:21:01] <dcaro>	 currently 0 stuck workers :)
[09:21:13] <taavi>	 don't jinx it :-)
[09:21:21] <godog>	 lolz and also \o/
[09:21:56] <taavi>	 godog: this went relatively well overall, thank you!!
[09:22:18] <godog>	 taavi: sure np! I'm glad it went relatively well
[09:22:37] <godog>	 "only" ~1h or low availability
[09:22:40] <taavi>	 I think the main thing to fix for the next time is to figure out why the initial mount failed
[09:22:45] <godog>	 s/or/of/
[09:23:26] <godog>	 yeah AFAICS there's a race now between the cookbook and prepare-volume, where /dev/sdb didn't appear yet
[09:23:41] <godog>	 will document more on the task, I'll take a ~10m break
[09:25:14] <dcaro>	 kudos godog!
[09:43:07] <godog>	 back
[09:43:11] <godog>	 dcaro: cheers
[09:43:18] <godog>	 thanks for the help folks, appreciate it
[10:07:49] <godog>	 mmhh my bad tools-nfs-3 is significantly smaller in cpu/mem than tools-nfs-2, I can resize the instance tomorrow though
[10:15:33] * taavi lunch
[10:18:26] * godog lunch too
[10:39:25] * volans errand + lunch
[13:07:11] <dcaro>	 I have some code increasing the log tailing in toolforge https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/997
[13:07:30] <dcaro>	 that will prevent us from losing logs due to rate limiting, and instead continue receiving them at the rate limit
[13:08:09] <dcaro>	 there's a chance that the alloy process will not be able to keep up if there's many logs, and will die from memory exhaustion (looking at options there), but when it restart it should come up again and continue where it left
[13:08:13] <dcaro>	 (if the logs are still around)
[13:08:59] <dcaro>	 there's some issue with one-off jobs though, as the logs are not around for very long, if the job is generating too many lines (ex. ~5000) with this settings it will not be able to send all of them before the log gets deleted
[13:09:05] <dcaro>	 also looking into that
[13:09:16] <dcaro>	 but for cronjobs and continuous jobs this should be quite helpful
[15:14:00] <dcaro>	 hmm... my loki tests killed lima-kilo xd
[15:14:22] <dcaro>	 starting 4 jobs with just a loop echoing constantly crashed something
[15:14:58] <dcaro>	 oh... 2-json.log: no space left on device" message=
[15:14:59] <dcaro>	 of course
[15:15:57] <dcaro>	 huh `/dev/sr0                                48M   48M     0 100% /mnt/lima-cidata`
[15:16:04] <dcaro>	 that's just 48M
[15:24:03] <dcaro>	 `/dev/vda1                               50G   49G     0 100% /`
[15:24:05] <dcaro>	 that's the isuse
[15:46:00] <volans>	 if there are no objections I'll reboot cloudcumin2001 (there is noone logged in nor screen/tmux or cookbook running)
[15:46:15] <dcaro>	 ack from me
[15:46:21] <godog>	 SGTM volans 
[15:47:33] <volans>	 k proceeding
[15:54:07] <volans>	 {done} lmk when could be a good time for cloudcumin1001, I see some people loggedin, although nothing really running right now 
[15:54:46] <godog>	 +1 from me to reboot volans 
[15:54:53] <godog>	 I logged out
[15:55:12] <volans>	 you've a vim open in a tmux :D
[15:56:17] <godog>	 doh! thank you, {{done}}
[15:56:31] <dcaro>	 feel free to kill anything I might have open
[15:57:06] <dcaro>	 lme brb
[15:59:40] <volans>	 thx, proceeding
[16:00:40] * godog off
[17:35:58] * dcaro off