[06:53:39] <jayme>	 akosiaris: AIUI the static ip thing does work on a per-pod basis, right? So that would probably not work as expected with deployments or alike
[06:53:44] <jayme>	 (https://docs.tigera.io/calico/latest/networking/ipam/use-specific-ip)
[08:13:41] <akosiaris>	 jayme: yeah, I want to test, but it does look like that if we go down that route, it will have to be a few hardcoded pods, not a deployment or anything like that.
[08:14:03] <jayme>	 hmm...that does not sound super nice either :/
[08:14:31] <jayme>	 worse even than pinning them to apiservers IMHO
[08:14:49] <akosiaris>	 there is no solution that is super nice overall (at least from any that we already discussed)
[08:14:56] <akosiaris>	 apiservers?
[08:15:09] <akosiaris>	 ah, k8s control plane
[08:15:17] <jayme>	 yes, sorry
[08:16:00] <akosiaris>	 well, neither are nice for sure. Which is worse... good question
[08:26:23] <elukey>	 hey folks, qq about dragonfly and docker-registry nodes - shall we upgrade to Bookworm? :D
[08:27:04] <elukey>	 dragonfly probably is the quickest
[08:29:20] <elukey>	 for the docker registry no idea, but we could probably upgrade the standby dc first, then failover etc..
[08:38:39] <jayme>	 dragonfly should indeed be easy - registry could be a mess
[08:40:06] <elukey>	 yeah but we are running on EOL, not great
[08:43:36] <jayme>	 yes, just stating the obvious - sorry 
[08:45:31] <elukey>	 no need, I got what you meant :)
[08:46:34] <elukey>	 for dragonfly, IIUC we just need to rebuild dragonfly-supernode and then create new VMs
[08:46:49] <jayme>	 I fear that we (as in serviceops) don't have time in the near future
[08:47:09] <elukey>	 I can try to set some time for Dragonfly
[08:47:13] <jayme>	 yes re: dragonfly. Although I think you already build the packages, right?
[08:47:23] <jayme>	 for bookworm k8s workers
[08:47:25] <elukey>	 for the workers IIRC, not the supernode
[08:47:33] <jayme>	 IIRC it's all from the same source package
[08:47:51] <elukey>	 then I may have not uploaded the supernode deb
[08:48:06] <jayme>	 I could also be wrong :)
[08:49:12] <jayme>	 I thing I would rather reimage the existing VMs with bookworm (outside a mw deployment window)
[08:49:18] <elukey>	 /var/cache/pbuilder/result/bookworm-amd64/dragonfly-supernode_1.0.6-2_amd64.deb
[08:49:21] <elukey>	 nope you are not :)
[08:49:37] <elukey>	 ahh right sometimes I forget that we can reimage VMs :D
[08:49:42] <elukey>	 yes yes then easier
[08:49:50] <jayme>	 no need to change config then and the clients will fall back to fetching directly in case they need to
[08:50:38] <elukey>	 dragonfly-supernode | 1.0.6-2 | bookworm-wikimedia | main | amd64
[08:50:47] <elukey>	 see past Luca is always better than present one
[08:50:51] <elukey>	 I always say that
[08:50:58] <jayme>	 seems like somebody did the right thing :)
[08:51:04] <elukey>	 ok then we just need to schedule the reimage
[08:53:48] <elukey>	 also found https://phabricator.wikimedia.org/T332011
[08:53:52] <elukey>	 I'll try to do it next week
[08:55:11] <jayme>	 <3 thanks
[12:37:31] <jayme>	 A new alert has been added to show workers that have been cordoned for >=24: https://alerts.wikimedia.org/?q=%40state%3Dactive&q=team%3Dsre&q=alertname%3DKubernetesWorkerUnschedulable
[12:48:19] <elukey>	 nice!
[12:53:09] <cdanis>	 jayme: akosiaris: maybe a daemonset of pods limited to just the control plane servers?
[12:53:28] <cdanis>	 none of this sounds nice but I think daemonset+nodeport is sounding nicer at this point :/
[12:54:18] <jayme>	 I think we have more than 3 coredns pods but yes, we could make them prefer the control-planes I think
[12:55:42] <claime>	 iiuc the need is to have *some* coredns pods be on reliably stable ips, not all?
[12:55:52] <jayme>	 yes
[12:56:24] <cdanis>	 yes
[12:58:01] <cdanis>	 I'm probably still waking up and need more coffee, but, I actually don't understand what Alex told me yesterday re: Calico doing BGP advertiements for service IPs with externalTrafficPolicy: Local
[12:58:33] <cdanis>	 if Calico is only advertising the /32 for the service IP just from nodes that have a running and ready pod, shouldn't that just work?
[12:59:08] <cdanis>	 and that avoids SNAT
[12:59:27] <cdanis>	 (https://docs.tigera.io/calico/latest/networking/configuring/advertise-service-ips)
[13:00:14] <jayme>	 yes, maybe. But I think we haven't tested that. And it also relies on ECMP which would, aiui, lead to unbalanced load
[13:00:28] <jayme>	 depending on "where" the caller is
[13:00:32] <cdanis>	 I don't think that's a problem at this scale
[13:00:48] <cdanis>	 we only have so many PTRs and they're cacheable
[13:01:02] * jayme nods
[13:09:45] <akosiaris>	 the probably with calico doing BGP advertisements for service IPs is that we don't have graceful restart in calico open core
[13:09:49] <akosiaris>	 problem*
[13:10:12] <akosiaris>	 so 1 node being rebooted, still receives traffic until the BGP timers expire
[13:10:26] <akosiaris>	 and that traffic gets blackholed
[13:10:29] <cdanis>	 it's DNS, it will retry
[13:10:49] <cdanis>	 do we have BFD?
[13:12:05] <akosiaris>	 no it's not in calico open core
[13:12:21] <cdanis>	 lol
[13:12:23] <akosiaris>	 IIRC there's a task asking exactly for that from yours truly in calico's github
[13:12:37] <cdanis>	 well I still think it's worth doing
[13:12:38] <akosiaris>	 I even had a patch
[13:12:50] <akosiaris>	 but it didn't align with their vision (or their business)
[13:13:09] <akosiaris>	 they weren't negative or anything btw. Even suggested an approach
[13:13:09] <cdanis>	 we don't need an ideal solution we need something to make debugging without jumping through horrible hoops possible
[13:15:07] <cdanis>	 like really https://phabricator.wikimedia.org/P67401 is the alternative :)
[13:20:45] <claime>	 !log homer lsw1-b6-codfw* commit 'T372878'
[13:20:45] <stashbot>	 claime: Not expecting to hear !log here
[13:20:45] <stashbot>	 T372878: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878
[13:20:49] <claime>	 soz