[07:49:14] <godog>	 greetings
[09:00:14] <dcaro>	 morning!
[09:04:00] <volans>	 hola
[09:33:32] <dcaro>	 Any +1 on https://phabricator.wikimedia.org/T419296 ?
[09:34:15] <volans>	 checking
[09:37:01] <dhinus>	 morning
[09:38:03] <volans>	 (as the task is NDA I suggest moving the discussion to private)
[09:40:10] <dcaro>	 👍
[11:25:29] <taavi>	 current review requests: https://gerrit.wikimedia.org/r/c/cloud/wmcs-cookbooks/+/1248798 https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/97
[11:27:03] <godog>	 {{done}}
[11:46:29] * dcaro lunch
[11:49:48] <taavi>	 hrm, apparently kubernetes(/kubeadm?) requires the conntrack cli to be installed, but it's not installed to new workers anymore for whatever reason
[11:50:10] <taavi>	 the apt history file on an older worker suggests that bin:kubelet used to depend on it:
[11:50:11] <taavi>	 Commandline: /usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install kubelet
[11:50:11] <taavi>	 Install: kubernetes-cni:amd64 (1.4.0-1.1, automatic), conntrack:amd64 (1:1.4.7-1+b2, automatic), kubelet:amd64 (1.30.14-1.1)
[11:50:13] <taavi>	 sigh
[11:55:59] <taavi>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/1249260
[12:01:36] <dhinus>	 taavi: +1d
[12:08:22] <godog>	 heads up, tomorrow EU morning I got T417393 scheduled for the switch reboot tests
[12:08:23] <stashbot>	 T417393: Carry out controlled network switch down tests in cloud - https://phabricator.wikimedia.org/T417393
[12:09:23] <godog>	 XioNoX topranks my understanding is that you'll be around, I'm happy to do the interface shutting myself and assess impact, and possibly poke you if disaster ensues, or if you'd rather shut the interfaces yourself that works for me too
[12:10:33] <topranks>	 godog: I’ll be around.  Fine for you to do it but ping me first so I don’t forget :)
[12:10:47] <topranks>	 well downtime the switches and hoses I presume in advance?
[12:11:21] <godog>	 topranks: ok! will ping you before I start, what's the recommended command to shut/unshut interfaces ?
[12:11:41] <godog>	 and yes I'll be downtiming the hosts, the switch shouldn't (hah) itself go down
[12:12:11] <topranks>	 yeah actually good point, we don’t alert on host facing ints so probably fine
[12:12:48] <topranks>	 “set interface x/x/x disable” is usually what I do
[12:14:06] <godog>	 ok! easy enough, and 'delete' instead of 'set' to undo IIRC?
[12:57:24] <dcaro>	 quick review? https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/81 it's blocking me from testing misctools-cli (currently the deb build is broken)
[13:05:49] <godog>	 LGTM
[13:07:41] <dcaro>	 thanks!
[13:16:25] <dcaro>	 that allowed me to test https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/80 :)
[13:16:34] <dcaro>	 another easy review (when anyone is in-between tasks)
[13:26:58] <godog>	 yep LGTM dcaro 
[13:41:03] <dcaro>	 Any +1 for T419182 ? The only thing I think would be missing from toolforge
[13:41:05] <stashbot>	 T419182: Request creation of lingualibre VPS project - https://phabricator.wikimedia.org/T419182
[14:27:05] <andrewbogott>	 dcaro: that request looks good to me, I +1'd
[14:27:12] <andrewbogott>	 I'm gunshy from T416483 but does anyone have a reason for me not to schedule the eqiad1 openstack upgrade? (codfw1dev is already running the new version, which is how we found T416483)
[14:27:12] <stashbot>	 T416483: openstack flamingo: "'enabled' is a required property" for LDAP-managed users - https://phabricator.wikimedia.org/T416483
[14:29:31] <dcaro>	 the fix looks a bit flaky :/ (if I understood it correctly), I'm ok as long as you are confident
[14:32:48] <andrewbogott>	 oh yes the fix is bad. We could wait on the upstream dev to help me figure out what's happening but I think they're as confused as I am at this point.
[14:34:07] <andrewbogott>	 hm, actually, can anyone log into labtesthorizon at all right now?  That seems important :)
[14:38:30] <andrewbogott>	 dcaro: ^ broken for you too? (you might need to try in a private window to get the login prompt)
[14:39:30] <dcaro>	 I get unauthorized
[14:40:26] <andrewbogott>	 after a suspiciously long pause?
[14:40:36] <dcaro>	 yep
[14:40:39] <andrewbogott>	 and then "Authentication Failure
[14:40:39] <andrewbogott>	 Authentication attempt has failed. "
[14:41:08] <andrewbogott>	 ok, so definitely not ready to upgrade eqiad1. And now I know how I'll be spending my day
[15:35:23] <taavi>	 `toolforge webservice restart` will now also provision a gatewayapi httproute object :P
[15:36:42] <dcaro>	 \o/
[15:37:04] <dcaro>	 it's not routed from the internet through that one yet no? Still using the nginx ingress?
[15:39:58] <taavi>	 correct
[15:53:10] <taavi>	 one more thing I'd like to get merged today: https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/88
[16:22:15] <dcaro>	 taavi: lgtm, there's a nit but feel free to ignore
[18:36:00] * dcaro off
[18:36:02] <dcaro>	 cya tomorrow!
[19:54:17] <andrewbogott>	 topranks or taavi, if either of you is still around... until a few weeks ago cloudidp2001-dev could reach ldap on cloudservices200[45]-dev (that's ports 389 and 636) but no longer. Do you know of anything that would have broken that?  Running ldap on those hosts is a stupid hack so I'm open to building standalone ganeti ldap servers if you think that's the right next step but I'm sure this was working recently.
[22:01:49] <taavi>	 uh, might be fallout from https://gerrit.wikimedia.org/r/plugins/gitiles/operations/homer/public/+/c47b41bb8e8e16ff43ff208e592ebddbd38db3d7%5E%21/#F0, will have a look tomorrow