[00:03:28] dduvall: I stubbed my toe on something today that you may or may not be aware of. In repos/releng/zuul/tofu-provisioning I am using kokkuri to build an image and then running later CI stages using that image. By default the image tag is ${CI_COMMIT_REF_NAME} which is generally the origin branch name. [00:05:16] What I stumbled on today was that when you are reusing the same branch name (force push or separate MR) the image can be cached on the gitlab runner and not fetch the latest build with that tag. I was debugging "live" in CI and this bit me hard until I figured out what was going on. [00:06:26] The fix seems to be to set `PUBLISH_IMAGE_TAG: $CI_COMMIT_SHORT_SHA` in the kokkuri variables. Then you will get a fresh tag on a force push or other branch reuse. [06:53:39] 10GitLab (Infrastructure), 06collaboration-services, 06Release-Engineering-Team, 13Patch-For-Review: Upgrade GitLab to major version 18 - https://phabricator.wikimedia.org/T394382#10987180 (10Jelto) [06:58:05] 10GitLab (Auth & Access), 10CAS-SSO, 06collaboration-services, 06Infrastructure-Foundations, 10Release-Engineering-Team (Radar): Add GitLab to offboarding workflow - https://phabricator.wikimedia.org/T339843#10987189 (10MoritzMuehlenhoff) >>! In T339843#10986537, @Dzahn wrote: > Since infrastructure-secu... [07:59:15] 10GitLab (Infrastructure), 06collaboration-services, 06Release-Engineering-Team, 13Patch-For-Review: Upgrade GitLab to major version 18 - https://phabricator.wikimedia.org/T394382#10987288 (10ops-monitoring-bot) Cookbook cookbooks.sre.gitlab.upgrade was started by jelto@cumin1003 on GitLab host gitlab2002.... [10:17:17] 10GitLab (Project Migration), 06cloud-services-team, 10Toolforge, 13Patch-For-Review: Migrate misctools package to GitLab - https://phabricator.wikimedia.org/T398202#10987556 (10A_smart_kitten) [10:18:42] 10GitLab (Project Migration), 06cloud-services-team, 10Toolforge, 13Patch-For-Review: Migrate misctools package to GitLab - https://phabricator.wikimedia.org/T398202#10987560 (10taavi) [10:18:50] 10GitLab (Project Migration), 06cloud-services-team, 10Toolforge, 13Patch-For-Review: Migrate misctools package to GitLab - https://phabricator.wikimedia.org/T398202#10987562 (10taavi) [11:12:59] GitLab needs a short maintenance restart at 12:00 UTC [12:09:00] GitLab maintenance done [12:09:30] 10GitLab (Infrastructure), 06collaboration-services, 06Release-Engineering-Team, 13Patch-For-Review: Upgrade GitLab to major version 18 - https://phabricator.wikimedia.org/T394382#10987824 (10ops-monitoring-bot) Cookbook cookbooks.sre.gitlab.upgrade started by jelto@cumin1003 on GitLab host gitlab2002.wiki... [12:10:37] 10GitLab (Infrastructure), 06collaboration-services, 06Release-Engineering-Team, 13Patch-For-Review: Upgrade GitLab to major version 18 - https://phabricator.wikimedia.org/T394382#10987825 (10Jelto) [12:15:03] 10GitLab (Infrastructure), 06collaboration-services, 06Release-Engineering-Team, 13Patch-For-Review: Upgrade GitLab to major version 18 - https://phabricator.wikimedia.org/T394382#10987839 (10Jelto) All hosts are updated successfully, I also enabled unattended upgrades in `devtools` and `gitlab-runners` ag... [12:26:22] 10GitLab (Infrastructure), 06collaboration-services, 06Release-Engineering-Team, 13Patch-For-Review: Upgrade GitLab to major version 18 - https://phabricator.wikimedia.org/T394382#10987878 (10Jelto) [13:09:00] 10GitLab (Infrastructure), 06collaboration-services, 06Release-Engineering-Team, 13Patch-For-Review: Increase GitLabs session timeout duration - https://phabricator.wikimedia.org/T398791#10987997 (10Jelto) 05Open→03Resolved I merged the change above and updated the settings on all GitLab hosts. The... [13:59:12] 10GitLab (Infrastructure), 06collaboration-services, 06Release-Engineering-Team, 13Patch-For-Review: Upgrade GitLab to major version 18 - https://phabricator.wikimedia.org/T394382#10988305 (10Jelto) 05Open→03Resolved I'll resolve the task, please re-open if there are any issues. [14:05:55] 10GitLab (Auth & Access), 06cloud-services-team, 10Cloud-VPS, 10Toolforge: Sync WMCS GitLab group membership from LDAP - https://phabricator.wikimedia.org/T398217#10988351 (10joanna_borun) p:05Triage→03Medium [14:21:22] 10GitLab (Auth & Access), 06cloud-services-team, 10Cloud-VPS, 10Toolforge: Sync WMCS GitLab group membership from LDAP - https://phabricator.wikimedia.org/T398217#10988440 (10Jelto) >>! In T398217#10959948, @thcipriani wrote: > FWIW, there are utilities that run in systemd (managed by puppet) to manage lda... [15:28:42] 10GitLab (Auth & Access), 06cloud-services-team, 10Cloud-VPS, 10Toolforge: Sync WMCS GitLab group membership from LDAP - https://phabricator.wikimedia.org/T398217#10988721 (10bd808) >>! In T398217#10988440, @Jelto wrote: > Yes that's right, we have a script and a systemd timer to sync users from ldap to Gi... [15:34:55] bd808: that makes sense. in kokkuri itself, we're using `PUBLISH_IMAGE_TAG: "pipeline-${CI_PIPELINE_ID}"` https://gitlab.wikimedia.org/repos/releng/kokkuri/-/blob/main/.gitlab-ci.yml?ref_type=heads#L169 [16:07:28] 10GitLab (Auth & Access), 10CAS-SSO, 06collaboration-services, 06Infrastructure-Foundations, 10Release-Engineering-Team (Radar): Add GitLab to offboarding workflow - https://phabricator.wikimedia.org/T339843#10988944 (10Dzahn) I am afraid the specific thing to GitLab is that the can of "private repos" ha... [17:01:05] dduvall: I wondered briefly if it would be helpful (and possible) to setup the runners use `imagePullPolicy: Always`, but that might make everything slower just to dodge an edge case. [17:32:01] bd808: i think a better long-term solution is to have kokkuri export the full ref (including digest) for the build-and-run-in-later-stage use case [17:32:49] that way the ref is unambiguous and will always resolve to the image that was just built [17:40:45] in fact, looks like the new `kokkuri image bake` command is already doing that, so i think we would just need to refactor the `kokkuri image build` command to use the same `process_buildx_build_metadata` function that bake uses [18:45:24] 10GitLab (CI & Job Runners), 10Beta-Cluster-Infrastructure, 06collaboration-services, 10m3api: Unable to connect to Beta Cluster from WMCS GitLab CI runners - https://phabricator.wikimedia.org/T397591#10989512 (10LucasWerkmeister) Can confirm that a build with the `wmcs` tag [works now](https://gitlab.wiki... [19:02:44] 10GitLab (CI & Job Runners), 10Beta-Cluster-Infrastructure, 06collaboration-services, 10m3api: Unable to connect to Beta Cluster from WMCS GitLab CI runners - https://phabricator.wikimedia.org/T397591#10989567 (10Dzahn) 05Open→03Resolved a:03Dzahn Thanks for confirming it works! Not that I know o... [19:14:46] 10GitLab (CI & Job Runners): [kokuri] Use a unique per CI run tag by default - https://phabricator.wikimedia.org/T399120 (10bd808) 03NEW [19:15:58] dduvall: I made T399120 to keep track of the idea of changing the default behavior [19:15:59] T399120: [kokuri] Use a unique per CI run tag by default - https://phabricator.wikimedia.org/T399120