[00:34:38] 10Continuous-Integration-Infrastructure (Zuul upgrade): Update ZuulDevOpsBot app credentials to include admin role access - https://phabricator.wikimedia.org/T399780#11011546 (10bd808) [00:39:10] 10Continuous-Integration-Infrastructure (Zuul upgrade): Update ZuulDevOpsBot app credentials to include admin role access - https://phabricator.wikimedia.org/T399780#11011551 (10bd808) [00:39:33] (03open) 10bd808: dev: Rotate application credential [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/30 (https://phabricator.wikimedia.org/T399780) [00:42:59] (03merge) 10bd808: dev: Rotate application credential [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/30 (https://phabricator.wikimedia.org/T399780) [03:40:12] 06Release-Engineering-Team, 06collaboration-services, 10function-evaluator, 10function-orchestrator, and 3 others: gitlab CI mem issue - https://phabricator.wikimedia.org/T399348#11011726 (10ecarg) Thank you, everyone~ @Jelto I tried using `wmcs` and unfortunately it wasn't enough πŸ₯² [06:04:00] 10Beta-Cluster-Infrastructure, 10MW-1.45-notes (1.45.0-wmf.9; 2025-07-08), 13Patch-For-Review: Move *.beta.wmflabs.org to *.beta.wmcloud.org - https://phabricator.wikimedia.org/T289318#11011841 (10Krinkle) p:05Triageβ†’03Medium a:03Krinkle [06:33:44] 10GitLab (Infrastructure), 06collaboration-services, 13Patch-For-Review: upgrade gitlab hosts to bookworm - https://phabricator.wikimedia.org/T399306#11011924 (10ops-monitoring-bot) Host gitlab1004.wikimedia.org rebooted by jelto@cumin1003 with reason: None [06:51:28] 06Release-Engineering-Team, 06collaboration-services, 10function-evaluator, 10function-orchestrator, and 3 others: gitlab CI mem issue - https://phabricator.wikimedia.org/T399348#11011950 (10Jelto) @ecarg can you link to the jobs which got killed on WMCS runners? I can't find any job which had memory issue... [07:39:48] (03PS1) 10Dreamrimmer: Zuul: remove Tusli's Zuul access [integration/config] - 10https://gerrit.wikimedia.org/r/1170236 [07:41:12] (03PS2) 10Dreamrimmer: Zuul: remove Tulsi's Zuul access [integration/config] - 10https://gerrit.wikimedia.org/r/1170236 [07:42:37] (03CR) 10CI reject: [V:04-1] Zuul: remove Tulsi's Zuul access [integration/config] - 10https://gerrit.wikimedia.org/r/1170236 (owner: 10Dreamrimmer) [07:46:39] (03PS3) 10Dreamrimmer: Zuul: remove Tulsi's Zuul access [integration/config] - 10https://gerrit.wikimedia.org/r/1170236 [07:49:46] (03CR) 10Hashar: [C:03+2] "And the account is already banned in Gerrit :)" [integration/config] - 10https://gerrit.wikimedia.org/r/1170236 (owner: 10Dreamrimmer) [07:51:10] (03Merged) 10jenkins-bot: Zuul: remove Tulsi's Zuul access [integration/config] - 10https://gerrit.wikimedia.org/r/1170236 (owner: 10Dreamrimmer) [07:52:47] (03CR) 10Dreamrimmer: "Thanks. I was sure he could not access it, but I did this for cleanup purpose." [integration/config] - 10https://gerrit.wikimedia.org/r/1170236 (owner: 10Dreamrimmer) [07:56:35] 10Beta-Cluster-Infrastructure, 10Pywikibot, 07Pywikibot-tests: Find a way for pywikibot GitHub Actions to avoid IP range blocks of Microsoft Azure hosted runners - https://phabricator.wikimedia.org/T399485#11012060 (10Xqt) Ah, this is not a http/https-proxy but SOCKS5 and requests needs `requests[socks]` whi... [07:57:10] (03CR) 10Hashar: [C:03+2] "+1 thank you very much for the cleanup!" [integration/config] - 10https://gerrit.wikimedia.org/r/1170236 (owner: 10Dreamrimmer) [09:08:02] 10Beta-Cluster-Infrastructure, 10Acme-chief, 07Beta-Cluster-reproducible: Warning about /etc/acmecerts/unified contents during puppet run on deployment-cache-text08 & deployment-cache-upload08 - https://phabricator.wikimedia.org/T399419#11012384 (10Vgutierrez) yes, I'm gonna add a systemd timer to take care... [09:58:45] Yeah. It's quite the dependency chain for UploadWizard to CodeMirror. But that's MediaWiki :D [11:03:37] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for [YOUR DEVELOPER ACCOUNT USERNAME HERE] - https://phabricator.wikimedia.org/T399834 (10SaintWolf) 03NEW [11:04:12] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation forSaintwolf - https://phabricator.wikimedia.org/T399835 (10SaintWolf) 03NEW [11:04:32] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for SaintWolf - https://phabricator.wikimedia.org/T399836 (10SaintWolf) 03NEW [11:08:59] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for SaintWolf - https://phabricator.wikimedia.org/T399836#11012842 (10hashar) [11:09:02] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation forSaintwolf - https://phabricator.wikimedia.org/T399835#11012845 (10hashar) β†’14Duplicate dup:03T399836 [11:09:03] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for [YOUR DEVELOPER ACCOUNT USERNAME HERE] - https://phabricator.wikimedia.org/T399834#11012846 (10hashar) β†’14Duplicate dup:03T399836 [11:10:30] (03Abandoned) 10Hashar: Use standalone jobs for Wikibase Selenium tests [integration/config] - 10https://gerrit.wikimedia.org/r/676107 (https://phabricator.wikimedia.org/T287582) (owner: 10Hashar) [11:10:33] (03Abandoned) 10Hashar: jjb: ensure npm script exists in wikibase-selenium job [integration/config] - 10https://gerrit.wikimedia.org/r/1112784 (https://phabricator.wikimedia.org/T287582) (owner: 10Hashar) [11:24:32] 10Scap: scap not complaining about dependencies only partially deployed with the train - https://phabricator.wikimedia.org/T397931#11012895 (10hashar) [11:25:31] 10Gerrit, 07Upstream: Gerrit trips on complicated string for PHP syntax highlighting - https://phabricator.wikimedia.org/T395324#11012900 (10hashar) [12:17:50] 10MediaWiki-Releasing, 10MediaWiki-Installer, 06MediaWiki-Stakeholders-Group, 05MW-1.45-release: Expand the set of bundled extensions and skins in MediaWiki 1.45 - https://phabricator.wikimedia.org/T392080#11013010 (10A_smart_kitten) [13:01:50] (03PS1) 10Hashar: Zuul: [mediawiki/extensions/ReaderExperiments] add CI [integration/config] - 10https://gerrit.wikimedia.org/r/1170337 [13:02:17] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/ReaderExperiments] add CI [integration/config] - 10https://gerrit.wikimedia.org/r/1170337 (owner: 10Hashar) [13:04:10] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/ReaderExperiments] add CI [integration/config] - 10https://gerrit.wikimedia.org/r/1170337 (owner: 10Hashar) [13:04:43] !log Reloaded Zuul to enable CI for mediawiki/extensions/ReaderExperiments | T1170337 [13:04:44] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:05:59] (03CR) 10Jforrester: Zuul: [mediawiki/extensions/ReaderExperiments] add CI (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1170337 (owner: 10Hashar) [13:19:44] 10Scap (SpiderPig πŸ•ΈοΈ): SpiderPig live job log view (terminal / console) sometimes freezes - https://phabricator.wikimedia.org/T399462#11013179 (10Lucas_Werkmeister_WMDE) Seems to be fixed indeed \o/ [13:26:00] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (Zuul upgrade): Simplify how GitHub oauth token is passed to composer - https://phabricator.wikimedia.org/T399846 (10hashar) 03NEW [13:47:42] (03PS1) 10Hashar: jjb: remove GitHub token from castor-save-workspace-cache [integration/config] - 10https://gerrit.wikimedia.org/r/1170347 [13:48:17] (03PS2) 10Hashar: jjb: remove GitHub token from castor-save-workspace-cache [integration/config] - 10https://gerrit.wikimedia.org/r/1170347 (https://phabricator.wikimedia.org/T399846) [13:49:10] !log jenkins: added composer-auth credentials which is a json blob intended to be passed as COMPOSER_AUTH environment variable # T399846 [13:49:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:49:12] T399846: Simplify how GitHub oauth token is passed to composer - https://phabricator.wikimedia.org/T399846 [13:51:54] (03PS1) 10Hashar: jjb: beta: remove COMPOSER_GITHUB_OAUTHTOKEN [integration/config] - 10https://gerrit.wikimedia.org/r/1170348 (https://phabricator.wikimedia.org/T399846) [13:56:14] (03PS1) 10Hashar: jjb: train-deploy-notes: rm COMPOSER_GITHUB_OAUTHTOKEN [integration/config] - 10https://gerrit.wikimedia.org/r/1170350 (https://phabricator.wikimedia.org/T399846) [13:56:43] hmm [13:56:50] I should try another approach [14:44:19] (03update) 10dancy: spiderpig: move error log viewer to its own tab [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/902 (https://phabricator.wikimedia.org/T399777) [14:46:26] (03merge) 10dancy: spiderpig: move error log viewer to its own tab [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/902 (https://phabricator.wikimedia.org/T399777) [14:47:30] (03open) 10dancy: spiderpig/api.py: Add missing /mediawiki/backport route [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/903 (https://phabricator.wikimedia.org/T392610) [14:47:33] (03update) 10dancy: spiderpig/api.py: Add missing /mediawiki/backport route [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/903 (https://phabricator.wikimedia.org/T392610) [14:48:42] (03approved) 10dancy: make-container-image: revert to mediawiki-httpd [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/193 (https://phabricator.wikimedia.org/T378128) (owner: 10swfrench) [14:49:29] (03merge) 10dancy: spiderpig/api.py: Add missing /mediawiki/backport route [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/903 (https://phabricator.wikimedia.org/T392610) [14:51:27] (03open) 10dancy: Release 4.189.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/904 [14:52:35] (03merge) 10dancy: Release 4.189.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/904 [15:05:00] 10Scap (SpiderPig πŸ•ΈοΈ), 13Patch-For-Review: Move error log view to its own tab - https://phabricator.wikimedia.org/T399777#11013791 (10dancy) 05In progressβ†’03Resolved Deployed [15:39:25] (03Abandoned) 10Hashar: jjb: remove GitHub token from castor-save-workspace-cache [integration/config] - 10https://gerrit.wikimedia.org/r/1170347 (https://phabricator.wikimedia.org/T399846) (owner: 10Hashar) [15:39:31] (03Abandoned) 10Hashar: jjb: beta: remove COMPOSER_GITHUB_OAUTHTOKEN [integration/config] - 10https://gerrit.wikimedia.org/r/1170348 (https://phabricator.wikimedia.org/T399846) (owner: 10Hashar) [15:39:38] (03Abandoned) 10Hashar: jjb: train-deploy-notes: rm COMPOSER_GITHUB_OAUTHTOKEN [integration/config] - 10https://gerrit.wikimedia.org/r/1170350 (https://phabricator.wikimedia.org/T399846) (owner: 10Hashar) [15:42:41] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation forSaintwolf - https://phabricator.wikimedia.org/T399865 (10SaintWolf) 03NEW [15:44:10] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for SaintWolf - https://phabricator.wikimedia.org/T399836#11014005 (10Bunnypranav) [15:44:12] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation forSaintwolf - https://phabricator.wikimedia.org/T399865#11014003 (10Bunnypranav) β†’14Duplicate dup:03T399836 [15:44:20] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation forSaintwolf - https://phabricator.wikimedia.org/T399865#11014006 (10Aklapper) @SaintWolf: Please stop creating more and more duplicates or your Phabricator account will get disabled (again). [15:45:26] 10Release-Engineering-Team (Yak Shaving πŸƒπŸͺ’), 10Scap: scap: Please provide a subcommand to deploy a new security patch in scap itself - https://phabricator.wikimedia.org/T361709#11014018 (10sbassett) >>! In T361709#11011155, @thcipriani wrote: > @sbassett what do you think of think of this? Yeah, it's probably... [15:48:56] (03PS1) 10Hashar: jjb: remove GitHub API token from all jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1170375 (https://phabricator.wikimedia.org/T399846) [16:03:59] 10Beta-Cluster-Infrastructure, 10Acme-chief, 07Beta-Cluster-reproducible: Warning about /etc/acmecerts/unified contents during puppet run on deployment-cache-text08 & deployment-cache-upload08 - https://phabricator.wikimedia.org/T399419#11014150 (10bd808) Testing the new timer service out on deployment-acme-... [16:33:48] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/ReaderExperiments] add CI (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1170337 (owner: 10Hashar) [16:34:15] 10Beta-Cluster-Infrastructure, 10Acme-chief, 07Beta-Cluster-reproducible: Warning about /etc/acmecerts/unified contents during puppet run on deployment-cache-text08 & deployment-cache-upload08 - https://phabricator.wikimedia.org/T399419#11014274 (10Vgutierrez) I like the idea of using two passes instead of `... [16:41:58] (03CR) 10Jforrester: Zuul: [mediawiki/extensions/ReaderExperiments] add CI (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1170337 (owner: 10Hashar) [16:44:05] 10Phabricator (Upstream), 10Release-Engineering-Team (Radar), 07Developer Productivity, 07Upstream: Add Open Graph support to Phabricator Maniphest Tasks to have link preview on Telegram, Slack, and other messaging apps - https://phabricator.wikimedia.org/T288117#11014325 (10Aklapper) Thanks for the in... [17:01:01] (03merge) 10swfrench: make-container-image: revert to mediawiki-httpd [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/193 (https://phabricator.wikimedia.org/T378128) [17:08:06] 10Beta-Cluster-Infrastructure, 10Acme-chief, 07Beta-Cluster-reproducible: Warning about /etc/acmecerts/unified contents during puppet run on deployment-cache-text08 & deployment-cache-upload08 - https://phabricator.wikimedia.org/T399419#11014450 (10ssingh) >>! In T399419#11014274, @Vgutierrez wrote: > I like... [17:10:24] 10Beta-Cluster-Infrastructure: 2025-07-11 traffic overload - https://phabricator.wikimedia.org/T399329#11014469 (10bd808) 05In progressβ†’03Resolved [17:15:53] 10Beta-Cluster-Infrastructure, 07Upstream: Fix upstream password manager domain config for beta - https://phabricator.wikimedia.org/T399879 (10Reedy) 03NEW [17:17:21] 10Beta-Cluster-Infrastructure, 07Upstream: Fix upstream password manager domain config for beta - https://phabricator.wikimedia.org/T399879#11014506 (10Reedy) [17:18:08] 10Beta-Cluster-Infrastructure, 07Upstream: Fix upstream password manager domain config for beta - https://phabricator.wikimedia.org/T399879#11014509 (10Reedy) [17:25:36] 10Continuous-Integration-Infrastructure (Zuul upgrade): Flannel networking broken in Magnum cluster because upstream containers are missing - https://phabricator.wikimedia.org/T399882 (10bd808) 03NEW [17:36:25] 10Beta-Cluster-Infrastructure, 07Upstream: Fix upstream password manager domain config for beta - https://phabricator.wikimedia.org/T399879#11014604 (10Reedy) p:05Triageβ†’03Low [17:43:12] 10Continuous-Integration-Infrastructure (Zuul upgrade): Flannel networking broken in Magnum cluster because upstream containers are missing - https://phabricator.wikimedia.org/T399882#11014618 (10bd808) `lang=irc [17:28] < bd808> andrewbogott: T399882 is a thing that I am now curious about in the PAWS and Qu... [18:17:25] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for khascall - https://phabricator.wikimedia.org/T399890 (10Khascall) 03NEW [18:34:33] 10Release-Engineering-Team (Yak Shaving πŸƒπŸͺ’), 10Scap: scap: Please provide a subcommand to deploy a new security patch in scap itself - https://phabricator.wikimedia.org/T361709#11014862 (10thcipriani) This is great info, thanks @sbassett ! Sounds like your ideal workflow might be: # One command that does init... [18:47:33] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 07Essential-Work, 05Release, 05Train Deployments: 1.45.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T392180#11014885 (10dancy) 05Openβ†’03Resolved All seems well. Easy train this week. [18:57:17] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for khascall - https://phabricator.wikimedia.org/T399890#11014902 (10Isaac) Chiming in to vouch for Kaylea -- she's contracting with us on the Research team and shared this task with me internally on slack [19:02:11] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for khascall - https://phabricator.wikimedia.org/T399890#11014916 (10Isaac) @brennen: this is for work that will be documented in {T399696} so if you get a chance to take a look at that task as well and approve or sen... [19:23:10] (03update) 10dduvall: docs: Generate markdown docs from examples [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/134 [19:23:14] (03update) 10dduvall: docs: Generate markdown docs from examples [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/134 [19:24:33] (03update) 10dduvall: docs: Generate markdown docs from examples [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/134 [19:28:48] (03merge) 10dduvall: docs: Generate markdown docs from examples [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/134 [19:31:30] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for khascall - https://phabricator.wikimedia.org/T399890#11015016 (10Aklapper) [19:36:52] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for khascall - https://phabricator.wikimedia.org/T399890#11015037 (10Aklapper) 05Openβ†’03Resolved a:03Aklapper GitLab account approved. (Meh, the GitLab account and the Phab account use different email addres... [19:55:31] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for khascall - https://phabricator.wikimedia.org/T399890#11015103 (10Isaac) Thanks @Aklapper! > (Meh, the GitLab account and the Phab account use different email addresses and the Phab account is linked to some pe... [20:10:12] (03open) 10swfrench: Draft: make-container-image: add mediawiki publish-83 flavour [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/194 (https://phabricator.wikimedia.org/T399884) [20:14:13] (03open) 10dancy: Train.vue: Add active train operation to deployment summary [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/905 [20:14:16] (03update) 10dancy: Train.vue: Add active train operation to deployment summary [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/905 [20:20:55] (03update) 10dancy: Train.vue: Add active train operation to deployment summary [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/905 [20:22:10] (03update) 10dancy: Train.vue: Add active train operation to deployment summary [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/905 [20:24:03] (03PS1) 10Umherirrender: build: Updating mediawiki/mediawiki-phan-config to 0.16.0 [integration/docroot] - 10https://gerrit.wikimedia.org/r/1170424 [20:26:17] 10Continuous-Integration-Infrastructure (Zuul upgrade): Update ZuulDevOpsBot app credentials to include admin role access - https://phabricator.wikimedia.org/T399780#11015189 (10bd808) [20:29:38] (03update) 10dancy: Train.vue: Add active train operation to train summary [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/905 [20:29:43] (03update) 10dancy: Train.vue: Add active train operation to train summary [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/905 [21:06:13] (03open) 10bd808: README: document runtime credential needs [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/31 (https://phabricator.wikimedia.org/T399780) [21:07:34] (03merge) 10bd808: README: document runtime credential needs [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/31 (https://phabricator.wikimedia.org/T399780) [21:12:44] 10Continuous-Integration-Infrastructure (Zuul upgrade), 13Patch-For-Review: Update ZuulDevOpsBot app credentials to include admin role access - https://phabricator.wikimedia.org/T399780#11015296 (10bd808) [21:15:17] 10Release-Engineering-Team (Yak Shaving πŸƒπŸͺ’), 10Scap: scap: Please provide a subcommand to deploy a new security patch in scap itself - https://phabricator.wikimedia.org/T361709#11015312 (10sbassett) >>! In T361709#11014862, @thcipriani wrote: > Sounds like your ideal workflow might be: > # One command that doe... [21:15:30] 10Continuous-Integration-Infrastructure (Zuul upgrade), 13Patch-For-Review: Update ZuulDevOpsBot app credentials to include admin role access - https://phabricator.wikimedia.org/T399780#11015315 (10bd808) > [] Tell @bd808 and @jnuche to update their local dev envvars for the new requirements Hey @jnuche, upda... [21:15:42] 10Continuous-Integration-Infrastructure (Zuul upgrade): Update ZuulDevOpsBot app credentials to include admin role access - https://phabricator.wikimedia.org/T399780#11015317 (10bd808) [21:15:47] 10Continuous-Integration-Infrastructure (Zuul upgrade): Update ZuulDevOpsBot app credentials to include admin role access - https://phabricator.wikimedia.org/T399780#11015319 (10bd808) 05Openβ†’03Resolved [21:38:13] 10Continuous-Integration-Infrastructure (Zuul upgrade), 13Patch-For-Review: Provision Kubernetes cluster and bastion using OpenTofu and Magnum - https://phabricator.wikimedia.org/T396936#11015359 (10bd808) [23:25:31] (03open) 10bd808: kubernetes: use calico network driver [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/32 (https://phabricator.wikimedia.org/T399768 https://phabricator.wikimedia.org/T399882) [23:29:57] (03update) 10bd808: kubernetes: use calico network driver [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/32 (https://phabricator.wikimedia.org/T399768 https://phabricator.wikimedia.org/T399882) [23:30:07] (03merge) 10bd808: kubernetes: use calico network driver [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/32 (https://phabricator.wikimedia.org/T399768 https://phabricator.wikimedia.org/T399882) [23:37:53] 10Continuous-Integration-Infrastructure (Zuul upgrade): Add helpers/instructions for ssh into a Kubernetes cluster node for debugging - https://phabricator.wikimedia.org/T399910 (10bd808) 03NEW [23:40:38] 10Continuous-Integration-Infrastructure (Zuul upgrade): Flannel networking broken in Magnum cluster because upstream containers are missing - https://phabricator.wikimedia.org/T399882#11015508 (10bd808) 05Openβ†’03In progress a:03bd808 [23:41:59] 10Continuous-Integration-Infrastructure (Zuul upgrade): Flannel networking broken in Magnum cluster because upstream containers are missing - https://phabricator.wikimedia.org/T399882#11015516 (10bd808) https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/jobs/565893 is rebuilding the cluster. I ex...