[07:48:13] 10Continuous-Integration-Infrastructure, 10MediaWiki-libs-Rdbms: Raise Quibble jobs' tested version of MariaDB to 10.6 - https://phabricator.wikimedia.org/T366646#10916851 (10Ladsgroup) I think this can be revisited again since the subticket is now resolved? [09:20:42] (03PS1) 10Bartosz Wójtowicz: inference-services: Remove unnecessary CI jobs from inference-services repository. [integration/config] - 10https://gerrit.wikimedia.org/r/1159377 (https://phabricator.wikimedia.org/T393865) [09:22:33] (03CR) 10CI reject: [V:04-1] inference-services: Remove unnecessary CI jobs from inference-services repository. [integration/config] - 10https://gerrit.wikimedia.org/r/1159377 (https://phabricator.wikimedia.org/T393865) (owner: 10Bartosz Wójtowicz) [09:24:37] (03PS2) 10Bartosz Wójtowicz: inference-services: Remove unnecessary CI jobs from inference-services repository. [integration/config] - 10https://gerrit.wikimedia.org/r/1159377 (https://phabricator.wikimedia.org/T393865) [09:26:35] (03CR) 10CI reject: [V:04-1] inference-services: Remove unnecessary CI jobs from inference-services repository. [integration/config] - 10https://gerrit.wikimedia.org/r/1159377 (https://phabricator.wikimedia.org/T393865) (owner: 10Bartosz Wójtowicz) [09:26:45] (03PS3) 10Bartosz Wójtowicz: inference-services: Remove unnecessary CI jobs from inference-services repo. [integration/config] - 10https://gerrit.wikimedia.org/r/1159377 (https://phabricator.wikimedia.org/T393865) [09:40:31] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Radar), 06collaboration-services: Investigate Zuul worker nodes for post-merge/trusted jobs - https://phabricator.wikimedia.org/T396655#10917526 (10hashar) > publish-to-doc -- Transfers documentation artifacts to `doc.wikimed... [09:41:21] (03open) 10jelto: rename elasticsearch to opensearch [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/115 [09:50:37] (03PS4) 10Hashar: inference-services: Remove unnecessary CI jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1159377 (https://phabricator.wikimedia.org/T393865) (owner: 10Bartosz Wójtowicz) [09:51:43] (03CR) 10Hashar: [C:03+1] "I have slightly tweaked the commit message and added the list of jobs being deleted (generated using `./utils/jjb-diff.sh`)." [integration/config] - 10https://gerrit.wikimedia.org/r/1159377 (https://phabricator.wikimedia.org/T393865) (owner: 10Bartosz Wójtowicz) [09:52:54] (03PS2) 10Hashar: zuul: Enable php-compile-php84 for Excimer repo [integration/config] - 10https://gerrit.wikimedia.org/r/1157104 (https://phabricator.wikimedia.org/T380748) (owner: 10Krinkle) [09:53:17] (03CR) 10Hashar: [C:03+2] "I have amended the commit message to link I581e6711433cd48c3c83fd624ecab5644b6695e7." [integration/config] - 10https://gerrit.wikimedia.org/r/1157104 (https://phabricator.wikimedia.org/T380748) (owner: 10Krinkle) [09:55:00] (03Merged) 10jenkins-bot: zuul: Enable php-compile-php84 for Excimer repo [integration/config] - 10https://gerrit.wikimedia.org/r/1157104 (https://phabricator.wikimedia.org/T380748) (owner: 10Krinkle) [10:02:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance deployment-puppetdb05 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [10:02:32] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-puppetdb05 on project deployment-prep - https://phabricator.wikimedia.org/T397012 (10wmcs-alerts) 03NEW [10:53:50] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 06serviceops, 06SRE, 06Traffic: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536#10917731 (10Clement_Goubert) [11:17:08] (03CR) 10Bartosz Wójtowicz: "Thank you for the review and upgrades to the commit message Antoine <3" [integration/config] - 10https://gerrit.wikimedia.org/r/1159377 (https://phabricator.wikimedia.org/T393865) (owner: 10Bartosz Wójtowicz) [11:17:28] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance deployment-puppetdb05 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [11:17:35] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-puppetdb05 on project deployment-prep - https://phabricator.wikimedia.org/T397012#10917795 (10wmcs-alerts) [11:39:49] (03CR) 10Hashar: [C:03+2] Support filtering root projects [integration/gerrit-admin] - 10https://gerrit.wikimedia.org/r/1155274 (owner: 10Hashar) [11:40:13] (03CR) 10Hashar: [C:03+2] Add emojis to differentiate mismatches [integration/gerrit-admin] - 10https://gerrit.wikimedia.org/r/1155264 (owner: 10Hashar) [11:40:41] (03Merged) 10jenkins-bot: Add emojis to differentiate mismatches [integration/gerrit-admin] - 10https://gerrit.wikimedia.org/r/1155264 (owner: 10Hashar) [11:40:43] (03Merged) 10jenkins-bot: Support filtering root projects [integration/gerrit-admin] - 10https://gerrit.wikimedia.org/r/1155274 (owner: 10Hashar) [11:53:45] 10Phabricator, 10Server-side-upload-request: Consider adding a global Herald rule to comment on #Server-side-upload-request tickets - https://phabricator.wikimedia.org/T397024 (10A_smart_kitten) 03NEW [11:55:10] 10Phabricator, 10Server-side-upload-request: Communicate in Phabricator that server-side upload tag is deprecated (project description; form 106) - https://phabricator.wikimedia.org/T391469#10917961 (10A_smart_kitten) >>! In T391469#10725916, @Aklapper wrote: > I updated https://phabricator.wikimedia.org/t... [12:07:30] (03PS1) 10Hashar: utils: docker-updates no more ask to apply no updates [integration/config] - 10https://gerrit.wikimedia.org/r/1159421 [12:11:46] 10Phabricator, 10Server-side-upload-request: Consider adding a global Herald rule to comment on #Server-side-upload-request tickets - https://phabricator.wikimedia.org/T397024#10917998 (10Aklapper) I admit I've also thought about this. The better approach however would be fixing the underlying issue and deploy... [12:18:54] (03CR) 10Jforrester: [C:03+2] utils: docker-updates no more ask to apply no updates [integration/config] - 10https://gerrit.wikimedia.org/r/1159421 (owner: 10Hashar) [12:19:03] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services, 06DBA: Prepare a database test for m3 - https://phabricator.wikimedia.org/T390034#10918025 (10Marostegui) >>! In T390034#10850978, @thcipriani wrote: >>>! In T390034#10820826, @Marostegui wrote: >> @brennen @Aklapper @Dzahn what is th... [12:20:57] (03Merged) 10jenkins-bot: utils: docker-updates no more ask to apply no updates [integration/config] - 10https://gerrit.wikimedia.org/r/1159421 (owner: 10Hashar) [12:44:27] Project beta-scap-sync-world build #211180: 04FAILURE in 2 min 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/211180/ [12:46:35] (03open) 10gkyziridis: machine-learning: Add ml-pipelines repo [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/116 (https://phabricator.wikimedia.org/T396495) [12:47:20] (03update) 10gkyziridis: machine-learning: Add ml-pipelines repo [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/116 (https://phabricator.wikimedia.org/T396495) [12:47:56] (03close) 10gkyziridis: machine-learning: Add ml-pipelines repo [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/116 (https://phabricator.wikimedia.org/T396495) [12:48:21] Yippee, build fixed! [12:48:21] Project beta-scap-sync-world build #211181: 09FIXED in 1 min 50 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/211181/ [12:49:12] (03open) 10gkyziridis: machine-learning: Add ml-pipelines repo [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/117 (https://phabricator.wikimedia.org/T396495) [12:54:16] (03PS1) 10Jforrester: Docker: [quibble-bullseye] Switch MariaDB to 10.6 Wikimedia package, again [integration/config] - 10https://gerrit.wikimedia.org/r/1159443 (https://phabricator.wikimedia.org/T366646) [12:59:22] 10Phabricator, 13Patch-For-Review: Deploy user style to reduce bot comments on Phabricator - https://phabricator.wikimedia.org/T393289#10918165 (10matmarex) @Aklapper What do I need to do next to get this deployed? [13:21:29] 10GitLab (Infrastructure), 10Ceph, 06collaboration-services, 10Data-Persistence-Backup, and 3 others: Migrate gitlab storage to apus (also: backups from S3?) - https://phabricator.wikimedia.org/T378922#10918326 (10Jelto) >>! In T378922#10848027, @jcrespo wrote: > I am working on setting up the dedicated gi... [13:39:46] 10GitLab (Infrastructure), 10Ceph, 06collaboration-services, 10Data-Persistence-Backup, and 2 others: Migrate gitlab storage to apus (also: backups from S3?) - https://phabricator.wikimedia.org/T378922#10918550 (10jcrespo) >>! In T378922#10918326, @Jelto wrote: > Thank you for the work on dedicated hardwar... [14:19:35] (03CR) 10Jforrester: [C:04-1] "cp not COPY" [integration/config] - 10https://gerrit.wikimedia.org/r/1159443 (https://phabricator.wikimedia.org/T366646) (owner: 10Jforrester) [14:28:53] (03PS2) 10Jforrester: Docker: [quibble-bullseye] Switch MariaDB to 10.6 Wikimedia package, again [integration/config] - 10https://gerrit.wikimedia.org/r/1159443 (https://phabricator.wikimedia.org/T366646) [14:29:21] (03CR) 10Jforrester: [C:03+2] Docker: [quibble-bullseye] Switch MariaDB to 10.6 Wikimedia package, again [integration/config] - 10https://gerrit.wikimedia.org/r/1159443 (https://phabricator.wikimedia.org/T366646) (owner: 10Jforrester) [14:31:44] (03Merged) 10jenkins-bot: Docker: [quibble-bullseye] Switch MariaDB to 10.6 Wikimedia package, again [integration/config] - 10https://gerrit.wikimedia.org/r/1159443 (https://phabricator.wikimedia.org/T366646) (owner: 10Jforrester) [14:31:59] !log Docker: [quibble-bullseye] Switch MariaDB to 10.6 Wikimedia package, again, for T366646 [14:32:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:32:02] T366646: Raise Quibble jobs' tested version of MariaDB to 10.6 - https://phabricator.wikimedia.org/T366646 [14:34:32] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure: Migrate all CI jobs from bookworm to trixie or later and drop bookworm testing support - https://phabricator.wikimedia.org/T397049 (10Jdforrester-WMF) 03NEW [14:34:48] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Migrate all CI jobs from bullseye to bookworm or later and drop bullseye testing support - https://phabricator.wikimedia.org/T383337#10918855 (10Jdforrester-WMF) [14:35:24] James_F: wanna do one build for mariadb + xdebug? [14:35:44] hashar: No, I'll do the mariadb one alone as it might have to be reverted again. :-( [14:36:03] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Migrate all CI jobs from bullseye to bookworm or later and drop bullseye testing support - https://phabricator.wikimedia.org/T383337#10918858 (10Jdforrester-WMF) [14:36:37] in which order do we do them? [14:36:44] MariaDB first. [14:36:48] +1 [14:36:49] As it's already landed. [14:36:51] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure: Migrate Quibble images from bookworm to trixie - https://phabricator.wikimedia.org/T397050 (10Jdforrester-WMF) 03NEW [14:36:57] I wll rebase the xdebug one on top of it [14:37:00] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure: Migrate Quibble images from bullseye to bookworm - https://phabricator.wikimedia.org/T362705#10918877 (10Jdforrester-WMF) [14:37:00] <3 [14:38:53] (03merge) 10dancy: rename elasticsearch to opensearch [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/115 (owner: 10jelto) [14:39:11] 10GitLab (Infrastructure), 10Ceph, 06collaboration-services, 10Data-Persistence-Backup, and 2 others: Migrate gitlab storage to apus (also: backups from S3?) - https://phabricator.wikimedia.org/T378922#10918887 (10Jelto) >>! In T378922#10918550, @jcrespo wrote: > > I'm sorry, but I thought that was an "ou... [14:39:38] codders: we will finally add Xdebug to the Quibble images [14:40:50] (03PS1) 10Jforrester: utils: Hint to docker-updates users the new --apply flag [integration/config] - 10https://gerrit.wikimedia.org/r/1159479 [14:40:50] (03PS1) 10Jforrester: jjb: Switch to Quibble images with MariaDB 10.6 [integration/config] - 10https://gerrit.wikimedia.org/r/1159480 (https://phabricator.wikimedia.org/T366646) [14:41:35] (03PS2) 10Hashar: Add xdebug back to Quibble PHP images [integration/config] - 10https://gerrit.wikimedia.org/r/1079250 (https://phabricator.wikimedia.org/T319495) (owner: 10Arthur taylor) [14:41:44] tadaa [14:42:00] (03CR) 10Hashar: [C:03+1] "Rebased :)" [integration/config] - 10https://gerrit.wikimedia.org/r/1079250 (https://phabricator.wikimedia.org/T319495) (owner: 10Arthur taylor) [14:43:05] 10Release-Engineering-Team (Priority Backlog 📥), 10MW-on-K8s, 06serviceops, 13Patch-For-Review: Provide an mwdebug functionality on kubernetes (mw-experimental) - https://phabricator.wikimedia.org/T276994#10918907 (10jijiki) [14:43:24] (03CR) 10CI reject: [V:04-1] Add xdebug back to Quibble PHP images [integration/config] - 10https://gerrit.wikimedia.org/r/1079250 (https://phabricator.wikimedia.org/T319495) (owner: 10Arthur taylor) [14:44:58] (03PS2) 10Jforrester: jjb: Switch to Quibble images with MariaDB 10.6 [integration/config] - 10https://gerrit.wikimedia.org/r/1159480 (https://phabricator.wikimedia.org/T366646) [14:44:58] (03PS1) 10Jforrester: utils: Add some --help [integration/config] - 10https://gerrit.wikimedia.org/r/1159481 [14:45:53] (03CR) 10Hashar: [C:03+2] utils: Add some --help [integration/config] - 10https://gerrit.wikimedia.org/r/1159481 (owner: 10Jforrester) [14:46:10] (03CR) 10Jforrester: [C:03+2] utils: Hint to docker-updates users the new --apply flag [integration/config] - 10https://gerrit.wikimedia.org/r/1159479 (owner: 10Jforrester) [14:46:11] (03CR) 10Hashar: [C:03+2] utils: Hint to docker-updates users the new --apply flag [integration/config] - 10https://gerrit.wikimedia.org/r/1159479 (owner: 10Jforrester) [14:46:24] I miss the old days of developing on small scripts :b [14:46:28] :-) [14:46:59] and I am happy we still have some without multiple layers of abstractions and remote calls over a rest api [14:47:25] (03Merged) 10jenkins-bot: utils: Hint to docker-updates users the new --apply flag [integration/config] - 10https://gerrit.wikimedia.org/r/1159479 (owner: 10Jforrester) [14:47:52] (03Merged) 10jenkins-bot: utils: Add some --help [integration/config] - 10https://gerrit.wikimedia.org/r/1159481 (owner: 10Jforrester) [14:48:03] Just wait. [14:48:41] (03approved) 10dancy: machine-learning: Add ml-pipelines repo [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/117 (https://phabricator.wikimedia.org/T396495) (owner: 10gkyziridis) [14:49:05] 10Phabricator, 13Patch-For-Review: Deploy user style to reduce bot comments on Phabricator - https://phabricator.wikimedia.org/T393289#10918942 (10Ottomata) Yes please! [14:49:13] scheduler-1 | 2025-06-16 14:48:29,405 DEBUG zuul.Pipeline.wikimedia.check: [e: 82ac1f9b13284f3aa53ef49f42b2e4f1] ] in check> is a failing item because ['it has an invalid configuration'] [14:49:14] bah [14:49:33] What's invalid about it? Helpful message. [14:50:09] the idea is Zuul reports back to Gerrit with the actual error [14:50:20] Alas. [14:50:25] but I haven't configured the reporter to avoid spamming the prod instance [14:50:27] of gerrit [14:52:17] Eurgh. [14:52:18] 00:04:17.934 FATAL ERROR: Could not find ./bin/my_print_defaults [14:52:34] Is this yet another mysql script that the mis-packaged WMF fork doesn't copy? [14:52:43] https://integration.wikimedia.org/ci/job/integration-quibble-fullrun-extensions-phpunit-php81/129/console [14:52:52] I guess so yeah [14:53:17] on contint.wikimedia.org I went to: [14:53:20] sudo docker run --user=root --rm -it --entrypoint=bash docker-registry.wikimedia.org/releng/quibble-bullseye-php81 [14:53:21] apt update [14:53:27] There's /opt/wmf-mariadb106/bin/my_print_defaults [14:54:12] apt purge mariadb-server-core-10.5 && apt install wmf-mariadb06 [14:54:16] then list files: [14:54:22] dpkg -L wmf-mariadb06|grep /bin/ [14:54:49] Do you have the results as a paste? [14:55:03] doing so [14:55:07] please hold [14:55:09] Thanks. [14:55:22] * hashar inserts some elevator music in the chat 🎶 [14:55:36] welcome to 1-800-CI_SYSTEM [14:55:38] please hold [14:55:40] an operator will answer [14:56:07] our customer service is opened from 6 UTC to 20 UTC from Monday to Friday [14:56:10] please hold [14:56:39] James_F: https://phabricator.wikimedia.org/P78061 [14:56:51] Do we use `mysql_config`? That's the only other script listed at https://dev.mysql.com/doc/refman/8.4/en/programs-development.html [14:57:05] that would be in quibble source code [14:57:32] Let's just add /opt/wmf-mariadb106/bin/ to the path. [14:57:35] it uses `mysqldump` as well [14:57:41] ahah [14:57:59] I don't know whether you can just add it to PATH [14:58:11] Hmm, yeah. [14:58:27] `ENV PATH "$PYENV_ROOT/shims:$PATH:$PYENV_ROOT/$TARGET_PYENV_VERSION/bin"` in python-all [14:58:38] So presumably that works, or you wouldn't have added it? [14:58:39] Right? :-) [14:58:48] well hmm [14:58:53] given I suggested COPY foo bar [14:58:57] instead of RUN cp foo bar [14:59:00] Indeed. [14:59:03] I can't say I am trusthworthy :b [15:00:36] Building to see. [15:05:40] 10GitLab (CI & Job Runners): Upgrade kube-prometheus-stack to 75.0.0 - https://phabricator.wikimedia.org/T397055 (10dancy) 03NEW [15:06:28] (03open) 10dancy: monitoring: Upgrade kube-prometheus-stack to 75.0.0 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/477 (https://phabricator.wikimedia.org/T397055) [15:06:30] (03update) 10dancy: monitoring: Upgrade kube-prometheus-stack to 75.0.0 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/477 (https://phabricator.wikimedia.org/T397055) [15:07:50] (03merge) 10dancy: monitoring: Upgrade kube-prometheus-stack to 75.0.0 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/477 (https://phabricator.wikimedia.org/T397055) [15:08:02] (03update) 10gkyziridis: machine-learning: Add ml-pipelines repo [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/117 (https://phabricator.wikimedia.org/T396495) [15:09:57] (03CR) 10Jforrester: Add xdebug back to Quibble PHP images (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1079250 (https://phabricator.wikimedia.org/T319495) (owner: 10Arthur taylor) [15:12:45] (03PS3) 10Jforrester: jjb: Switch to Quibble images with MariaDB 10.6 [integration/config] - 10https://gerrit.wikimedia.org/r/1159480 (https://phabricator.wikimedia.org/T366646) [15:12:45] (03PS1) 10Jforrester: Docker: [quibble-bullseye] Add the MariaDB binaries to our path [integration/config] - 10https://gerrit.wikimedia.org/r/1159501 (https://phabricator.wikimedia.org/T366646) [15:12:53] (03CR) 10Jforrester: [C:03+2] jjb: Switch to Quibble images with MariaDB 10.6 [integration/config] - 10https://gerrit.wikimedia.org/r/1159480 (https://phabricator.wikimedia.org/T366646) (owner: 10Jforrester) [15:12:58] (03CR) 10Jforrester: jjb: Switch to Quibble images with MariaDB 10.6 [integration/config] - 10https://gerrit.wikimedia.org/r/1159480 (https://phabricator.wikimedia.org/T366646) (owner: 10Jforrester) [15:13:02] (03CR) 10Jforrester: [C:03+2] Docker: [quibble-bullseye] Add the MariaDB binaries to our path [integration/config] - 10https://gerrit.wikimedia.org/r/1159501 (https://phabricator.wikimedia.org/T366646) (owner: 10Jforrester) [15:13:38] 10MediaWiki-Releasing, 05MW-1.44-notes, 05MW-1.44-release: Release MW 1.44.0-rc.0 - https://phabricator.wikimedia.org/T390697#10919085 (10Atieno) a:03Arlolra [15:15:14] (03Merged) 10jenkins-bot: Docker: [quibble-bullseye] Add the MariaDB binaries to our path [integration/config] - 10https://gerrit.wikimedia.org/r/1159501 (https://phabricator.wikimedia.org/T366646) (owner: 10Jforrester) [15:15:26] !log Docker: [quibble-bullseye] Add the MariaDB binaries to our path T366646 [15:15:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:15:29] T366646: Raise Quibble jobs' tested version of MariaDB to 10.6 - https://phabricator.wikimedia.org/T366646 [15:15:31] 10MediaWiki-Releasing, 05MW-1.44-notes, 05MW-1.44-release: Release MW 1.44.0-rc.0 - https://phabricator.wikimedia.org/T390697#10919118 (10Atieno) [15:15:58] (03merge) 10dancy: machine-learning: Add ml-pipelines repo [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/117 (https://phabricator.wikimedia.org/T396495) (owner: 10gkyziridis) [15:16:10] 10MediaWiki-Releasing, 05MW-1.44-notes, 05MW-1.44-release: Release MW 1.44.0-rc.0 - https://phabricator.wikimedia.org/T390697#10919123 (10Atieno) a:05Arlolra→03Atieno [15:16:32] 10MediaWiki-Releasing, 05MW-1.44-notes, 05MW-1.44-release, 10MW-Interfaces-Team (MWI-Sprint-11 (2025-06-03 to 2025-06-17)): Release MW 1.44.0-rc.0 - https://phabricator.wikimedia.org/T390697#10919125 (10Atieno) [15:25:11] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-puppetdb05 on project deployment-prep - https://phabricator.wikimedia.org/T397012#10919182 (10bd808) a:03dcaro Likely fixed by https://gerrit.wikimedia.org/r/c/operations/puppet/+/1159406 `lang=shell-session bd808@deployment-puppet... [15:27:50] (03open) 10dancy: prometheus-stack.yaml: enable CRDs upgradeJob [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/478 [15:27:53] (03update) 10dancy: prometheus-stack.yaml: enable CRDs upgradeJob [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/478 [15:28:56] (03merge) 10dancy: prometheus-stack.yaml: enable CRDs upgradeJob [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/478 [15:34:49] FFS. [15:35:04] Not enough because something called by backend.py hard-codes bin/my_print_defaults [15:38:27] (03PS1) 10Dreamrimmer: zuul: Add EggRoll97 to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1159506 [15:39:15] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 10Data-Persistence-Backup: Backups for new m1 database: zuul - https://phabricator.wikimedia.org/T396322#10919253 (10Jelto) p:05Triage→03Medium [15:40:44] 10Release-Engineering-Team (Priority Backlog 📥), 10MW-on-K8s, 06serviceops, 13Patch-For-Review: Provide an mwdebug functionality on kubernetes (mw-experimental) - https://phabricator.wikimedia.org/T276994#10919257 (10jijiki) [15:41:24] 10Release-Engineering-Team (Priority Backlog 📥), 10MW-on-K8s, 06serviceops, 13Patch-For-Review: Allow hostPath mounting of /srv/mediawiki in MediaWiki Helm chart - https://phabricator.wikimedia.org/T395284#10919260 (10jijiki) 05Open→03Resolved p:05Triage→03Medium [15:42:04] 10GitLab (CI & Job Runners): Upgrade kube-prometheus-stack to 75.0.0 - https://phabricator.wikimedia.org/T397055#10919274 (10dancy) 05Open→03Resolved [15:43:56] 10Gerrit, 10Release-Engineering-Team (Radar), 06collaboration-services, 06Traffic, 13Patch-For-Review: Separate Gerrit https and ssh/git hostnames - https://phabricator.wikimedia.org/T394271#10919284 (10Jelto) a:03Jelto [15:44:57] (03PS1) 10Jforrester: Docker: [quibble-bullseye] Copy the my_print_defaults script to main bin too [integration/config] - 10https://gerrit.wikimedia.org/r/1159509 (https://phabricator.wikimedia.org/T366646) [15:46:15] (03open) 10dancy: Release 4.175.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/838 [15:47:26] thanks for the manual assist on T395697 andre [15:47:27] T395697: Update StrikerBot Developer, SUL, and related accounts to email folks besides just bd808 - https://phabricator.wikimedia.org/T395697 [15:47:37] yw [15:48:42] (03merge) 10dancy: Release 4.175.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/838 [15:55:36] 10GitLab: Create an account for performing gitlab-trusted-runner configuration operations - https://phabricator.wikimedia.org/T397065 (10dancy) 03NEW [15:57:56] 10GitLab, 06collaboration-services: Create an account for performing gitlab-trusted-runner configuration operations - https://phabricator.wikimedia.org/T397065#10919388 (10Jelto) [16:02:57] 10GitLab, 06collaboration-services: Create an account for performing gitlab-trusted-runner configuration operations - https://phabricator.wikimedia.org/T397065#10919407 (10Jelto) Thanks for opening the task! A bit more context: I tried to replace my personal token with a project access token in https://gitlab... [16:06:49] (03PS3) 10Hashar: Add xdebug back to Quibble PHP images [integration/config] - 10https://gerrit.wikimedia.org/r/1079250 (https://phabricator.wikimedia.org/T319495) (owner: 10Arthur taylor) [16:07:58] (03CR) 10CI reject: [V:04-1] Add xdebug back to Quibble PHP images [integration/config] - 10https://gerrit.wikimedia.org/r/1079250 (https://phabricator.wikimedia.org/T319495) (owner: 10Arthur taylor) [16:14:44] 10Phabricator: MFA issue with Phabricator - https://phabricator.wikimedia.org/T397069 (10Mstyles) 03NEW [16:15:10] (03PS1) 10Hashar: Enable check pipeline to report back to Gerrit [integration/config] (zuul3) - 10https://gerrit.wikimedia.org/r/1159522 [16:17:45] 10Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.45.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T392176#10919481 (10brennen) [16:19:45] 10GitLab, 06collaboration-services: Create an account for performing gitlab-trusted-runner configuration operations - https://phabricator.wikimedia.org/T397065#10919488 (10dancy) The existing [[ https://gitlab.wikimedia.org/admin/users/configure-projects-bot | configure-projects-bots ]] looks like a good candi... [16:23:48] hashar: Eurgh. How about we just move quibble images to bookworm now? That comes with MariaDB 10.11… [16:24:25] 10GitLab: configure-projects bot impersonation token expires June 17, 2025 - https://phabricator.wikimedia.org/T397071 (10dancy) 03NEW [16:27:58] (03PS2) 10Jforrester: Docker: [quibble-bullseye] Revert MardiaDB to 10.5, Wikimedia package lacks all scripts [integration/config] - 10https://gerrit.wikimedia.org/r/1159509 (https://phabricator.wikimedia.org/T366646) [16:28:06] (03CR) 10Jforrester: "I'm giving up. :-(" [integration/config] - 10https://gerrit.wikimedia.org/r/1159509 (https://phabricator.wikimedia.org/T366646) (owner: 10Jforrester) [16:28:55] 10GitLab: configure-projects bot impersonation token expires June 17, 2025 - https://phabricator.wikimedia.org/T397071#10919566 (10dancy) I created a new non-expiring token named `configure-projects-bot-2025`. Contact me for the token. I will be using it to resolve T397065. [16:29:02] (03CR) 10CI reject: [V:04-1] Docker: [quibble-bullseye] Revert MardiaDB to 10.5, Wikimedia package lacks all scripts [integration/config] - 10https://gerrit.wikimedia.org/r/1159509 (https://phabricator.wikimedia.org/T366646) (owner: 10Jforrester) [16:29:31] (03PS1) 10Hashar: Have all projects added to the check pipeline [integration/config] (zuul3) - 10https://gerrit.wikimedia.org/r/1159529 [16:29:45] (03PS3) 10Jforrester: Docker: [quibble-bullseye] Revert MardiaDB to 10.5, Wikimedia package lacks [integration/config] - 10https://gerrit.wikimedia.org/r/1159509 (https://phabricator.wikimedia.org/T366646) [16:29:49] (03CR) 10Jforrester: [C:03+2] Docker: [quibble-bullseye] Revert MardiaDB to 10.5, Wikimedia package lacks [integration/config] - 10https://gerrit.wikimedia.org/r/1159509 (https://phabricator.wikimedia.org/T366646) (owner: 10Jforrester) [16:29:54] 10Release-Engineering-Team (Priority Backlog 📥), 10MW-on-K8s, 06serviceops, 13Patch-For-Review: Make mw-experimental production ready - https://phabricator.wikimedia.org/T396767#10919572 (10jijiki) [16:31:05] (03Merged) 10jenkins-bot: Docker: [quibble-bullseye] Revert MardiaDB to 10.5, Wikimedia package lacks [integration/config] - 10https://gerrit.wikimedia.org/r/1159509 (https://phabricator.wikimedia.org/T366646) (owner: 10Jforrester) [16:35:28] 10GitLab, 06collaboration-services: Create an account for performing gitlab-trusted-runner configuration operations - https://phabricator.wikimedia.org/T397065#10919604 (10dancy) 05Open→03Resolved In T397071 I created a fresh token for configure-projects-bots and used it for the `GITLAB_TOKEN` variable... [17:02:10] 10Release-Engineering-Team (Priority Backlog 📥), 10MW-on-K8s, 06serviceops, 13Patch-For-Review: Make mw-experimental production ready - https://phabricator.wikimedia.org/T396767#10919751 (10jijiki) [17:32:56] (03PS1) 10Hashar: Fix up log URL reported by upload-logs-s3 [integration/config] (zuul3) - 10https://gerrit.wikimedia.org/r/1159544 [17:42:28] (03update) 10thcipriani: Add "xlab" window to deployments calendar [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/183 (https://phabricator.wikimedia.org/T395025) (owner: 10phuedx) [17:45:10] 10Phabricator (Upstream), 10Release-Engineering-Team (Doing 😎), 07Upstream: Option to measure WIP limits based on card count instead of points, to more closely adhere to Kanban standards. - https://phabricator.wikimedia.org/T120042#10919927 (10Aklapper) I proposed an upstream patch in https://we.phorge.it/D2... [17:55:33] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06cloud-services-team, 10Cloud-VPS: Magnum created instances failing to talk to OpenStack user_data service - https://phabricator.wikimedia.org/T396935#10919962 (10bd808) 05Open→03Invalid This seems to have been a transient failure. Let's shake... [18:02:05] 10Continuous-Integration-Infrastructure (Zuul upgrade): Provision Kubernetes cluster and bastion using OpenTofu and Magnum - https://phabricator.wikimedia.org/T396936#10919979 (10bd808) I got to a new failure point. `lines=10 $ sudo wmcs-openstack stack resource list b899cc99-bec8-47d7-8eb5-b5f31027bfb4 +-------... [18:09:29] 10Phabricator: MFA issue with Phabricator - https://phabricator.wikimedia.org/T397069#10920008 (10Aklapper) Yeah, this is surprising and unfortunate behavior. I added a warning in upstream's https://we.phorge.it/D26028 Someone with CLI access should verify this request out of band (ideally via a video call) and... [18:09:43] 10Phabricator: Reset Phabricator MFA of Mstyles - https://phabricator.wikimedia.org/T397069#10920009 (10Aklapper) a:05Aklapper→03None [18:09:56] 10Phabricator: Reset Phabricator MFA of Mstyles - https://phabricator.wikimedia.org/T397069#10920012 (10Aklapper) (Not sure why this was assigned to me) [18:32:30] 10GitLab (Infrastructure), 10Ceph, 06collaboration-services, 10Data-Persistence-Backup, and 2 others: Migrate gitlab storage to apus (also: backups from S3?) - https://phabricator.wikimedia.org/T378922#10920067 (10jcrespo) Thanks, that's more insightful and helpful, I will give it a think and maybe talk to... [18:34:42] (03approved) 10thcipriani: Add "xlab" window to deployments calendar [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/183 (https://phabricator.wikimedia.org/T395025) (owner: 10phuedx) [18:34:49] (03merge) 10thcipriani: Add "xlab" window to deployments calendar [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/183 (https://phabricator.wikimedia.org/T395025) (owner: 10phuedx) [18:39:39] 10Phabricator: Reset Phabricator MFA of Mstyles - https://phabricator.wikimedia.org/T397069#10920082 (10Mstyles) I assigned it to you @Aklapper because it looked like you did this in other situations such as T331170. I'll follow up with my team to find out if anyone has CLI access [19:14:02] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [19:14:03] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [19:14:08] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [19:16:07] (03approved) 10dduvall: lib/image.py: Add wmf-built dockerfile-frontend as default [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/129 (owner: 10dancy) [19:52:51] 10Scap: scap backport should warn if it knows it will take a long time - https://phabricator.wikimedia.org/T397089 (10thcipriani) 03NEW [20:00:28] (03update) 10brennen: tox.ini: add flake8 settings [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/837 [20:03:21] 10Scap: scap backport should warn if it knows it will take a long time - https://phabricator.wikimedia.org/T397089#10920330 (10dancy) > But as soon as scap knows that we're going to do a full image rebuild, warning the users would be great. Do you mean writing a (likely to be overlooked) line of log output whil... [20:03:35] (03update) 10brennen: tox.ini: add flake8 settings [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/837 [20:05:35] (03approved) 10dancy: tox.ini: add flake8 settings [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/837 (owner: 10brennen) [20:07:08] (03merge) 10dancy: lib/image.py: Add wmf-built dockerfile-frontend as default [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/129 [20:10:12] (03PS4) 10Hashar: Add xdebug back to Quibble PHP images [integration/config] - 10https://gerrit.wikimedia.org/r/1079250 (https://phabricator.wikimedia.org/T319495) (owner: 10Arthur taylor) [20:11:18] (03CR) 10CI reject: [V:04-1] Add xdebug back to Quibble PHP images [integration/config] - 10https://gerrit.wikimedia.org/r/1079250 (https://phabricator.wikimedia.org/T319495) (owner: 10Arthur taylor) [20:11:31] 06Release-Engineering-Team, 10Scap (SpiderPig 🕸️): SpiderPig log view: Make table sortable by column, default to descending by count - https://phabricator.wikimedia.org/T397091 (10brennen) 03NEW [20:13:58] 06Release-Engineering-Team, 10Scap (SpiderPig 🕸️): SpiderPig log view: Display temporal info to the user - https://phabricator.wikimedia.org/T397092 (10brennen) 03NEW [20:16:35] 06Release-Engineering-Team, 10Scap (SpiderPig 🕸️): SpiderPig log view: Logstash links should apply expected filters - https://phabricator.wikimedia.org/T397093 (10brennen) 03NEW [20:19:09] 06Release-Engineering-Team, 10Scap (SpiderPig 🕸️), 10Phatality: SpiderPig log view: Integrate Phatality-style search & reporting features for errors - https://phabricator.wikimedia.org/T397094 (10brennen) 03NEW [20:21:07] 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10Scap (SpiderPig 🕸️), 10logspam-watch, 10observability, and 2 others: Add a log view to SpiderPig - https://phabricator.wikimedia.org/T391005#10920397 (10brennen) A handful of followups: - {T397091} - {T397092} - {T397093} - {T397094} [20:33:24] 10Phabricator, 10Release-Engineering-Team (Doing 😎): Vanishing request for User Snowyamur9889 - https://phabricator.wikimedia.org/T396053#10920482 (10Aklapper) 05Open→03Resolved a:03Aklapper I do not know what "vanish" exactly implies and which problem that solves given the technical constraints and... [20:37:05] 10Scap (SpiderPig 🕸️): SpiderPig live job log view is broken - https://phabricator.wikimedia.org/T397097 (10dancy) 03NEW [20:37:35] 10Scap (SpiderPig 🕸️): SpiderPig live job log view is broken - https://phabricator.wikimedia.org/T397097#10920498 (10dancy) [20:38:20] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Cloud-VPS (Quota-requests): Increase volume storge quota for zuul project - https://phabricator.wikimedia.org/T397098 (10bd808) 03NEW [20:38:43] 10Continuous-Integration-Infrastructure (Zuul upgrade): Provision Kubernetes cluster and bastion using OpenTofu and Magnum - https://phabricator.wikimedia.org/T396936#10920511 (10bd808) [20:38:44] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Cloud-VPS (Quota-requests): Increase volume storge quota for zuul project - https://phabricator.wikimedia.org/T397098#10920512 (10bd808) [20:42:31] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Cloud-VPS (Quota-requests): Increase volume storge quota for zuul project - https://phabricator.wikimedia.org/T397098#10920523 (10Andrew) +1 [20:44:24] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Cloud-VPS (Quota-requests): Increase volume storge quota for zuul project - https://phabricator.wikimedia.org/T397098#10920527 (10bd808) [20:44:54] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06cloud-services-team, 10Cloud-VPS (Quota-requests): Increase volume storge quota for zuul project - https://phabricator.wikimedia.org/T397098#10920529 (10bd808) [20:49:23] (03update) 10thcipriani: Update scap beta release scripts [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/827 [20:49:28] (03update) 10thcipriani: Update scap beta release scripts [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/827 [20:50:45] 10Release-Engineering-Team (Radar), 10MediaWiki-extensions-VipsScaler, 10Structured Data Engineering, 06Structured-Data-Backlog, and 3 others: Undeploy VipsScaler from Wikimedia wikis - https://phabricator.wikimedia.org/T290759#10920540 (10ABreault-WMF) [21:00:47] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [21:00:55] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [21:00:55] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [21:04:35] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [21:04:38] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [21:04:38] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [21:06:59] 10GitLab (Infrastructure), 10Phabricator, 06Release-Engineering-Team: Reset Phabricator MFA of Mstyles - https://phabricator.wikimedia.org/T397069#10920595 (10sbassett) [21:20:30] (03open) 10dduvall: Allow repos/releng/zuul/zuul to build from a Dockerfile [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/118 (https://phabricator.wikimedia.org/T396245) [21:26:41] (03update) 10dduvall: Allow repos/releng/zuul/zuul to build from a Dockerfile [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/118 (https://phabricator.wikimedia.org/T396245) [21:26:46] (03update) 10dduvall: Allow repos/releng/zuul/zuul to build from a Dockerfile [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/118 (https://phabricator.wikimedia.org/T396245) [21:35:48] 10Continuous-Integration-Infrastructure (Zuul upgrade): Provision Kubernetes cluster and bastion using OpenTofu and Magnum - https://phabricator.wikimedia.org/T396936#10920678 (10bd808) I got past the quota problem temporarily by adjusting the Magnum template. The 2 node test cluster //almost// provisioned this... [21:54:48] 10Continuous-Integration-Infrastructure (Zuul upgrade): OpenTofu erroring out when trying to fetch data to build kubeconfig - https://phabricator.wikimedia.org/T397106 (10bd808) 03NEW [22:05:25] 10GitLab (Infrastructure), 10Phabricator, 06Release-Engineering-Team: Reset Phabricator MFA of Mstyles - https://phabricator.wikimedia.org/T397069#10920756 (10Mstyles) 05Open→03Resolved a:03Mstyles Thank you so much @brennen! [22:06:14] (03approved) 10dancy: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 (owner: 10dduvall) [22:06:33] (03approved) 10thcipriani: Drop several php_fpm* config parameters [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/829 (https://phabricator.wikimedia.org/T396166) (owner: 10dancy) [22:07:08] 10Continuous-Integration-Infrastructure (Zuul upgrade): OpenTofu erroring out when trying to fetch data to build kubeconfig - https://phabricator.wikimedia.org/T397106#10920760 (10bd808) p:05Triage→03High Turning on more verbose logging with `OS_DEBUG=1` shows: `lang=go,COUNTEREXAMPLE,lines=10 2025-06-16T22:... [22:07:25] (03update) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [22:07:43] (03merge) 10dduvall: image.py: Support buildx bake based builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/125 [22:33:18] 10Continuous-Integration-Infrastructure (Zuul upgrade): OpenTofu erroring out when trying to fetch data to build kubeconfig - https://phabricator.wikimedia.org/T397106#10920818 (10bd808) 05Open→03In progress a:03bd808 [22:53:22] (03merge) 10brennen: tox.ini: add flake8 settings [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/837 [23:05:25] 10Continuous-Integration-Infrastructure (Zuul upgrade): OpenTofu erroring out when trying to fetch data to build kubeconfig - https://phabricator.wikimedia.org/T397106#10920881 (10bd808) Downgrading the registry.opentofu.org/terraform-provider-openstack/openstack plugin to the 1.51.1 version used in https://gitl... [23:13:44] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for eggroll97 - https://phabricator.wikimedia.org/T397110 (10EggRoll97) 03NEW [23:13:50] 10GitLab (Infrastructure), 10Phabricator, 06Release-Engineering-Team: Reset Phabricator MFA of Mstyles - https://phabricator.wikimedia.org/T397069#10920895 (10bd808) a:05Mstyles→03brennen [23:32:00] (03update) 10dduvall: release: Use a multi stage release process [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/130 [23:32:04] (03update) 10dduvall: release: Use a multi stage release process [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/130 [23:32:08] (03update) 10dduvall: release: Use a multi stage release process [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/130 [23:46:10] 10Continuous-Integration-Infrastructure (Zuul upgrade): OpenTofu erroring out when trying to fetch data to build kubeconfig - https://phabricator.wikimedia.org/T397106#10920941 (10bd808) >>! In T397106#10920881, @bd808 wrote: > I think the issue is somewhere in the gophercloud client library that the provider us...