[06:16:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance deployment-mwmaint03 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [06:16:40] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-mwmaint03 on project deployment-prep - https://phabricator.wikimedia.org/T407596 (10wmcs-alerts) 03NEW [06:46:00] 10Gerrit, 06collaboration-services, 13Patch-For-Review: gerrit2003 is trying to backup incrementally 3.5 million files every hour, clogging backus and filling in available disk space - https://phabricator.wikimedia.org/T406762#11284043 (10ABran-WMF) 05Open→03Resolved I think that can be closed now, f... [09:07:19] 10GitLab: Wikimedia GitLab SSH does not use post-quantum key exchange algorithm - https://phabricator.wikimedia.org/T407557#11284514 (10Lucas_Werkmeister_WMDE) >>! In T407557#11282964, @LucasWerkmeister wrote: > So potentially this affects not just GitLab but also production SSH… Sure does: ` $ ssh deployment.... [09:08:38] 10GitLab, 06SRE: OpenSSH 10.1+ warns that Wikimedia SSH does not use post-quantum key exchange algorithm - https://phabricator.wikimedia.org/T407557#11284517 (10Lucas_Werkmeister_WMDE) [09:10:54] 10GitLab, 06SRE: OpenSSH 10.1+ warns that Wikimedia SSH does not use post-quantum key exchange algorithm - https://phabricator.wikimedia.org/T407557#11284532 (10Lucas_Werkmeister_WMDE) >>! In T407557#11282964, @LucasWerkmeister wrote: > `lang=shell-session > me@host operations-puppet $ git grep curve25519-sha2... [09:13:50] 10GitLab, 06SRE: OpenSSH 10.1+ warns that Wikimedia SSH does not use post-quantum key exchange algorithm - https://phabricator.wikimedia.org/T407557#11284538 (10cmooney) p:05Triage→03Low I'm not really sure this is a massive issue right now. It's not clear to me that ssh sessions logs from now will be hug... [09:36:28] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance deployment-mwmaint03 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [10:10:36] 10GitLab, 06SRE: OpenSSH 10.1+ warns that Wikimedia SSH does not use post-quantum key exchange algorithm - https://phabricator.wikimedia.org/T407557#11284644 (10Ladsgroup) (Wrong Mortiz) [10:22:40] bd808: thank you for https://phabricator.wikimedia.org/T407403#11279689 , I can't believe I missed running the serialization with PHP 8.1 :] I was left wondering why I was unexpectedly not seeing any serialization differences between php versions hence why I eventually gave up. I guess I was tired / not focusing [10:22:56] and for sure the PHP 8.1 shows the `have_special_relative` property (which is what I was expecting [10:22:58] while 8.3 does not [10:23:05] and that is the source of the bug :] [10:23:07] kudos! [10:24:04] sometime I feel we should split the caches between php versions [10:24:39] (which I would cause other issues cause we would have a split brain) [10:25:34] 10GitLab, 06collaboration-services: OpenSSH 10.1+ warns that Wikimedia SSH does not use post-quantum key exchange algorithm - https://phabricator.wikimedia.org/T407557#11284684 (10Ladsgroup) Assuming gitlab ssh config needs changing, this needs attention of #collaboration-services team. Production seems to be... [10:29:51] 10GitLab, 06collaboration-services: OpenSSH 10.1+ warns that Wikimedia SSH does not use post-quantum key exchange algorithm - https://phabricator.wikimedia.org/T407557#11284688 (10Lucas_Werkmeister_WMDE) >>! In T407557#11284684, @Ladsgroup wrote: > Production seems to be handled already. No it isn’t, as shown... [11:21:51] 10GitLab, 06collaboration-services, 06Infrastructure-Foundations: OpenSSH 10.1+ warns that Wikimedia SSH does not use post-quantum key exchange algorithm - https://phabricator.wikimedia.org/T407557#11284841 (10LSobanski) [12:22:51] 10Diffusion, 10Phabricator (2025-10-07), 06Release-Engineering-Team (Doing 😎), 07Essential-Work: In the Diffusion "Clone" dialog, show gerrit URI instead of gerrit-replica URI - https://phabricator.wikimedia.org/T406553#11284997 (10Aklapper) [12:23:01] 10Diffusion, 10Phabricator, 06Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services, 13Patch-For-Review: Disable IO for diffusion repositories - https://phabricator.wikimedia.org/T405596#11284998 (10Aklapper) [12:23:50] 06Release-Engineering-Team (Doing 😎), 06cloud-services-team, 10Striker: Striker: Always expose observed external URI when creating Diffusion repositories - https://phabricator.wikimedia.org/T407624 (10Aklapper) 03NEW p:05Triage→03Low [14:26:42] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.45.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T405679#11285403 (10thcipriani) 05Open→03Resolved [14:46:59] 10Gerrit, 10GitLab, 06collaboration-services, 06Infrastructure-Foundations: OpenSSH 10.1+ warns that Wikimedia SSH does not use post-quantum key exchange algorithm - https://phabricator.wikimedia.org/T407557#11285495 (10Lucas_Werkmeister_WMDE) Also affects Gerrit: ` $ git fetch ** WARNING: connection is n... [16:21:44] 10Gerrit, 10GitLab, 06collaboration-services, 06Infrastructure-Foundations: OpenSSH 10.1+ warns that Wikimedia SSH does not use post-quantum key exchange algorithm - https://phabricator.wikimedia.org/T407557#11285825 (10jhathaway) deploy2002 is running bullseye, which has ssh 1:8.4p1-5+deb11u5, so it does... [16:32:07] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.45.0-wmf.24 deployment blockers - https://phabricator.wikimedia.org/T405680#11285845 (10Lucas_Werkmeister_WMDE) [16:36:52] 06Project-Admins, 10Tool-enderbot-dev: Workboard request for #tool-enderbot-dev - https://phabricator.wikimedia.org/T407251#11285871 (10Anderlaxe) 05Resolved→03Open yuhlo, please add the following columns after the "Backlog" column of #tool-enderbot-dev's workboard: - Needs R&D - Design review - Code revi... [16:42:16] 10Gerrit, 10GitLab, 06collaboration-services, 06Infrastructure-Foundations: OpenSSH 10.1+ warns that Wikimedia SSH does not use post-quantum key exchange algorithm - https://phabricator.wikimedia.org/T407557#11285886 (10Lucas_Werkmeister_WMDE) >>! In T407557#11285825, @jhathaway wrote: > gerrit1003 is also... [16:45:43] 06Project-Admins, 07Tracking-Neverending: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706#11285906 (10Anderlaxe) Hello, Could you please add me to #acl_project-admins (or other as appropriate) to allow me to create sprint milestones and projects fo... [17:37:08] 10Gerrit, 10GitLab, 06collaboration-services, 06Infrastructure-Foundations: OpenSSH 10.1+ warns that Wikimedia SSH does not use post-quantum key exchange algorithm - https://phabricator.wikimedia.org/T407557#11286112 (10Paladox) >>! In T407557#11285825, @jhathaway wrote: > deploy2002 is running bullseye, w... [18:17:56] (03PS1) 10Zabe: phan: Add Notifications as dependency for WikimediaMaintenance [integration/config] - 10https://gerrit.wikimedia.org/r/1196956 [18:21:58] (03PS3) 10Zabe: phan: Add Echo as dependency for WikimediaMaintenance [integration/config] - 10https://gerrit.wikimedia.org/r/1196956 [18:34:00] 10Gerrit: Bicho - https://phabricator.wikimedia.org/T407657 (10Izai187) 03NEW [18:36:35] 06Project-Admins, 10Tool-enderbot-dev: Workboard request for #tool-enderbot-dev - https://phabricator.wikimedia.org/T407251#11286375 (10A_smart_kitten) 05Open→03Resolved Sure :) I added them before the "In progress" column, but let me know if you'd prefer them to be moved around in any way. [18:47:37] 10Diffusion, 10Phabricator, 06Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services, 13Patch-For-Review: Disable IO for diffusion repositories - https://phabricator.wikimedia.org/T405596#11286410 (10Aklapper) **:** I disabled I/O for any non-observed, non-mirrored `http` clone UR... [18:56:49] 06Project-Admins, 10Tool-enderbot-dev: Workboard request for #tool-enderbot-dev - https://phabricator.wikimedia.org/T407251#11286441 (10Anderlaxe) >>! In T407251#11286375, @A_smart_kitten wrote: > Sure :) I added them before the "In progress" column, but let me know if you'd prefer them to be moved around... [20:08:23] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-mwmaint03 on project deployment-prep - https://phabricator.wikimedia.org/T407596#11286610 (10bd808) 05Open→03Invalid Already resolved by the time I manually checked the report: `lang=shell-session bd808@deployment-mwmaint03:~... [20:10:00] 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-webperf21 in project deployment-prep - https://phabricator.wikimedia.org/T407205#11286625 (10bd808) 05Open→03Invalid Already resolved by the time I manually checked the report: `lang=shell-session bd808@deployment-webperf... [20:16:16] 10Gerrit, 10GitLab, 06collaboration-services, 06Infrastructure-Foundations: OpenSSH 10.1+ warns that Wikimedia SSH does not use post-quantum key exchange algorithm - https://phabricator.wikimedia.org/T407557#11286637 (10jhathaway) ` debug1: Remote protocol version 2.0, remote software version GerritCodeRev... [22:04:26] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 13Patch-For-Review: puppetize setup of new zuul VMs - https://phabricator.wikimedia.org/T395938#11286775 (10Dzahn) We compared config of zookeeper in the staging environment with the zookeeper we have in production and but ran... [22:05:24] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 13Patch-For-Review: puppetize setup of new zuul VMs - https://phabricator.wikimedia.org/T395938#11286776 (10Dzahn) Given this, I am now tempted to reimage one of the VMs to trixie. [22:25:38] (03open) 10dduvall: image: Dynamically enforce published image repo [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/154 (https://phabricator.wikimedia.org/T407294) [22:28:45] (03update) 10dduvall: image: Dynamically enforce published image repo [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/154 (https://phabricator.wikimedia.org/T407294) [22:28:46] (03update) 10dduvall: image: Dynamically enforce published image repo [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/154 (https://phabricator.wikimedia.org/T407294) [22:28:49] (03update) 10dduvall: image: Dynamically enforce published image repo [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/154 (https://phabricator.wikimedia.org/T407294) [22:28:54] (03update) 10dduvall: image: Dynamically enforce published image repo [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/154 (https://phabricator.wikimedia.org/T407294) [22:28:58] (03update) 10dduvall: image: Dynamically enforce published image repo [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/154 (https://phabricator.wikimedia.org/T407294)