[00:04:15] 06cloud-services-team, 06Wikimedia Enterprise, 10Wikimedia Enterprise Volunteer Request: Toolforge no longer has IP-based access to Wikimedia Enterprise - https://phabricator.wikimedia.org/T410994#11407958 (10bd808) I wondered if this was possibly related to the Toolforge bastions getting IPv6, but that does... [00:18:07] 10VPS-project-Codesearch, 06collaboration-services: "error: No space left on device" for codesearch9:/srv - https://phabricator.wikimedia.org/T411047#11407980 (10Dzahn) The command once found by bd808 and used by legoktm: ` ran `mkfs.ext4 -T news /dev/mapper/vd-second--local--disk` (h/t b... [00:59:30] 10VPS-project-Codesearch, 06collaboration-services: "error: No space left on device" for codesearch9:/srv - https://phabricator.wikimedia.org/T411047#11408053 (10Dzahn) We talked about how to determine if the usage type "news" (aka "small") was already set because the system has likely been recreated since the... [01:09:14] 10VPS-project-Codesearch, 06collaboration-services: "error: No space left on device" for codesearch9:/srv - https://phabricator.wikimedia.org/T411047#11408054 (10Dzahn) It does not seem possible though to run tune2fs and modify this on an existing file system / no way to add more inodes to existing ext4 fs. S... [04:15:42] 06cloud-services-team, 10Cloud-VPS: [tofu-infra] "tofu plan" failing in codfw - https://phabricator.wikimedia.org/T410265#11408167 (10Andrew) Tests suggest that the 'NoneType' error happens 100% of the time when cloudcontrol2005-dev is the radosgw backend, and 0% of the time with either other cloudcontrol as t... [07:31:04] (03open) 10marostegui: task_template*: Remove references to groups [toolforge-repos/switchmaster] - 10https://gitlab.wikimedia.org/toolforge-repos/switchmaster/-/merge_requests/16 (https://phabricator.wikimedia.org/T408663) [09:14:35] (03open) 10volans: tools/toolsbeta: add CNAME for infra-tracing-loki [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/103 (https://phabricator.wikimedia.org/T399313) [09:22:03] (03approved) 10fnegri: tools/toolsbeta: add CNAME for infra-tracing-loki [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/103 (https://phabricator.wikimedia.org/T399313) (owner: 10volans) [09:41:47] 06cloud-services-team, 10Quarry, 07CSS, 13Patch-For-Review: Improve maintenance message CSS - https://phabricator.wikimedia.org/T343644#11408607 (10Framawiki) Hey @taavi , pinging you as you are the last person that have updated Quarry. When you have a minute, if it could be possible to merge this small PR... [09:45:24] (03CR) 10D3r1ck01: [C:03+2] write_config: Add wikimedia/discovery/ repos to "MediaWiki & services at WMF" [labs/codesearch] - 10https://gerrit.wikimedia.org/r/828055 (owner: 10Chad) [09:46:23] (03Merged) 10jenkins-bot: write_config: Add wikimedia/discovery/ repos to "MediaWiki & services at WMF" [labs/codesearch] - 10https://gerrit.wikimedia.org/r/828055 (owner: 10Chad) [09:49:56] FIRING: SystemdUnitDown: The service unit disable-tool.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [09:54:10] 10VPS-project-Codesearch, 06collaboration-services: Stop pulling netbox-exported-dns repo from Phabricator Diffusion (which mirrors netbox-exports.wikimedia.org) - https://phabricator.wikimedia.org/T410624#11408668 (10Aklapper) 05Resolved→03Open Hmm, https://phabricator.wikimedia.org/diffusion/pulllog/?rep... [09:54:56] RESOLVED: SystemdUnitDown: The service unit disable-tool.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [10:00:50] (03update) 10fnegri: Update OpenStack provider [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/91 [10:05:07] (03approved) 10volans: Update OpenStack provider [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/91 (owner: 10fnegri) [10:05:26] (03merge) 10fnegri: Update OpenStack provider [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/91 [10:07:52] (03update) 10fnegri: tools/toolsbeta: add CNAME for infra-tracing-loki [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/103 (https://phabricator.wikimedia.org/T399313) (owner: 10volans) [10:22:24] (03merge) 10fnegri: projects: added project prove [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/283 (https://phabricator.wikimedia.org/T408387) (owner: 10group_199_bot_f98be072172e323ae6d1441939d3e461) [10:23:12] !log fnegri@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for main branch [10:23:32] 10Cloud-VPS, 06tools-infrastructure-team: Improve how virt networks are configured in cloudgw - https://phabricator.wikimedia.org/T411081 (10taavi) 03NEW [10:23:33] !log fnegri@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch [10:23:49] !log fnegri@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch (T408387) [10:23:49] 10Cloud-VPS, 06tools-infrastructure-team: Improve how virt networks are configured in cloudgw - https://phabricator.wikimedia.org/T411081#11408746 (10taavi) [10:23:53] 10Toolforge, 06tools-infrastructure-team, 06Infrastructure-Foundations, 10netops: Plan networking for Toolforge-on-Metal experiment - https://phabricator.wikimedia.org/T407140#11408747 (10taavi) [10:23:55] T408387: CloudVPS instance for ProVe - https://phabricator.wikimedia.org/T408387 [10:24:41] !log fnegri@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch (T408387) [10:25:25] 10Cloud-VPS, 06tools-infrastructure-team: Improve how virt networks are configured in cloudgw - https://phabricator.wikimedia.org/T411081#11408755 (10taavi) /cc @cmooney @fgiunchedi, just wanted to get your opinion on whether this looks reasonable or whether I'm missing something obvious here before I start wr... [10:32:12] 10Cloud-VPS, 06tools-infrastructure-team: Improve how virt networks are configured in cloudgw - https://phabricator.wikimedia.org/T411081#11408772 (10fgiunchedi) Looks good to me! Definitely good to refactor these bits [10:32:48] !log fnegri@cloudcumin1001 prove START - Cookbook wmcs.vps.create_project for project prove in eqiad1 (T408387) [10:32:51] T408387: CloudVPS instance for ProVe - https://phabricator.wikimedia.org/T408387 [10:33:49] !log fnegri@cloudcumin1001 prove END (PASS) - Cookbook wmcs.vps.create_project (exit_code=0) for project prove in eqiad1 (T408387) [10:39:27] 06cloud-services-team (FY2025/26-Q1-Q2), 10Cloud-VPS (Project-requests): CloudVPS instance for ProVe - https://phabricator.wikimedia.org/T408387#11408794 (10fnegri) 05In progress→03Resolved All done! [10:40:20] 06cloud-services-team (FY2025/26-Q1-Q2), 10Toolforge: [toolsdb] Automatically terminate long transactions - https://phabricator.wikimedia.org/T409857#11408801 (10fnegri) [10:41:16] 06cloud-services-team, 10Toolforge: [toolsdb] Automatically terminate long transactions - https://phabricator.wikimedia.org/T409857#11408803 (10fnegri) 05In progress→03Open [10:43:31] (03PS1) 10Muehlenhoff: Update secrets for tilerator->tegola rename [labs/private] - 10https://gerrit.wikimedia.org/r/1211625 (https://phabricator.wikimedia.org/T381565) [10:48:01] (03CR) 10Muehlenhoff: [V:03+2 C:03+2] Update secrets for tilerator->tegola rename [labs/private] - 10https://gerrit.wikimedia.org/r/1211625 (https://phabricator.wikimedia.org/T381565) (owner: 10Muehlenhoff) [10:59:26] !log fnegri@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for main branch [10:59:46] !log fnegri@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch [11:01:31] (03approved) 10ladsgroup: task_template*: Remove references to groups [toolforge-repos/switchmaster] - 10https://gitlab.wikimedia.org/toolforge-repos/switchmaster/-/merge_requests/16 (https://phabricator.wikimedia.org/T408663) (owner: 10marostegui) [11:01:39] (03merge) 10ladsgroup: task_template*: Remove references to groups [toolforge-repos/switchmaster] - 10https://gitlab.wikimedia.org/toolforge-repos/switchmaster/-/merge_requests/16 (https://phabricator.wikimedia.org/T408663) (owner: 10marostegui) [11:04:59] (03merge) 10volans: tools/toolsbeta: add CNAME for infra-tracing-loki [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/103 (https://phabricator.wikimedia.org/T399313) [11:50:10] 06cloud-services-team, 10Cloud-VPS: [tofu-infra] Remove vendored openstack provider binary - https://phabricator.wikimedia.org/T411087 (10fnegri) 03NEW [11:52:12] 06cloud-services-team, 10Cloud-VPS: [tofu-infra] Remove vendored openstack provider binary - https://phabricator.wikimedia.org/T411087#11409039 (10fnegri) [11:55:59] 06cloud-services-team, 10Cloud-VPS: [tofu-infra] Remove vendored openstack provider binary - https://phabricator.wikimedia.org/T411087#11409056 (10taavi) I'd prefer to keep the vendored copy. Otherwise we'll be relying on the uptime of both registry.opentofu.org and GitHub to keep operating our own cloud. [12:02:41] 06cloud-services-team, 10Cloud-VPS: Support nftables in Cloud VPS - https://phabricator.wikimedia.org/T411089 (10taavi) 03NEW [12:07:31] 06cloud-services-team, 10Cloud-VPS: [tofu-infra] [wmcs-cookbooks] Allow running tofu on a single cluster - https://phabricator.wikimedia.org/T411090 (10fnegri) 03NEW [12:07:47] 06cloud-services-team, 10Cloud-VPS, 07Epic: Cloud VPS: extend tofu-infra coverage - https://phabricator.wikimedia.org/T370037#11409111 (10fnegri) >>! In T370037#11305611, @fnegri wrote: >> deployments (codfw1dev and eqiad1) are currently coupled, so if tofu can't apply in codfw1dev then we also can't change... [13:11:09] 10Cloud-VPS, 06tools-infrastructure-team: Improve how virt networks are configured in cloudgw - https://phabricator.wikimedia.org/T411081#11409336 (10cmooney) @taavi broadly this looks good to me, nicely done. I wonder how binding interfaces to VRFs works here? Might it make sense to also have a key in //pro... [14:10:47] 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE, 06Data-Platform-SRE (2025.11.07 - 2025.11.28): eqiad row C/D cloud hosts pending migration - https://phabricator.wikimedia.org/T411025#11409537 (10Gehel) For the cloudelastic* nodes, it should be ok to unplug them for a few minutes. Ideally, we want to... [14:17:47] (03PS1) 10Muehlenhoff: Remove obsolete stub secrets [labs/private] - 10https://gerrit.wikimedia.org/r/1211684 (https://phabricator.wikimedia.org/T381565) [14:26:25] 06cloud-services-team, 10Cloud-VPS: [tofu-infra] Remove vendored openstack provider binary - https://phabricator.wikimedia.org/T411087#11409587 (10fnegri) > Otherwise we'll be relying on the uptime of both registry.opentofu.org and GitHub to keep operating our own cloud. Would we though? We would have a local... [14:37:27] 06cloud-services-team, 10Cloud-VPS: [tofu-infra] [wmcs-cookbooks] Allow running tofu on a single cluster - https://phabricator.wikimedia.org/T411090#11409643 (10fnegri) [14:37:28] 06cloud-services-team, 10Cloud-VPS, 07Epic: Cloud VPS: extend tofu-infra coverage - https://phabricator.wikimedia.org/T370037#11409644 (10fnegri) [14:39:31] 06cloud-services-team (FY2025/26-Q1-Q2), 10Cloud-VPS: tofu-infra: update to opentofu 1.8.x - https://phabricator.wikimedia.org/T376704#11409656 (10fnegri) 05Open→03Resolved a:03fnegri I upgraded tofu to the latest version 1.10.7 on all `cloudcontrol*` hosts: https://debmonitor.wikimedia.org/packages/... [14:48:59] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Support nftables in Cloud VPS - https://phabricator.wikimedia.org/T411089#11409690 (10taavi) [14:52:10] 06cloud-services-team, 10Cloud-VPS, 10VPS-Projects: [trove] Database quota values are not updating correctly - https://phabricator.wikimedia.org/T373348#11409700 (10fnegri) This is still happening: `lang=shell-session,lines=10 fnegri@cloudcontrol1007:~$ sudo OS_PROJECT_ID=tofuinfratest wmcs-openstack databa... [14:56:42] 06cloud-services-team, 10Cloud-VPS, 10VPS-Projects: [trove] Database quota values are not updating correctly - https://phabricator.wikimedia.org/T373348#11409706 (10fnegri) I deleted all the instances in the project with `openstack database instance delete {id}`, and reset the quotas with `update quota_usage... [16:07:18] 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE, 06Data-Platform-SRE (2025.11.07 - 2025.11.28): eqiad row C/D cloud hosts pending migration - https://phabricator.wikimedia.org/T411025#11409981 (10Andrew) [16:19:44] (03open) 10mormegil: Simplify Czech verb forms [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/23 [16:28:48] 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE, 06Data-Platform-SRE (2025.11.07 - 2025.11.28): eqiad row C/D cloud hosts pending migration - https://phabricator.wikimedia.org/T411025#11410092 (10fnegri) clouddb10[17-20] are depooled and downtimed, I mistakenly posted comments about those in the pare... [16:35:36] (03PS1) 10FNegri: tofu: Allow running apply on a single cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1211722 (https://phabricator.wikimedia.org/T411090) [16:36:07] 06cloud-services-team (FY2025/26-Q1-Q2), 10Cloud-VPS, 13Patch-For-Review: [tofu-infra] [wmcs-cookbooks] Allow running tofu on a single cluster - https://phabricator.wikimedia.org/T411090#11410120 (10fnegri) 05Open→03In progress a:03fnegri [16:36:28] 06cloud-services-team (FY2025/26-Q1-Q2), 10Cloud-VPS, 13Patch-For-Review: [tofu-infra] [wmcs-cookbooks] Allow running tofu on a single cluster - https://phabricator.wikimedia.org/T411090#11410125 (10fnegri) p:05Triage→03Medium [16:36:41] 06cloud-services-team (FY2025/26-Q1-Q2), 10Cloud-VPS, 13Patch-For-Review: [tofu-infra] [wmcs-cookbooks] Allow running "tofu apply" on a single cluster - https://phabricator.wikimedia.org/T411090#11410126 (10fnegri) [16:44:13] (03CR) 10Volans: [C:03+1] "Makes sense to me but I'll leave it to those with more context." [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1211722 (https://phabricator.wikimedia.org/T411090) (owner: 10FNegri) [16:47:18] 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE, 06Data-Platform-SRE (2025.11.07 - 2025.11.28): eqiad row C/D cloud hosts pending migration - https://phabricator.wikimedia.org/T411025#11410170 (10Andrew) [17:10:28] FIRING: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [17:11:30] FIRING: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [17:14:03] 06cloud-services-team (FY2025/26-Q1-Q2), 10Cloud-VPS, 10Toolforge: If the inactive clouddumps host goes down, it causes a ripple effect on Cloud VPS and Toolforge - https://phabricator.wikimedia.org/T391369#11410304 (10fnegri) > Hmm... would have been interesting to see where the containers were getting stuc... [17:21:30] RESOLVED: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [17:23:28] 06cloud-services-team (FY2025/26-Q1-Q2), 10Cloud-VPS, 10Toolforge: If the inactive clouddumps host goes down, it causes a ripple effect on Cloud VPS and Toolforge - https://phabricator.wikimedia.org/T391369#11410336 (10fnegri) clouddumps1002 is back and pending pods recovered automatically. [17:31:03] 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE, 06Data-Platform-SRE (2025.11.07 - 2025.11.28): eqiad row C/D cloud hosts pending migration - https://phabricator.wikimedia.org/T411025#11410395 (10fnegri) clouddb10[17-20] are now repooled and working fine. [17:52:58] RESOLVED: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [19:06:40] 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE, 06Data-Platform-SRE (2025.11.07 - 2025.11.28): eqiad row C/D cloud hosts pending migration - https://phabricator.wikimedia.org/T411025#11410757 (10Andrew) 05Open→03Resolved [19:13:07] 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE, 06Data-Platform-SRE (2025.11.07 - 2025.11.28): eqiad row C/D cloud hosts pending migration - https://phabricator.wikimedia.org/T411025#11410786 (10Jclark-ctr) a:05Andrew→03Jclark-ctr cloudelastic1009 clouddb1017 clouddumps1002 clouddb1018 cloud... [19:22:17] FIRING: JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [19:26:02] 06cloud-services-team (FY2025/26-Q1-Q2), 10Cloud-VPS (Project-requests): CloudVPS instance for ProVe - https://phabricator.wikimedia.org/T408387#11410855 (10komla) @Odinaldo this has been done. Thanks, @fnegri [19:37:27] 10Tool-schedule-deployment: schedule-deployment tool doesn't recognize mediawiki-vendor as a "MediaWiki repo" - https://phabricator.wikimedia.org/T411128 (10cscott) 03NEW [19:58:53] !log tools.cluebotng-review Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/19715882591 (https://github.com/cluebotng/component-configs/commits/18c2bc79b5f0023e682a9245197cf87c5cc76943) [19:58:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [20:12:17] RESOLVED: JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [20:14:17] FIRING: JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [20:17:32] RESOLVED: JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [21:04:35] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: [tofu-infra] "tofu plan" failing in codfw - https://phabricator.wikimedia.org/T410265#11411057 (10Andrew) >>! In T410265#11408167, @Andrew wrote: > Tests suggest that the 'NoneType' error happens 100% of the time when cloudcontrol2005-dev is the radosgw... [22:01:29] (03update) 10bd808: phorge: Use user.search Conduit endpoint [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/63 (https://phabricator.wikimedia.org/T402454) [22:23:53] (03merge) 10bd808: phorge: Use user.search Conduit endpoint [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/63 (https://phabricator.wikimedia.org/T402454) [22:51:27] (03open) 10bd808: ci(deploy:toolforge): Use HTTP/1.1 to work around protocol violation [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/64 [22:55:04] (03merge) 10bd808: ci(deploy:toolforge): Use HTTP/1.1 to work around protocol violation [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/64 [23:06:21] 10Wikibugs, 10Phabricator: Replace deprecated (frozen) Phabricator Conduit API calls with their stable equivalents - https://phabricator.wikimedia.org/T402454#11411471 (10bd808) [23:13:52] 10Wikibugs, 10Phabricator: Replace deprecated (frozen) Phabricator Conduit API calls with their stable equivalents - https://phabricator.wikimedia.org/T402454#11411490 (10bd808) 05In progress→03Open I got the easy one `{{done}}`, but the other two are going to be quite a bit more work. The application is u... [23:13:57] 10Wikibugs, 10Phabricator: Replace deprecated (frozen) Phabricator Conduit API calls with their stable equivalents - https://phabricator.wikimedia.org/T402454#11411493 (10bd808) a:05bd808→03None [23:17:44] 06cloud-services-team, 10Cloud-VPS: [tofu-infra] "tofu plan" failing in codfw - https://phabricator.wikimedia.org/T410265#11411496 (10Andrew) This is probably unrelated, but it /is/ a concern with Ceph and trixie (right now the ceph hosts themselves are running bookworm but the radosgw is on Trixie.) https://... [23:43:46] 10Wikibugs, 10Phabricator: Replace deprecated (frozen) Phabricator Conduit API calls with their stable equivalents - https://phabricator.wikimedia.org/T402454#11411549 (10Aklapper) Thanks! Already a little bit less noise in the logs :)