[00:00:07] <wikibugs>	 10Tool-Phabricator-bug-status: Adopt and move phabricator-bug-status to the Toolforge Jobs Framework - https://phabricator.wikimedia.org/T142237#11527257 (10bd808)
[08:17:19] <wikibugs>	 06cloud-services-team, 06collaboration-services, 06SRE Observability, 13Patch-For-Review: Move https://status.wikimedia.org/ away from rackspace - https://phabricator.wikimedia.org/T414098#11527810 (10Dzahn)
[08:49:35] <wikibugs>	 10VPS-project-Codesearch: Codesearch is down/unreachable (2026-01-16) - https://phabricator.wikimedia.org/T414776 (10A_smart_kitten) 03NEW
[09:17:07] <wmcs-alerts>	 FIRING: ToolsDBReplicationMissing: ToolsDB replication is not running - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBReplicationMissing
[09:17:07] <wmcs-alerts>	 FIRING: ToolsDBReplicationError: ToolsDB replication is broken - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBReplicationError
[09:19:28] <wikibugs>	 10VPS-project-Codesearch: Codesearch is down/unreachable (2026-01-16) - https://phabricator.wikimedia.org/T414776#11527915 (10Dzahn) Could not connect to instance. Rebooted it via Horizon.  Can see UI again.. Indices coming up:  https://codesearch.wmcloud.org/_health/
[09:21:33] <wikibugs>	 (03PS1) 10Akaza24: Fix request timeout issue by adding timeouts to Flask requests [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1227712 (https://phabricator.wikimedia.org/T414776)
[09:21:57] <wikibugs>	 (03PS2) 10Akaza24: Fix request timeout issue by adding timeouts to Flask requests [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1227712 (https://phabricator.wikimedia.org/T414776)
[09:27:07] <wikibugs>	 10VPS-project-Codesearch, 13Patch-For-Review: Codesearch is down/unreachable (2026-01-16) - https://phabricator.wikimedia.org/T414776#11527929 (10Dzahn) 05Open→03Resolved a:03Dzahn
[09:27:50] <wikibugs>	 10VPS-project-Codesearch, 13Patch-For-Review: Codesearch is down/unreachable (2026-01-16) - https://phabricator.wikimedia.org/T414776#11527936 (10A_smart_kitten) Thanks @Dzahn :D
[10:51:07] <wmcs-alerts>	 FIRING: ToolsDBReplicationLagIsTooHigh: ToolsDB replication is lagging - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBReplicationLagIsTooHigh
[10:51:28] <wikibugs>	 06cloud-services-team (FY2023/2024-Q1-Q2), 10Data-Services: [toolsdb] Replication stopped because of invalid event - https://phabricator.wikimedia.org/T351457#11528209 (10fnegri) One more today:  ` Jan 16 09:11:13 tools-db-7 mysqld[2328]: 2026-01-16  9:11:13 5389930 [ERROR] Read invalid event from master:...
[10:52:07] <wmcs-alerts>	 RESOLVED: ToolsDBReplicationError: ToolsDB replication is broken - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBReplicationError
[10:52:07] <wmcs-alerts>	 RESOLVED: ToolsDBReplicationMissing: ToolsDB replication is not running - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBReplicationMissing
[10:56:07] <wmcs-alerts>	 RESOLVED: ToolsDBReplicationLagIsTooHigh: ToolsDB replication is lagging - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBReplicationLagIsTooHigh
[11:53:52] <wikibugs>	 (03open) 10aghirelli: Add beta suffix to version format for linter [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/8
[11:54:15] <wikibugs>	 (03update) 10aghirelli: Add beta suffix to version format for linter [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/8
[11:54:21] <wikibugs>	 (03update) 10aghirelli: Add beta suffix to version format for linter [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/8
[11:54:35] <wikibugs>	 (03update) 10aghirelli: Add beta suffix to version format for linter [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/8
[11:56:43] <wikibugs>	 (03update) 10aghirelli: Add beta suffix to version format for linter [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/8
[12:09:58] <wikibugs>	 (03update) 10aghirelli: Add beta suffix to version format for linter [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/8
[14:08:36] <wikibugs>	 (03PS1) 10Dpogorzelski: ml_builder: add missing prod_build_password [labs/private] - 10https://gerrit.wikimedia.org/r/1227800
[14:09:43] <wikibugs>	 (03PS2) 10Dpogorzelski: ml_builder: add missing prod_build_password [labs/private] - 10https://gerrit.wikimedia.org/r/1227800
[14:19:07] <wikibugs>	 (03CR) 10Elukey: ml_builder: add missing prod_build_password (031 comment) [labs/private] - 10https://gerrit.wikimedia.org/r/1227800 (owner: 10Dpogorzelski)
[14:33:56] <wikibugs>	 (03CR) 10Elukey: "Once removed the extra settings feel free to merge!" [labs/private] - 10https://gerrit.wikimedia.org/r/1227800 (owner: 10Dpogorzelski)
[14:48:16] <wikibugs>	 10Tool-paulina, 10Outreachy (Round 31): Outreachy 31: Features to edit author and work data on Wikidata directly from Paulina - https://phabricator.wikimedia.org/T392429#11529120 (10Nurah_Wakili) Weekly Internship Report  Week 6: January 12 – January 16  Work Completed:    - Updated Wikidata edit comments to u...
[15:23:51] <wikibugs>	 (03PS3) 10Dpogorzelski: ml_builder: add missing prod_build_password [labs/private] - 10https://gerrit.wikimedia.org/r/1227800
[15:24:14] <wikibugs>	 (03CR) 10Dpogorzelski: [C:03+2] ml_builder: add missing prod_build_password (031 comment) [labs/private] - 10https://gerrit.wikimedia.org/r/1227800 (owner: 10Dpogorzelski)
[15:24:17] <wikibugs>	 (03CR) 10Dpogorzelski: [V:03+2 C:03+2] ml_builder: add missing prod_build_password [labs/private] - 10https://gerrit.wikimedia.org/r/1227800 (owner: 10Dpogorzelski)
[15:48:14] <wikibugs>	 (03PS2) 10Spinster: Correction gerrit > phabricator; various cosmetic updates [labs/tools/commons-recon-service] - 10https://gerrit.wikimedia.org/r/763952 (https://phabricator.wikimedia.org/T294196)
[16:23:36] <wikibugs>	 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge: [lima-kilo] SSH fails after stopping and starting the VM - https://phabricator.wikimedia.org/T414824 (10fnegri) 03NEW
[16:23:45] <wikibugs>	 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge: [lima-kilo] SSH fails after stopping and starting the VM - https://phabricator.wikimedia.org/T414824#11529612 (10fnegri) 05Open→03In progress
[16:27:07] <wikibugs>	 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge: [lima-kilo] SSH fails after stopping and starting the VM - https://phabricator.wikimedia.org/T414824#11529624 (10fnegri) p:05Triage→03Medium
[16:35:57] <wikibugs>	 (03update) 10fnegri: Don't chmod the home directory [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/304 (https://phabricator.wikimedia.org/T414824)
[16:38:39] <wikibugs>	 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge: [lima-kilo] SSH fails after stopping and starting the VM - https://phabricator.wikimedia.org/T414824#11529682 (10fnegri) > we change the permissions of the home directory from 755 in 775 in main/roles/userconfig/tasks/main.yaml  I created https://gitlab.w...
[16:38:57] <wikibugs>	 (03update) 10fnegri: Don't chmod the home directory [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/304 (https://phabricator.wikimedia.org/T414824)
[16:39:07] <wikibugs>	 (03update) 10fnegri: Don't chmod the home directory [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/304 (https://phabricator.wikimedia.org/T414824)
[16:59:02] <wikibugs>	 06cloud-services-team, 10Toolforge: resource consumption on issue toolforge tool wpcleaner - https://phabricator.wikimedia.org/T412081#11529758 (10fnegri) p:05Triage→03Medium
[17:00:39] <wikibugs>	 06cloud-services-team, 10Toolforge: Output of "toolforge jobs list" is wrong or incomplete - https://phabricator.wikimedia.org/T412081#11529773 (10fnegri)
[17:01:17] <wikibugs>	 06cloud-services-team, 10Toolforge: Output of "toolforge jobs list" is wrong or incomplete - https://phabricator.wikimedia.org/T412081#11529785 (10fnegri)
[17:02:36] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation), 10Toolforge: Upgrade Toolsbeta Harbor instance to Trixie to match tools - https://phabricator.wikimedia.org/T414475#11529789 (10fnegri) p:05Triage→03Medium
[17:02:47] <wikibugs>	 06cloud-services-team, 10Toolforge: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.34 - https://phabricator.wikimedia.org/T414595#11529790 (10fnegri) p:05Triage→03Medium
[17:02:54] <wikibugs>	 06cloud-services-team, 10Toolforge: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.35 - https://phabricator.wikimedia.org/T414596#11529791 (10fnegri) p:05Triage→03Medium
[17:04:34] <wikibugs>	 06cloud-services-team, 10Toolforge: Remove remaining uses of ingress-nginx specific annotations - https://phabricator.wikimedia.org/T414674#11529811 (10fnegri) p:05Triage→03High Triaging as High as the EOL date is soon:  > In March 2026, Ingress NGINX maintenance will be halted, and the project will be ret...
[17:08:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[17:35:36] <wikibugs>	 06cloud-services-team, 10Toolforge: Remove remaining uses of ingress-nginx specific annotations - https://phabricator.wikimedia.org/T414674#11529957 (10bd808) >>! In T414674#11524714, @taavi wrote: > The current list of tools relying on this functionality is at P87548. Of those, a couple are simple redirects t...
[17:40:25] <wikibugs>	 06cloud-services-team, 10Toolforge: Remove remaining uses of ingress-nginx specific annotations - https://phabricator.wikimedia.org/T414674#11529972 (10taavi) >>! In T414674#11529957, @bd808 wrote: > I will try to make a reverse proxy container in the same spirit as the redirect container. I can borrow from th...
[17:46:57] <wikibugs>	 (03approved) 10wikigit: Add beta suffix to version format for linter [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/8 (owner: 10aghirelli)
[17:47:17] <wikibugs>	 (03merge) 10wikigit: Add beta suffix to version format for linter [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/8 (owner: 10aghirelli)
[18:13:18] <wikibugs>	 (03Abandoned) 10Shivaansh Singh: Add one-click bug reporting system [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1193969 (https://phabricator.wikimedia.org/T288418) (owner: 10Shivaansh Singh)
[18:14:57] <wikibugs>	 06cloud-services-team, 10Toolforge: Remove remaining uses of ingress-nginx specific annotations - https://phabricator.wikimedia.org/T414674#11530067 (10bd808) >>! In T414674#11529972, @taavi wrote: >>>! In T414674#11529957, @bd808 wrote: >> I will try to make a reverse proxy container in the same spirit as the...
[18:29:59] <wikibugs>	 10Tool-containers: Create a reusable container to replace nginx ingress anonymizing reverse proxy setups - https://phabricator.wikimedia.org/T414836 (10bd808) 03NEW
[18:35:40] <wikibugs>	 10Tool-containers: Create a reusable container to replace nginx ingress anonymizing reverse proxy setups - https://phabricator.wikimedia.org/T414836#11530124 (10bd808) p:05Triage→03High a:03bd808
[19:45:48] <wikibugs>	 10Tool-curator: Omit heading in Location template and SDC when it is 0 - https://phabricator.wikimedia.org/T414438#11530310 (10DaxServer) 05Open→03Resolved
[21:08:56] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[21:48:41] <jinxer-wm>	 RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[22:22:39] <wikibugs>	 10Tool-containers: Create a reusable container to replace nginx ingress anonymizing reverse proxy setups - https://phabricator.wikimedia.org/T414836#11530695 (10bd808)
[22:22:47] <wikibugs>	 06cloud-services-team, 10Toolforge: Replace ingress-nginx before upstream EOL date - https://phabricator.wikimedia.org/T392356#11530696 (10bd808)
[22:34:03] <wikibugs>	 10Tool-centralnotice-banner-editor, 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10MediaWiki-extensions-CentralNotice, 0725 Dec Q3 Planning: Operationalize Outreachy banner editor project - https://phabricator.wikimedia.org/T412454#11530724 (10aude) Wikimedia NYC is interested in the outreachy tool...