[00:04:07] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO:Switch refresh diagram - https://phabricator.wikimedia.org/T408511#11407956 (10Papaul) [00:09:23] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO:Switch refresh diagram - https://phabricator.wikimedia.org/T408511#11407960 (10Papaul) @RobH I update the task description with all the connections that we need for phase 1 in December. Please don't forget the Cable ID's. Please... [07:55:39] 06Traffic, 10Wikimedia-Apache-configuration, 06Abstract Wikipedia team, 10MediaWiki-Action-API, and 3 others: wikifunctions.org API no longer works via that URL (without 'www.') - https://phabricator.wikimedia.org/T411066#11408330 (10Reedy) [08:51:35] 06Traffic, 10Wikimedia-Apache-configuration, 06Abstract Wikipedia team, 10MediaWiki-Action-API, and 3 others: wikifunctions.org API no longer works via that URL (without 'www.') - https://phabricator.wikimedia.org/T411066#11408456 (10aaron) Maybe related to https://gerrit.wikimedia.org/r/c/operations/puppe... [08:52:27] 06Traffic, 10Wikimedia-Apache-configuration, 06Abstract Wikipedia team, 10MediaWiki-Action-API, and 3 others: wikifunctions.org API no longer works via that URL (without 'www.') - https://phabricator.wikimedia.org/T411066#11408459 (10aaron) @Clement_Goubert and @hnowlan would know more. [08:52:43] FIRING: HaproxyKafkaNoMessages: Unexpected rate of produced HaproxyKafka messages by cp7001 - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaNoMessages - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=magru&var-instance=cp7001 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaNoMessages [09:02:43] RESOLVED: [2x] HaproxyKafkaNoMessages: Unexpected rate of produced HaproxyKafka messages by cp7001 - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaNoMessages - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=magru&var-instance=cp7001 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaNoMessages [10:23:53] 10netops, 06Infrastructure-Foundations, 10Toolforge, 06tools-infrastructure-team: Plan networking for Toolforge-on-Metal experiment - https://phabricator.wikimedia.org/T407140#11408747 (10taavi) [10:53:49] 06Traffic, 06serviceops, 06MediaWiki-Platform-Team (Kanban Board), 07OKR-Work: api-gateway helm chart: rest routes should return retry-after when a rate limit applies. - https://phabricator.wikimedia.org/T405636#11408840 (10daniel) a:03Hokwelum [11:28:32] 06Traffic, 10Wikimedia-Apache-configuration, 06Abstract Wikipedia team, 10MediaWiki-Action-API, and 3 others: wikifunctions.org API no longer works via that URL (without 'www.') - https://phabricator.wikimedia.org/T411066#11408959 (10Clement_Goubert) >>! In T411066#11408456, @aaron wrote: > Maybe related t... [11:29:19] 06Traffic, 10Wikimedia-Apache-configuration, 06Abstract Wikipedia team, 10MediaWiki-Action-API, and 4 others: wikifunctions.org API no longer works via that URL (without 'www.') - https://phabricator.wikimedia.org/T411066#11408962 (10Clement_Goubert) p:05Triage→03High a:03Clement_Goubert [12:26:07] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, 10Thumbor: FY 25/26 WE 5.4.7 Standardize thumbnail sizes - https://phabricator.wikimedia.org/T408062#11409175 (10Vgutierrez) We are now rate-limiting non thumbnail steps requests for cache misses when certain X-Is-Browser thresholds are met [12:35:31] 06Traffic, 10Wikimedia-Apache-configuration, 06Abstract Wikipedia team, 10MediaWiki-Action-API, and 4 others: wikifunctions.org API no longer works via that URL (without 'www.') - https://phabricator.wikimedia.org/T411066#11409199 (10Clement_Goubert) 05Open→03Resolved Deployed and tested quickly, l... [14:17:51] 06Traffic, 06SRE: Deprecate low-traffic proxoid service and O:hcaptcha_proxy for the older hcaptcha proxy setup - https://phabricator.wikimedia.org/T411097 (10ssingh) 03NEW [14:24:56] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Move sretest1006 to rack D8 and connect to lswtest-d8-eqiad - https://phabricator.wikimedia.org/T411098 (10cmooney) 03NEW p:05Triage→03Medium [14:25:09] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Move sretest1006 to rack D8 and connect to lswtest-d8-eqiad - https://phabricator.wikimedia.org/T411098#11409583 (10cmooney) [14:38:07] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Move sretest1006 to rack D8 and connect to lswtest-d8-eqiad - https://phabricator.wikimedia.org/T411098#11409646 (10Jclark-ctr) a:03Jclark-ctr Relocated sretest1006 to D8 U37. Connected to lswtest-d8-eqiad Port 1 [15:40:39] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: rows C/D Upgrade Tracking - https://phabricator.wikimedia.org/T404609#11409838 (10fnegri) @Jclark-ctr `clouddb10[17-20]` are now depooled, but not downtimed. Can you please downtime them yourself when you migrate them? Otherwise... [15:48:56] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: rows C/D Upgrade Tracking - https://phabricator.wikimedia.org/T404609#11409855 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=80e83414-993e-4a63-b612-9625174481c7) set by fnegri@cumin1003 for 2:00:00 on 4 ho... [17:35:54] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: rows C/D Upgrade Tracking - https://phabricator.wikimedia.org/T404609#11410416 (10RobH) Day 11 Update: * 8 hosts moved, 5 remain out of 308 total hosts. * John did all the moves today working with Andrew. * Migrated 6 of the 8 W... [17:36:23] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: rows C/D Upgrade Tracking - https://phabricator.wikimedia.org/T404609#11410427 (10RobH) [18:37:25] 06Traffic, 06SRE: Deprecate low-traffic proxoid service and O:hcaptcha_proxy for the older hcaptcha proxy setup - https://phabricator.wikimedia.org/T411097#11410637 (10Raine) [18:48:30] 06Traffic, 06SRE: Deprecate low-traffic proxoid service and O:hcaptcha_proxy for the older hcaptcha proxy setup - https://phabricator.wikimedia.org/T411097#11410676 (10Raine) [18:53:04] 06Traffic, 10DNS, 06serviceops, 06SRE, 07Language codes: Redirect legacy language codes for Toki Pona to tok.wikipedia.org - https://phabricator.wikimedia.org/T404507#11410708 (10Tamzin) This technically wasn't stalled, but there wasn't much reason to get around to it till now, so, noting that T404457 ha... [19:21:15] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Eqiad C/D refresh: 2 x test hosts for config validation - https://phabricator.wikimedia.org/T405560#11410830 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin1003 for host sretest1006.eqiad.wm... [19:54:08] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Eqiad C/D refresh: 2 x test hosts for config validation - https://phabricator.wikimedia.org/T405560#11410946 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmooney@cumin1003 for host sretest1006.eqiad.wmnet... [19:56:33] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Eqiad C/D refresh: 2 x test hosts for config validation - https://phabricator.wikimedia.org/T405560#11410949 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin1003 for host sretest1006.eqiad.wm... [20:37:57] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Eqiad C/D refresh: 2 x test hosts for config validation - https://phabricator.wikimedia.org/T405560#11411029 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmooney@cumin1003 for host sretest1006.eqiad.wmnet... [23:58:01] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Remove lvs1018 L2 link to ssw1-e1-eqiad - https://phabricator.wikimedia.org/T405499#11411590 (10VRiley-WMF) Hey @cmooney It has been reused for that purpose, however it's still being worked on to update the connection in netbox [23:59:37] 06Traffic: Cover where/how to report in "If you report this error to the Wikimedia System Administrators, please include the details below." - https://phabricator.wikimedia.org/T410202#11411591 (10Aklapper)