[09:16:40] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Netbox: remove old cr2-codfw Switch Control Board inventory items - https://phabricator.wikimedia.org/T398940#10994586 (10ayounsi) We can remove them from Netbox if they're not in the device anymore. and add them to the spare tracking... [10:27:55] 10netops, 06Infrastructure-Foundations, 06SRE: Cloudcephosd: migrate to single network uplink - https://phabricator.wikimedia.org/T399180#10994796 (10cmooney) [13:00:23] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: Q3:test NIC for lvs1017 - https://phabricator.wikimedia.org/T387145#10995251 (10VRiley-WMF) I will look into this. I believe it may be due to lvs1017's nic being misconfigured. I will update it and test it out [13:21:01] 06Traffic, 13Patch-For-Review: Googlebot Commons 429 throttling - https://phabricator.wikimedia.org/T398668#10995384 (10Joe) p:05Triage→03Medium [13:34:40] FIRING: VarnishHighThreadCount: Varnish's thread count on cp5017:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/wiU3SdEWk/cache-host-drilldown?viewPanel=99&var-site=eqsin&var-instance=cp5017 - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [13:34:58] please no it's Friday! [13:44:40] RESOLVED: VarnishHighThreadCount: Varnish's thread count on cp5017:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/wiU3SdEWk/cache-host-drilldown?viewPanel=99&var-site=eqsin&var-instance=cp5017 - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [13:54:11] sukhe LOL [15:16:45] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Netbox: remove old cr2-codfw Switch Control Board inventory items - https://phabricator.wikimedia.org/T398940#10995790 (10RobH) >>! In T398940#10994586, @ayounsi wrote: > We can remove them from Netbox if they're not in the device anym... [15:30:14] 06Traffic, 13Patch-For-Review: provide x-analytics data for http frontend in haproxy - https://phabricator.wikimedia.org/T399167#10995842 (10Fabfur) a:03Fabfur [15:54:49] 06Traffic, 06DC-Ops, 10ops-codfw, 13Patch-For-Review: Q4:rack/setup/install cp20[43-58] codfw - https://phabricator.wikimedia.org/T392851#10995939 (10elukey) @Jhancock.wm I tried with 2045 since I wasn't able to log in on 2044, I get the same failures in provisioning: no nics reported. As Riccardo pointed... [15:58:59] purged lag firing expected as it is happening again (site depooled) [16:06:59] 06Traffic, 06DC-Ops, 10ops-codfw, 13Patch-For-Review: Q4:rack/setup/install cp20[43-58] codfw - https://phabricator.wikimedia.org/T392851#10996028 (10wiki_willy) Hi @elukey - can you or @Volans send me an email summarizing everything you need from Dell? I'll add the Technical Account Rep to the email thre... [16:07:36] 06Traffic, 06MediaWiki-Platform-Team, 06MW-Interfaces-Team, 06serviceops, and 3 others: API Rate Limiting Architecture - https://phabricator.wikimedia.org/T399291#10996031 (10daniel) >>! In T399291#10995626, @Aklapper wrote: >>>! In T399291#10994879, @daniel wrote: >> Tagging #traffic and #serviceops for v... [17:15:42] 06Traffic: eqsin purged consumers lag - https://phabricator.wikimedia.org/T399221#10996289 (10cmooney) a:03cmooney So I did some more testing this evening to try and substantiate the exact problem and get some stats to go to the carrier with. We depooled eqsin in order to do this, and un-drained the Arelion t... [21:49:03] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw: eqsin purged consumers lag - https://phabricator.wikimedia.org/T399221#10996713 (10cmooney) Arelion came back to say they did move a path but that they see CRC errors inbound from us in codfw: ` 2025-07-11 19:48 Hello Team, We ha... [21:49:14] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw: eqsin purged consumers lag - https://phabricator.wikimedia.org/T399221#10996717 (10cmooney) p:05Triage→03High