[06:59:49] 06Traffic, 13Patch-For-Review: Consider using a dedicated TLS certificate for upload.w.o - https://phabricator.wikimedia.org/T394484#10982074 (10Vgutierrez) @CDanis I just realized that measure-$site.wikimedia.org is using the upload cluster: ` wikimedia.org/operations/dns$ git grep measure- templates/wikimedi... [08:00:54] 06Traffic, 06SRE: Benthos - remove the kafka output module - https://phabricator.wikimedia.org/T398916 (10Fabfur) 03NEW [09:03:31] 06Traffic: Too Many Requests Error on certain useragents (QtWebEngine, outdated chromium based browsers, Safari 605) - https://phabricator.wikimedia.org/T398753#10982473 (10Joe) I hope you can all understand we can't be particularly helpful in our rejection messages for traffic that is both automated and excessi... [09:18:57] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Codfw: management down to racks D3 and D8 (switch port down) - https://phabricator.wikimedia.org/T398598#10982612 (10cmooney) 05Open→03Resolved >>! In T398598#10980766, @Jhancock.wm wrote: > reset the tripped breaker in D3. On... [09:31:42] 06Traffic, 06Experimentation Lab, 13Patch-For-Review: Block requests to /evt-103e/v2/events with no Edge Unique - https://phabricator.wikimedia.org/T398181#10982707 (10Vgutierrez) 05Open→03In progress [10:57:25] FIRING: SystemdUnitCrashLoop: varnish-frontend-fetcherr.service crashloop on cp3071:9100 - TODO - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitCrashLoop [11:02:25] RESOLVED: SystemdUnitCrashLoop: varnish-frontend-fetcherr.service crashloop on cp3071:9100 - TODO - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitCrashLoop [11:17:00] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Netbox: remove old cr2-codfw Switch Control Board inventory items - https://phabricator.wikimedia.org/T398940 (10cmooney) 03NEW p:05Triage→03Medium [13:03:09] FIRING: LVSHighRX: Excessive RX traffic on lvs5005:9100 (ens1f0np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs5005 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [13:08:09] RESOLVED: LVSHighRX: Excessive RX traffic on lvs5005:9100 (ens1f0np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs5005 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [13:14:41] :D [13:34:19] 06Traffic, 06SRE, 05FY2025-26 WE3.3 Engaging core audiences: [Reading Lists] Monitor potential performance impact of Reading Lists for Web - https://phabricator.wikimedia.org/T397526#10983824 (10Jdrewniak) [13:35:01] 06Traffic, 06SRE, 05FY2025-26 WE3.3 Engaging core audiences: [Reading Lists] Monitor potential performance impact of Reading Lists for Web - https://phabricator.wikimedia.org/T397526#10983830 (10Jdrewniak) [15:48:52] 06Traffic: Upgrade to haproxy 2.8.15 - https://phabricator.wikimedia.org/T398720#10984740 (10Vgutierrez) a:03SLyngshede-WMF [16:16:10] cdanis: I'm talking with ChrisDobbins901_ about the NetworkProbeLimit cookie [16:16:20] cdanis: which code uses its value client side? [16:16:32] I'm failing to find it on https://gitlab.wikimedia.org/repos/sre/probenet [16:16:55] https://gerrit.wikimedia.org/g/mediawiki/extensions/WikimediaEvents/+/e35e77a8054bbfbaf2225b1109cf7844bc4520a4/modules/ext.wikimediaEvents/networkProbe.js [16:17:01] it's part of WikimediaEvents extension [16:17:04] in Mediawiki [16:17:43] awesome, thanks :D [16:17:55] any changes to the client need a MW train or backport [16:26:38] 06Traffic: Upgrade to haproxy 2.8.15 - https://phabricator.wikimedia.org/T398720#10984974 (10Vgutierrez)