[09:58:51] FIRING: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2014 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [10:03:51] RESOLVED: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2014 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [11:42:21] 10netops, 06Infrastructure-Foundations, 06SRE: Rancid network backups not being synced to git properly - https://phabricator.wikimedia.org/T409217#11344196 (10cmooney) Yup that seemed to fix it: ` cmooney@netmon1003:/var/log/rancid$ tail -f core.20251105.112816 starting: Wed Nov 5 11:28:16 AM UTC 2025... [11:42:25] 10netops, 06Infrastructure-Foundations, 06SRE: Rancid network backups not being synced to git properly - https://phabricator.wikimedia.org/T409217#11344199 (10cmooney) 05Open→03Resolved [13:14:51] 10netops, 06Infrastructure-Foundations, 06SRE: Nokia SR-Linux ARP resolution bug on v24.10.x+ - https://phabricator.wikimedia.org/T409178#11344450 (10cmooney) [13:27:23] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Connect spare Nokia D2L switches to Spines in eqiad - https://phabricator.wikimedia.org/T409288 (10cmooney) 03NEW p:05Triage→03Low [13:27:37] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Connect spare Nokia D2L switches to Spines in eqiad - https://phabricator.wikimedia.org/T409288#11344516 (10cmooney) [13:45:14] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Connect spare Nokia D2L switches to Spines in eqiad - https://phabricator.wikimedia.org/T409288#11344589 (10Jclark-ctr) a:03Jclark-ctr [13:48:55] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Connect spare Nokia D2L switches to Spines in eqiad - https://phabricator.wikimedia.org/T409288#11344599 (10cmooney) [14:26:52] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Connect spare Nokia D2L switches to Spines in eqiad - https://phabricator.wikimedia.org/T409288#11344803 (10Jclark-ctr) @cmooney lswtest1 Racked / cabled /. Netbox has been updated with Temp cableid https://netbox.wikimedia.org/dcim... [15:17:22] 10netops, 06cloud-services-team, 06Infrastructure-Foundations, 10Toolforge: Create new VRF and networks for Toolforge-on-Metal - https://phabricator.wikimedia.org/T409309 (10taavi) 03NEW [15:19:37] 10netops, 06cloud-services-team, 06Infrastructure-Foundations, 10Toolforge: Create new VRF and networks for Toolforge-on-Metal - https://phabricator.wikimedia.org/T409309#11345168 (10taavi) [15:19:38] 10netops, 06Infrastructure-Foundations, 10Toolforge, 06tools-infrastructure-team: Plan networking for Toolforge-on-Metal experiment - https://phabricator.wikimedia.org/T407140#11345169 (10taavi) [15:28:12] 10netops, 06Infrastructure-Foundations, 10Toolforge, 06tools-infrastructure-team: Plan networking for Toolforge-on-Metal experiment - https://phabricator.wikimedia.org/T407140#11345217 (10cmooney) Just to note down what we discussed on our meeting about this: # We will need option 2 ** To have connectivit... [15:29:52] 10netops, 06cloud-services-team, 06Infrastructure-Foundations, 10Toolforge: Create new VRF and networks for Toolforge-on-Metal - https://phabricator.wikimedia.org/T409309#11345226 (10cmooney) Thanks for the task @taavi. I'll need to discuss this with @ayounsi on his return, but in principal it makes sense... [15:47:40] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: Downgrade pfw1-codfw to Junos 23.4R2-S3 - https://phabricator.wikimedia.org/T393996#11345299 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=0cf41cdd-05b0-49da-ab43-1e9132f58a47) set by pt1979@cumin2002 for 2:00:00 on 1 host(... [15:50:02] 06Traffic: [Search Console Verification DNS Request] - wiktionary.org and wikibooks.org - https://phabricator.wikimedia.org/T409314 (10JKelsoteel-WMF) 03NEW [16:24:12] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Connect spare Nokia D2L switches to Spines in eqiad - https://phabricator.wikimedia.org/T409288#11345485 (10cmooney) Awesome @Jclark-ctr thank you! We can probably close this task for now I think, I can set up a new one for the actual... [16:24:52] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Connect spare Nokia D2L switches to Spines in eqiad - https://phabricator.wikimedia.org/T409288#11345488 (10Jclark-ctr) 05Open→03Resolved [16:43:36] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: Downgrade pfw1-codfw to Junos 23.4R2-S3 - https://phabricator.wikimedia.org/T393996#11345582 (10Papaul) 05Open→03Resolved Bother firewalls are not running Junos: 23.4R2-S5.5. Thanks to @Jgreen and @Dwisehaupt. Closing this task now [16:48:04] 10netops, 06Traffic, 06Infrastructure-Foundations: Upgrade End Of Support Junos - https://phabricator.wikimedia.org/T390813#11345599 (10Papaul) @ssingh @Vgutierrez planning on doing this on Nov 19th @10:am CT. Thank you [16:56:29] hello traffic friends o/ any concerns or conflicts if I deploy an haproxy config change to A:cp in a few minutes? (i.e., disable-puppet, test, rolling run-puppet-agent) [16:57:19] swfrench-wmf: no prob, let me know if you need help w/ testing [16:57:32] I'm an expert in forgetting the confd stuff :D [16:58:26] awesome, thanks fabfur! :) [18:12:30] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: Transport link saturation not alerting - https://phabricator.wikimedia.org/T409330 (10ssingh) 03NEW [18:12:32] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: Transport link saturation not alerting - https://phabricator.wikimedia.org/T409330#11345967 (10ssingh) p:05Triage→03High [18:18:00] 10netops, 06Traffic, 06Infrastructure-Foundations: Upgrade End Of Support Junos - https://phabricator.wikimedia.org/T390813#11345978 (10ssingh) >>! In T390813#11345599, @Papaul wrote: > @ssingh @Vgutierrez planning on doing this on Nov 19th @10:am CT. Thank you Thanks @Papaul, that works for us. [18:56:07] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: Transport link saturation not alerting - https://phabricator.wikimedia.org/T409330#11346176 (10cmooney) Thanks for the task @ssingh ! I agree this is definitely a major gap. In terms of the alertmanager rule you list it does make sense we should h... [19:11:38] 10Wikimedia-Apache-configuration, 06All-and-every-Wikisource: Move wikisource.org to www.wikisource.org - https://phabricator.wikimedia.org/T409335 (10Tgr) 03NEW [19:13:44] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: Transport link saturation not alerting - https://phabricator.wikimedia.org/T409330#11346272 (10ssingh) >>! In T409330#11346176, @cmooney wrote: > Thanks for the task @ssingh ! > > I agree this is definitely a major gap. In terms of the alertmanage... [19:25:30] 06Traffic, 10DNS: Request to create the donate.wikipedia25.org domain + 301 redirect to a donate.wiki page - https://phabricator.wikimedia.org/T408168#11346328 (10ssingh) >>! In T408168#11342081, @SCampos-WMF wrote: > Hi @BCornwall thank you for this. I have one last update request, apologies for the back... [20:52:53] 06Traffic, 10DNS: Request to create the donate.wikipedia25.org domain + 301 redirect to a donate.wiki page - https://phabricator.wikimedia.org/T408168#11346728 (10SCampos-WMF) Thank you so much @ssingh! Looks good, I appreciate your support with this task. [21:25:30] 06Traffic, 06collaboration-services, 06SRE, 13Patch-For-Review, 06Release-Engineering-Team (Radar): Deploy a TCP proxy across all DCs - https://phabricator.wikimedia.org/T408532#11346868 (10Dzahn) We are debugging why things work from SOME of the VMs but not from others..in this pattern: {P84954} [21:54:34] 06Traffic, 10Hiddenparma, 13Patch-For-Review: Introduce known-client identity objects and integrate with requestctl - https://phabricator.wikimedia.org/T403220#11347031 (10Scott_French) Alright, pilot round #2 worked as expected: `x-requestctl` logging was retained, even in the identification-enabled case. F... [22:25:29] 10netops, 06Infrastructure-Foundations, 06SRE: Netbox: Update server provision script to support Nokia switches - https://phabricator.wikimedia.org/T405637#11347169 (10cmooney) 05Open→03Resolved This one is complete, was not much to do.