[03:44:01] 10netops, 06Infrastructure-Foundations, 06SRE: Update esams network pop diagrams - https://phabricator.wikimedia.org/T368084#11671668 (10Papaul) I update the diagram [03:48:49] 10netops, 06Infrastructure-Foundations, 06SRE: Update esams network pop diagrams - https://phabricator.wikimedia.org/T368084#11671673 (10cmooney) Looks great, nice work! [04:39:39] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO: New switch configuration - https://phabricator.wikimedia.org/T408892#11671699 (10Papaul) @ayounsi prior of deleting the sandbox1-ulsfo range 198.35.26.240/28 I will have to delete the interfaces et-0/0/1.1221 on both routers. D... [04:54:14] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO: Update ULSFO LVS service IP's - https://phabricator.wikimedia.org/T418971 (10Papaul) 03NEW [05:01:16] 10netops, 06Infrastructure-Foundations, 10Prod-Kubernetes, 06ServiceOps new, 06SRE: Eqiad: lsw1-d7-eqiad BGP maintenance - https://phabricator.wikimedia.org/T418772#11671717 (10Papaul) @jcrespo @Marostegui @MatthewVernon can you please let us know if backup1007, dbprov1004 and ms-be1093 need depool befo... [06:31:18] gnmic 0.45.0 is out ! https://gnmic.openconfig.net/changelog/ [06:37:31] nice :) [08:25:44] 10netops, 06Infrastructure-Foundations, 10Prod-Kubernetes, 06ServiceOps new, 06SRE: Eqiad: lsw1-d7-eqiad BGP maintenance - https://phabricator.wikimedia.org/T418772#11671968 (10jcrespo) @papaul for backup1007, dbprov1004, while they are a production host with important content, a small network interrupti... [08:26:14] 10netops, 06Infrastructure-Foundations, 10Prod-Kubernetes, 06ServiceOps new, 06SRE: Eqiad: lsw1-d7-eqiad BGP maintenance - https://phabricator.wikimedia.org/T418772#11671969 (10jcrespo) [08:36:52] 10netops, 06Infrastructure-Foundations, 10Prod-Kubernetes, 06ServiceOps new, 06SRE: Eqiad: lsw1-d7-eqiad BGP maintenance - https://phabricator.wikimedia.org/T418772#11671989 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=cd8c8777-0916-4a5b-b6f5-55f2535990f4) set by jynus@cumin1003 fo... [08:37:05] 10netops, 06Infrastructure-Foundations, 10ops-magru, 06SRE: cr2-magru <-> asw1-b3-magru link down March 2026 - https://phabricator.wikimedia.org/T418978 (10cmooney) 03NEW p:05Triage→03High [08:38:15] 10netops, 06Infrastructure-Foundations, 10Prod-Kubernetes, 06ServiceOps new, 06SRE: Eqiad: lsw1-d7-eqiad BGP maintenance - https://phabricator.wikimedia.org/T418772#11672002 (10jcrespo) [09:20:19] 10netops, 06Infrastructure-Foundations, 06SRE: Nokia SR-Linux DHCP Relay Bug - https://phabricator.wikimedia.org/T411054#11672136 (10cmooney) @ayounsi thanks for following up on this. I've done some testing to see if there may be a better way to force a tunnel teardown/re-establishment today. The reason cl... [09:34:28] 10netops, 06Infrastructure-Foundations, 10Prod-Kubernetes, 06ServiceOps new, 06SRE: Eqiad: lsw1-d7-eqiad BGP maintenance - https://phabricator.wikimedia.org/T418772#11672217 (10MatthewVernon) [09:36:07] 10netops, 06Infrastructure-Foundations, 10Prod-Kubernetes, 06ServiceOps new, 06SRE: Eqiad: lsw1-d7-eqiad BGP maintenance - https://phabricator.wikimedia.org/T418772#11672223 (10MatthewVernon) Is this maintenance happening at 15:00 UTC today? @Papaul ms-be1093 needs no action taking, but it'd be worth co... [12:04:01] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO: Update ULSFO LVS service IP's - https://phabricator.wikimedia.org/T418971#11672840 (10ayounsi) [12:04:10] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO: New switch configuration - https://phabricator.wikimedia.org/T408892#11672843 (10ayounsi) [12:04:21] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO: New switch configuration - https://phabricator.wikimedia.org/T408892#11672845 (10ayounsi) [12:04:26] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO: Update ULSFO LVS service IP's - https://phabricator.wikimedia.org/T418971#11672844 (10ayounsi) [13:20:25] FIRING: [3x] SystemdUnitFailed: user@100982.service on build2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:10:25] FIRING: [3x] SystemdUnitFailed: user@100982.service on build2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:37:03] 10Mail, 06Infrastructure-Foundations, 10vrts: VRT replies to Hotmail/Outlook bounce - https://phabricator.wikimedia.org/T418700#11673516 (10Xaosflux) Still occurring, VRT ticket 2026030410007057 error data: <_REDACTED_@outlook.com>: host outlook-com.olc.protection.outlook.com[52.101.10.0] said: 550 5.... [14:39:34] 10netops, 06Infrastructure-Foundations, 10Prod-Kubernetes, 06ServiceOps new, 06SRE: Eqiad: lsw1-d7-eqiad BGP maintenance - https://phabricator.wikimedia.org/T418772#11673532 (10Papaul) @MatthewVernon thank you. Yes it will be at 15:00 UTC [14:42:44] 10Mail, 06Infrastructure-Foundations, 10vrts: VRT replies to Hotmail/Outlook bounce - https://phabricator.wikimedia.org/T418700#11673547 (10Xaosflux) SPF: v=spf1 include:_cidrs.wikimedia.org ~all SPF record resolution: wikipedia.org include:_cidrs.wikimedia.org ip4:208.80.152.0/22 ip6:2620:0:860::/56 ip6:... [14:43:38] 10Mail, 06Infrastructure-Foundations, 10vrts: VRT replies to Hotmail/Outlook bounce - https://phabricator.wikimedia.org/T418700#11673549 (10Xaosflux) see also: T418803 [15:22:04] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, and 2 others: ULSFO: Update ULSFO LVS service IP's - https://phabricator.wikimedia.org/T418971#11673768 (10ssingh) Per T410411, we no longer need at least `pybal-high-traffic1-ulsfo.wikimedia.org` and `pybal-high-traffic2-ulsfo.wikimedia.org` i... [15:46:59] 10netops, 06Infrastructure-Foundations, 10Prod-Kubernetes, 06ServiceOps new, 06SRE: Eqiad: lsw1-d7-eqiad BGP maintenance - https://phabricator.wikimedia.org/T418772#11673879 (10ayounsi) cirrussearch repooled [16:26:42] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO: New switch configuration - https://phabricator.wikimedia.org/T408892#11674086 (10ayounsi) >>! In T408892#11671699, @Papaul wrote: > @ayounsi prior of deleting the sandbox1-ulsfo range 198.35.26.240/28 I will have to delete the... [18:10:40] FIRING: [2x] SystemdUnitFailed: user@100982.service on build2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:22:03] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad A/B switch cabling documentation - https://phabricator.wikimedia.org/T418018#11674997 (10RobH) >>! In T418018#11648884, @Papaul wrote: > @RobH like @ayounsi mentioned today everything for row A/B should be QSFP-100G CWMD4 like in... [20:16:29] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch upgrade - https://phabricator.wikimedia.org/T418012#11675191 (10RobH) [20:26:49] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad A/B switch cabling documentation - https://phabricator.wikimedia.org/T418018#11675231 (10RobH) >>! In T418018#11674997, @RobH wrote: >>>! In T418018#11648884, @Papaul wrote: >> @RobH like @ayounsi mentioned today everything for r... [21:46:19] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch upgrade - https://phabricator.wikimedia.org/T418012#11675615 (10RobH) [21:48:23] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch upgrade - https://phabricator.wikimedia.org/T418012#11675636 (10RobH) [22:10:40] FIRING: [2x] SystemdUnitFailed: user@100982.service on build2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:30:17] 10Mail, 06Infrastructure-Foundations, 10vrts: VRT replies to Hotmail/Outlook bounce - https://phabricator.wikimedia.org/T418700#11675813 (10jhathaway) @Xaosflux thanks for reporting this issue. I did some tests with the info-en queue, but I wasn't able to reproduce the issue. Questions: # What is the From... [23:04:44] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad A/B switch cabling documentation - https://phabricator.wikimedia.org/T418018#11675991 (10Papaul) Please see below for the spine to spine port information |Switch|Interface|Switch|Interface| |ssw1-a1-eqiad|ethernet-1/31|ssw1-f1-e... [23:43:51] 10Mail, 06Infrastructure-Foundations, 10vrts: VRT replies to Hotmail/Outlook bounce - https://phabricator.wikimedia.org/T418700#11676074 (10Xaosflux) @jhathaway Here is the reject data: -------------------------- Return-Path: Received: from vrts1003.eqiad.wmnet (vrts1003.eqi...