[02:47:17] 10netops, 06Infrastructure-Foundations, 10ops-magru, 06SRE: cr2-magru <-> asw1-b3-magru link down March 2026 - https://phabricator.wikimedia.org/T418978#11690710 (10Papaul) Looks like changing the module on the switch side fixed the issue. ` sw1-b3-magru> show interfaces et-0/0/50 descriptions Interface... [03:50:41] FIRING: SystemdUnitFailed: bitu-permission-request.service on idm1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:54:31] 10netops, 06Infrastructure-Foundations, 10ops-magru, 06SRE: cr2-magru <-> asw1-b3-magru link down March 2026 - https://phabricator.wikimedia.org/T418978#11690822 (10cmooney) Showing as down right now both sides, lane 3 RX still poor on cr2-magru: ` cmooney@cr2-magru> show interfaces diagnostics optics et-0... [07:50:41] FIRING: SystemdUnitFailed: bitu-permission-request.service on idm1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:19:58] slyngs: ^ [08:20:20] looking [08:22:42] moritzm fixed that yesterday, but the service need to be rerun [08:25:41] RESOLVED: SystemdUnitFailed: bitu-permission-request.service on idm1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:26:13] Weeee [08:27:12] nice! [09:42:19] 10netops, 06Infrastructure-Foundations, 10ops-magru, 06SRE: cr2-magru <-> asw1-b3-magru link down March 2026 - https://phabricator.wikimedia.org/T418978#11691068 (10ayounsi) Link is down again: https://alerts.wikimedia.org/?q=scope%3Dnetwork&q=site%3Dmagru&q=%40state%3Dsuppressed Thanks rob for leading on... [13:25:32] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO:Switch refresh diagram - https://phabricator.wikimedia.org/T408511#11691843 (10Papaul) @ayounsi factory reset the switch same issue. [14:17:02] 10SRE-tools, 06Infrastructure-Foundations, 13Patch-For-Review: Cookbook for rack depool - https://phabricator.wikimedia.org/T327300#11692131 (10ayounsi) Example output in its current state with `--show --teams` : ` an-master1004: Couldn't get or parse depool Hiera key an-worker1152: skipping host (no depool... [16:47:33] 10netops, 06Infrastructure-Foundations, 10ops-magru, 06SRE: cr2-magru <-> asw1-b3-magru link down March 2026 - https://phabricator.wikimedia.org/T418978#11693404 (10RobH) > Support, > > You have swapped the optic on the router side, and the MPO patch cable. The link is still down, so we'd like you to swa... [19:35:24] 10netops, 06Infrastructure-Foundations, 10ops-magru, 06SRE: cr2-magru <-> asw1-b3-magru link down March 2026 - https://phabricator.wikimedia.org/T418978#11694408 (10RobH) They swapped the optic GT3AAG00314 out of the switch for optic GT3AAG00316 and now the link shows up: router: ` et-0/0/1 up... [20:06:30] 10netops, 06Infrastructure-Foundations, 10ops-magru, 06SRE: cr2-magru <-> asw1-b3-magru link down March 2026 - https://phabricator.wikimedia.org/T418978#11694501 (10RobH) Sent an email to investigate the return/repair of GT3AAG00314 & GT3AAG00321