[03:11:41] FIRING: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [04:11:41] RESOLVED: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [09:30:19] 10netops, 06Infrastructure-Foundations, 06SRE: Cloudcephosd: migrate to single network uplink - https://phabricator.wikimedia.org/T399180#11431822 (10ayounsi) From that comment : T410989#11429115 cloudcephosd1052 still needs to be migrated. Both interfaces are still doing significant traffic : https://libre... [10:51:13] 10netops, 06Infrastructure-Foundations, 06SRE: Cloudcephosd: migrate to single network uplink - https://phabricator.wikimedia.org/T399180#11432052 (10fgiunchedi) I took a look at why cloudcephosd1052 still has second nic up, currently: ` 4: ens1f1np1: mtu 9000 qdisc mq stat... [11:58:56] 10netops, 06Infrastructure-Foundations, 06SRE: Cloudcephosd: migrate to single network uplink - https://phabricator.wikimedia.org/T399180#11432253 (10cmooney) >>! In T399180#11432052, @fgiunchedi wrote: > I think the easiest would be to: > > * Remove the spurious `enp13s0f1np1` config, run puppet to verify... [13:29:50] elukey: FYI I don't see a corresponding spicerack patch for this patch from filippo: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1214473 [13:51:10] volans: ack thanks! [13:55:29] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: lvs1018: remove cross-rack link to asw2-c2-eqiad xe-2/0/13 - https://phabricator.wikimedia.org/T411781 (10cmooney) 03NEW p:05Triage→03Medium [13:56:54] volans: something like https://gerrit.wikimedia.org/r/c/operations/software/spicerack/+/1215162/1/spicerack/service.py ? [13:57:09] I don't recall exactly how soon the spicerack release needs to be done in these cases [13:58:30] the puppet define doesn't have a default, so maybe just empty string? [13:59:04] or you want to replicate the logic in the metrics file [13:59:19] I think we should reflect what's in the catalog, so if not defined empty [13:59:50] but yeah the patch is like that [14:00:35] as for the release, IIRC the cookbooks using the catalog will probably break once in puppet there is one service that defines it [14:01:08] I have vague memories regarding the fact that we looked for alternatives to make it less strict but it was a nbit of a mess, but I can't recall the details right now [14:01:29] 10netops, 06cloud-services-team, 10Horizon, 06Infrastructure-Foundations, 10Striker: Move cloudweb hosts to cloud racks? - https://phabricator.wikimedia.org/T411783 (10taavi) 03NEW [14:01:41] what define are you talking about? [14:03:03] in hieradata/common/service.yaml [14:03:13] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: lvs1018: remove cross-rack link to asw2-c2-eqiad xe-2/0/13 - https://phabricator.wikimedia.org/T411781#11432684 (10Vgutierrez) the assessment is OK and the link can be removed safely [14:06:08] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 2 others: lvs1020: move primary uplink from asw2-d7-eqiad to lsw1-d7-eqiad and remove link to asw2-c2-eqiad - https://phabricator.wikimedia.org/T405609#11432701 (10Jclark-ctr) [14:10:22] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: lvs1018: remove cross-rack link to asw2-c2-eqiad xe-2/0/13 - https://phabricator.wikimedia.org/T411781#11432739 (10Jclark-ctr) https://netbox.wikimedia.org/dcim/interfaces/29150/trace/ https://netbox.wikimedia.org/dcim/interfaces/29151... [14:10:56] elukey: https://etherpad.wikimedia.org/p/volans-tmp3 [14:11:18] where I added team: test to the kibana service in a copy of the catalog file [14:11:57] there might be cookbooks that loop over the catalog looking for services matching values, those will probably break if the team key is set in service.yaml [14:45:35] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 2 others: lvs1020: move primary uplink from asw2-d7-eqiad to lsw1-d7-eqiad and remove link to asw2-c2-eqiad - https://phabricator.wikimedia.org/T405609#11432921 (10Jclark-ctr) 05Open→03Resolved a:05cmooney→03Jclark-ctr [15:02:16] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: lvs1018: remove cross-rack links to rows A, C and D - https://phabricator.wikimedia.org/T411781#11433019 (10cmooney) [15:02:42] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: lvs1018: remove cross-rack links to rows A, C and D - https://phabricator.wikimedia.org/T411781#11433025 (10cmooney) [15:02:43] 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic: Eqiad row C/D switch refresh: LVS changes to support migration - https://phabricator.wikimedia.org/T405602#11433026 (10cmooney) [15:08:09] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: lvs1018: remove cross-rack links to rows A, C and D - https://phabricator.wikimedia.org/T411781#11433060 (10cmooney) [15:16:56] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO: New switch configuration - https://phabricator.wikimedia.org/T408892#11433083 (10cmooney) >>! In T408892#11330727, @cmooney wrote: > Additionally for the rebuild we should aim to: > > # Convert the existing ganeti hosts to rout... [16:25:01] 10Mail, 06Infrastructure-Foundations, 10Notifications (Echo): E-mail doesn't get confirmed and gets unlinked constantly - https://phabricator.wikimedia.org/T411799 (10Niepodkoloryzowany) 03NEW [17:06:51] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 3 others: lvs1019: move primary uplink from asw2-c7-eqiad to lsw1-c7-eqiad and remove link to asw2-d2-eqiad - https://phabricator.wikimedia.org/T405628#11433554 (10BCornwall) [17:14:48] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 3 others: lvs1019: move primary uplink from asw2-c7-eqiad to lsw1-c7-eqiad and remove link to asw2-d2-eqiad - https://phabricator.wikimedia.org/T405628#11433580 (10BCornwall) [17:22:43] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 3 others: lvs1019: move primary uplink from asw2-c7-eqiad to lsw1-c7-eqiad and remove link to asw2-d2-eqiad - https://phabricator.wikimedia.org/T405628#11433608 (10cmooney) [17:28:50] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 3 others: lvs1019: move primary uplink from asw2-c7-eqiad to lsw1-c7-eqiad and remove link to asw2-d2-eqiad - https://phabricator.wikimedia.org/T405628#11433621 (10BCornwall) [17:29:57] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: rows C/D Upgrade Tracking - https://phabricator.wikimedia.org/T404609#11433626 (10RobH) [17:30:02] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: lvs1018: remove cross-rack links to rows A, C and D - https://phabricator.wikimedia.org/T411781#11433627 (10RobH) [17:30:47] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 2 others: lvs1019: move primary uplink from asw2-c7-eqiad to lsw1-c7-eqiad and remove link to asw2-d2-eqiad - https://phabricator.wikimedia.org/T405628#11433629 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by bret... [17:32:09] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: rows C/D Upgrade Tracking - https://phabricator.wikimedia.org/T404609#11433638 (10RobH) Day 13 Update: * all hosts in rows C and D migrated ** lvs1018 in row B has links into C and D need removal via T411781 before we can kill... [17:47:29] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: rows C/D Upgrade Tracking - https://phabricator.wikimedia.org/T404609#11433701 (10RobH) [18:04:00] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 2 others: lvs1019: move primary uplink from asw2-c7-eqiad to lsw1-c7-eqiad and remove link to asw2-d2-eqiad - https://phabricator.wikimedia.org/T405628#11433761 (10Jclark-ctr) [18:06:01] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 2 others: lvs1019: move primary uplink from asw2-c7-eqiad to lsw1-c7-eqiad and remove link to asw2-d2-eqiad - https://phabricator.wikimedia.org/T405628#11433764 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cu... [18:16:10] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 2 others: lvs1019: move primary uplink from asw2-c7-eqiad to lsw1-c7-eqiad and remove link to asw2-d2-eqiad - https://phabricator.wikimedia.org/T405628#11433805 (10cmooney) [18:16:55] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 2 others: lvs1019: move primary uplink from asw2-c7-eqiad to lsw1-c7-eqiad and remove link to asw2-d2-eqiad - https://phabricator.wikimedia.org/T405628#11433810 (10cmooney) [18:17:25] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 2 others: lvs1019: move primary uplink from asw2-c7-eqiad to lsw1-c7-eqiad and remove link to asw2-d2-eqiad - https://phabricator.wikimedia.org/T405628#11433811 (10Jclark-ctr) [18:17:36] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 2 others: lvs1020: move primary uplink from asw2-d7-eqiad to lsw1-d7-eqiad and remove link to asw2-c2-eqiad - https://phabricator.wikimedia.org/T405609#11433812 (10cmooney) [18:18:46] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 2 others: lvs1019: move primary uplink from asw2-c7-eqiad to lsw1-c7-eqiad and remove link to asw2-d2-eqiad - https://phabricator.wikimedia.org/T405628#11433817 (10cmooney) 05Open→03Resolved [18:21:59] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 2 others: lvs1019: move primary uplink from asw2-c7-eqiad to lsw1-c7-eqiad and remove link to asw2-d2-eqiad - https://phabricator.wikimedia.org/T405628#11433829 (10BCornwall)