[08:31:27] Hello, reprepo checkupdate is broken because node20 upstream repo is broken. Is it fine to to remove this package temporarily from updates? https://gerrit.wikimedia.org/r/c/operations/puppet/+/1217456 [08:31:27] elukey: I think you had a similar problem two month ago in T406623 [08:31:28] T406623: nodesource node22 apt mirror is broken - https://phabricator.wikimedia.org/T406623 [08:57:08] jelto: o/ checking [09:02:00] okok makes sense, I think it is the same problem that I had with node22 [09:03:25] jelto: if you are not in a rush, I can try to fix it [09:03:25] thank you! Then I'll merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/1217456 and see if that unblocks the gitlab update [09:04:09] not in a rush but https://phabricator.wikimedia.org/T412276 should be addressed soon :) It's a security update [09:04:50] but delaying it for an hour should be fine imho [09:06:47] jelto: yeah I found the fix, but now I see that hp-mcp-bookworm also complains about expired GPG key [09:10:19] ok so https://gerrit.wikimedia.org/r/c/operations/puppet/+/1217465 fixes the node20 [09:10:20] I think mutante removed the key in https://gerrit.wikimedia.org/r/c/operations/puppet/+/1217278 ? [09:12:13] ah interesting, lemme check [09:16:23] running puppet on apt with the patch to see if the previous error still remains [09:20:09] jelto: it still complains about C208ADDE26C2B797 being expired [09:21:48] jelto: when you have a minute could you ping me to fix https://phabricator.wikimedia.org/T370018 ? [09:21:57] tried to manually delete it from gpg, not it complains about the missing key [09:21:58] weird [09:22:59] XioNoX: I'll do! but first I'd like to rollout the security update [09:23:22] no rush, when the time is right [09:27:28] jelto: I commented some stuff, you are unblocked right now, please update the gitlab files, I'll fix the rest later on (puppet is disabled) [09:27:55] great thank you, I'll let you know when I'm done on the apt host [09:30:51] elukey: I'm done on apt1002, I bumped the gitlab-ce package version successfully [09:32:06] oook [11:13:17] filed https://gerrit.wikimedia.org/r/c/operations/puppet/+/1217483 for the hp stuff, jelto if you have time could you please sanity check it? [11:36:01] elukey: I added a comment, I think you added the wrong expired key. 26C2B797 should be used. They linked the wrong key in the hyperlink "(extended through 2027)" afaict [11:37:00] B1275EA3 looks definitely expired to me [13:16:00] jelto: I am glad that I asked for a sanity check :) updated, thanks! [13:22:21] in a meeting, I look in a sec [13:34:44] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: rows C/D Upgrade Tracking - https://phabricator.wikimedia.org/T404609#11451533 (10RobH) [13:35:31] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: rows C/D Upgrade Tracking - https://phabricator.wikimedia.org/T404609#11451534 (10RobH) a:05RobH→03Jclark-ctr lvs1018's links were removed from use yesterday, so this project is now on the steps: [] [john or valerie] move a... [13:49:58] jelto: sorry I self-merged since I'll leave soon for the on-call compensation time and I didn't want to leave things broken :( [13:51:52] np, I +1ed anyways. The old key file for B1275EA3 can probably be removed. But that can happen next week as well. Enjoy your weekend [13:56:09] jelto: all good it works! Thanks :) [15:49:33] 10netops, 06Infrastructure-Foundations, 10Observability-Logging: ~5k/logs/sec from netdev - https://phabricator.wikimedia.org/T412143#11451960 (10ayounsi) Opened Juniper case 2025-1211-979319. [16:06:16] Hi, we now have another issue when constructing the service catalog. `Error exporting discovery types: Service.__init__() got an unexpected keyword argument 'exclude_from_switchover'` - this probably happens after https://gerrit.wikimedia.org/r/c/operations/puppet/+/1217189 got merged [16:07:53] I don't know if the issue I mentioned yesterday (keyword argument `team` - https://www.irccloud.com/pastebin/No3OgYaS/ ) is still present or not [16:40:49] ^ let me know if you prefer me to file a bug for this [17:33:06] matthieulec: hi [17:33:32] sorry for the lack of effort on this, L uca who is probably best placed to take a look was on call this week and is now not back until Monday [17:33:43] can you file a Phabricator task with the detail? [17:33:54] is this blocking your work currently? can it wait until next week? [17:56:14] I'll file a bug. My immediate task can wait until next week but it may block others (anyone using service_catalog) [18:05:14] ok thanks [18:05:31] I can have a look tomorrow and see if I can work it out but I fear it may need to wait for the experts [19:01:15] 10netops, 06Infrastructure-Foundations, 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Handle `network_flows_internal` data growth - https://phabricator.wikimedia.org/T412443 (10JAllemandou) 03NEW [19:14:25] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, and 2 others: ULSFO: switch refresh - https://phabricator.wikimedia.org/T408510#11452637 (10RobH) [19:36:19] 10netops, 06Infrastructure-Foundations, 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Handle `network_flows_internal` data growth - https://phabricator.wikimedia.org/T412443#11452701 (10ayounsi) Thanks, basically there was a firewall rule that prevented the ingestion of the SFLOW traffic from... [19:39:18] 10netops, 06Infrastructure-Foundations, 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Handle `network_flows_internal` data growth - https://phabricator.wikimedia.org/T412443#11452708 (10cmooney) @JAllemandou thanks for the task. And apologies this has just hit you out of the blue - we should... [20:03:41] 10netops, 06Infrastructure-Foundations, 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Handle `network_flows_internal` data growth - https://phabricator.wikimedia.org/T412443#11452763 (10xcollazo) ( For now, we have paused the pipeline, as it is continually failing: https://airflow.wikimedia.or...