[09:18:03] morning [09:25:11] greetings [10:03:14] morning [11:54:06] hello! I stumbled upon a latency when upgrading gerrit in devtools https://www.irccloud.com/pastebin/4EX7sP4u/oof.txt should I just try to turn it off and on again or is it something worth creating a tracking task for? [11:54:55] note: the latency happens after the bastion [12:01:40] arnaudb: it's fast for me using the root user... let me try with a regular user [12:04:27] fast with a regular user too: 0m2.768s [12:04:31] is it still slow for you? [12:05:02] * dhinus lunch, bbl [13:23:15] dhinus: sorry I missed your messages! I've retried and still get the same latency, with -vvv it's after the `ssh_init_stdio_forwarding: gitlab-1002.devtools.eqiad1.wikimedia.cloud:22` steps that the slowing starts. I'll let it unfold until the end to see if there is something meaningful that helps me snipe what looks like a pebcak [13:27:36] it is indeed, some other keys are tried and tested before, sorry for the disturbance :) [13:27:43] tried and failed* [13:27:57] arnaudb: so it seems like that instance is missing an ipv6 default route for whatever reason [13:28:56] which means your ssh connection tries to use ipv6, times out, and falls back to ipv4, which is where that delay comes from. d.hinus is using the restricted bastion, which I haven't yet replaced with one that supports v6, so it never tries with v6 [13:31:34] networkctl is showing that interface as 'failed', with 'Could not set route: Connection timed out' (but not much else) in the logs [13:34:45] not seeing anything in the firewall logs either [13:35:04] arnaudb: a networkd restart seems to have fixed it, try now? [13:47:25] works 100% thanks taavi! [13:49:08] thanks taavi! [13:49:28] in T415239 a new toolforge user is having issues with ssh [13:49:28] T415239: Toolforge SSH login: connection closed after publickey authentication - https://phabricator.wikimedia.org/T415239 [13:49:37] I think the user is missing some groups? https://ldap.toolforge.org/user/JacobHung [13:53:43] I don't remember if there's a way to fix that without manually editing LDAP [13:57:00] wait I don't see the request in https://toolsadmin.wikimedia.org/tools/membership/ so maybe the user has not applied for membership [14:42:48] volans: did you see T415199 which I noticed yesterday? [14:42:48] T415199: Toolforge NFS tracing misses some dumps events - https://phabricator.wikimedia.org/T415199 [14:43:17] taavi: yes, I was catching up with backlogs and stuff. [14:43:47] so, the original code from da.vid had this optimization that bails out of any path that doesn't start with /mnt/nfs/ [14:44:15] so I assumed that was the only path things were accessible [14:44:33] and/or that would already resolve symlinks [14:47:13] there are some standard links or other paths tools use to access NFS? [14:49:28] yes, various. including (but probably not limited to) stuff under /data, /public and /home [14:50:25] * volans checking docs right now [14:52:35] https://wikitech.wikimedia.org/wiki/Help:Shared_storage is the user-facing doc [14:52:52] nice to see a lot of `This part of the docs is incomplete, contributions are very welcome` in ebpf docs :/ [15:09:14] I'll update the task shortly [16:46:12] just noticed that the gateway-api docs have a specific page for people migrating away from ingress-nginx :D [16:46:45] nice! [16:47:35] https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests are the patches I mentioned that need review. the allowlist patch does currently list the ingress-nginx annotations in use, so we can merge+deploy that now and remove those from the allowlist once the use cases have been migrated away [16:54:47] do we have by any chance gitlab repo for tooling to be run from our laptop? [16:55:09] I have a clinic duty start shift helper script that I'd like to publish somewhere [17:33:10] I don't think we have a repo yet but you could make one! [17:59:40] "Cloud-Native API and AI Gateway" ... [17:59:54] ^ that's how one k8s gateway-api implementation describes itself [18:06:13] :barf: [18:35:02] * dhinus off