[11:09:52] hey folks, it seems that the docker registry now serves the whole catalog way faster without swift's ratelimit :D [11:10:35] cc: jayme (credits as always to Janis for listening to my endless rumbling and providing great feedback) [11:11:08] oh, it did work?! Nice! 🎉 [11:11:21] now it fails for a new thing but it is docker-report-related [11:14:57] basically https://gerrit.wikimedia.org/r/c/operations/docker-images/docker-report/+/1078345 [11:15:18] it surfaced with the new filter to include "/repos/etc.." [11:15:26] I can easily exclude the image for the moment [11:15:34] (that causes the issue I mean) [11:44:37] > now it fails for a new thing but it is docker-report-related [11:44:38] lol [11:44:47] congrats btw, nice [11:55:11] elukey: that change is WIP, right? Your last comment led me to believe that at least (that's why I did not look as of now) [12:30:03] elukey: wowow [12:33:48] also jayme -- re https://gerrit.wikimedia.org/r/c/operations/puppet/+/1078719 -- *something* is stopping TCP from being accepted on port 53 (it times out) and afaict it isn't network policies [12:34:41] cdanis: I did dig -x 10.192.75.58 +nsid +tcp @kubestagemaster2003.codfw.wmnet [12:34:45] that seemed okay [12:34:52] uh it didn't work for me yesterday :D [12:37:25] jayme: ok TCP doesn't work for e.g. wikikube-ctrl1002, but UDP does [12:37:59] oh. [12:38:17] - name: dns-tcp [12:38:18] nodePort: 52337 [13:00:51] the TCP nodePort is a different random number on every cluster; given how the helm chart looks I don't really understand [13:05:54] editing the service by hand doesn't work either haha [13:06:00] the edit is accepted by kubectl but it's a no-op [13:10:15] jayme: re change is WIP, yes yes I need to figure out some other things, not ready yet