[09:26:25] Morning :) does this change look sane? https://gerrit.wikimedia.org/r/c/operations/puppet/+/1220352 [10:02:15] o/ [10:14:57] to recap - the idea is to create a specific /ml prefix in the docker registry's namespace, to better isolate it and potentially switch docker registry's backend anytime (for example, using s3/ceph) [10:16:08] the only thing that we need to verify is the authentication to the /v2 prefix (see line 120 in https://gerrit.wikimedia.org/r/c/operations/puppet/+/1220352/4/modules/docker_registry/templates/registry-nginx.conf.erb) [10:17:02] the config on registry2004 has the restricted password in both places (under /v2 and /v2/restricted), since I believe docker authenticates first using /v2 before pushing [10:17:11] (I am not 100% sure about it) [10:17:22] anyway, it should be easy to add it [10:18:25] this work is really useful for https://phabricator.wikimedia.org/T412951 in my opinion - it would be nice if the /v2/ml prefix used the s3/ceph backend, so we could test it properly [10:19:30] we'd need to add an extra "upstream" config at the top, but it would help a lot since /ml images wouldn't face the 5GB barrier for each layer (that is the maximum object size for swift) [10:20:33] we also have "client_max_body_size 4608m; # 4.5GB https://phabricator.wikimedia.org/T404742#11197688" in the server config, that is tied to the nginx's tmpfs mountpoint size (basically where POSTs get buffered) that will need to be tuned, but we can override it in location blocks so the /ml prefix could have a higher limit (if needed) [10:20:58] (it would require more memory assigned to the registry's vm though) [14:03:23] commented in the code review with the first bit [14:09:24] akosiaris: o/ when you have a moment - the docker registry instance with the ceph backend is named "restricted", tailored for the MediaWiki use case [14:09:41] we have also the ML one, that follows the same pattern, and it would be great to test both at the same time [14:10:02] what was your idea about the instances? Keep multiple ones, with different buckets etc.. ? [14:11:04] the registry daemons seem not requiring a huge cpu/memory footprint afaics [14:11:36] maybe having multiple buckets will allow us to perform GC (if it will work without swift) more easily [14:19:04] partially self-answering - in https://phabricator.wikimedia.org/T394476 an apus s3 account was created, and the bucket was created manually for the restricted use case [14:19:15] we could create an ML bucket as well, same account [14:27:16] <_joe_> +1 [14:27:25] <_joe_> elukey: alex is out today [14:28:14] oh ok thanks! [14:47:16] filed https://gerrit.wikimedia.org/r/c/operations/puppet/+/1224091