[09:46:44] > dump wrong_size 1 day, 2 hours ago 221.2 GB -9.9 % The previous backup had a size of 245.5 GB, a change larger than 5.0%. [09:46:53] rev_sha1 drop ^ [09:48:15] 👍 [14:11:39] Emperor: o/ [14:12:31] I am investigating a weird hang between the docker registry and the ceph's s3 api, that it is difficult to pin point. One thing that I'd like to check is the permissions that the docker-registry user has on the bucket [14:13:05] [meeting, back to you afterwards] [14:13:15] okok! [14:31:55] elukey: what's the bucket name? [14:33:25] registry-ml is owned by docker-registry [14:33:49] likewise registry-restricted [14:34:22] I think the former is empty currently? [14:34:53] yeah I am using the latter [14:35:41] from s3cmd I see FULL_CONTROL, I was checking if everything was right on this side [14:36:17] for some reason docker hangs on the last operation after pushing all layers, that seems to be a PUSH [14:36:33] the registry's go code seems to hang on the s3 operation [14:36:51] but I didn't find much in apus' logs [14:37:14] in the past days sometimes it just worked after a few tries [14:37:17] that it is even more strange [14:42:33] AFAICT the permissions are straightforward (and correct), docker-registry owns the bucket. [14:43:20] and it can delete etc.. so all operations [14:43:36] yes [14:47:16] ack super.. the other doubt that I have is related to replication - could it happen that a certain write gets stalled because replication between dcs is in progress? [14:48:39] It /shouldn't/ be [14:53:23] because replication is async [15:07:06] where are the ceph settings listed? I just want to review them, because what I see is that eventually the push succeeds, and the ops that were hanging terminate quick [15:07:28] as if something was blocking them before, but I don't see high latency or similar in the apus-fe logs [15:13:04] They're basically inspected with assorted runes to radosgw-admin e.g. [15:13:12] sudo cephadm shell -- radosgw-admin bucket stats --bucket=registry-restricted [15:13:47] https://docs.ceph.com/en/reef/man/8/radosgw-admin/ are the docs for that not-especially-user-friendly tool [15:17:04] okok - on what host I should run those? Forgive me for the questions, if there is a wikitech page I'll read that instead [15:17:58] something like this I imagine https://wikitech.wikimedia.org/wiki/Ceph/Cephadm [15:18:51] * elukey reads [15:21:11] so moss-be1001, got it :) [15:30:28] indeed