[11:19:27] Emperor: o/ qq - is there any bw/rate-limit applied on apus for the docker-registry client? I am asking since I tried to upload some docker layers (via the docker registry) and objects weighting around 3/4 GBs seem to take quite some time to complete (mins). [11:21:59] No. [11:22:49] it is quite a small and spinning-disk based cluster, but you don't look to have stressed it very hard from the dashboards [11:25:13] okok good to know, I wanted to rule out some extra configs that I may missed [11:26:18] I did do some benchmarking way back when, but I think I no longer have the answers [11:28:31] is there an expansion plan to also use ssds etc.. ? [11:28:36] (just for my knowledge) [11:46:44] Depends a bit on cost/budget, and my not having even-higher-priority-tasks to do. I think in an ideal world we'd at least look at NVME for bucket indexes (which would improve performance for buckets with lots of objects in); but also scaling out the cluster will improve performance - currently it's still pretty small. [11:47:42] sorry, that's a bit of a non-answer :( [12:07:54] it is a start, I think we can definitely have a chat with service ops in Lisbon and figure out next steps. Ideally apus will become the main storage for the Docker image binaries, deprecating Swift, so we should plan for it :) [12:08:31] 👍 [15:06:48] marostegui: https://gerrit.wikimedia.org/r/c/operations/alerts/+/1220640 is fixed and merged [15:07:16] federico3: excellent, I will wait for it to be spread and on monday we can test it [15:17:53] ok! [15:53:07] Emperor: sorry if I bother you again - are the apus ceph logs available somewhere? [15:53:42] elukey: on the frontends themselves, I'm afraid. [15:54:59] journalctl (with -S to pin down the timescale and -g username to grep out the relevant bits) on the ceph-[clusterid]@rgw.apus... service [16:13:04] ah perfect thanks! [16:15:28] [I inherited this pattern from the ms frontends, where trying to ship all the logs elsewhere was considered likely to be too disruptive] [16:15:57] tab-completion is likely your friend to find the right service name