[16:24:57] <wikibugs>	 10GitLab (CI & Job Runners): kokkuri doesn't seem to work from WMCS runners - https://phabricator.wikimedia.org/T396924#10928939 (10dancy) Only trusted runners are allowed to publish to the production docker registry.  The `wmcs` runners are not trusted.
[16:26:18] <wikibugs>	 10GitLab (CI & Job Runners): kokkuri doesn't seem to work from WMCS runners - https://phabricator.wikimedia.org/T396924#10928942 (10dancy) To clarify, the gitlab-cloud-runners do have a registry available for hosting temporary images.  There is no corresponding thing in wmcs.
[16:41:42] <wikibugs>	 10GitLab (CI & Job Runners): kokkuri cannot publish "public" images from WMCS runners due to a lack of a local registry - https://phabricator.wikimedia.org/T396924#10929014 (10bd808)
[18:12:24] <bd808>	 Disk is apparently full on runner-1023.gitlab-runners.eqiad1.wikimedia.cloud -- https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/jobs/540644
[18:12:48] <bd808>	 Do we have a runbook for cleaning those boxes up?
[18:12:53] * bd808 goes looking on wikitech
[18:16:30] <bd808>	 `du -sh /var/lib/docker/overlay2` 73G
[18:19:55] <bd808>	 `docker system prune --all` only got 5G back
[19:45:52] <dancy>	 bd808: Did you find something further to do? If not, it looks like the issue resolved itself.
[20:14:47] <bd808>	 dancy: I pruned those 5G of layers. The system seemed to think the rest were still in use. It does look like something that was hanging on to a lot of layers finished up. There are 35G free in /var/lib/docker now.
[20:15:26] * bd808 now wonders how a 40G mount reported as 73G in-use
[20:18:07] <dancy>	 I'm guessing hard links are involved.
[20:18:35] <bd808>	 :nod: that would make some sense.