[08:00:05] Cteam: welcome to today 🦄! Don’t forget to post your update in thread. [08:00:05] Feel free to include: [08:00:05] 1. 🕫 Anything you'd like to share about your work [08:00:05] 2. ☏ Anything you'd like to get help with [08:00:05] 3. ⚠ Anything you're currently blocked on [08:00:05] (this message is from a toolforge job under the admin project) [09:48:12] Done: [09:48:14] * Wrote the first draft of the incident report https://wikitech.wikimedia.org/wiki/Incidents/2025-07-11_WMCS_Ceph_issues_causing_Toolforge_and_Cloud_VPS_failures [09:48:16] * T399464 restarted a Trove instance that was stuck [09:48:18] Working on: [09:48:20] * T399481 Swift container endpoints are unavailable [09:48:22] * T399212 nf_conntrack_max is not set at boot in cloudvirts [09:48:24] * Creating follow-up tasks for T399281 (Ceph incident) [13:34:27] Yesterday: [13:34:27] * PTO [13:34:28] Today: [13:34:28] * This is tedious, but I'm going to downgrade a codfw1 cephosd node back to bullseye, then... [13:34:28] * Check side-by-side RAM usage between the bullseye and bookworm hosts, and... [13:34:28] * eventually upgrade to Q and see if that performs better on bookworm [17:52:51] Done: [17:52:51] * [toolforge] admin stuff (notes, boards, task wrangling, ...) [17:52:52] * [push-to-deploy] prepared demo for tomorrow's devex meet [17:52:52] Doing: [17:52:52] * [ceph] looking into last friday's issue, just fooling around really, will try to play with the bookworm node tomoorow (thanks andrewbogott for preparing it!) [17:52:52] * [openstack-browser] started trying to make it work locally, saw lots of errors in the logs (rabbit hole alert) [17:52:52] * [jobs-api] have to focus on the reviews [17:52:52] Blockers: [17:52:53] * None [17:53:09] * dcaro of [17:54:32] *off