[08:00:26] Cteam: welcome to today 🦄! Don’t forget to post your update in thread. [08:00:26] Feel free to include: [08:00:26] 1. 🕫 Anything you'd like to share about your work [08:00:26] 2. ☏ Anything you'd like to get help with [08:00:26] 3. ⚠ Anything you're currently blocked on [08:00:26] (this message is from a toolforge job under the admin project) [08:01:06] from yesterday, as messages for the record [08:01:08] Done: [08:01:08] * [lima-kilo] fixed the foxtrot-ldap issue, now lima-kilo should work well, there's some improvement patches left but at least it does not block creating your dev env anymore [08:01:08] * [toolforge] presented pust-to-deploy to the devex group [08:01:08] Doing: [08:01:08] * [ceph] tomorrow will pick up the ceph investigation again, try to make it fail in eqiad (codfw is not failing :l ) [08:01:08] * [jobs-api] continue with the reviews, now that lima-kilo is unblocked I'll start testing [08:01:09] Blockers: [08:01:10] * None [08:47:25] Done: [08:47:27] * T399212 nf_conntrack_max is not set at boot in cloudvirts [08:47:29] Working on: [08:47:31] * Creating follow-up tasks for T399281 (Ceph incident) [08:47:33] * T396724 trove disk full issue [16:56:44] Done: [16:56:44] * Investigated some the ceph outage [16:56:44] Doing: [16:56:44] * [jobs-api] Tested the MR repos/cloud/toolforge/jobs-api!182, to allow doing the job diff at the model level Raymond_Ndibe feel free to test and merge into yours if you don't find any issues [16:56:44] * [jobs-cli] sent an mr to remove unset fields when creating a job, this is something that helps differentiate between a request to set a field to a value that turns out is the default, and a request that does not care about the value and should fall back to the default. [16:56:44] Blockers: [16:56:45] * None [16:59:25] andrewbogott: I'm setting the osd_target_memory to 5G for all the osds in cloudcephosd1006 [16:59:36] see if that forces it not to grow so much or it keeps kinda ignoring it [16:59:50] ok! [17:00:43] I used `ceph config set osd.$i osd_memory_target 5368709120` (for reference)