[07:03:28] backups ran a bit faster today [07:08:15] federico3: db2240 is depooled and with 0 weight, is this a leftover from https://phabricator.wikimedia.org/T397163 [07:08:38] federico3: Similarely, db2179 has not been removed from API (but depooled from there) [07:08:49] So I think the last steps from that switchover werent completed [07:09:07] And they are indeed not marked as done, can you please review and fix that? [07:13:52] looking [07:16:40] thank you [07:18:37] the last step was skipped in order to do a os update and a schema change, and at the end of it there was a pool-in https://phabricator.wikimedia.org/T391056#10950729 but evidently it's not enough [07:32:30] Yeah, I'd suggest not to close the task till those steps are done [07:32:36] So you can use it as a reminder that it is pending [08:21:33] marostegui: BTW weeks ago I put together this page to help me understand how the weight are set https://zarcillo.wikimedia.org/ui/weights [10:01:55] marostegui: regarding https://gitlab.wikimedia.org/repos/data_persistence/dbtools/scripts/-/merge_requests/7#note_152183 what is the purpose of this "They key here for the RO es* hosts is to detect which host is the master" ? [10:34:49] federico3: Basically that you cannot depool a master [10:34:55] so the script will have to stop on those [10:35:10] federico3: Did you see the private data emails? Can you please take a look and check what's going on? Thanks [10:37:39] I saw them [10:50:30] federico3: And the second question, can you check what is going on? [10:52:14] I was looking : there are the same 2 tables from nl_subscriptions as "Non-public tables that are present" across the hosts [10:54:55] federico3: I think you probably have to run the redact_sanitarium script [10:54:57] To clean them up [10:55:32] I believe it is part of the catalog work amir was doing [11:03:01] Amir1: it's the nl_subscriptions table in the mediawikiwiki and testwiki tables and they contain a small amount of data with columns : nls_newsletter_id nls_subscriber_id [11:04:31] federico3: I think Amir.1 is AFK at the moment (see -private) [11:05:08] thanks Emperor [11:24:16] marostegui: I can try running sanitize-wiki with --check-only https://wikitech.wikimedia.org/wiki/MariaDB/PII [11:24:44] federico3: you'll have to restart sanitarium to pick up the filters and then sanitize it, in that order [11:26:36] federico3: Which also reminds me, if there's a sanitarium host involved, maybe double check if the kernel needs to be upgraded there [11:30:38] sre.mysql.sanitarium_restart is listing only db1154 and db1155 and the latter is not in the alert emails so I can run the sanitarium restart only on 1154 [11:31:36] Sure, if that's the only one affected [11:33:15] the kernel needs updating [11:35:55] federico3: Maybe you can take the opportunity to upgrade it too [11:36:28] REmember you will have to silence all clouddb* that hang from that hosts, which has multi instance, and stop mariadb on all the instances before reboot [11:38:10] the cookbook does the instance restarts automatically but no host reboot etc so I'd rather let this run and fix the leak but without making too many changes on the host at the same time [11:40:17] ok! [11:45:36] the restarts are done, I can run the sanitize in check mode [11:46:52] ok, but you still have to run the sanitization [12:05:57] hm, no, the sanitize-wiki cookbook is only meant for s5 so far [12:07:07] federico3: Wasn't it customizable? If not, we can run the old script [12:07:34] it is but we never tested it on other sections [12:10:55] It is time then to do it [12:11:00] It should be the same really [12:11:05] Nothing really changes on a section level [12:14:11] I'm testing the manual script but it looks safe to run the cookbook [12:14:22] ok, go for it [12:18:46] I'll have to leave in a bit so I'll continue later on today unless it's super urgent [12:19:54] the socket detection in the cookbook has a glitch and needs a small fix [12:19:59] federico3: it is fine to finish it later