[01:35:40] <jinxer-wm>	 FIRING: SystemdUnitFailed: swift_rclone_sync.service on ms-be2069:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[05:35:40] <jinxer-wm>	 FIRING: SystemdUnitFailed: swift_rclone_sync.service on ms-be2069:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[06:18:08] <marostegui>	 I am switching s4 primary master
[08:25:25] <jinxer-wm>	 RESOLVED: SystemdUnitFailed: swift_rclone_sync.service on ms-be2069:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[10:09:35] <jynus>	 Last dump for s1 at codfw (db2141) taken on 2026-01-20 00:00:02 is 153 GiB, but the previous one was 183 GiB, a change of -16.5 %
[10:10:15] <marostegui>	 yep, that's the revision work
[10:10:22] <marostegui>	 https://phabricator.wikimedia.org/T411164ç
[10:10:28] <marostegui>	 https://phabricator.wikimedia.org/T411164
[10:10:51] <jynus>	 👍
[14:00:12] <Amir1>	 s8 will be even bigger
[15:00:46] <brouberol>	 Hi folks! Just to let you know, we've dropped the `mpic_{staging,production}` tables from an-mariadb1001 and added the `test_kitchen_{staging,production}` tables. I don't know whether you care, but I just wanted to let y'all know out of caution 
[15:01:04] <brouberol>	 s/tables/databases/g
[15:02:56] <jynus>	 is that the section analytics_meta?
[15:04:30] <brouberol>	 I'm not sure about the section name, but this is very much an analytics mariadb instance, yes
[15:04:49] <jynus>	 is it staging ?
[15:05:11] <jynus>	 because I don't think we backup that, should we?
[15:07:11] <brouberol>	 the instance holds both (small) staging and production databases, for tools such as superset, datahub and test-kitchen. Most of the databases are actually un-used now, except these 3. I wouldn't hurt to back them up, especially because the DBs are quite small. IIRC it has been discussed here https://phabricator.wikimedia.org/T377820
[15:17:22] <brouberol>	 generally, when working on removing obsolete DBs from these servers (cf https://phabricator.wikimedia.org/T369612), should we let y'all know?
[18:14:37] <Amir1>	 https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&refresh=5m&var-server=db2152&var-datasource=000000026&var-cluster=mysql&viewPanel=panel-28&from=now-2d&to=now&timezone=utc 150GB for s8 rev_sha1
[18:39:07] <marostegui>	 Amir1: it's not finished yet on db2152
[18:39:25] <Amir1>	 this is at least that much
[18:39:33] <Amir1>	 I think it'll be bigger