[05:59:25] <jinxer-wm>	 FIRING: SystemdUnitFailed: swift_rclone_sync.service on ms-be1069:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[06:00:40] <marostegui>	 I am going to switch m1 master
[09:59:40] <jinxer-wm>	 FIRING: SystemdUnitFailed: swift_rclone_sync.service on ms-be1069:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[10:12:48] <marostegui>	 [12:08:45]  <+icinga-wm> PROBLEM - Uncommitted dbctl configuration changes- check dbctl config diff on cumin2002 is CRITICAL: CRITICAL - Uncommitted dbctl configuration changes, check dbctl config diff https://wikitech.wikimedia.org/wiki/Dbctl%23Uncommitted_dbctl_diffs
[10:12:50] <marostegui>	 federico3: ^
[10:13:31] <federico3>	 huh?!
[10:13:46] <marostegui>	 That arrived to wikimedia-operations
[10:15:06] <federico3>	 but I committed it at https://phabricator.wikimedia.org/T393296#10999610
[10:15:27] <marostegui>	 yes, but you never added it to instances.yaml
[10:15:35] <marostegui>	 So it is being removed as soon as puppet runs on cumin
[10:20:40] <federico3>	 ok I'll rerun puppet and check dbctl after   https://gerrit.wikimedia.org/r/c/operations/puppet/+/1169071
[10:38:44] <federico3>	 wow, power outage
[10:39:52] <marostegui>	 what?
[10:39:56] <marostegui>	 where?
[10:40:04] <marostegui>	 in which dc?
[10:40:19] <marostegui>	 federico3: ^
[10:40:31] <federico3>	 at my home 
[10:40:36] <marostegui>	 ah ok :-!
[10:41:18] <marostegui>	 you scared me to death
[10:41:35] <marostegui>	 🤌🤌
[10:42:26] <federico3>	 :D 
[10:42:52] <federico3>	 there should be a page storm (I hope) if we had a dc power outage :))
[10:43:08] <marostegui>	 yeah, but maybe you saw before it arrived or who knows
[11:28:09] <federico3>	 marostegui: the cloning is running, meanwhile can I start "flipping" es1031 and updating it?
[11:29:24] <marostegui>	 federico3: sure thing
[11:55:01] <federico3>	 marostegui: pushed an update to https://gitlab.wikimedia.org/repos/data_persistence/dbtools/scripts/-/merge_requests/7 
[11:57:09] <marostegui>	 thanks, will check when I have a min
[13:55:56] <marostegui>	 jynus: so if it involves redistributing things, it is probably better to think of an any other approach 
[13:56:14] <jynus>	 I pmed you
[13:56:14] <marostegui>	 Maybe you can leave a couple of sections prepared in advance
[13:56:22] <marostegui>	 ah sure
[13:59:40] <jinxer-wm>	 FIRING: SystemdUnitFailed: swift_rclone_sync.service on ms-be1069:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[14:06:52] <jynus>	 So Manuel and I discussed a way to move fix the dependencies- it won't be perfect, but it will be something
[16:40:29] <federico3>	 can I reset the dbctl diff ?
[17:59:40] <jinxer-wm>	 FIRING: SystemdUnitFailed: swift_rclone_sync.service on ms-be1069:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[21:59:40] <jinxer-wm>	 FIRING: SystemdUnitFailed: swift_rclone_sync.service on ms-be1069:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed