[01:59:40] <jinxer-wm>	 FIRING: SystemdUnitFailed: swift_rclone_sync.service on ms-be1069:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[05:48:02] <marostegui>	 I am going to start s5 switchover
[05:59:40] <jinxer-wm>	 FIRING: SystemdUnitFailed: swift_rclone_sync.service on ms-be1069:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[06:30:24] <marostegui>	 I was bitten by the semi-sync bug, as the s5 primary wasn't patched yet and I had to finish the switchover manually because it got stuck while inverting replication
[08:41:39] <federico3>	 what are the next steps for handling grants on https://phabricator.wikimedia.org/T393990#10983486 ?
[08:44:23] <marostegui>	 federico3: I think you'll simply have to duplicate the existing ouput of show grants for 'root'@'$cumin1002'; and place the same with cumin1003's ip on those listed hosts
[08:45:19] <marostegui>	 I will post that on the task
[08:48:25] <marostegui>	 federico3: the depooling/repooling script for pc is done? it can be handy for:https://phabricator.wikimedia.org/T399540#11003450
[08:48:35] <federico3>	 pc or es?
[08:48:41] <marostegui>	 the pc cookbook
[08:50:42] <federico3>	 yes, it can be used using test-cookbook - the optional downtime in https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1165546/comment/8bcebbb8_6404cd75/ is not added yet
[08:51:34] <marostegui>	 ok, that's the only pending thing then?
[08:52:11] <federico3>	 yep -  if you can test it and feed back the outcome in the pr it would be useful
[08:52:23] <marostegui>	 ok, I will use it now
[08:52:38] <marostegui>	 I think from what I can remember that was my only last thing pending, as the pooling/repooling issues were fixed right?
[08:58:25] <federico3>	 reviewing...
[09:00:04] <federico3>	 it should be ok
[09:01:05] <marostegui>	 yeah it worked fine
[09:01:16] <marostegui>	 I will use it a few times today and I will report back on the patch
[09:01:25] <marostegui>	 But for now only the downtime thing is needed
[09:01:58] <federico3>	 d'you need it now?
[09:03:52] <federico3>	 currently the script also removes the downtime before pool-in
[09:04:52] <marostegui>	 no, it is fine for now
[09:05:13] <marostegui>	 It is not blocking me, but I will comment at the end of the usage, to make it optional so we can merge and close the task, one less hanging fruit
[09:10:12] <marostegui>	 federico3: db1259 isn't pooled in dump/vslow, please pool it there too
[09:19:25] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: ferm.service on es1043:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[09:20:54] <federico3>	 marostegui will do - meanwhile the es* upgrade is running: https://phabricator.wikimedia.org/P79064   
[09:21:41] <marostegui>	 nice
[09:21:51] <marostegui>	 please check the above alert which I assume is related to those?
[09:24:25] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: ferm.service on es1043:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[09:24:38] <federico3>	 yes it was just updated - I'm a bit puzzled as the host is silenced during the reboot and it does wait for icinga to be optimal
[09:25:41] <federico3>	 icinga is now showing   for a raid controller CHECK_NRPE STATE UNKNOWN: Socket timeout after 10 seconds    
[09:26:26] <marostegui>	 it will need to recheck 
[09:28:08] <federico3>	 the ferm service is running ok, so maybe we should trigger a full recheck and also give it a 10 minutes bake in time for good measure
[09:28:57] <marostegui>	 sure
[09:28:59] <marostegui>	 that works
[09:39:17] <federico3>	 marostegui: is 1m a safe enough time to drain teh RO hosts after depool?
[09:39:52] <marostegui>	 normally yes, but we can give it 3 just to be fully sure
[09:39:58] <federico3>	 itś a bit slow at showing the traffic drop on prometehus
[09:40:09] <marostegui>	 just check via show processlist
[09:55:23] <federico3>	 aha, again ferm is failing to restart quickly
[10:03:44] <federico3>	 hum, there's another issue with ferm: ERROR ferm input drop default policy not set, ferm might not have been started correctly 
[10:11:21] <marostegui>	 federico3: if you run puppet it will most likely take caer of it
[10:17:50] <federico3>	 marostegui: thanks it worked
[10:54:18] <federico3>	 marostegui: I'm doing fully hand-off es* upgrades with  https://gitlab.wikimedia.org/repos/data_persistence/dbtools/scripts/-/merge_requests/7  with master flip and flip back. If you want to try it I can stop it and leave some hosts 
[10:59:36] <marostegui>	 federico3: I'm a bit busy at the moment but I can try tomorrow 
[11:30:57] <jynus>	 one question for the dbas, I heard yesterday you are expanding es6 and es7, will they have more replicas per cluster?
[11:38:56] <marostegui>	 One more
[11:39:00] <marostegui>	 They are already installed 
[11:39:18] <marostegui>	 Es6 one is serving, the es7 will be pooled later today 
[11:41:39] <jynus>	 thanks
[12:29:39] <Amir1>	 marostegui: once you pooled the extra replica, would you mind setting the weight on master to zero? It has been causing a lot of warnings (connection to master on GET) in logstash 
[12:50:51] <marostegui>	 Amir1: yep
[12:50:55] <marostegui>	 Also, you were on holidays?
[12:55:28] <Amir1>	 I am. Until end of July but annoying you knows no boundaries <3
[13:29:07] <jinxer-wm>	 FIRING: SystemdUnitFailed: swift_rclone_sync.service on ms-be1069:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[13:41:40] <jynus>	 I will be disabling alerts on m3 replicas before the phab maintenance and stop replication
[13:43:43] <jynus>	 maybe on the full cluster (?) as if they do alerts it could lag
[15:06:21] <sobanski>	 Thanks jynus
[15:15:31] <jynus>	 I have a backup from July 15, 2025, 5:48 a.m., July 15, 2025, 1:01 p.m. and July 15, 2025, 2:32 p.m. (still ongoing)
[15:20:16] <jynus>	 and I still can recover any time to the second (or transaction) in the last 3 months
[15:42:47] <sobanski>	 jynus: if you're still around the upgrade has finished and we're OK to restart replication
[15:44:59] <jynus>	 ok
[15:45:02] <jynus>	 let me do it
[15:49:07] <brennen>	 jynus: ty!
[15:49:18] <jynus>	 once they catch up I will remove the downtime
[15:55:49] <jynus>	 lots of file attachment deletions, that I can see?
[16:25:37] <federico3>	 lookng
[16:29:07] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter.service on es1032:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[17:05:14] <federico3>	 (again I see   Not all services are recovered: es1032:MariaDB read only es1,MegaRAID  )
[17:08:17] <mutante>	 yea, but it's depooled, there are no errors for MW and the downtime has been extended. DBA can look in their morning
[17:08:43] <mutante>	 the wmf_auto_restart thing is the usual indirect consequence
[17:08:52] <mutante>	 which we can also ACK.. doing
[17:10:28] <federico3>	 I'm finishing an update and it's healthy now
[17:10:55] <mutante>	 ok, could not even find it in that alertmanager link
[17:11:11] <mutante>	 update of what?
[17:11:27] <federico3>	 an OS/kernel update
[17:11:46] <mutante>	 oh, so this is what started the alert in the first place? interesting
[17:13:11] <federico3>	 the update is part of https://phabricator.wikimedia.org/T395241  now automated with https://gitlab.wikimedia.org/repos/data_persistence/dbtools/scripts/-/merge_requests/7    - the MegaRAID glitch seems to show up occasionally but it's not an issue
[17:13:35] <mutante>	 alright then. maybe just a tiny follow-up; downtime before planned maintenance would be ideal to avoid pages
[17:13:54] <mutante>	 thanks for the task link
[17:16:28] <federico3>	 yes, sorry for the noise (the script does the downtiming and downtime removal, I missed the expiration of the manual silence) 
[17:19:10] <mutante>	 no worries. thanks
[17:19:36] <mutante>	 glad we know the reason and nothing to worry now
[20:29:07] <jinxer-wm>	 FIRING: SystemdUnitFailed: swift_rclone_sync.service on ms-be1069:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed