[02:35:00] FIRING: SystemdUnitFailed: swift_dispersion_stats_lowlatency.service on ms-fe2009:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:45:00] RESOLVED: SystemdUnitFailed: swift_dispersion_stats_lowlatency.service on ms-fe2009:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:31:01] thanks for the quick turnaround on cleaning up sockpuppet! <3 [10:42:15] I am retrying some backups that failed during the night (probably due to extra load due to schema changes?) [12:17:47] hnowlan: we are always happy when it comes to delete things [12:26:20] Delete All The Things XD [14:11:50] FYI (mostly for InfraFun) Ive installed a couple of extra packages on a production host to test binary diffs against production data [14:12:01] to see if they would benefit some pipelines [14:12:30] also in case someone would be interested on my findings [16:00:00] FIRING: [6x] SystemdUnitFailed: dup-uefi.service on ms-be1087:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:05:00] FIRING: [16x] SystemdUnitFailed: dup-uefi.service on ms-be1087:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:06:44] o.O [16:07:40] jhathaway: the new dup-uefi service is broken on ms-be* [16:07:59] jhathaway: Dec 03 15:53:56 ms-be1087 dup-uefi[2030488]: lsblk: unrecognized option '--shell' [16:08:42] Emperor: thanks, looks like the current version doesn't work on bullseye, working on patch [16:08:48] *a patch [16:09:01] jhathaway: cool, thanks. I'll silence the alerts for a bit. [16:10:00] FIRING: [21x] SystemdUnitFailed: dup-uefi.service on ms-be1087:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:12:16] {{done}}, hopefully [16:41:19] Emperor: should now be fixed [16:47:10] thanks, checking [16:48:14] yeah, recovers once puppet runs, thanks.