[00:32:25] <jinxer-wm>	 FIRING: SystemdUnitFailed: prometheus-nft-throttling-denylist.service on durum7003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[04:32:40] <jinxer-wm>	 FIRING: SystemdUnitFailed: prometheus-nft-throttling-denylist.service on durum7003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[04:39:10] <wikibugs>	 06Traffic, 10MobileFrontend, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: MobileFrontend should declare "X-Subdomain" variance via "Vary" response header - https://phabricator.wikimedia.org/T390929#10930411 (10Krinkle)
[05:36:58] <_joe_>	 looks like ocsp stapling doesn't work
[07:21:59] <vgutierrez>	 _joe_: hmmm care to provide more details?
[07:22:02] <vgutierrez>	 DC?
[07:22:51] <vgutierrez>	 if it's a US DC using Let's Encrypt that's more than expected, Let's Encrypt decommissioned their OCSP infrastructure a few months ago, see https://letsencrypt.org/2024/12/05/ending-ocsp/
[07:29:06] <vgutierrez>	 and in non-US DCs where we are using Google Trust Services OCSP stapling is working as expected:  https://www.irccloud.com/pastebin/LNAaTkdM/
[07:45:39] <vgutierrez>	 oh, it triggered a new alert during the night
[07:56:50] <_joe_>	 vgutierrez: yes
[08:26:44] <vgutierrez>	 I got a CR ready to address it, I'll wait till sukhe gets online to review it
[08:26:50] <vgutierrez>	 fab.fur is OoO today :)
[08:27:11] <vgutierrez>	 sukhe: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1161397 that's for you :) 
[08:28:46] <vgutierrez>	 and I'll simplify all the certificate picking logic soon given that we are getting rid of digicert so all the certs will be managed by acmechief
[08:32:40] <jinxer-wm>	 FIRING: SystemdUnitFailed: prometheus-nft-throttling-denylist.service on durum7003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[08:45:36] <fabfur>	 no need to wait for sukhe, I reviewed it (plumbers finished and I'm waiting for the dentist appointment)
[08:49:33] <vgutierrez>	 fabfur: thx <3
[08:49:45] <vgutierrez>	 plumbers and dentist on the same day...
[08:49:49] <vgutierrez>	 that's torture
[08:52:00] <wikibugs>	 06Traffic, 06SRE, 13Patch-For-Review: Research and respond to Let's Encrypt's intent to deprecate OCSP in favour of CRLs - https://phabricator.wikimedia.org/T370821#10931172 (10Vgutierrez) 05In progress→03Resolved a:03Vgutierrez
[08:59:43] <fabfur>	 vgutierrez: the important thing is that the plumber doesn't put it's hands in my mouth
[09:00:07] <vgutierrez>	 [SFW filter blocked this response]