[11:35:10] <hnowlan>	 herron: is there a specific log stream that's causing that impact? 
[11:35:17] <hnowlan>	 seems like something specific is running away with it
[12:31:53] <effie>	 one thing I can see is that there are many translation problems https://logstash.wikimedia.org/goto/d67b3d9660ccfb0a020e8a7a0c2900ad
[12:32:14] <effie>	 but this volume seems is not enough for what you are observing 
[12:32:50] <effie>	 but if it is only in codfw, we could first maybe look at mw-crons or mw-script?
[12:35:39] <effie>	 for example https://logstash.wikimedia.org/goto/6eafed83016ae7c7b4e8ee668293de24
[15:38:03] <herron>	 csp-report-only looks to match the rough timeframe https://usercontent.irccloud-cdn.com/file/dEi6eDdQ/Screenshot%202026-02-06%20at%2010.35.58%20AM.png
[16:28:22] <hnowlan>	 woooow heh
[16:57:30] <hnowlan>	 I can't see any changes that would cause such a significant increase, we really only configure csp-report-only in a handful of places 
[17:07:37] <hnowlan>	 I've asked #mediawiki-engineering-teams about it
[18:37:00] <herron>	 thanks!
[18:38:48] <cdanis>	 if anyone is in need of queries that make titan1002 unhappy, I have no shortage of them
[18:51:42] <herron>	 haha, shall we start a paste?
[18:52:24] <cdanis>	 i think it's basically always associated with high cardinality (or massively-high cardinality once you're looking at a long time window)
[18:52:30] <cdanis>	 which I don't think is anything we didn't already know?
[20:56:12] <jinxer-wm>	 FIRING: ThanosCompactHalted: Thanos Compact has failed to run and is now halted. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/651943d05a8123e32867b4673963f42b/thanos-compact - https://alerts.wikimedia.org/?q=alertname%3DThanosCompactHalted
[21:16:12] <jinxer-wm>	 RESOLVED: ThanosCompactHalted: Thanos Compact has failed to run and is now halted. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/651943d05a8123e32867b4673963f42b/thanos-compact - https://alerts.wikimedia.org/?q=alertname%3DThanosCompactHalted
[21:17:02] <herron>	 ^^ https://phabricator.wikimedia.org/T416745
[21:36:12] <jinxer-wm>	 FIRING: ThanosCompactHalted: Thanos Compact has failed to run and is now halted. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/651943d05a8123e32867b4673963f42b/thanos-compact - https://alerts.wikimedia.org/?q=alertname%3DThanosCompactHalted
[21:56:12] <jinxer-wm>	 RESOLVED: ThanosCompactHalted: Thanos Compact has failed to run and is now halted. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/651943d05a8123e32867b4673963f42b/thanos-compact - https://alerts.wikimedia.org/?q=alertname%3DThanosCompactHalted