[14:27:37] For the past couple of weeks, the data-platform SRE team has received alerts when an admin_ng change hits HEAD and goes undeployed to the dse-k8s-eqiad k8s cluster for > 24h. The alert is configured here [14:27:37] https://gerrit.wikimedia.org/r/plugins/gitiles/operations/alerts/+/9fdb19d32b27da5061e6bda8b87b65604bd7f257/team-data-platform/helmfile_admin_ng_pending_changes.yaml [14:28:14] The reason why I mention his is because jayme and _.joe_ mentioned we might want to setup such an alert for all k8s clusters [14:29:14] I'm not 100% sure what the best way to do this would be: either each team sets up a similar alert for "their" k8s clusters, or maybe we have a smarter way to do this. If any of you has an idea, I'm all ears! [14:30:52] the core idea is: we probably don't want to let pending changes accumulate for too long, as the more we wait, the more difficult it might be to properly evaluate the impact of changes, as we switch context, and potentially the bigger the diff size, and thus the blast radius [14:35:16] very nice! [14:35:45] probably an alert for each team would make life easier since we could independently "tune" how soon it alerts [14:35:56] it is the only thing that would be nice that I can think of [15:13:34] oh that's nice! [15:13:37] thanks! [15:31:23] yw :) [15:31:31] Can I let each team self-organize? [15:44:52] interpolating from historic data I'd say it won't happen then :p [15:45:40] can we have a generic alert and let teams override it when they want to? That's probably not possible, right? [15:54:15] hahaha, that's fair. As a compromise, I can try to replicate that alert for all clusters/team tuple as a starting point [15:54:26] and let people chime in [15:55:41] whether we can get a generic alert, I think the idea was to be able to route the alert wherever each team wanted, but we could have a generic alert going to, say, #-sre or wherever, and exclude some clusters from the generic alert scope when some teams want to route "their" alert somewhere else? [16:53:42] sgtm, thanks!