[15:33:05] gilles: one min :) [16:22:31] Krinkle: I amended https://gerrit.wikimedia.org/r/c/mediawiki/core/+/661452 [17:00:11] Amir1: I added a "relative change" panel to https://grafana-rw.wikimedia.org/d/BvWJlaDWk/startup-manifest-size [17:00:23] it's not great in terms of what it shows, but at least it helps find which one changed [17:00:34] I couldn't find a better way to diff them, but maybe you know of one :) [17:01:08] I used offsetToZero and then maximumAbove(0) [17:01:37] I try to think of something [17:02:20] there's also a confusing 0 -> 1 at the start when I do it this way because the data is not every minute [17:02:48] offsetToZero makes the first null a zero [17:05:02] commonswiki still significantly bigger (+20% or so to enwiki) [17:05:59] 50k vs 60k [17:06:03] about 7k is Wikibase [17:06:08] 4k [17:06:19] if you want to be over time, the derivative would be nice https://grafana.wikimedia.org/d/000000156/wikidata-dispatch?viewPanel=20&orgId=1&refresh=1m&from=now-6h&to=now [17:06:26] 4k Wikibase, 1k Translate, 0.5k WikibaseMediaInfo [17:06:37] of course in a bigger timeframe [17:06:45] hm.. derivative should work, Ill try that [17:06:53] I was searching for diff and didn't find this one, makes sense [17:07:12] diffSeriesLists looked usable, but derivative should be it yeah thats what I'm looking for [18:07:41] Amir1: nice that worked out quite well! [18:08:16] did it as one instead of two (no pos/neg split) so that there's not two entries given this is per-component [18:08:40] \o/ [21:52:29] phedenskog: There is a "Source" link at the bottom of the alertmanager emails [21:52:38] which goes to the specific Grafana alert panel [21:52:46] e.g. https://grafana.wikimedia.org/d/000000402/resourceloader-alerts?tab=alert&viewPanel=22&orgId=1 [21:53:00] took me a while to notice :) [21:53:11] but that means dashboard probably doesn't need to be specified by hand [21:54:21] did we talk with sre about, or file a task for, to have the subject not vary between fire and resolve? [21:54:25] so that it is one thread