[00:08:35] RESOLVED: DiskSpace: Disk space centrallog1002:9100:/srv 3.99% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=centrallog1002 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [16:58:38] FIRING: StatsvThroughput: StatsV is not ingesting metrics - https://wikitech.wikimedia.org/wiki/Performance.wikimedia.org/Runbook#statsv - https://grafana.wikimedia.org/d/ba06cb37-dfab-40ae-8e61-0710522881e0/statsv - https://alerts.wikimedia.org/?q=alertname%3DStatsvThroughput [19:09:00] cjd91, blblack: could one of you have a look at statsv on webperf1003? When ^^ happens, the systemd service needs restarted. If you're inclined, a peek at the journald logs would be helpful. I'm away from the laptop right now or I'd do it. [19:11:20] Only escalating it because this is a metrics data loss event for a subset of metrics. [20:58:53] FIRING: StatsvThroughput: StatsV is not ingesting metrics - https://wikitech.wikimedia.org/wiki/Performance.wikimedia.org/Runbook#statsv - https://grafana.wikimedia.org/d/ba06cb37-dfab-40ae-8e61-0710522881e0/statsv - https://alerts.wikimedia.org/?q=alertname%3DStatsvThroughput [23:13:35] FIRING: DiskSpace: Disk space centrallog1002:9100:/srv 3.965% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=centrallog1002 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace