[00:21:40] FIRING: [2x] SystemdUnitFailed: curator_actions_cluster_wide.service on logging-sd2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:46:25] FIRING: [2x] SystemdUnitFailed: curator_actions_cluster_wide.service on logging-sd2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:11:25] RESOLVED: SystemdUnitFailed: mw-log-cleanup.service on mwlog2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:50:35] FIRING: DiskSpace: Disk space mwlog1003:9100:/srv 3.971% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=mwlog1003 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [19:05:35] RESOLVED: DiskSpace: Disk space mwlog1003:9100:/srv 3.864% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=mwlog1003 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [20:30:34] FIRING: DiskSpace: Disk space mwlog2003:9100:/srv 3.971% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=mwlog2003 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [20:50:34] FIRING: [2x] DiskSpace: Disk space mwlog1003:9100:/srv 3.961% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [21:05:34] FIRING: [2x] DiskSpace: Disk space mwlog1003:9100:/srv 3.923% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [21:25:34] RESOLVED: DiskSpace: Disk space mwlog1003:9100:/srv 3.968% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=mwlog1003 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [22:05:34] FIRING: DiskSpace: Disk space mwlog1003:9100:/srv 3.976% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=mwlog1003 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [22:35:35] FIRING: [2x] DiskSpace: Disk space mwlog1003:9100:/srv 3.841% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [23:03:48] FIRING: PuppetFailure: Puppet has failed on logging-hd2005:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [23:13:48] RESOLVED: PuppetFailure: Puppet has failed on logging-hd2005:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure