[00:00:08] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-esams, and 3 others: ESAMS 502 broken pipe connection issues - https://phabricator.wikimedia.org/T415473#11552156 (10ssingh) Thank for reporting. Can you please try again from your home connection (without VPN) and let us know if that works? We depool... [00:01:28] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-esams, and 3 others: ESAMS 502 broken pipe connection issues - https://phabricator.wikimedia.org/T415473#11552158 (10AlexisJazz) Works now, thanks! [00:07:36] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-esams, and 3 others: ESAMS 502 broken pipe connection issues - https://phabricator.wikimedia.org/T415473#11552161 (10AlexisJazz) {F71607077} There was a visible dip in editing and surge in error responses. [01:19:27] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-esams, and 3 others: ESAMS 502 broken pipe connection issues - https://phabricator.wikimedia.org/T415473#11552172 (10ssingh) We had a transient link failure between eqiad and esams that resulted in this issue. It should be resolved now, and esams is p... [01:29:41] FIRING: [9x] SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:33:42] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-esams, and 3 others: ESAMS 502 broken pipe connection issues - https://phabricator.wikimedia.org/T415473#11552173 (10Peachey88) [05:29:41] FIRING: [9x] SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:29:41] FIRING: [9x] SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:29:41] FIRING: [9x] SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:29:41] FIRING: [9x] SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:29:41] FIRING: [9x] SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed