[12:08:25] FIRING: SystemdUnitFailed: arclamp_generate_svgs.service on arclamp2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:18:25] RESOLVED: SystemdUnitFailed: arclamp_generate_svgs.service on arclamp2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:22:25] FIRING: [2x] SystemdUnitFailed: arclamp_compress_logs.service on arclamp2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:32:25] FIRING: [2x] SystemdUnitFailed: arclamp_compress_logs.service on arclamp2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:37:25] FIRING: [2x] SystemdUnitFailed: arclamp_compress_logs.service on arclamp2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:47:25] FIRING: [2x] SystemdUnitFailed: arclamp_compress_logs.service on arclamp2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:02:25] RESOLVED: [2x] SystemdUnitFailed: arclamp_compress_logs.service on arclamp2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:04:51] Hi! We'd like to enable ingestion of the w3c.reportingapi.network_error stream into hive, and the production of canary events into that stream. ottomata alluded to the fact that it would require a logstash filter dropping all events with `meta.domain == 'canary'` (cf https://phabricator.wikimedia.org/T304373#11229932). I started having a look in [13:04:51] https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/production/modules/profile/files/logstash/filters/, but I'm not sure how to proceed. Would anyone be available to assist? Thanks! [14:59:31] brouberol: I can help with that. How does NEL data get into Hive? [15:01:33] Possibly I misunderstand. Are canary events synthetic and would pollute the NEL data if they weren't dropped by logstash? [15:02:11] that's exactly right [15:03:09] the way data is ingested into hive is through https://airflow.wikimedia.org/dags/refine_to_hive_hourly/grid?search=refine_to_hive_hourly, that collects the streams to ingest by performing a request on meta.wikimedia.org, which basically returns all streams with analytics_hive_ingestion.enabled => true in [15:03:09] https://gerrit.wikimedia.org/r/plugins/gitiles/operations/mediawiki-config/%2B/refs/heads/master/wmf-config/ext-EventStreamConfig.php#1188 [15:03:44] you'll notice that there's a `canary_events_enabled' => false/true` config for each stream, that ottomata recommends we enable for w3c.reportingapi.network_error [15:04:02] however, as you mention, it would pollute the logstash data if we didn't drop it through a logstash filter [15:04:35] which is supposed to be dropping all events with `meta.domain == 'canary'` [15:06:59] Understood. Thanks! I'll put together a patch. :) [15:16:00] Thanks! [15:25:55] cwhite: could you attach the patch to T304373? [15:25:56] T304373: Also intake Network Error Logging events into the Analytics Data Lake - https://phabricator.wikimedia.org/T304373 [15:26:08] Will do :) [15:45:49] tx! [15:46:23] cdanis: when ^this^ is done, you should be able to enable hive ingestion and canary events for the stream