[01:01:00] FIRING: MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ...
[01:01:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ...
[01:01:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag
[01:06:00] RESOLVED: MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ...
[01:06:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ...
[01:06:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag
[02:25:41] FIRING: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ...
[02:25:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ...
[02:25:44] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag
[02:30:00] FIRING: [2x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ...
[02:30:00] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ...
[02:30:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag
[02:30:41] RESOLVED: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ...
[02:30:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ...
[02:30:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag
[02:35:00] FIRING: [4x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ...
[02:35:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ...
[02:35:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag
[02:40:00] FIRING: [4x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ...
[02:40:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ...
[02:40:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag
[02:45:01] FIRING: [4x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ...
[02:45:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ...
[02:45:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag
[03:00:01] FIRING: [3x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ...
[03:00:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ...
[03:00:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag
[03:05:00] RESOLVED: [2x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ...
[03:05:00] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ...
[03:05:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag
[05:36:05] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly
[05:41:05] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly
[06:27:20] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Add new API rate limiting fields from webrequest_logs to Turnilo view - https://phabricator.wikimedia.org/T419736#11897045 (10JAllemandou) @amastilovic Please note the latest patch from CDanis: field names have changed! we need to update the webrequest table...
[06:50:35] 06Data-Engineering, 10Pageviews-Anomaly: "Nahui Ollin" is enwiki's #1 article. Never heard of it? That's the problem - https://phabricator.wikimedia.org/T425600#11897055 (10Joe) This is clearly some bot (or set of bots) requesting that page again and again. The requests are almost all coming from ISPs notoriou...
[07:11:43] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10MWH-Incremental: Accelerate sqoop landing for MediaWiki History private tables - https://phabricator.wikimedia.org/T424355#11897098 (10APizzata-WMF) >I can test this and come back with the results! this took roughly 2 hours (started at `2026-05-06T15:55:...
[07:25:34] (03CR) 10Joal: "Comments about naming-consistency and caonoical table usage" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1279651 (https://phabricator.wikimedia.org/T421743) (owner: 10Snwachukwu)
[09:07:17] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: Relative Trending - Flink app for page_view - https://phabricator.wikimedia.org/T425624 (10JMonton-WMF) 03NEW
[09:42:46] 06Data-Engineering, 10Pageviews-Anomaly: "Nahui Ollin" is enwiki's top-viewed article in April 2026 - https://phabricator.wikimedia.org/T425600#11897512 (10Aklapper)
[10:54:02] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Product Safety and Integrity (Sprint lily-of-the-valley (May 4 - May 22)): Backfill via AirFlow for time_to_revert_bad_faith_edits - https://phabricator.wikimedia.org/T425526#11897868 (10Tchanders)
[11:03:05] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly
[11:13:05] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly
[11:59:29] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Product-Analytics, 06Product Safety and Integrity (Sprint lily-of-the-valley (May 4 - May 22)): Backfill via AirFlow for time_to_revert_bad_faith_edits - https://phabricator.wikimedia.org/T425526#11898236 (10kostajh)
[12:20:58] 06Data-Engineering, 07Epic: [Epic] KAPOW: The next generation of bot detection in the Data Platform. - https://phabricator.wikimedia.org/T425661 (10GGoncalves-WMF) 03NEW
[12:23:49] 06Data-Engineering: Client-side signals for 100% of enwiki traffic are usable in for bot detection. - https://phabricator.wikimedia.org/T425663 (10GGoncalves-WMF) 03NEW
[12:26:22] 06Data-Engineering: Generate input KAPOW signals tables. - https://phabricator.wikimedia.org/T425664 (10GGoncalves-WMF) 03NEW
[12:27:50] 06Data-Engineering: Develop the new classification algorithm. - https://phabricator.wikimedia.org/T425665 (10GGoncalves-WMF) 03NEW
[14:30:05] !log Deploying Refinery at 4734c67 for weekly deployment train
[14:30:07] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[15:01:17] !log Deployed refinery using scap, then deployed onto hdfs
[15:01:18] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[15:17:55] 06Data-Engineering, 06Data-Engineering-Radar, 06Growth-Team, 10MediaWiki-extensions-WikimediaEvents, and 4 others: Could not hoist data into experiment.subject_id for event - https://phabricator.wikimedia.org/T421152#11899286 (10phuedx) I've moved this into **BLOCKED** because {T425096} and {T419513} are r...
[15:27:06] 06Data-Engineering, 06Data-Engineering-Radar, 06Growth-Team, 10MediaWiki-extensions-WikimediaEvents, and 4 others: Could not hoist data into experiment.subject_id for event - https://phabricator.wikimedia.org/T421152#11899322 (10phuedx)
[15:47:52] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: mediawiki.page_change.v1 event - add a page type field - https://phabricator.wikimedia.org/T409462#11899446 (10Ahoelzl) p:05Triage→03High a:03tchin
[16:11:32] (03PS1) 10Aleksandar Mastilovic: Add two new rate-limiting value fields to Druid's Kafka supervisor JSON [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1284700 (https://phabricator.wikimedia.org/T419736)
[17:25:34] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): `mw_content_reconcile_mw_content_history_daily`: NoClassDefFoundError(EventStreamFactory) in spark_emit_reconcile_events_to_kafka - https://phabricator.wikimedia.org/T425569#11899927 (10xcollazo) a:03AKhatun_WMF
[17:30:25] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10MWH-Incremental, 13Patch-For-Review: Accelerate sqoop landing for MediaWiki History private tables - https://phabricator.wikimedia.org/T424355#11899937 (10xcollazo) >>! In T424355#11895295, @JAllemandou wrote: > From the DB graphs, it seems we could pa...
[17:33:42] (03CR) 10Joal: [V:03+2 C:03+2] "LGTM! Merging" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1284700 (https://phabricator.wikimedia.org/T419736) (owner: 10Aleksandar Mastilovic)
[17:38:55] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 13Patch-For-Review: Add new API rate limiting fields from webrequest_logs to Turnilo view - https://phabricator.wikimedia.org/T419736#11899952 (10JAllemandou) >>! In T419736#11898635, @CDanis wrote: > FWIW I kept the names unchanged on the past-Benthos sid...
[18:08:46] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests: Requesting access to analytics_privatedata_users and SQL Lab for AnnieKim_WMDE - https://phabricator.wikimedia.org/T420500#11900078 (10Dzahn)
[18:10:12] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests: Requesting access to analytics_privatedata_users and SQL Lab for AnnieKim_WMDE - https://phabricator.wikimedia.org/T420500#11900085 (10Dzahn)
[18:20:20] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests: Requesting access to analytics_privatedata_users and SQL Lab for AnnieKim_WMDE - https://phabricator.wikimedia.org/T420500#11900140 (10Dzahn) >>! In T420500#11886631, @AnnieKim_WMDE wrote: > Hello! Is there anything else I can or n...
[18:28:16] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests: Requesting access to analytics_privatedata_users and SQL Lab for AnnieKim_WMDE - https://phabricator.wikimedia.org/T420500#11900153 (10Dzahn) >>! In T420500#11832838, @Martyn.ranyard wrote: > I think kerberos is automatically enabl...
[18:30:34] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests: Requesting access to analytics_privatedata_users and SQL Lab for AnnieKim_WMDE - https://phabricator.wikimedia.org/T420500#11900170 (10Dzahn) 05Open→03In progress
[18:30:59] (03PS1) 10Aleksandar Mastilovic: Add two new rate-limiting fields to Druid load webrequest_sampled [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1284776 (https://phabricator.wikimedia.org/T419736)
[18:31:25] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Migrate generated-data-platform-aqs Docker images away from Debian Bullseye - https://phabricator.wikimedia.org/T425310#11900179 (10Snwachukwu) The following AQS services docker images have been updated to Trixie: ` Edit-analytics Geo-analytics Device-anal...
[18:31:35] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Migrate generated-data-platform-aqs Docker images away from Debian Bullseye - https://phabricator.wikimedia.org/T425310#11900181 (10Snwachukwu) Done
[19:30:22] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10MWH-Incremental: Implement MERGE INTO writers for mediawiki_history_incremental_v1 - https://phabricator.wikimedia.org/T425729 (10xcollazo) 03NEW
[19:36:38] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10MWH-Incremental: Airflow DAGs for mediawiki_history_incremental_v1 writers - https://phabricator.wikimedia.org/T425730 (10xcollazo) 03NEW
[19:37:55] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10MWH-Incremental: Spike: can analytics-refinery-source build Spark 3.5 artifacts? - https://phabricator.wikimedia.org/T425474#11900361 (10xcollazo) 05Open→03Resolved
[19:45:03] (03CR) 10Joal: [C:03+1] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1284776 (https://phabricator.wikimedia.org/T419736) (owner: 10Aleksandar Mastilovic)
[19:51:01] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10MWH-Incremental: Quality verification for mediawiki_history_incremental_v1 using Iceberg time travel - https://phabricator.wikimedia.org/T425734 (10xcollazo) 03NEW
[19:51:39] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10MWH-Incremental: Quality verification for mediawiki_history_incremental_v1 using Iceberg time travel - https://phabricator.wikimedia.org/T425734#11900439 (10xcollazo) a:05xcollazo→03None
[19:56:02] 06Data-Engineering, 06Movement-Insights: interwiki imports and its effects on revision data - https://phabricator.wikimedia.org/T425735 (10CMyrick-WMF) 03NEW
[19:58:58] (03CR) 10Aleksandar Mastilovic: [V:03+2 C:03+2] Add two new rate-limiting fields to Druid load webrequest_sampled [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1284776 (https://phabricator.wikimedia.org/T419736) (owner: 10Aleksandar Mastilovic)
[20:36:37] (03PS1) 10Xcollazo: Add MWHistoryDeltaWriter and MWHistorySnapshotMerger to refinery-job-35 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1284858 (https://phabricator.wikimedia.org/T424350)
[20:38:00] (03PS2) 10Xcollazo: Add MWHistoryDeltaWriter and MWHistorySnapshotMerger to refinery-job-35 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1284858 (https://phabricator.wikimedia.org/T425729)
[20:40:12] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10MWH-Incremental: Implement MERGE INTO writers for mediawiki_history_incremental_v1 - https://phabricator.wikimedia.org/T425729#11900653 (10xcollazo) CC @JAllemandou
[20:52:18] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10MWH-Incremental: Implement MERGE INTO writers for mediawiki_history_incremental_v1 - https://phabricator.wikimedia.org/T425729#11900670 (10xcollazo) For some reason first patchset is not attaching, but here it is: https://gerrit.wikimedia.org/r/c/analyti...