[01:01:52] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform, 13Patch-For-Review: mediawiki.page_change.v1 event - Add revision is revert field - https://phabricator.wikimedia.org/T423583#11892496 (10Ottomata) For my own reference: https://meta.wikimedia.org/wiki/Research:Revert [01:05:31] FIRING: [2x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [01:05:31] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [01:05:31] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [01:10:34] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10MWH-Incremental, 07Epic: Incremental MediaWiki History - https://phabricator.wikimedia.org/T424350#11892523 (10Ottomata) [01:10:35] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10MWH-Incremental, 10Event-Platform: Incremental MWH - MediaWiki event data source improvements - https://phabricator.wikimedia.org/T423935#11892524 (10Ottomata) [01:10:36] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform, 13Patch-For-Review: mediawiki.page_change.v1 event - Add revision is revert field - https://phabricator.wikimedia.org/T423583#11892522 (10Ottomata) [01:12:23] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform, 13Patch-For-Review: mediawiki.page_change.v1 event - Add revision is revert field - https://phabricator.wikimedia.org/T423583#11892528 (10Ottomata) @xcollazo I'm not 100% sure, but my brief understanding is: - if a revert was done via... [01:45:31] RESOLVED: MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [01:45:31] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [01:45:31] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [06:37:42] (03CR) 10Joal: [C:03+1] "LGTM! Merge at will" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1283113 (owner: 10Xcollazo) [06:39:55] (03CR) 10Joal: [C:03+1] "LGTM! Thanks for this :)" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1283087 (https://phabricator.wikimedia.org/T425474) (owner: 10Xcollazo) [07:18:22] 06Data-Engineering, 06Data-Platform-SRE (2026-04-24 - 2026-05-15), 07Epic, 13Patch-For-Review: Upgrade Spark to a version with long term Iceberg support, and with fixes to support Dumps 2.0 - https://phabricator.wikimedia.org/T338057#11892881 (10JAllemandou) After talking with @BTullis yesterday, I confirm... [07:35:34] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10AQS2.0: Introduce a new AQS endpoint to expose video plays - https://phabricator.wikimedia.org/T415202#11892923 (10simon04) The underlying analytics endpoint seems to be working nicely, for example https://wikimedia.org/api/rest_v1/metrics/video_plays/v3... [07:38:16] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10MWH-Incremental, 10Event-Platform, 13Patch-For-Review: Create mediawiki.user_change event stream - https://phabricator.wikimedia.org/T423952#11892934 (10JAllemandou) I commented on the schema, but I'm not comfortable reviewing the mediawiki code :) [07:52:47] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Data-Platform-SRE (2026-04-24 - 2026-05-15): Investigate Gobblin failures - https://phabricator.wikimedia.org/T419436#11893045 (10JAllemandou) 05Open→03Resolved No recent failures. Closing for now. [07:56:23] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10AQS2.0: Introduce a new AQS endpoint to expose video plays - https://phabricator.wikimedia.org/T415202#11893066 (10simon04) Hi @Ladsgroup, I debugged your tool a bit: You seem to be using the `imageinfo...url` to compute the filepath. But the newly added... [08:45:35] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: Relative Trending - Design document - https://phabricator.wikimedia.org/T425421#11893296 (10JMonton-WMF) Here is a proposal for the implementation: https://docs.google.com/document/d/1tTRH3lGaHWFJIgdTx4W4dsV7b42ejF2gnpXG001Qmj0/edit?tab=t... [09:54:55] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Product Safety and Integrity (Sprint lily-of-the-valley (May 4 - May 22)): Backfill via AirFlow for time_to_revert_bad_faith_edits - https://phabricator.wikimedia.org/T425526 (10Tchanders) 03NEW [09:55:33] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Product Safety and Integrity (Sprint lily-of-the-valley (May 4 - May 22)): Backfill via AirFlow for time_to_revert_bad_faith_edits - https://phabricator.wikimedia.org/T425526#11893529 (10Tchanders) [10:04:35] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10MWH-Incremental: Accelerate sqoop landing for MediaWiki History private tables - https://phabricator.wikimedia.org/T424355#11893543 (10APizzata-WMF) The run of `/usr/local/bin/refinery-sqoop-mediawiki-production-history` took about **2 hours**, from `202... [10:16:48] 06Data-Engineering, 06Data-Engineering-Icebox, 06DBA, 13Patch-For-Review: Move Mostcategories computation to Hadoop - https://phabricator.wikimedia.org/T413362#11893593 (10Zabe) @xcollazo the DAG failed. Could you tell me what the underlying error is? ` [2026-05-04, 18:44:58 UTC] {taskinstance.py:3337} ER... [11:43:29] 06Data-Engineering, 06Data-Engineering-Radar, 10Event-Platform, 06Machine-Learning-Team (Q4 FY2025-26), 13Patch-For-Review: Add Multilingual RevertRisk predictions to mediawiki.page_revert_risk_prediction_change - https://phabricator.wikimedia.org/T415892#11893754 (10isarantopoulos) @gkyziridis since the... [12:54:13] (03PS1) 10Gkyziridis: expand_event_sanitized_analytics_allowlist: Add revertrisk-multilingual predictions to allowlist. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1283749 (https://phabricator.wikimedia.org/T415892) [13:21:54] 06Data-Engineering, 06Wikimedia Enterprise: PageViews S3 Data Transfer MR [Enterprise] - https://phabricator.wikimedia.org/T425543 (10LDlulisa-WMF) 03NEW [13:22:05] 06Data-Engineering, 06Wikimedia Enterprise: PageViews S3 Data Transfer MR [Enterprise] - https://phabricator.wikimedia.org/T425543#11894127 (10LDlulisa-WMF) [13:24:34] (03CR) 10Xcollazo: [C:03+2] Add .DS_Store, logs/, and *.bak to .gitignore [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1283113 (owner: 10Xcollazo) [13:24:45] (03CR) 10Xcollazo: [V:03+2 C:03+2] Add .DS_Store, logs/, and *.bak to .gitignore [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1283113 (owner: 10Xcollazo) [13:25:20] (03CR) 10Xcollazo: [V:03+2 C:03+2] Spike: refinery-job-35 submodule compiles against Spark 3.5.8 + Iceberg 1.10.1 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1283087 (https://phabricator.wikimedia.org/T425474) (owner: 10Xcollazo) [13:37:04] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data-Engineering-Jupyter, 06Data-Platform-SRE, 06Product-Analytics: Remove anaconda-wmf package from the cluster - https://phabricator.wikimedia.org/T337963#11894187 (10BTullis) 05Open→03Resolved a:03BTullis This has already been achieved through... [13:37:17] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data-Engineering-Jupyter, 06Product-Analytics, 06Data-Platform-SRE (2026-04-24 - 2026-05-15): Remove anaconda-wmf package from the cluster - https://phabricator.wikimedia.org/T337963#11894191 (10BTullis) [13:41:15] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10MWH-Incremental, 10Event-Platform, 13Patch-For-Review: Create mediawiki.user_change event stream - https://phabricator.wikimedia.org/T423952#11894199 (10xcollazo) >>! In T423952#11892934, @JAllemandou wrote: > I commented on the schema, but I'm not c... [13:41:32] 06Data-Engineering, 06Data-Engineering-Radar, 10Event-Platform, 06Machine-Learning-Team (Q4 FY2025-26), 13Patch-For-Review: Add Multilingual RevertRisk predictions to mediawiki.page_revert_risk_prediction_change - https://phabricator.wikimedia.org/T415892#11894201 (10gkyziridis) - I configured the rest o... [13:43:20] (03Merged) 10jenkins-bot: Add .DS_Store, logs/, and *.bak to .gitignore [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1283113 (owner: 10Xcollazo) [13:45:02] (03Merged) 10jenkins-bot: Spike: refinery-job-35 submodule compiles against Spark 3.5.8 + Iceberg 1.10.1 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1283087 (https://phabricator.wikimedia.org/T425474) (owner: 10Xcollazo) [13:50:14] (03CR) 10Ottomata: [C:03+1] expand_event_sanitized_analytics_allowlist: Add revertrisk-multilingual predictions to allowlist. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1283749 (https://phabricator.wikimedia.org/T415892) (owner: 10Gkyziridis) [14:04:21] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10MWH-Incremental: Accelerate sqoop landing for MediaWiki History private tables - https://phabricator.wikimedia.org/T424355#11894365 (10xcollazo) > `content`: 261.6 G Surprising size considering `content` actually [[ https://www.mediawiki.org/wiki/Manual:... [14:11:44] 06Data-Engineering, 06Data-Platform-SRE (2026-04-24 - 2026-05-15), 07Epic, 13Patch-For-Review: Upgrade Spark to a version with long term Iceberg support, and with fixes to support Dumps 2.0 - https://phabricator.wikimedia.org/T338057#11894463 (10xcollazo) >>! In T338057#11892881, @JAllemandou wrote: > Afte... [14:16:27] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Data-Engineering-Radar, 06ServiceOps new, 10ServiceOps-Services-Oids, and 2 others: Make eventstreams-internal available to WMF staff without an ssh tunnel - https://phabricator.wikimedia.org/T348763#11894502 (10atsuko) a:05JAllemandou→03atsuko [14:30:18] !log Test Kitchen edge-unique experiments (poll 204398) - adds: none; removes: logged-out-retention-round7; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [14:30:21] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:31:34] 06Data-Engineering, 10Observability-Logging, 06SRE, 10Wikimedia-Logstash, and 2 others: Produce ECS formatted logstash logs to Event Platform, allowing them to be queried in the WMF Data Lake with SQL - https://phabricator.wikimedia.org/T291645#11894623 (10BTullis) a:03BTullis Assigning this to myself, s... [14:40:27] 06Data-Engineering, 06Data-Engineering-Icebox, 06DBA, 13Patch-For-Review: Move Mostcategories computation to Hadoop - https://phabricator.wikimedia.org/T413362#11894694 (10xcollazo) >>! In T413362#11893592, @Zabe wrote: > @xcollazo the DAG failed. Could you tell me what the underlying error is? DAG link:... [15:05:13] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10MWH-Incremental: Accelerate sqoop landing for MediaWiki History private tables - https://phabricator.wikimedia.org/T424355#11894875 (10APizzata-WMF) > Given this data, one idea is to split the tables into two groups by size and measure how much time we c... [15:23:26] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10MWH-Incremental: Accelerate sqoop landing for MediaWiki History private tables - https://phabricator.wikimedia.org/T424355#11895011 (10JAllemandou) > Questions are: what are the blockers to split /usr/local/bin/refinery-sqoop-mediawiki-history in paralle... [15:26:00] FIRING: [2x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [15:26:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [15:26:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [15:42:12] 06Data-Engineering: `mw_content_reconcile_*`: NoClassDefFoundError(EventStreamFactory) in spark_emit_reconcile_events_to_kafka - https://phabricator.wikimedia.org/T425569 (10AKhatun_WMF) 03NEW [15:43:03] 06Data-Engineering: `mw_content_reconcile_mw_content_history_daily`: NoClassDefFoundError(EventStreamFactory) in spark_emit_reconcile_events_to_kafka - https://phabricator.wikimedia.org/T425569#11895242 (10AKhatun_WMF) [15:51:00] RESOLVED: [2x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [15:51:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [15:51:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [15:51:53] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10MWH-Incremental: Accelerate sqoop landing for MediaWiki History private tables - https://phabricator.wikimedia.org/T424355#11895268 (10JAllemandou) After analyzing the graph I pasted above a bit more, I found that sqooping the `content` tables from cloud... [15:52:49] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10MWH-Incremental: Accelerate sqoop landing for MediaWiki History private tables - https://phabricator.wikimedia.org/T424355#11895270 (10APizzata-WMF) > I recommend we sqoop this table from the analytics-replicas. I can test this and come back with the res... [15:53:19] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Backfill datasets affected by Nov 2025 automated traffic incident - https://phabricator.wikimedia.org/T421735#11895271 (10mforns) [15:57:34] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10MWH-Incremental: Accelerate sqoop landing for MediaWiki History private tables - https://phabricator.wikimedia.org/T424355#11895291 (10JAllemandou) Note about what it means to change the sqooping from clouddb to analytics-replicas: * Change the sqoop la... [15:59:45] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10MWH-Incremental: Accelerate sqoop landing for MediaWiki History private tables - https://phabricator.wikimedia.org/T424355#11895295 (10JAllemandou) From the DB graphs, it seems we could parallelize the sqooping of small wikis more. Our problem will then... [16:01:38] 06Data-Engineering, 06Data-Platform-SRE (2026-04-24 - 2026-05-15), 07Essential-Work: ERROR AsyncEventQueue: Listener DatahubSparkListener threw an exception - https://phabricator.wikimedia.org/T400207#11895298 (10BTullis) Hmm. This is interesting. ` Caused by: datahub.shaded.org.apache.kafka.common.config.Co... [16:15:04] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10MWH-Incremental: mediawiki_history_incremental_v1: schema specification for stakeholder review - https://phabricator.wikimedia.org/T425573 (10xcollazo) 03NEW [16:21:00] FIRING: [2x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [16:21:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [16:21:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [16:26:38] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10MWH-Incremental: mediawiki_history_incremental_v1: schema specification for stakeholder review - https://phabricator.wikimedia.org/T425573#11895393 (10xcollazo) # Schema Specification: `mediawiki_history_incremental_v1` > **Purpose of this document:** S... [16:27:41] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10MWH-Incremental: mediawiki_history_incremental_v1: schema specification for stakeholder review - https://phabricator.wikimedia.org/T425573#11895399 (10xcollazo) [16:30:04] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10MWH-Incremental: mediawiki_history_incremental_v1: schema specification for stakeholder review - https://phabricator.wikimedia.org/T425573#11895402 (10xcollazo) CC DPE folks involved in this effort: @Ottomata @JAllemandou @mforns @Milimetric @APizzata-WM... [16:37:01] 06Data-Engineering, 06MediaWiki-Platform-Team, 05FY2025-26 KR 5.1, 07OKR-Work: redioscope: periodically publish top clients to the data lake - https://phabricator.wikimedia.org/T424823#11895414 (10Ahoelzl) @daniel can you provide background information how this supports KR or metrics work? [16:41:41] FIRING: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [16:41:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [16:41:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [16:44:21] (03PS1) 10Aleksandar Mastilovic: Add two new rate-limiting fields to webrequest_sampled [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1283833 (https://phabricator.wikimedia.org/T419736) [16:46:00] FIRING: [4x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [16:46:00] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [16:46:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [16:46:41] RESOLVED: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [16:46:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [16:46:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [16:48:41] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): `mw_content_reconcile_mw_content_history_daily`: NoClassDefFoundError(EventStreamFactory) in spark_emit_reconcile_events_to_kafka - https://phabricator.wikimedia.org/T425569#11895484 (10Ahoelzl) p:05Triage→03High [17:01:01] FIRING: [4x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [17:01:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [17:01:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [17:06:00] FIRING: [4x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [17:06:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [17:06:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [17:17:30] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10AQS2.0: Introduce a new AQS endpoint to expose video plays - https://phabricator.wikimedia.org/T415202#11895630 (10Ladsgroup) oh I'm so sorry you had to see that code. I'm planning to fully rewrite it. I apply your change ASAP. [17:31:01] FIRING: [2x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [17:31:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [17:31:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [17:36:00] RESOLVED: [2x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [17:36:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [17:36:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [17:36:30] (03CR) 10Joal: [C:03+1] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1283833 (https://phabricator.wikimedia.org/T419736) (owner: 10Aleksandar Mastilovic) [17:37:55] (03CR) 10Aleksandar Mastilovic: [V:03+2 C:03+2] Add two new rate-limiting fields to webrequest_sampled [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1283833 (https://phabricator.wikimedia.org/T419736) (owner: 10Aleksandar Mastilovic) [17:44:00] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): `mw_content_reconcile_mw_content_history_daily`: NoClassDefFoundError(EventStreamFactory) in spark_emit_reconcile_events_to_kafka - https://phabricator.wikimedia.org/T425569#11895704 (10xcollazo) Although we should fix this properly, as a stop gap to fix the... [17:47:25] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10AQS2.0: Introduce a new AQS endpoint to expose video plays - https://phabricator.wikimedia.org/T415202#11895714 (10TheDJ) speaking of media metrics. i noticed there is a beacon that is used by MultimediaViewer, to log how long a image was viewed in the m... [17:51:22] 06Data-Engineering, 06DBA, 07Schema-change-in-production: DB schema change in production - for ce_event_contributions - https://phabricator.wikimedia.org/T425582 (10cmelo) 03NEW [17:52:00] 06Data-Engineering, 06DBA, 07Schema-change-in-production: DB schema change in production - for ce_event_contributions - https://phabricator.wikimedia.org/T425582#11895749 (10cmelo) [18:02:18] 06Data-Engineering, 06DBA, 07Schema-change-in-production: DB schema change in production - for ce_event_contributions - https://phabricator.wikimedia.org/T425582#11895773 (10cmelo) [18:04:15] 06Data-Engineering, 06Data-Persistence, 06DBA, 06Connection-Team (Connection-Q4-27Apr-8May-2026), 07Schema-change-in-production: DB schema change in production - for ce_event_contributions - https://phabricator.wikimedia.org/T425582#11895784 (10cmelo) [18:12:38] 06Data-Engineering, 06Data-Persistence, 06DBA, 06Connection-Team (Connection-Q4-27Apr-8May-2026), 07Schema-change-in-production: DB schema change in production - for ce_event_contributions - https://phabricator.wikimedia.org/T425582#11895798 (10cmelo) [18:15:22] 06Data-Engineering, 06Data-Persistence, 06DBA, 06Connection-Team (Connection-Q4-27Apr-8May-2026), 07Schema-change-in-production: DB schema change in production - for ce_event_contributions - https://phabricator.wikimedia.org/T425582#11895812 (10Ladsgroup) 05Open→03Resolved a:03Ladsgroup [18:16:35] 06Data-Engineering, 06Data-Persistence, 06DBA, 06Connection-Team (Connection-Q4-27Apr-8May-2026), 07Schema-change-in-production: DB schema change in production - for ce_event_contributions - https://phabricator.wikimedia.org/T425582#11895826 (10Ladsgroup) I applied it as emergency but you need to fo... [18:26:57] 06Data-Engineering, 06Data-Persistence, 06DBA, 06Connection-Team (Connection-Q4-27Apr-8May-2026), 07Schema-change-in-production: DB schema change in production - for ce_event_contributions - https://phabricator.wikimedia.org/T425582#11895841 (10cmelo) [18:28:09] 06Data-Engineering, 06Data-Persistence, 06DBA, 06Connection-Team (Connection-Q4-27Apr-8May-2026), 07Schema-change-in-production: DB schema change in production - for ce_event_contributions - https://phabricator.wikimedia.org/T425582#11895842 (10cmelo) >>! In T425582#11895812, @Ladsgroup wrote: > I a... [18:31:25] 06Data-Engineering, 06Data-Persistence, 06DBA, 06Connection-Team (Connection-Q4-27Apr-8May-2026), 07Schema-change-in-production: DB schema change in production - for ce_event_contributions - https://phabricator.wikimedia.org/T425582#11895853 (10Ladsgroup) No worries. It happens! [18:57:33] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Mediawiki History Failure [2026-04] - https://phabricator.wikimedia.org/T425443#11895918 (10Ahoelzl) **Additional info** The problem occurs only for a very small portions of events, where: `event_entity = 'user' AND event_type = 'altergroups'` Within those... [19:02:52] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Backfill video play request data for the past 3 months - https://phabricator.wikimedia.org/T425121#11895934 (10Snwachukwu) Backfill from 1st Jan to March 31st is complete for the following job: `cassandra_load_mediarequest_per_file_daily ` `cassandra_load_m... [19:03:24] (03CR) 10Snwachukwu: [V:03+2] mediarequest_hourly: use file/filetypes as media_classification ground truth [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1279651 (https://phabricator.wikimedia.org/T421743) (owner: 10Snwachukwu) [19:06:48] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 13Patch-For-Review: Migrate generated-data-platform-aqs Docker images away from Debian Bullseye - https://phabricator.wikimedia.org/T425310#11895961 (10Snwachukwu) a:03Snwachukwu [19:39:12] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 13Patch-For-Review: `mw_content_reconcile_mw_content_history_daily`: NoClassDefFoundError(EventStreamFactory) in spark_emit_reconcile_events_to_kafka - https://phabricator.wikimedia.org/T425569#11896062 (10AKhatun_WMF) After some debugging with @JAllemando... [21:54:53] !log Test Kitchen mw-user experiment (poll 205709) - adds: fy25-26-we-1-7-8-suggestion-mode-beta; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [21:54:54] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log