[01:17:51] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [01:17:51] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [01:43:45] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines, 10Wikidata, 10Wikidata Analytics: NEW FEATURE REQUEST: sqoop (all) user properties from mariadb to wmf_raw.mediawiki_user_properties - https://phabricator.wikimedia.org/T323456#11581655 (10Ahoelzl) @AndrewTavis_WMDE are there any privacy co... [01:53:36] 06Data-Engineering, 06Traffic: Request for a new request dataset for caching research - https://phabricator.wikimedia.org/T401331#11581666 (10Ahoelzl) @yazhuoz can you help us with the requirements for the data set? The old request from [[ https://phabricator.wikimedia.org/T225538 | 2019 ]] has some informati... [05:17:51] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [05:17:51] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [06:19:45] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Update imagelinks primary key on wmf production - https://phabricator.wikimedia.org/T415786#11581896 (10Marostegui) [09:16:55] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Update imagelinks primary key on wmf production - https://phabricator.wikimedia.org/T415786#11582241 (10Marostegui) [09:17:51] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [09:17:51] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [09:18:34] !log Test Kitchen edge-unique experiments (poll 89045) - adds: none; removes: synth-test-new-external-path; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [09:18:36] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [09:23:36] !log Test Kitchen edge-unique experiments (poll 89060) - adds: synth-test-new-external-path; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [09:23:38] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [09:57:03] 06Data-Engineering, 06Data-Platform-SRE (2026.01.23 - 2026.02.13): Airflow devenv BashOperator image is lacking libssl1.1 - https://phabricator.wikimedia.org/T415667#11582355 (10Gehel) 05In progress→03Resolved [09:58:12] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Migrate cleanup jobs for snapshot datasets from systemd timers to Airflow - https://phabricator.wikimedia.org/T411999#11582361 (10Antoine_Quhen) On this ticket, we have consulted both SREs and our team. We have agreed on the following details: * extract a... [10:06:03] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Update imagelinks primary key on wmf production - https://phabricator.wikimedia.org/T415786#11582387 (10Marostegui) [10:06:45] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Update imagelinks primary key on wmf production - https://phabricator.wikimedia.org/T415786#11582388 (10Marostegui) [10:33:52] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines, 10Wikidata, 10Wikidata Analytics: NEW FEATURE REQUEST: sqoop (all) user properties from mariadb to wmf_raw.mediawiki_user_properties - https://phabricator.wikimedia.org/T323456#11582489 (10JAllemandou) In terms of PII, we don't yet have hav... [11:25:10] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Data-Platform-SRE (2026.01.23 - 2026.02.13), 07OKR-Work, 13Patch-For-Review: Provide a Spark-on-k8s access for sql tools (dbt) - https://phabricator.wikimedia.org/T410017#11582699 (10JAllemandou) [11:32:53] (03CR) 10Aqu: [C:03+1] "Looking good." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1236347 (https://phabricator.wikimedia.org/T361210) (owner: 10Snwachukwu) [12:10:22] 06Data-Engineering, 10DPE-Mediawiki-Content, 06Data-Platform-SRE (2026.01.23 - 2026.02.13), 07Essential-Work: When wikis cannot be exported due to SiteInfo, don't fail them - https://phabricator.wikimedia.org/T408819#11582847 (10brouberol) a:03xcollazo [12:41:28] 06Data-Engineering, 06Data-Platform-SRE, 10FR-Tech-Analytics: Enable greater integration between the DPE and FR-tech analytics stacks - https://phabricator.wikimedia.org/T416457#11582952 (10BTullis) [12:48:41] 06Data-Engineering, 06Data-Platform-SRE, 10FR-Tech-Analytics, 07Epic: Enable greater integration between the DPE and FR-tech analytics stacks - https://phabricator.wikimedia.org/T416457#11583005 (10BTullis) [13:06:08] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Dan and Thomas can deploy backports - https://phabricator.wikimedia.org/T416470 (10Milimetric) 03NEW [13:15:55] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Send client signals in various ways to understand new data - https://phabricator.wikimedia.org/T416472 (10Milimetric) 03NEW [13:16:05] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Send client signals in various ways to understand new data - https://phabricator.wikimedia.org/T416472#11583184 (10Milimetric) p:05Triage→03Medium a:03Milimetric [13:17:51] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [13:17:51] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [13:20:17] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Migrate cleanup jobs for snapshot datasets from systemd timers to Airflow - https://phabricator.wikimedia.org/T411999#11583193 (10JAllemandou) If you go and extract the python libs from refinery, I think it'd be worth also extracting all python scripts in... [13:23:18] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Send client signals in various ways to understand new data - https://phabricator.wikimedia.org/T416472#11583200 (10Milimetric) new instrument activated as: https://test-kitchen.wikimedia.org/instrument/bot-detection-2026-02 [14:17:28] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Update imagelinks primary key on wmf production - https://phabricator.wikimedia.org/T415786#11583413 (10Marostegui) [14:17:58] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Update imagelinks primary key on wmf production - https://phabricator.wikimedia.org/T415786#11583414 (10Marostegui) [14:32:55] 06Data-Engineering, 06Traffic: Request for a new request dataset for caching research - https://phabricator.wikimedia.org/T401331#11583485 (10yazhuoz) @Ahoelzl Thanks for getting back to me! Here are the detailed requirements for the new CDN caching dataset. **Data fields:** We would like to retain the prev... [14:33:06] 06Data-Engineering: Adapt Sqoop for imagelinks schema changes - https://phabricator.wikimedia.org/T416481 (10GGoncalves-WMF) 03NEW [14:44:39] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Update imagelinks primary key on wmf production - https://phabricator.wikimedia.org/T415786#11583551 (10Marostegui) [14:53:47] 06Data-Engineering, 06DBA, 13Patch-For-Review, 07Schema-change-in-production: Update imagelinks primary key on wmf production - https://phabricator.wikimedia.org/T415786#11583680 (10Marostegui) [14:57:46] 06Data-Engineering, 10AQS2.0: Introduce a new AQS endpoint to expose video plays - https://phabricator.wikimedia.org/T415202#11583694 (10GGoncalves-WMF) Having spoken to @Ladsgroup this morning (thanks!), here's my notes on this task. There are three potential quality issues in how we surface video plays: 1.... [15:01:13] 06Data-Engineering, 10Dumps-Generation: SQL metadata files problem - https://phabricator.wikimedia.org/T416416#11583731 (10xcollazo) > The specific SQL files I use are enwiki-[date]-page.sql.gz and enwiki-[date]-categorylinks.sql.gz, as can be found for example here: https://dumps.wikimedia.org/enwiki/20260101... [15:02:52] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10AQS2.0: Introduce a new AQS endpoint to expose video plays - https://phabricator.wikimedia.org/T415202#11583739 (10GGoncalves-WMF) [15:05:12] 06Data-Engineering, 10Observability-Metrics: [Data Quality] Sending Apache Spark metrics to PushGateway - https://phabricator.wikimedia.org/T297231#11583746 (10Aklapper) a:05Antoine_Quhen→03None @Antoine_Quhen Removing task assignee as this open task has been assigned for more than two years - See the emai... [15:05:39] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Missing reconciliation for MWCH - https://phabricator.wikimedia.org/T416491 (10APizzata-WMF) 03NEW [15:06:15] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10DPE-Mediawiki-Content: Missing reconciliation for MWCH - https://phabricator.wikimedia.org/T416491#11583771 (10APizzata-WMF) [15:06:47] 06Data-Engineering, 06Data-Engineering-Radar, 10Dumps-Generation: Get Data Engineering folks access to hosts and systems needed for maintenance of the existing dumps system - https://phabricator.wikimedia.org/T341045#11583773 (10Aklapper) @ArielGlenn: Is that last open item still to-do and still needed after... [15:07:03] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Dumps-Generation: SQL metadata files problem - https://phabricator.wikimedia.org/T416416#11583774 (10xcollazo) [15:07:13] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Dumps-Generation: SQL metadata files problem - https://phabricator.wikimedia.org/T416416#11583775 (10xcollazo) a:03xcollazo [15:11:38] 06Data-Engineering, 06Data-Engineering-Radar, 10Dumps-Generation: Get Data Engineering folks access to hosts and systems needed for maintenance of the existing dumps system - https://phabricator.wikimedia.org/T341045#11583802 (10ArielGlenn) a:05ArielGlenn→03None I do not know, but the data engineering gr... [15:14:45] 06Data-Engineering, 06Data-Engineering-Radar, 10Dumps-Generation: Get Data Engineering folks access to hosts and systems needed for maintenance of the existing dumps system - https://phabricator.wikimedia.org/T341045#11583814 (10xcollazo) 05Open→03Resolved a:03xcollazo Right, we do not need this ti... [15:16:07] 06Data-Engineering, 06Data-Platform-SRE, 10Event-Platform: [Event Platform] Define Flink k8s operator SLO - https://phabricator.wikimedia.org/T345914#11583820 (10Aklapper) a:05gmodena→03None @gmodena Removing task assignee as this open task has been assigned for more than two years - See the email sent t... [15:48:18] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Update imagelinks primary key on wmf production - https://phabricator.wikimedia.org/T415786#11583952 (10Marostegui) [15:49:04] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Update imagelinks primary key on wmf production - https://phabricator.wikimedia.org/T415786#11583953 (10ops-monitoring-bot) Start pool of db1236 gradually with 4 steps - After schema change - marostegui@cumin1003 [16:14:45] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 07ci-test-error (WMF-deployed Build Failure): CI error: mediawiki.base/track trackError: unexpected "{\"exception\":{},\"module\":\"mw.cx.eventlogging\",\"source\":\"module-execute\"}" - https://phabricator.wikimedia.org/T413202#11584103 (10Lucas_Werkme... [16:18:45] 06Data-Engineering, 10ContentTranslation, 10MediaWiki-extensions-EventLogging, 07ci-test-error (WMF-deployed Build Failure): CI error: mediawiki.base/track trackError: unexpected "{\"exception\":{},\"module\":\"mw.cx.eventlogging\",\"source\":\"module-exec... - https://phabricator.wikimedia.org/T413202#11584107 [16:34:24] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Update imagelinks primary key on wmf production - https://phabricator.wikimedia.org/T415786#11584170 (10ops-monitoring-bot) Completed pool of db1236 gradually with 4 steps - After schema change - marostegui@cumin1003 [17:17:51] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [17:17:51] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [17:29:37] 06Data-Engineering, 06Data-Platform-SRE, 10FR-Tech-Analytics, 07Epic: Enable greater integration between the DPE and FR-tech analytics stacks - https://phabricator.wikimedia.org/T416457#11584359 (10BTullis) We have been discussing a few short-term and long-term options around this. In the short-term, we c... [18:03:41] 14Analytics, 06Data-Engineering, 06Test Kitchen, 13Patch-For-Review: Count the number of video plays - https://phabricator.wikimedia.org/T198628#11584479 (10Ladsgroup) >>! In T198628#11552043, @New_York-air wrote: > @Ladsgroup Wow, thank you so much for having a look into this! > I can't imagine that it's... [18:04:10] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10AQS2.0: Introduce a new AQS endpoint to expose video plays - https://phabricator.wikimedia.org/T415202#11584482 (10Ladsgroup) Regarding 2 and 3, it'd be T373546 [18:39:35] (03CR) 10Snwachukwu: "Thank you!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1236347 (https://phabricator.wikimedia.org/T361210) (owner: 10Snwachukwu) [18:40:16] (03CR) 10Snwachukwu: [V:03+2 C:03+2] Migrate cu_changes table to use cuua_text in new cu_usergent table. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1236347 (https://phabricator.wikimedia.org/T361210) (owner: 10Snwachukwu) [18:51:21] 06Data-Engineering, 06DBA: Move Mostcategories computation to Hadoop - https://phabricator.wikimedia.org/T413362#11584745 (10Ladsgroup) I‌ think I‌ have admin rights and I‌ think I‌ deleted the record. I‌ keep a copy here in case I‌ break things: ` platform_eng/dags/querypage/querypage_most_categories_monthly_... [19:45:50] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines, 10Wikidata, 10Wikidata Analytics: NEW FEATURE REQUEST: sqoop (all) user properties from mariadb to wmf_raw.mediawiki_user_properties - https://phabricator.wikimedia.org/T323456#11584921 (10Ottomata) [19:46:45] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines, 10Wikidata, 10Wikidata Analytics: NEW FEATURE REQUEST: sqoop (all) user properties from mariadb to wmf_raw.mediawiki_user_properties - https://phabricator.wikimedia.org/T323456#11584926 (10Ottomata) Too bad we don't have a `mediawiki.user_p... [20:08:01] 06Data-Engineering, 06Data-Platform-SRE, 10FR-Tech-Analytics, 07Epic: Enable greater integration between the DPE and FR-tech analytics stacks - https://phabricator.wikimedia.org/T416457#11584999 (10Ottomata) > https://gravitino.apache.org/ Interesting! So for our existent HMS use cases, gravitino is a met... [21:13:03] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 13Patch-For-Review: Send client signals in various ways to understand new data - https://phabricator.wikimedia.org/T416472#11585138 (10Milimetric) To send a more complete URL we'd have to checksum it to keep it within a certain string length limit. It... [21:17:51] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [21:17:52] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [23:07:42] 06Data-Engineering, 06Data-Platform-SRE, 10FR-Tech-Analytics, 07Epic: Enable greater integration between the DPE and FR-tech analytics stacks - https://phabricator.wikimedia.org/T416457#11585519 (10BTullis)