[00:12:40] 06Data-Engineering: Refactor pingback analytics pipeline - https://phabricator.wikimedia.org/T415283#11556121 (10Pppery) [01:07:50] 10Data-Engineering-Roadmap, 06Data-Platform-SRE, 06Movement-Insights, 07Epic, 13Patch-For-Review: Provide a dbt-core development environment and production setup in the data-platform - https://phabricator.wikimedia.org/T406764#11556206 (10Mayakp.wiki) The parent task T408146 will be closed this week. @Ah... [01:25:19] FIRING: GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest_sampled ingested an unexpected number of records for a Kafka topic partition. ... [01:25:20] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest_sampled&var-kafka_topic=webrequest_sampled&viewPanel=24 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [01:54:28] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Content-Transform-Team, 06MW-Interfaces-Team, 10Event-Platform: Common event data model for data derived from parsed page revision content - https://phabricator.wikimedia.org/T415158#11556272 (10Ottomata) cc @xcollazo in case you have any thoughts... [02:01:57] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Research, 10Event-Platform, 13Patch-For-Review: Implement stream of HTML content on mw.page_change event - https://phabricator.wikimedia.org/T360794#11556277 (10Ottomata) I think we can make some progress on this while we bikeshed the data model i... [02:10:43] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 07OKR-Work: SDS 2.2.6 Improve experiment event data data lake management - https://phabricator.wikimedia.org/T414105#11556304 (10Ottomata) I have a couple of questions if you don't mind! > Various experiments stream data into tables that follow these s... [05:25:20] FIRING: GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest_sampled ingested an unexpected number of records for a Kafka topic partition. ... [05:25:20] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest_sampled&var-kafka_topic=webrequest_sampled&viewPanel=24 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [08:42:15] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop ar_sha1 from archive table in wmf production - https://phabricator.wikimedia.org/T411163#11556741 (10Marostegui) [09:25:20] FIRING: GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest_sampled ingested an unexpected number of records for a Kafka topic partition. ... [09:25:20] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest_sampled&var-kafka_topic=webrequest_sampled&viewPanel=24 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [09:37:52] (03CR) 10Joal: [V:03+2 C:03+2] "Comment only" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1229112 (https://phabricator.wikimedia.org/T414467) (owner: 10Santiago Faci) [09:51:59] (03Merged) 10jenkins-bot: Updated description of a TestKitchen contextual attribute [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1229112 (https://phabricator.wikimedia.org/T414467) (owner: 10Santiago Faci) [10:08:50] 06Data-Engineering, 06Research: Content history dataset issues - https://phabricator.wikimedia.org/T415311#11556910 (10Miriam) [10:35:38] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Make canary-events for the `resource_change` stream - https://phabricator.wikimedia.org/T415638 (10JAllemandou) 03NEW [10:37:31] 06Data-Engineering, 06Data-Engineering-Icebox, 06Product-Analytics: Analyze differences between checksum-based and revert-tag based reverts in mediawiki_history - https://phabricator.wikimedia.org/T266374#11556984 (10JAllemandou) Flagging for @Ahoelzl : This could be something we wish to consider. [10:47:24] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Optimize canary event generation resources consumption on Airflow - https://phabricator.wikimedia.org/T411989#11557025 (10Antoine_Quhen) First deploy crashed because and reverted. I was blocked by missing connection from k8s to eventgates. It was fixed by... [11:09:35] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Data-Platform-SRE (2026.01.23 - 2026.02.13), 07Essential-Work: Refine generates very large XCOM values - https://phabricator.wikimedia.org/T414953#11557151 (10Antoine_Quhen) * Option 1: - downstreaming xcoms (keeping only the necessary fields) -... [11:54:31] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 06Test Kitchen, 07Essential-Work, 05Goal: [GOAL] Tidy up EventLogging - https://phabricator.wikimedia.org/T408059#11557388 (10Sfaci) [11:54:39] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 06Test Kitchen, 07Essential-Work, 05Goal: [GOAL] Tidy up EventLogging - https://phabricator.wikimedia.org/T408059#11557399 (10Sfaci) [12:12:52] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Research, 10Event-Platform, 13Patch-For-Review: Implement stream of HTML content on mw.page_change event - https://phabricator.wikimedia.org/T360794#11557504 (10JMonton-WMF) @Ottomata I've [[ https://gitlab.wikimedia.org/repos/data-engineering/med... [12:22:34] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 07Essential-Work: Airflow-main scheduler loop sometimes slows down markedly - https://phabricator.wikimedia.org/T412003#11557522 (10brouberol) [12:26:26] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 07Essential-Work: Airflow-main scheduler loop sometimes slows down markedly - https://phabricator.wikimedia.org/T412003#11557541 (10brouberol) [13:13:26] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Migrate cleanup jobs for snapshot datasets from systemd timers to Airflow - https://phabricator.wikimedia.org/T411999#11557696 (10Antoine_Quhen) With T415357 I’ve already started extracting the Python conda environment build for analytics/refinery into Gi... [13:25:20] FIRING: GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest_sampled ingested an unexpected number of records for a Kafka topic partition. ... [13:25:20] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest_sampled&var-kafka_topic=webrequest_sampled&viewPanel=24 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [14:01:13] 06Data-Engineering: Airflow devenv BashOperator image is lacking libssl1.1 - https://phabricator.wikimedia.org/T415667 (10awight) 03NEW [14:05:18] 06Data-Engineering, 06Data-Platform-SRE: Airflow devenv BashOperator image is lacking libssl1.1 - https://phabricator.wikimedia.org/T415667#11558011 (10awight) [14:05:43] 06Data-Engineering, 06Data-Platform-SRE: Airflow devenv BashOperator image is lacking libssl1.1 - https://phabricator.wikimedia.org/T415667#11558015 (10awight) [14:21:44] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Data-Platform-SRE (2026.01.23 - 2026.02.13), 07Essential-Work: Refine generates very large XCOM values - https://phabricator.wikimedia.org/T414953#11558062 (10JAllemandou) Thanks @aqu for the summary above. My 2 cents: I would be ok to time-bound-tr... [14:28:27] 06Data-Engineering: [Iceberg Migration] Extend Iceberg table maintenance mechanism to support multiple Airflow instances - https://phabricator.wikimedia.org/T373693#11558097 (10xcollazo) [14:47:57] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 07OKR-Work: SDS 2.2.6 Improve experiment event data data lake management - https://phabricator.wikimedia.org/T414105#11558179 (10AKhatun_WMF) @Ottomata > IIUC, the desire is to have a single Iceberg table with specific partitioning containing the data... [14:57:56] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Make canary-events for the `resource_change` stream - https://phabricator.wikimedia.org/T415638#11558275 (10Ottomata) The canary event is generated from the first [[ https://gitlab.wikimedia.org/repos/data-engineering/schemas-event-primary/-/blob/master/j... [15:01:05] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Make canary-events for the `resource_change` stream - https://phabricator.wikimedia.org/T415638#11558299 (10Jgiannelos) I think it doesn't matter on the event consumer side to have a canary event like that. Worst case our invalidation is going to point to... [15:08:10] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 07OKR-Work: SDS 2.2.6 Improve experiment event data data lake management - https://phabricator.wikimedia.org/T414105#11558319 (10Ottomata) > All that would be left is to combine everything in one table (which would be done at the input level in the near... [15:24:13] (03PS1) 10Aqu: Remove datahub-cli package env dir [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1233738 (https://phabricator.wikimedia.org/T415357) [15:29:06] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Research: Content history dataset issues - https://phabricator.wikimedia.org/T415311#11558363 (10Ahoelzl) [15:31:11] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Make canary-events for the `resource_change` stream - https://phabricator.wikimedia.org/T415638#11558370 (10Ottomata) Well great then let's do it! [15:40:15] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Migrate cleanup jobs for snapshot datasets from systemd timers to Airflow - https://phabricator.wikimedia.org/T411999#11558390 (10JAllemandou) >>! In T411999#11557696, @Antoine_Quhen wrote: > With T415357 I’ve already started extracting the Python conda e... [15:41:23] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 13Patch-For-Review: Upgrade DataHub CLI virtualenv used by metadata_ingest_daily to restore Druid ingestion - https://phabricator.wikimedia.org/T415357#11558395 (10Antoine_Quhen) Build creation has been moved here: https://gitlab.wikimedia.org/repos/dat... [15:59:36] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 13Patch-For-Review: Upgrade DataHub CLI virtualenv used by metadata_ingest_daily to restore Druid ingestion - https://phabricator.wikimedia.org/T415357#11558447 (10Antoine_Quhen) 05Open→03Resolved [17:15:18] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Research, 10Event-Platform, 13Patch-For-Review: Implement stream of HTML content on mw.page_change event - https://phabricator.wikimedia.org/T360794#11558818 (10Ottomata) Sweet! Just left some comments on the MR. [17:25:20] FIRING: GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest_sampled ingested an unexpected number of records for a Kafka topic partition. ... [17:25:20] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest_sampled&var-kafka_topic=webrequest_sampled&viewPanel=24 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [17:49:02] 06Data-Engineering, 06Data-Engineering-Radar, 10GrowthExperiments, 06MediaWiki-Engineering, and 7 others: mw.track: support for histogram metrics - https://phabricator.wikimedia.org/T383563#11558962 (10Michael) [17:58:35] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Data-Platform-SRE (2026.01.23 - 2026.02.13), 07Essential-Work: Refine generates very large XCOM values - https://phabricator.wikimedia.org/T414953#11559057 (10brouberol) Just FYI, we're mitigating the effect of these large XCOM values in T415635 and... [20:33:02] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Content-Transform-Team, 06MW-Interfaces-Team, 10Event-Platform: Common event data model for data derived from parsed page revision content - https://phabricator.wikimedia.org/T415158#11559799 (10xcollazo) Regarding data layout, I favor Option A be... [21:06:34] 06Data-Engineering, 06Infrastructure-Foundations, 06Traffic, 13Patch-For-Review: Export development_network_probe data to Puppet servers for CDN deployment - https://phabricator.wikimedia.org/T402512#11559899 (10brouberol) > I am wondering if there is a way in external-services or similar to pick the known... [21:19:20] (03PS1) 10Xcollazo: Remove mediawiki_wikitext_* from refinery-drop-mediawiki-snapshots [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1233834 (https://phabricator.wikimedia.org/T396031) [21:21:52] 06Data-Engineering, 06Infrastructure-Foundations, 06Traffic, 13Patch-For-Review: Export development_network_probe data to Puppet servers for CDN deployment - https://phabricator.wikimedia.org/T402512#11560006 (10brouberol) That being said, if that information is defined in a hiera value, we can export it t... [21:25:20] FIRING: GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest_sampled ingested an unexpected number of records for a Kafka topic partition. ... [21:25:20] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest_sampled&var-kafka_topic=webrequest_sampled&viewPanel=24 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [21:36:30] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 13Patch-For-Review, 07User-notice: Publish Dumps 2 to dumps.wikimedia.org and provide only monthly dumps - https://phabricator.wikimedia.org/T414389#11560041 (10xcollazo) [23:32:53] !log Test Kitchen edge-unique experiments (poll 57285) - adds: none; removes: none; fields: synth-aa-test-traffic-impact - xLab/MPIC/TK tips at https://w.wiki/FwuD [23:32:55] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log