[01:25:19] FIRING: GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest_sampled ingested an unexpected number of records for a Kafka topic partition. ... [01:25:19] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest_sampled&var-kafka_topic=webrequest_sampled&viewPanel=24 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [05:25:19] FIRING: GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest_sampled ingested an unexpected number of records for a Kafka topic partition. ... [05:25:19] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest_sampled&var-kafka_topic=webrequest_sampled&viewPanel=24 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [05:35:37] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop ar_sha1 from archive table in wmf production - https://phabricator.wikimedia.org/T411163#11547693 (10Marostegui) [05:35:39] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop rev_sha1 from revision table in wmf production - https://phabricator.wikimedia.org/T411164#11547694 (10Marostegui) [07:19:50] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 07OKR-Work: Update Blunderbuss-bugler - https://phabricator.wikimedia.org/T415338 (10JMonton-WMF) 03NEW [08:13:58] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop ar_sha1 from archive table in wmf production - https://phabricator.wikimedia.org/T411163#11547866 (10Marostegui) s1 is done apart from masters. I won't switch them on a Friday. [08:14:20] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop rev_sha1 from revision table in wmf production - https://phabricator.wikimedia.org/T411164#11547867 (10Marostegui) s1 is done apart from masters. I won't switch them on a Friday. [09:08:08] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 06Data-Platform-SRE (2026.01.05 - 2026.01.23), 07Essential-Work: Grant Access to analytics-privatedata-users for hmonroy - https://phabricator.wikimedia.org/T414375#11547916 (10Gehel) [09:25:19] FIRING: GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest_sampled ingested an unexpected number of records for a Kafka topic partition. ... [09:25:19] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest_sampled&var-kafka_topic=webrequest_sampled&viewPanel=24 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [09:48:39] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): aggregate_for_fundraising_hourly failing for last 24 hours - https://phabricator.wikimedia.org/T415267#11547981 (10amastilovic) The issue has now been fixed: https://airflow-platform-eng.wikimedia.org/dags/aggregate_for_fundraising_hourly/grid [10:03:18] 06Data-Engineering, 06Data-Engineering-Radar, 06cloud-services-team, 06Data-Persistence, and 3 others: Create wiki replicas views for globaljsonlinks tables - https://phabricator.wikimedia.org/T387419#11548009 (10Gehel) [10:03:39] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Data-Platform-SRE (2026.01.23 - 2026.02.13), 07OKR-Work, 13Patch-For-Review: Provide a Spark production access for dbt with Airflow - https://phabricator.wikimedia.org/T410017#11548023 (10Gehel) [10:04:03] 06Data-Engineering, 06Discovery-Search, 06Java-Scala-Standardization, 06Data-Platform-SRE (2026.01.23 - 2026.02.13), and 2 others: [Epic] Replace Archiva with Gitlab artifact repositories - https://phabricator.wikimedia.org/T367315#11548034 (10Gehel) [10:05:26] 06Data-Engineering, 10DPE-Mediawiki-Content, 06Data-Platform-SRE (2026.01.23 - 2026.02.13), 07Essential-Work: When wikis cannot be exported due to SiteInfo, don't fail them - https://phabricator.wikimedia.org/T408819#11548074 (10Gehel) [10:05:50] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Data-Platform-SRE (2026.01.23 - 2026.02.13), 07Essential-Work: Carry out end-user testing of spark on kubernetes - https://phabricator.wikimedia.org/T412925#11548094 (10Gehel) [10:06:47] 06Data-Engineering, 06Data-Engineering-Radar, 10superset.wikimedia.org, 06Data-Platform-SRE (2026.01.23 - 2026.02.13), 07Essential-Work: Thanos / Prometheus metrics names with uppercase characters not accessible from Superset SQL Lab or Presto CLI - https://phabricator.wikimedia.org/T409874#11548118 (10Ge... [10:07:05] 06Data-Engineering, 06cloud-services-team, 06Data-Persistence, 10Data-Services, and 3 others: Set up x1 replication to Wiki Replicas - https://phabricator.wikimedia.org/T395881#11548110 (10Gehel) [10:07:15] 06Data-Engineering, 06cloud-services-team, 06Data-Persistence, 10Data-Services, and 3 others: Set up x1 replication to an-redacteddb1001 - https://phabricator.wikimedia.org/T407485#11548116 (10Gehel) [10:08:32] 06Data-Engineering, 06Data-Platform-SRE (2026.01.23 - 2026.02.13), 07Essential-Work: ERROR AsyncEventQueue: Listener DatahubSparkListener threw an exception - https://phabricator.wikimedia.org/T400207#11548153 (10Gehel) [10:09:18] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE (2026.01.23 - 2026.02.13), 07Essential-Work: Blunderbuss: Move Hadoop/HDFS XML configuration into Helm deployment chart - https://phabricator.wikimedia.org/T402323#11548168 (10Gehel) [10:09:32] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2026.01.23 - 2026.02.13), 07Essential-Work: Move the dumps_v1 DAGs from the Airflow test_k8s instance to the main instance - https://phabricator.wikimedia.org/T404084#11548174 (10Gehel) [10:09:38] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Data-Platform-SRE (2026.01.23 - 2026.02.13), 07Essential-Work: Refine generates very large XCOM values - https://phabricator.wikimedia.org/T414953#11548177 (10Gehel) [10:09:48] 06Data-Engineering, 10BetaFeatures, 06cloud-services-team, 10Data-Services, and 2 others: Create view for betafeatures_user_counts table in wiki replicas - https://phabricator.wikimedia.org/T402145#11548158 (10Gehel) [10:10:08] 06Data-Engineering, 10Dumps-Generation, 06Data-Platform-SRE (2026.01.23 - 2026.02.13), 07Essential-Work: Certain *recombine tasks in dumps_v1 are non-idempotent and can generate corrupt files - https://phabricator.wikimedia.org/T404859#11548184 (10Gehel) [10:10:38] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2026.01.23 - 2026.02.13), 07Essential-Work: Provide an access to MaxMind GeoIP in DSE K8S pods - https://phabricator.wikimedia.org/T405509#11548194 (10Gehel) [10:10:45] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Wikimedia Enterprise, 10Wikimedia Enterprise - Content Integrity, 06Data-Platform-SRE (2026.01.23 - 2026.02.13), 07Essential-Work: Implement an Airflow operator for moving data from point A to B - https://phabricator.wikimedia.org/T405360#11548191... [10:11:01] 06Data-Engineering, 10Technical-blog-posts, 06Data-Platform-SRE (2026.01.23 - 2026.02.13), 07Essential-Work: Write a blog post about the recent Airflow migration to Kubernetes - https://phabricator.wikimedia.org/T393603#11548200 (10Gehel) [10:12:03] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Data-Platform-SRE (2026.01.23 - 2026.02.13), 07Essential-Work: Create alert on Airflow scheduler slow down - https://phabricator.wikimedia.org/T411405#11548225 (10Gehel) [10:12:13] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Movement-Insights, 06Data-Platform-SRE (2026.01.23 - 2026.02.13), 07OKR-Work, 13Patch-For-Review: Run dbt from Airflow - https://phabricator.wikimedia.org/T410268#11548219 (10Gehel) [12:44:32] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Upgrade DataHub CLI virtualenv used by metadata_ingest_daily to restore Druid ingestion - https://phabricator.wikimedia.org/T415357 (10Antoine_Quhen) 03NEW [13:21:45] 06Data-Engineering, 07Essential-Work, 06Test Kitchen (Test Kitchen (Experiment Platform Sprint 18)): [Renaming TestKitchen] Update custom-data-monitor - https://phabricator.wikimedia.org/T414451#11548750 (10Sfaci) >>! In T414451#11546211, @Milimetric wrote: > @Sfaci are you going to make the changes? Yes!.... [13:21:53] 06Data-Engineering, 07Essential-Work, 06Test Kitchen (Test Kitchen (Experiment Platform Sprint 18)): [Renaming TestKitchen] Update custom-data-monitor - https://phabricator.wikimedia.org/T414451#11548751 (10Sfaci) a:03Sfaci [13:25:19] FIRING: GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest_sampled ingested an unexpected number of records for a Kafka topic partition. ... [13:25:19] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest_sampled&var-kafka_topic=webrequest_sampled&viewPanel=24 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [13:41:52] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 06Test Kitchen, 07Technical-Debt: Deprecate and remove mw.eventLog.submitInteraction() - https://phabricator.wikimedia.org/T415362 (10Sfaci) 03NEW [13:42:10] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 06Test Kitchen, 07Technical-Debt: Deprecate and remove mw.eventLog.submitInteraction() - https://phabricator.wikimedia.org/T415362#11548826 (10Sfaci) [13:42:12] 06Data-Engineering, 06Data-Platform-SRE, 10ServiceOps-Datastores, 06SRE, 10Event-Platform: DRY kafka broker declaration in helmfiles - https://phabricator.wikimedia.org/T253058#11548824 (10MLechvien-WMF) Removing our tag, please add it back if anything is needed from our end [13:42:16] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 06Test Kitchen, 07Essential-Work, 05Goal: [GOAL] Tidy up EventLogging - https://phabricator.wikimedia.org/T408059#11548827 (10Sfaci) [13:44:52] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Content-Transform-Team, 06MW-Interfaces-Team, 10Event-Platform: Common event data model for data derived from parsed page revision content - https://phabricator.wikimedia.org/T415158#11548829 (10Isaac) I think the `render_id` is useful as a concep... [13:55:51] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Upgrade DataHub CLI virtualenv used by metadata_ingest_daily to restore Druid ingestion - https://phabricator.wikimedia.org/T415357#11548889 (10Antoine_Quhen) a:03Antoine_Quhen [14:03:14] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 06Test Kitchen, 07Essential-Work: Deprecate and remove EventLogging::getMetricsPlatformClient() - https://phabricator.wikimedia.org/T415246#11548974 (10Sfaci) @phuedx are usages like [[https://gerrit.wikimedia.org/g/mediawiki/extensions/CheckUser/+/1... [14:15:58] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 06Test Kitchen, 07Technical-Debt: Deprecate and remove mw.eventLog.submitInteraction() - https://phabricator.wikimedia.org/T415362#11549025 (10Sfaci) [14:17:53] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 06Test Kitchen, 07Technical-Debt: Deprecate and remove mw.eventLog.submitInteraction() - https://phabricator.wikimedia.org/T415362#11549034 (10Sfaci) [14:43:07] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 13Patch-For-Review: Support for Java 25 and Flink 2 - https://phabricator.wikimedia.org/T412978#11549092 (10JMonton-WMF) [15:28:58] 06Data-Engineering, 06Infrastructure-Foundations, 06Traffic: Export development_network_probe data to Puppet servers for CDN deployment - https://phabricator.wikimedia.org/T402512#11549215 (10elukey) Replying to my own question - in `helmfile.d/dse-k8s-services/mediawiki-dumps-legacy/values-dumps.yaml` I see... [16:12:58] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Content-Transform-Team, 06MW-Interfaces-Team, 10Event-Platform: Common event data model for data derived from parsed page revision content - https://phabricator.wikimedia.org/T415158#11549399 (10daniel) > I think the render_id is useful as a conce... [16:48:02] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Inventory of SystemD timer based jobs and pipelines - https://phabricator.wikimedia.org/T414107#11549475 (10AKhatun_WMF) The doc that groups the timers and does an initial assessment of moving the timers to Airflow: [Google Doc](https://docs.google.com/do... [16:51:00] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): 100% Sample a small geographically concentrated wiki - https://phabricator.wikimedia.org/T415384 (10tchin) 03NEW [16:56:38] 06Data-Engineering, 07Essential-Work, 06Test Kitchen (Test Kitchen (Experiment Platform Sprint 18)): [Renaming TestKitchen] Update custom-data-monitor - https://phabricator.wikimedia.org/T414451#11549540 (10Sfaci) [17:25:19] FIRING: GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest_sampled ingested an unexpected number of records for a Kafka topic partition. ... [17:25:19] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest_sampled&var-kafka_topic=webrequest_sampled&viewPanel=24 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [18:03:45] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Data-Platform-SRE (2026.01.23 - 2026.02.13), 07Essential-Work: Blunderbuss: Move Hadoop/HDFS XML configuration into Helm deployment chart - https://phabricator.wikimedia.org/T402323#11549741 (10Ahoelzl) [18:04:17] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Update Blunderbuss wikitech documentation - https://phabricator.wikimedia.org/T402290#11549742 (10Ahoelzl) [18:05:52] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Trust and Safety Product Team, 06Product-Analytics (Kanban): Add mediawiki_product_metrics_incident_reporting_system_interaction to the sanitization allowlist - https://phabricator.wikimedia.org/T384650#11549743 (10Ahoelzl) [18:07:02] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Do not set WMF-Last-Access cookie when Sec-Fetch-Dest is not 'document' - https://phabricator.wikimedia.org/T403897#11549758 (10Ahoelzl) [18:07:55] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10MediaWiki-Page-derived-data, 07OKR-Work: Global Editor Metrics - Data Pipeline - edit_per_editor_per_page_daily - https://phabricator.wikimedia.org/T407559#11549772 (10Ahoelzl) 05Open→03Resolved a:03Ahoelzl [18:27:21] 06Data-Engineering, 10Observability-Tracing, 10Event-Platform, 13Patch-For-Review: EventGate: Enable OpenTelemetry Propagation - https://phabricator.wikimedia.org/T391353#11549795 (10Ahoelzl) a:05tchin→03None [18:27:35] 06Data-Engineering, 10ChangeProp, 06MW-Interfaces-Team, 10Observability-Tracing, and 2 others: Implement tracing across changeprop-jobqueue, kafka, eventgate - https://phabricator.wikimedia.org/T395038#11549797 (10Ahoelzl) [18:36:01] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Fix reconcile bug where user_id is not being populated correctly. - https://phabricator.wikimedia.org/T411803#11549809 (10xcollazo) Ran the following to deploy scale down: ` ssh deployment.eqiad.wmnet cd /srv/deployment-charts/helmfile.d/dse-k8s-services... [19:13:56] 06Data-Engineering, 07OKR-Work (WE1 FY2025-26): Productize Data for Monthly Active Moderator Actions - https://phabricator.wikimedia.org/T410940#11549870 (10ldelench_wmf) [20:19:21] 06Data-Engineering, 10Data-Platform, 06Moderator-Tools-Team, 06Product-Analytics (Kanban): Personal Dashboard Instrumentation Superset Dashboard - https://phabricator.wikimedia.org/T412137#11550019 (10MNeisler) @Kgraessle @DMburugu - Have the events needed for this dashboard been instrumented? Or is that s... [20:26:53] 06Data-Engineering, 10Data-Platform, 06Moderator-Tools-Team, 06Product-Analytics (Kanban): Personal Dashboard Instrumentation Superset Dashboard - https://phabricator.wikimedia.org/T412137#11550030 (10Kgraessle) [20:30:24] 06Data-Engineering, 10Data-Platform, 06Moderator-Tools-Team, 06Product-Analytics (Kanban): Personal Dashboard Instrumentation Superset Dashboard - https://phabricator.wikimedia.org/T412137#11550038 (10Kgraessle) Hi @MNeisler Nope, they are still in progress/code review. Though I realize this ticket is m... [20:32:07] 06Data-Engineering, 10Data-Platform, 06Moderator-Tools-Team, 06Product-Analytics (Kanban): Personal Dashboard Instrumentation Superset Dashboard - https://phabricator.wikimedia.org/T412137#11550069 (10Kgraessle) > Once the instrumentation is complete, I can start to set up the dashboard with the metrics i... [21:25:19] FIRING: GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest_sampled ingested an unexpected number of records for a Kafka topic partition. ... [21:25:19] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest_sampled&var-kafka_topic=webrequest_sampled&viewPanel=24 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [23:42:05] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 07OKR-Work: SDS 2.2.6 Improve experiment event data data lake management - https://phabricator.wikimedia.org/T414105#11550613 (10AKhatun_WMF) @mpopov, I went over the doc and previous meeting. Wanted to summarize my understanding and get some clarificat...