[05:33:01] 06Data-Engineering, 10CampaignEvents, 06DBA, 06Connection-Team (Connection-Current-Sprint), 07Schema-change-in-production: Apply ce_event_contributions schema changes in production (x1) - https://phabricator.wikimedia.org/T407587#11283941 (10Marostegui) a:03Marostegui @Daimona can you ping us this has... [05:33:06] 06Data-Engineering, 10CampaignEvents, 06DBA, 06Connection-Team (Connection-Current-Sprint), 07Schema-change-in-production: Apply ce_event_contributions schema changes in production (x1) - https://phabricator.wikimedia.org/T407587#11283943 (10Marostegui) p:05Triage→03Medium [05:51:18] 06Data-Engineering, 06cloud-services-team, 06Data-Persistence, 06Data-Platform-SRE, and 2 others: Set up x1 replication to Wiki Replicas - https://phabricator.wikimedia.org/T395881#11283979 (10Marostegui) >>! In T395881#11280914, @BTullis wrote: > I created T407485 to track the work required to add this se... [06:29:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [06:34:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [07:49:14] 06Data-Engineering, 10DPE-Mediawiki-Content, 10Dumps-Generation, 06SRE, 07Epic: Dumps generation cause disruption to the production environment - https://phabricator.wikimedia.org/T368098#11284163 (10Marostegui) I don't recall any - let's keep an eye on them though [08:28:10] 06Data-Engineering, 10BetaFeatures, 06cloud-services-team, 10Data-Services, 06Data-Platform-SRE (2025.09.26 - 2025.10.17): Create view for betafeatures_user_counts table in wiki replicas - https://phabricator.wikimedia.org/T402145#11284219 (10Gehel) [08:28:28] 06Data-Engineering, 10BetaFeatures, 06cloud-services-team, 10Data-Services, and 2 others: Create view for betafeatures_user_counts table in wiki replicas - https://phabricator.wikimedia.org/T402145#11284220 (10Gehel) p:05Triage→03Medium [08:48:07] 06Data-Engineering, 06Infrastructure-Foundations, 06Data-Platform-SRE (2025.10.17 - 2025.11.07), 13Patch-For-Review: Also intake Network Error Logging events into the Analytics Data Lake - https://phabricator.wikimedia.org/T304373#11284259 (10Gehel) [08:48:22] 06Data-Engineering, 06Data-Engineering-Radar, 10Observability-Logging, 06serviceops, 06Data-Platform-SRE (2025.10.17 - 2025.11.07): Fix Kafka replicas skew - https://phabricator.wikimedia.org/T407185#11284261 (10Gehel) [08:48:25] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2025.10.17 - 2025.11.07), 07Essential-Work: Do performance testing of a big Hadoop Table hosted by Ceph - https://phabricator.wikimedia.org/T381416#11284271 (10Gehel) [08:48:50] 06Data-Engineering, 06Data-Engineering-Radar, 06cloud-services-team, 06Data-Persistence, and 3 others: Create wiki replicas views for globaljsonlinks tables - https://phabricator.wikimedia.org/T387419#11284279 (10Gehel) [08:49:25] 06Data-Engineering, 06Discovery-Search, 06Java-Scala-Standardization, 06Data-Platform-SRE (2025.10.17 - 2025.11.07), and 2 others: [Epic] Replace Archiva with Gitlab artifact repositories - https://phabricator.wikimedia.org/T367315#11284291 (10Gehel) [08:50:27] 06Data-Engineering, 10Dumps-Generation, 06Wikibase Reuse Team, 10Wikidata, and 3 others: No Wikidata dumps for Week 40 of 2025 (recurring issue) - https://phabricator.wikimedia.org/T406429#11284311 (10Gehel) [08:50:37] 06Data-Engineering, 06Data-Engineering-Radar, 06Discovery-Search, 06Infrastructure-Foundations, and 3 others: Elasticsearch dependency upgrade in spicerack - https://phabricator.wikimedia.org/T390860#11284321 (10Gehel) [08:52:52] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE (2025.10.17 - 2025.11.07): Add dbt related packages to conda-analytics - https://phabricator.wikimedia.org/T406766#11284426 (10Gehel) [08:52:58] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE (2025.10.17 - 2025.11.07): Add dbt related packages to conda-analytics - https://phabricator.wikimedia.org/T406767#11284425 (10Gehel) [08:53:04] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE (2025.10.17 - 2025.11.07): Create a dbt Docker container - https://phabricator.wikimedia.org/T406636#11284427 (10Gehel) [08:53:10] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE (2025.10.17 - 2025.11.07): Set up a working, usable dbt installation on stat boxes - https://phabricator.wikimedia.org/T406634#11284429 (10Gehel) [08:53:16] 06Data-Engineering, 06cloud-services-team, 06Data-Persistence, 10Data-Services, and 2 others: Set up x1 replication to an-redacteddb1001 - https://phabricator.wikimedia.org/T407485#11284424 (10Gehel) [08:53:55] 06Data-Engineering, 10Technical-blog-posts, 06Data-Platform-SRE (2025.10.17 - 2025.11.07), 07Essential-Work: Write a blog post about the recent Airflow migration to Kubernetes - https://phabricator.wikimedia.org/T393603#11284445 (10Gehel) [08:56:17] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE (2025.10.17 - 2025.11.07): Add dbt related packages to conda-analytics - https://phabricator.wikimedia.org/T406767#11284471 (10BTullis) →14Duplicate dup:03T406766 [08:56:18] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE (2025.10.17 - 2025.11.07): Add dbt related packages to conda-analytics - https://phabricator.wikimedia.org/T406766#11284473 (10BTullis) [11:41:05] (03PS1) 10Ottomata: Add HQL for user_edited_pages_daily [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1196892 (https://phabricator.wikimedia.org/T407559) [11:42:14] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10MediaWiki-Page-derived-data, 07OKR-Work, 13Patch-For-Review: Global Editor Metrics - Data Pipeline - user_edited_pages - https://phabricator.wikimedia.org/T407559#11284868 (10Ottomata) @mforns let me know if this ticket and [Add HQL for user_ed... [11:47:03] 06Data-Engineering, 10CampaignEvents, 06DBA, 06Connection-Team (Connection-Current-Sprint), 07Schema-change-in-production: Apply ce_event_contributions schema changes in production (x1) - https://phabricator.wikimedia.org/T407587#11284879 (10Daimona) Sure! Moving this to in progress so it remains visible. [13:45:57] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE (2025.10.17 - 2025.11.07), 07OKR-Work: Add dbt related packages to conda-analytics - https://phabricator.wikimedia.org/T406767#11285262 (10Gehel) [13:45:59] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE (2025.10.17 - 2025.11.07), 07OKR-Work: Add dbt related packages to conda-analytics - https://phabricator.wikimedia.org/T406766#11285264 (10Gehel) [13:46:16] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE (2025.10.17 - 2025.11.07), 07OKR-Work: Set up a working, usable dbt installation on stat boxes - https://phabricator.wikimedia.org/T406634#11285273 (10Gehel) [13:46:22] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE (2025.10.17 - 2025.11.07), 07OKR-Work: Create a dbt Docker container - https://phabricator.wikimedia.org/T406636#11285272 (10Gehel) [13:46:54] 06Data-Engineering, 06Data-Engineering-Radar, 10AbuseFilter, 06Product Safety and Integrity (Sprint Oct 20 - Nov 7), and 2 others: AbuseFilter abuse_filter_log table: Store IP addresses as hex values - https://phabricator.wikimedia.org/T395612#11285281 (10OKryva-WMF) [14:25:21] 06Data-Engineering, 06Data-Engineering-Radar, 06Discovery-Search, 06Infrastructure-Foundations, and 3 others: Elasticsearch dependency upgrade in spicerack - https://phabricator.wikimedia.org/T390860#11285384 (10elukey) I was about to cut a new spicerack release but I realized that https://gerrit.wikimedia... [14:26:40] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Prepare data engineering infrastructure for drop of rev_sha1 - https://phabricator.wikimedia.org/T405503#11285400 (10xcollazo) >>! In T405503#11213627, @Ladsgroup wrote: > We are planning to do the drop in 30 days. Would that be enough time for the tea... [14:51:15] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10DPE-Mediawiki-Content, 13Patch-For-Review: Modify code to dump all slots - https://phabricator.wikimedia.org/T384945#11285530 (10xcollazo) Ran the following as `analytics` to remove the existing `mediawiki_content_current` dump, to be rerun with... [15:02:59] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 07OKR-Work: SDS 1.3.2 Conduct Analysis on Alerting for changes in automated traffic distribution - https://phabricator.wikimedia.org/T406882#11285596 (10Snwachukwu) > Do the quantile values capture the spikes before May 28th? I ask because May 28th u... [15:14:18] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Prepare data engineering infrastructure for drop of rev_sha1 - https://phabricator.wikimedia.org/T405503#11285642 (10Ladsgroup) Thanks. Yeah. We need to clean up some stuff and will do that after Oct 25. Will you encounter issues if the columns still e... [15:16:40] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Adapt mediawiki_history to the removal of mediawiki revision.rev_sha1 - https://phabricator.wikimedia.org/T406000#11285666 (10xcollazo) 05Open→03In progress [15:18:08] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Prepare data engineering infrastructure for drop of rev_sha1 - https://phabricator.wikimedia.org/T405503#11285679 (10xcollazo) >Will you encounter issues if the columns still exists for a while? We should be good. We have already adapted the two import... [15:28:57] 06Data-Engineering, 06Infrastructure-Foundations, 06Data-Platform-SRE (2025.10.17 - 2025.11.07), 07Essential-Work, 13Patch-For-Review: Also intake Network Error Logging events into the Analytics Data Lake - https://phabricator.wikimedia.org/T304373#11285707 (10CDanis) 05In progress→03Resolved >>!... [15:43:32] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10DPE-Mediawiki-Content, 13Patch-For-Review: Modify code to dump all slots - https://phabricator.wikimedia.org/T384945#11285757 (10xcollazo) [[ https://airflow.wikimedia.org/dags/mw_content_xml_export_current_mid_month/grid?dag_run_id=scheduled__20... [16:44:45] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10MediaWiki-Page-derived-data, 07OKR-Work: Global Editor Metrics - Data Pipeline - https://phabricator.wikimedia.org/T405039#11285903 (10amastilovic) @Ottomata Regarding the naming of `user_edited_pages` as described in T407559 - why do we need the... [17:07:06] 06Data-Engineering, 10DPE-Mediawiki-Content, 07Epic: Production-level file export (aka dump) of MW Content in XML - https://phabricator.wikimedia.org/T384382#11285957 (10xcollazo) [17:07:07] 10Data-Engineering-Roadmap, 10DPE-Mediawiki-Content, 07Epic: Dumps 2.0 Phase III: Production level dumps - https://phabricator.wikimedia.org/T366752#11285958 (10xcollazo) [17:07:08] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 13Patch-For-Review: Optimize metrics computation for the MW Content Pipeline - https://phabricator.wikimedia.org/T401010#11285956 (10xcollazo) [17:07:23] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 13Patch-For-Review: Optimize metrics computation for the MW Content Pipeline - https://phabricator.wikimedia.org/T401010#11285960 (10xcollazo) (Moved out of critical path for File Export) [17:09:12] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10DPE-Mediawiki-Content, 13Patch-For-Review: Rewrite wmf_content.mediawiki_content_*_v1 tables with a new column for origin_rev_id - https://phabricator.wikimedia.org/T405944#11285971 (10xcollazo) 05Open→03Resolved [17:09:24] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10DPE-Mediawiki-Content: Wait till november wmf_raw.mediawiki_slots sqoop table is available, and apply origin_rev_id fix to mw_content tables - https://phabricator.wikimedia.org/T407237#11285976 (10xcollazo) [17:09:26] 06Data-Engineering, 10DPE-Mediawiki-Content, 07Epic: Production-level file export (aka dump) of MW Content in XML - https://phabricator.wikimedia.org/T384382#11285977 (10xcollazo) [17:09:28] 10Data-Engineering-Roadmap, 10DPE-Mediawiki-Content, 07Epic: Dumps 2.0 Phase III: Production level dumps - https://phabricator.wikimedia.org/T366752#11285978 (10xcollazo) [17:14:51] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10DPE-Mediawiki-Content: Compare the exported content between File Export and DumpV1 - https://phabricator.wikimedia.org/T407649 (10xcollazo) 03NEW [17:14:54] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10DPE-Mediawiki-Content: Compare the exported content between File Export and DumpV1 - https://phabricator.wikimedia.org/T407649#11286008 (10xcollazo) [17:14:56] 06Data-Engineering, 10DPE-Mediawiki-Content, 07Epic: Production-level file export (aka dump) of MW Content in XML - https://phabricator.wikimedia.org/T384382#11286009 (10xcollazo) [17:17:55] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Wikimedia Enterprise, 10Wikimedia Enterprise - Content Integrity, 06Data-Platform-SRE (2025.10.17 - 2025.11.07), 07Essential-Work: Implement an Airflow operator for moving data from point A t... - https://phabricator.wikimedia.org/T405360#11286013 [17:29:46] 06Data-Engineering, 06Data-Engineering-Radar, 06Discovery-Search, 06Infrastructure-Foundations, and 3 others: Elasticsearch dependency upgrade in spicerack - https://phabricator.wikimedia.org/T390860#11286083 (10bking) @elukey just wanted to pipe in and offer my assistance as well, since we have slightly m... [17:31:20] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11286092 (10achou) **Summary for yesterday's... [17:46:52] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE, 06Movement-Insights, 07Epic: Exlore the use of dbt-core and appropriate adapters in the data-platform environment - https://phabricator.wikimedia.org/T406764#11286133 (10Mayakp.wiki) [18:06:10] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines, 06SRE, 06Traffic-Icebox: Mobile redirects drop provenance parameters - https://phabricator.wikimedia.org/T252227#11286186 (10LucasWerkmeister) 05Open→03Resolved I believe this task can now be closed (not sure which status is best, l... [18:51:00] (03PS1) 10CDanis: WIP: traffic_signals schema draft [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1196959 [18:58:07] (03PS2) 10CDanis: WIP: traffic_signals schema draft [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1196959 [19:54:02] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10DPE-Mediawiki-Content: Compare the exported content between File Export and DumpV1 - https://phabricator.wikimedia.org/T407649#11286584 (10xcollazo) A quick check of `abwiki`, one of our smallest wikis, already yields a discrepancy: Note how this... [20:57:17] (03PS8) 10Aleksandar Mastilovic: Add user_central_id to the mediawiki_history dataset(s) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1194951 (https://phabricator.wikimedia.org/T406263) [21:02:00] (03PS2) 10Aleksandar Mastilovic: Add user central ID columns to mediawiki_history tables [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1195348 (https://phabricator.wikimedia.org/T406263) [22:25:20] (03PS3) 10Aleksandar Mastilovic: Add user central ID columns to mediawiki_history tables [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1195348 (https://phabricator.wikimedia.org/T406263)