[00:37:37] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to Analytics_Privatedata for Chandra-WMDE - https://phabricator.wikimedia.org/T409707#11403718 (10RLazarus) @Milimetric @Ahoelzl Ping - can you approve for Data Engineering please? The requester is not a WMF or WMDE emp... [02:05:57] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10Event-Platform, 13Patch-For-Review: Add CI step to event schema repositories to test to fail if a schema is deleted - https://phabricator.wikimedia.org/T377023#11403869 (10amastilovic) Ditto what @xcollazo said above. In order to have the desired... [03:44:16] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Persistence, 13Patch-For-Review: Provision Global Editor Metrics tables & endpoints - https://phabricator.wikimedia.org/T410962#11403977 (10Eevans) The more I look at this top k endpoint, the more I think I may have misunderstood what was in... [06:46:15] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop rc_type from recentchanges in wmf production - https://phabricator.wikimedia.org/T410531#11404102 (10Marostegui) [06:47:01] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop rc_type from recentchanges in wmf production - https://phabricator.wikimedia.org/T410531#11404104 (10Marostegui) [08:13:23] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 13Patch-For-Review: Implement a new pipeline and table with reconciled historical revision data - https://phabricator.wikimedia.org/T410688#11404175 (10JAllemandou) >>! In T410688#11397421, @xcollazo wrote: > Since we are going to need to backfill `w... [08:25:10] 06Data-Engineering, 06Data-Platform-SRE (2025.11.07 - 2025.11.28), 13Patch-For-Review: Bump hadoop container maximum memory size - https://phabricator.wikimedia.org/T410966#11404188 (10Gehel) [08:25:17] 06Data-Engineering, 06Data-Platform-SRE (2025.11.07 - 2025.11.28), 13Patch-For-Review: Bump hadoop container maximum memory size - https://phabricator.wikimedia.org/T410966#11404189 (10Gehel) p:05Triage→03Medium [08:43:51] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 10mwcli: Update eventlogging image for MWCLI - https://phabricator.wikimedia.org/T406317#11404255 (10Addshore) >>! In T406317#11396486, @SuzanneWood-WMDE wrote: > I tried adding the setting of `EVENTLOGGING_IMAGE=docker-registry.wikimedia.org/wikimedia... [08:45:41] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 13Patch-For-Review: Implement a new pipeline and table with reconciled historical revision data - https://phabricator.wikimedia.org/T410688#11404258 (10JAllemandou) Some data validation on joining for `user_central_id`: ` WITH centralauth AS ( SE... [09:12:00] 06Data-Engineering, 06Data-Platform-SRE (2025.11.07 - 2025.11.28), 13Patch-For-Review: Bump hadoop container maximum memory size - https://phabricator.wikimedia.org/T410966#11404371 (10brouberol) a:03brouberol [09:19:15] 06Data-Engineering, 06Data-Platform-SRE (2025.11.07 - 2025.11.28), 13Patch-For-Review: Bump hadoop container maximum memory size - https://phabricator.wikimedia.org/T410966#11404408 (10brouberol) [09:19:45] 06Data-Engineering, 06Data-Platform-SRE (2025.11.07 - 2025.11.28), 13Patch-For-Review: Bump hadoop container maximum memory size - https://phabricator.wikimedia.org/T410966#11404409 (10brouberol) a:05brouberol→03JAllemandou [10:00:43] 06Data-Engineering, 06Data-Platform-SRE (2025.11.07 - 2025.11.28), 13Patch-For-Review: Bump hadoop container maximum memory size - https://phabricator.wikimedia.org/T410966#11404586 (10brouberol) Deployed to both `an-master` hosts, and I restarted `hadoop-yarn-resourcemanager.service` as well. [10:01:00] 06Data-Engineering, 06Data-Platform-SRE (2025.11.07 - 2025.11.28), 13Patch-For-Review: Bump hadoop container maximum memory size - https://phabricator.wikimedia.org/T410966#11404588 (10brouberol) 05Open→03Resolved [10:07:57] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10MediaWiki-Page-derived-data, 07OKR-Work: Global Editor Metrics - Druid mediawiki_history_reduced changes - https://phabricator.wikimedia.org/T406069#11404649 (10JAllemandou) After talking with @mforns this morning: * The 4 metrics defined in the... [10:33:01] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10MediaWiki-Page-derived-data, 07OKR-Work: Global Editor Metrics - Druid mediawiki_history_reduced changes - https://phabricator.wikimedia.org/T406069#11404736 (10mforns) Makes sense @JAllemandou! [11:48:08] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop rc_type from recentchanges in wmf production - https://phabricator.wikimedia.org/T410531#11405012 (10Marostegui) [11:49:57] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10Event-Platform, 13Patch-For-Review: Add CI step to event schema repositories to test to fail if a schema is deleted - https://phabricator.wikimedia.org/T377023#11405018 (10JMonton-WMF) Thanks both! [[ https://gitlab.wikimedia.org/repos/data-engin... [11:50:57] 06Data-Engineering: Productize Data for Monthly Active Moderator Actions - https://phabricator.wikimedia.org/T410940#11405025 (10GGoncalves-WMF) Looking at the attached sheet, I see the following must-have actions listed as "complicated": |**Action**|**How common?**|**Ease of measuring**|**Where's the data?**|*... [12:08:46] 06Data-Engineering, 10Data-Engineering-Wikistats, 06Movement-Insights: NEW FEATURE REQUEST: Temp Accounts on Wikistats - https://phabricator.wikimedia.org/T410796#11405064 (10GGoncalves-WMF) Thanks for checking! What annotation do you have in mind? Something like, //"Here's when we enabled temp accounts, an... [12:52:14] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to Analytics_Privatedata for Chandra-WMDE - https://phabricator.wikimedia.org/T409707#11405197 (10Milimetric) Approved sorry to miss the previous ping [13:44:58] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10MediaWiki-Page-derived-data, 07OKR-Work: Add user_central_id to mediawiki_content_history_v1 (and mediawiki_content_current_v1) - https://phabricator.wikimedia.org/T406515#11405334 (10xcollazo) I attempted to backfill this table yesterday, just f... [13:47:08] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 13Patch-For-Review: Implement a new pipeline and table with reconciled historical revision data - https://phabricator.wikimedia.org/T410688#11405340 (10xcollazo) >>! In T410688#11404175, @JAllemandou wrote: >>>! In T410688#11397421, @xcollazo wrote:... [15:14:30] 06Data-Engineering: Productize Data for Monthly Active Moderator Actions - https://phabricator.wikimedia.org/T410940#11405854 (10fkaelin) Great to hear. @GGoncalves-WMF your educated guess is indeed such. Here some additional context **Content diff** - [[ https://gitlab.wikimedia.org/repos/research/research-dat... [15:23:23] 06Data-Engineering, 06Data-Engineering-Radar, 10Observability-Logging, 06serviceops, and 2 others: Fix Kafka replicas skew - https://phabricator.wikimedia.org/T407185#11405911 (10Clement_Goubert) Waiting until {T405950} is done with moving the `kafka-main` nodes so we don't run into a network blip if the r... [16:11:59] 06Data-Engineering, 06Experimentation Lab, 10MediaWiki-extensions-EventLogging, 07Essential-Work: Remove mw.eventLog.id - https://phabricator.wikimedia.org/T408179#11406235 (10Milimetric) p:05Triage→03Medium [16:20:57] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Persistence, 13Patch-For-Review: Provision Global Editor Metrics tables & endpoints - https://phabricator.wikimedia.org/T410962#11406277 (10Ahoelzl) > The more I look at this top k endpoint, the more I think I may have misunderstood what was... [16:36:56] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE, 06SRE, 10Event-Platform: Discovery for Kafka cluster brokers - https://phabricator.wikimedia.org/T213561#11406384 (10brouberol) a:05brouberol→03None [16:54:24] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Persistence, 13Patch-For-Review: Provision Global Editor Metrics tables & endpoints - https://phabricator.wikimedia.org/T410962#11406480 (10Eevans) >>! In T410962#11406277, @Ahoelzl wrote: >> The more I look at this top k endpoint, the more... [17:55:29] 06Data-Engineering, 10Event-Platform: EventBus tests fail without EventStreamConfig - https://phabricator.wikimedia.org/T410056#11406744 (10Ottomata) [18:02:36] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to Analytics_Privatedata for Chandra-WMDE - https://phabricator.wikimedia.org/T409707#11406782 (10RLazarus) [18:10:28] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to Analytics_Privatedata for Chandra-WMDE - https://phabricator.wikimedia.org/T409707#11406826 (10RLazarus) 05In progress→03Resolved a:03RLazarus Thanks @Milimetric! Added to `nda`: ` rzl@ldap-maint1001:~$ ld... [18:32:27] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10MediaWiki-Page-derived-data, 07OKR-Work: Global Editor Metrics - Druid mediawiki_history_reduced changes - https://phabricator.wikimedia.org/T406069#11406936 (10Ottomata) For my own (out of the loop) understanding, here are the changes to previou... [18:33:15] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 13Patch-For-Review: Implement a new pipeline and table with reconciled historical revision data - https://phabricator.wikimedia.org/T410688#11406937 (10JAllemandou) Something else needed in order to get the data in Druid with all the needed dimension... [20:17:59] 06Data-Engineering, 10DPE-Mediawiki-Content: Consult Dumps users and other community members about the future of Dumps - https://phabricator.wikimedia.org/T337887#11407307 (10Quiddity) a:05Quiddity→03None Removing myself as assignee. I believe this task was handed off to folks with more topical expertise,... [20:44:33] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 13Patch-For-Review: Implement a new pipeline and table with reconciled historical revision data - https://phabricator.wikimedia.org/T410688#11407372 (10xcollazo) DDL as of now: https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-content-pip... [20:49:30] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 13Patch-For-Review: Implement a new pipeline and table with reconciled historical revision data - https://phabricator.wikimedia.org/T410688#11407379 (10xcollazo) Hmm, the conditions must be wrong, as a count check is missing 1B rows: ` spark-sql (def... [20:58:36] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 13Patch-For-Review: Implement a new pipeline and table with reconciled historical revision data - https://phabricator.wikimedia.org/T410688#11407437 (10xcollazo) Moved the predicates to the ON condition so that they only apply to the right table. ` .... [21:55:41] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Update thresholds configuration for MediaWiki History Reduced error checks - https://phabricator.wikimedia.org/T409782#11407599 (10amastilovic) 05Open→03Resolved