[00:10:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=codfw%2Bprometheus/k8s&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [00:15:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=codfw%2Bprometheus/k8s&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [03:55:08] 06Data-Engineering, 10Dumps-Generation, 06Wikibase Reuse Team, 10Wikidata, and 4 others: No Wikidata dumps for Week 40 of 2025 (recurring issue) - https://phabricator.wikimedia.org/T406429#11279782 (10Hannah_Bast) @BTullis @dcausse In the meantime, would it be an option to just ask Blazegraph the query `CO... [07:14:48] 07Analytics-Data-Problem, 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Unique devices data uses non-standard domains for Wikidata, Wikifunctions, and MediaWiki.org - https://phabricator.wikimedia.org/T405533#11280075 (10JAllemandou) Patch has been deployed, last 2 days of backfill ongoing, call... [08:10:08] 06Data-Engineering, 06Data-Engineering-Radar, 10Observability-Logging, 06serviceops, 06Data-Platform-SRE (2025.09.26 - 2025.10.17): Fix Kafka replicas skew - https://phabricator.wikimedia.org/T407185#11280200 (10Gehel) [08:12:24] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE (2025.09.26 - 2025.10.17): Create a dbt Docker container - https://phabricator.wikimedia.org/T406636#11280207 (10Gehel) p:05Triage→03Medium [08:13:03] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE (2025.09.26 - 2025.10.17): Set up a working, usable dbt installation on stat boxes - https://phabricator.wikimedia.org/T406634#11280211 (10Gehel) p:05Triage→03Medium [08:13:15] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE (2025.09.26 - 2025.10.17): Add dbt related packages to conda-analytics - https://phabricator.wikimedia.org/T406766#11280213 (10Gehel) [08:13:22] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE (2025.09.26 - 2025.10.17): Add dbt related packages to conda-analytics - https://phabricator.wikimedia.org/T406766#11280214 (10Gehel) p:05Triage→03Medium [08:13:56] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE (2025.09.26 - 2025.10.17): Add dbt related packages to conda-analytics - https://phabricator.wikimedia.org/T406767#11280217 (10Gehel) p:05Triage→03Medium [08:16:36] 06Data-Engineering, 10BetaFeatures, 06cloud-services-team, 06Data-Platform-SRE, 10Data-Services: Create view for betafeatures_user_counts table in wiki replicas - https://phabricator.wikimedia.org/T402145#11280232 (10Gehel) I pinged #data-engineering on slack to get validation. [08:31:19] 06Data-Engineering, 06Data-Persistence, 06DBA, 10MediaWiki-Core-Revision-backend, and 6 others: Rethink rev_sha1 field - https://phabricator.wikimedia.org/T389026#11280280 (10JAllemandou) Hi Mediawiki-core and Data-Persistence folks, while working on adapting data-engineering pipelines to the `rev_sha1` re... [08:34:48] 06Data-Engineering, 10BetaFeatures, 06cloud-services-team, 06Data-Platform-SRE, 10Data-Services: Create view for betafeatures_user_counts table in wiki replicas - https://phabricator.wikimedia.org/T402145#11280297 (10JAllemandou) I have reviewed the table, it contains only a features list with how many u... [09:27:13] 06Data-Engineering, 10ChangeProp, 10Citoid, 06Editing-team, and 19 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#11280488 (10Mvolz) [10:17:52] 06Data-Engineering, 06Data-Persistence, 06DBA, 10MediaWiki-Core-Revision-backend, and 6 others: Rethink rev_sha1 field - https://phabricator.wikimedia.org/T389026#11280655 (10Ladsgroup) Thanks for the catch. I try to make sure it's properly fixed after a good cry. [10:38:54] (03CR) 10Aqu: [C:03+1] Improve mediawiki_history previous patch [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1196518 (https://phabricator.wikimedia.org/T406000) (owner: 10Joal) [10:58:18] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=codfw%2Bprometheus/k8s&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [11:01:01] joal: hii, the user I shared the doc with is zabe :) [11:02:22] Hi Amir1 - ok! let me know how I can help [11:03:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=codfw%2Bprometheus/k8s&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [11:05:07] thanks [11:15:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=codfw%2Bprometheus/k8s&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [11:20:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=codfw%2Bprometheus/k8s&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [11:38:34] 06Data-Engineering, 06cloud-services-team, 06Data-Persistence, 10Data-Services, and 2 others: Set up x1 replication to an-redacteddb1001 - https://phabricator.wikimedia.org/T407485 (10BTullis) 03NEW [11:38:37] 06Data-Engineering, 06cloud-services-team, 06Data-Persistence, 10Data-Services, and 2 others: Set up x1 replication to an-redacteddb1001 - https://phabricator.wikimedia.org/T407486 (10BTullis) 03NEW [11:39:24] 06Data-Engineering, 06cloud-services-team, 06Data-Persistence, 10Data-Services, and 2 others: Set up x1 replication to an-redacteddb1001 - https://phabricator.wikimedia.org/T407486#11280899 (10BTullis) →14Duplicate dup:03T407485 [11:39:25] 06Data-Engineering, 06cloud-services-team, 06Data-Persistence, 10Data-Services, and 2 others: Set up x1 replication to an-redacteddb1001 - https://phabricator.wikimedia.org/T407485#11280901 (10BTullis) [11:50:26] 06Data-Engineering, 06cloud-services-team, 06Data-Persistence, 06Data-Platform-SRE, and 2 others: Set up x1 replication to Wiki Replicas - https://phabricator.wikimedia.org/T395881#11280914 (10BTullis) I created T407485 to track the work required to add this section to `an-redacteddb1001` and set up the in... [11:56:23] (03CR) 10Mforns: [C:03+1] "LGMT!" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1196518 (https://phabricator.wikimedia.org/T406000) (owner: 10Joal) [12:07:09] 06Data-Engineering, 06cloud-services-team, 06Data-Persistence, 06Data-Platform-SRE, and 2 others: Set up x1 replication to Wiki Replicas - https://phabricator.wikimedia.org/T395881#11280962 (10Ladsgroup) It's worth noting that because of the table catalog (T363581), now all tables that should be fully filt... [12:12:08] 06Data-Engineering, 10DPE-Mediawiki-Content, 10Dumps-Generation, 06SRE, 07Epic: Dumps generation cause disruption to the production environment - https://phabricator.wikimedia.org/T368098#11280973 (10BTullis) Hello again. It looks like the wikibase dumps performance issue described in {T389199} may have... [12:49:49] 06Data-Engineering, 10DPE-Mediawiki-Content, 10Dumps-Generation, 06SRE, 07Epic: Dumps generation cause disruption to the production environment - https://phabricator.wikimedia.org/T368098#11281067 (10Ladsgroup) Doing it temporarily and specially on the wikibase dumpers should be fineTM (we had a lot of i... [13:04:27] (03CR) 10Snwachukwu: [C:03+2] Improve mediawiki_history previous patch [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1196518 (https://phabricator.wikimedia.org/T406000) (owner: 10Joal) [13:04:34] (03CR) 10CI reject: [V:04-1] Improve mediawiki_history previous patch [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1196518 (https://phabricator.wikimedia.org/T406000) (owner: 10Joal) [13:05:40] (03CR) 10Snwachukwu: [C:03+1] Improve mediawiki_history previous patch [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1196518 (https://phabricator.wikimedia.org/T406000) (owner: 10Joal) [13:39:25] (03PS2) 10Joal: Improve mediawiki_history previous patch [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1196518 (https://phabricator.wikimedia.org/T406000) [13:46:39] (03CR) 10CI reject: [V:04-1] Improve mediawiki_history previous patch [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1196518 (https://phabricator.wikimedia.org/T406000) (owner: 10Joal) [14:18:46] (03PS2) 10Snwachukwu: Update changelog.md for v0.3.4 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1195272 [14:21:20] (03PS3) 10Snwachukwu: Update changelog.md for v0.3.4 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1195272 [14:22:32] (03PS4) 10Snwachukwu: Update changelog.md for v0.3.4 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1195272 [14:29:24] (03CR) 10Xcollazo: Update changelog.md for v0.3.4 (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1195272 (owner: 10Snwachukwu) [15:19:15] (03PS3) 10Joal: Improve mediawiki_history previous patch [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1196518 (https://phabricator.wikimedia.org/T406000) [15:42:10] (03CR) 10Xcollazo: [C:03+2] Improve mediawiki_history previous patch [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1196518 (https://phabricator.wikimedia.org/T406000) (owner: 10Joal) [15:44:02] (03CR) 10Xcollazo: [C:03+2] Update changelog.md for v0.3.4 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1195272 (owner: 10Snwachukwu) [15:59:17] (03Merged) 10jenkins-bot: Improve mediawiki_history previous patch [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1196518 (https://phabricator.wikimedia.org/T406000) (owner: 10Joal) [16:00:06] (03Merged) 10jenkins-bot: Update changelog.md for v0.3.4 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1195272 (owner: 10Snwachukwu) [16:12:19] 06Data-Engineering, 06Discovery-Search: Ignore MacOS .DS_Store in parent pom - https://phabricator.wikimedia.org/T407514 (10TJones) 03NEW [16:16:41] Starting build #49 for job analytics-refinery-maven-release [16:34:25] (03PS1) 10Joal: Nullify sha1 from revision and archive in sqoop [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1196716 (https://phabricator.wikimedia.org/T406000) [16:36:29] (03CR) 10Mforns: [C:03+2] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1196716 (https://phabricator.wikimedia.org/T406000) (owner: 10Joal) [16:36:35] (03CR) 10Mforns: [V:03+2 C:03+2] Nullify sha1 from revision and archive in sqoop [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1196716 (https://phabricator.wikimedia.org/T406000) (owner: 10Joal) [16:40:20] Project analytics-refinery-maven-release build #49: 09SUCCESS in 23 min: https://integration.wikimedia.org/ci/job/analytics-refinery-maven-release/49/ [16:42:00] Starting build #34 for job analytics-refinery-update-jars [16:42:43] (03PS1) 10Maven-release-user: Add refinery-source jars for v0.3.4 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1196717 [16:42:44] Project analytics-refinery-update-jars build #34: 09SUCCESS in 44 sec: https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars/34/ [16:44:36] (03CR) 10Mforns: [C:03+2] Add refinery-source jars for v0.3.4 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1196717 (owner: 10Maven-release-user) [16:44:38] (03CR) 10Mforns: [V:03+2 C:03+2] Add refinery-source jars for v0.3.4 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1196717 (owner: 10Maven-release-user) [16:46:09] !log Deployed refinery-source 0.3.4 (https://github.com/wikimedia/analytics-refinery-source/commit/c3536f39c76923edd9c80b2a58b5d49de21d60e0) [16:46:11] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:48:20] 06Data-Engineering, 06Discovery-Search, 07Essential-Work: Ignore MacOS .DS_Store in parent pom - https://phabricator.wikimedia.org/T407514#11282037 (10TJones) [16:53:17] !log Starting refinery deployment at 2c3754c5e02458859c4310df5b92b42522727752 Nullify sha1 from revision and archive in sqoop [16:53:19] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:58:51] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10DPE-Mediawiki-Content, 13Patch-For-Review: Adapt MW Content pipelines to the removal of upstream revision.rev_sha1 - https://phabricator.wikimedia.org/T405641#11282056 (10xcollazo) [16:59:23] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10DPE-Mediawiki-Content, 13Patch-For-Review: Adapt MW Content pipelines to the removal of upstream revision.rev_sha1 - https://phabricator.wikimedia.org/T405641#11282057 (10xcollazo) Just added this decision to the description of this task: **Deci... [17:23:10] !log Finished refinery deployment at 2c3754c5e02458859c4310df5b92b42522727752 Nullify sha1 from revision and archive in sqoop [17:23:11] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:52:19] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10MediaWiki-Page-derived-data, 07OKR-Work: Add user_central_id to mediawiki_content_history_v1 (and mediawiki_content_current_v1) - https://phabricator.wikimedia.org/T406515#11282359 (10xcollazo) a:05amastilovic→03xcollazo As agreed with @amast... [18:24:13] 06Data-Engineering, 10BetaFeatures, 06cloud-services-team, 06Data-Platform-SRE, 10Data-Services: Create view for betafeatures_user_counts table in wiki replicas - https://phabricator.wikimedia.org/T402145#11282523 (10Ahoelzl) Thanks @Gehel for asking for clarification. I agree with Joseph and approve the... [18:25:28] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10MediaWiki-Page-derived-data, 07OKR-Work: Add user_central_id to mediawiki_content_history_v1 (and mediawiki_content_current_v1) - https://phabricator.wikimedia.org/T406515#11282530 (10xcollazo) It doesn't look like the [[ https://www.mediawiki.or... [18:28:10] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10MediaWiki-Page-derived-data, 07OKR-Work: Add user_central_id to mediawiki_content_history_v1 (and mediawiki_content_current_v1) - https://phabricator.wikimedia.org/T406515#11282546 (10Ottomata) Oh hm yes indeed. > reconcile is looking difficul... [18:36:58] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10MediaWiki-Page-derived-data, 07OKR-Work: Global Editor Metrics - Data Pipeline - https://phabricator.wikimedia.org/T405039#11282571 (10Ottomata) [22:02:16] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data-Engineering-Wikistats, 07Russian-Sites: "Active editors by country" doesn't display numbers for Belarus, Kazakhstan, Russia - https://phabricator.wikimedia.org/T333716#11283536 (10Iniquity) What's the plan for implementing the new data? I just noticed t... [22:05:24] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data-Engineering-Wikistats, 06Wikimedia Enterprise, 07Russian-Sites: "Active editors by country" doesn't display numbers for Belarus, Kazakhstan, Russia - https://phabricator.wikimedia.org/T333716#11283540 (10Iniquity) [23:45:51] 06Data-Engineering, 10CampaignEvents, 06DBA, 06Connection-Team (Connection-Current-Sprint), 07Schema-change-in-production: Apply ce_event_contributions schema changes in production (x1) - https://phabricator.wikimedia.org/T407587 (10Daimona) 03NEW [23:46:00] 06Data-Engineering, 10CampaignEvents, 06DBA, 06Connection-Team (Connection-Current-Sprint), 07Schema-change-in-production: Apply ce_event_contributions schema changes in production (x1) - https://phabricator.wikimedia.org/T407587#11283649 (10Daimona)