[03:18:33] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [06:59:54] 06Data-Engineering, 10Dumps-Generation, 06Growth-Team, 10Wikidata: No wikidata dumps this week (20250630) - https://phabricator.wikimedia.org/T398756#10977525 (10Michael) [07:18:33] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [07:28:52] 06Data-Engineering, 10Dumps-Generation, 06Growth-Team, 10Wikidata: No wikidata dumps this week (20250630) - https://phabricator.wikimedia.org/T398756#10977584 (10Lydia_Pintscher) p:05Triage→03High [07:37:10] 06Data-Engineering, 10Dumps-Generation, 06Growth-Team, 10Wikidata: No wikidata dumps this week (20250630) - https://phabricator.wikimedia.org/T398756#10977596 (10brouberol) This is on the Data Platform SRE team radar. [07:37:32] 06Data-Engineering, 10Dumps-Generation, 06Growth-Team, 10Wikidata, 10Data-Platform-SRE (2025.07.05 - 2025.07.25): No wikidata dumps this week (20250630) - https://phabricator.wikimedia.org/T398756#10977599 (10brouberol) 05Open→03In progress [07:37:33] 06Data-Engineering, 10Dumps-Generation, 06Growth-Team, 10Wikidata, 10Data-Platform-SRE (2025.07.05 - 2025.07.25): No wikidata dumps this week (20250630) - https://phabricator.wikimedia.org/T398756#10977598 (10brouberol) [09:19:38] 06Data-Engineering, 10Dumps-Generation, 06Growth-Team, 10Wikidata, 10Data-Platform-SRE (2025.07.05 - 2025.07.25): No wikidata dumps this week (20250630) - https://phabricator.wikimedia.org/T398756#10978033 (10BTullis) a:03BTullis [09:37:27] 06Data-Engineering, 10Dumps-Generation, 06Growth-Team, 10Wikidata, 10Data-Platform-SRE (2025.07.05 - 2025.07.25): No wikidata dumps this week (20250630) - https://phabricator.wikimedia.org/T398756#10978140 (10BTullis) Hello, Apologies for this. The dumps have been created, but they have been inadvertentl... [09:42:58] 06Data-Engineering, 06Data-Engineering-Radar, 10ConfirmEdit (CAPTCHA extension), 10MediaWiki-extensions-Campaigns, and 3 others: Send hCaptcha API response data to event platform - https://phabricator.wikimedia.org/T379179#10978210 (10kostajh) [09:56:39] 06Data-Engineering, 10Dumps-Generation, 10Wikidata, 10Data-Platform-SRE (2025.07.05 - 2025.07.25), 13Patch-For-Review: No wikidata dumps this week (20250630) - https://phabricator.wikimedia.org/T398756#10978297 (10Michael) (Turns out, image suggestions, and thus Growth and Apps, don't actually depend on... [09:59:14] 06Data-Engineering, 10Dumps-Generation, 10Wikidata, 10Data-Platform-SRE (2025.07.05 - 2025.07.25), 13Patch-For-Review: No wikidata dumps this week (20250630) - https://phabricator.wikimedia.org/T398756#10978312 (10BTullis) The reason that the `wikidatawiki/entities` link works is because of this symlink.... [10:51:27] 06Data-Engineering, 10Data-Platform-SRE (2025.07.05 - 2025.07.25): Errors from refinery-sqoop-whole-mediawiki.service - July 2025 - https://phabricator.wikimedia.org/T398602#10978438 (10BTullis) I manually executed the command and the error is: `Table 'etwiki.ipblocks' doesn't exist` ` analytics@an-launcher100... [10:54:12] 06Data-Engineering, 10Data-Platform-SRE (2025.07.05 - 2025.07.25): Errors from refinery-sqoop-whole-mediawiki.service - July 2025 - https://phabricator.wikimedia.org/T398602#10978447 (10BTullis) It looks like the `ipblocks` table was completely dropped in {T367632}. [11:15:43] 06Data-Engineering, 10Dumps-Generation, 10Wikidata, 10Data-Platform-SRE (2025.07.05 - 2025.07.25), 13Patch-For-Review: No wikidata dumps this week (20250630) - https://phabricator.wikimedia.org/T398756#10978518 (10BTullis) >>! In T398756#10978297, @Michael wrote: > (Turns out, image suggestions, and thus... [11:18:33] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [11:30:53] 06Data-Engineering, 10Dumps-Generation, 10Wikidata, 10Data-Platform-SRE (2025.07.05 - 2025.07.25): No wikidata dumps this week (20250630) - https://phabricator.wikimedia.org/T398756#10978548 (10BTullis) OK, the code is fixed. I have manually triggered [[https://airflow-test-k8s.wikimedia.org/dags/sync_wiki... [11:47:09] 06Data-Engineering, 10Dumps-Generation, 10Wikidata, 10Data-Platform-SRE (2025.07.05 - 2025.07.25): No wikidata dumps this week (20250630) - https://phabricator.wikimedia.org/T398756#10978568 (10BTullis) That seems fine now. {F63346957,width=60%} I will manually trigger the wikibase/wikidatawiki sync job. [12:58:09] 06Data-Engineering, 10Data-Platform-SRE (2025.07.05 - 2025.07.25), 13Patch-For-Review: Errors from refinery-sqoop-whole-mediawiki.service - July 2025 - https://phabricator.wikimedia.org/T398602#10978822 (10BTullis) Now executing `/usr/local/bin/refinery-sqoop-mediawiki-production-not-history` again on an-lau... [13:25:42] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10MediaWiki-DomainEvents, 07Epic, 10Event-Platform, and 3 others: Testing the domain event refactoring with production data - https://phabricator.wikimedia.org/T394899#10978908 (10gmodena) [13:26:03] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Data-Platform-SRE (2025.07.05 - 2025.07.25), 13Patch-For-Review: deployment-charts: remove deprecated mediawiki-content-history config - https://phabricator.wikimedia.org/T396593#10978910 (10brouberol) [13:26:06] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10MediaWiki-DomainEvents, 07Epic, 10Event-Platform, and 3 others: EventBus: replace ArticleRevisionVisibilitySetHook with a RevisionVisibilityChangedEvent listener - https://phabricator.wikimedia.org/T396213#10978912 (10gmodena) [13:26:10] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Data-Platform-SRE (2025.07.05 - 2025.07.25), 13Patch-For-Review: deployment-charts: remove deprecated mediawiki-content-history config - https://phabricator.wikimedia.org/T396593#10978913 (10brouberol) 05Open→03In progress [13:26:39] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10MediaWiki-DomainEvents, 07Epic, 10Event-Platform, and 3 others: PageChangeEventSerializer should deprecate WikiPage and adopt PageIdentity - https://phabricator.wikimedia.org/T396453#10978922 (10gmodena) [13:26:54] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Event-Platform, 10MW-1.45-notes (1.45.0-wmf.8; 2025-07-01): mediawiki.page_change.v1 should not contain events for undelete into existing pages. - https://phabricator.wikimedia.org/T395327#10978925 (10gmodena) [13:27:22] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10MediaWiki-DomainEvents, 07Epic, 10Event-Platform, and 3 others: EventBus: replace PageMoveCompleteHook with PageMovedEvent - https://phabricator.wikimedia.org/T393890#10978927 (10gmodena) [13:27:51] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10MediaWiki-DomainEvents, 07Epic, 10Event-Platform, and 3 others: EventBus: Replace PageUndeleteCompleteHook with PageRevisionUpdated - https://phabricator.wikimedia.org/T393891#10978929 (10gmodena) [13:40:05] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Data-Platform-SRE (2025.07.05 - 2025.07.25): deployment-charts: remove deprecated mediawiki-content-history config - https://phabricator.wikimedia.org/T396593#10978991 (10brouberol) Both apps were removed: ` brouberol@deploy1003:/srv/deployment-charts/helmf... [13:41:05] 06Data-Engineering, 06DBA, 10MediaWiki-Page-derived-data: Normalize categorylinks table - https://phabricator.wikimedia.org/T299951#10979008 (10Zabe) [13:41:20] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Data-Platform-SRE (2025.07.05 - 2025.07.25): deployment-charts: remove deprecated mediawiki-content-history config - https://phabricator.wikimedia.org/T396593#10979009 (10brouberol) The following `admin_ng` patch was applied: `lang=diff flink-operator,... [13:41:23] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Data-Platform-SRE (2025.07.05 - 2025.07.25): deployment-charts: remove deprecated mediawiki-content-history config - https://phabricator.wikimedia.org/T396593#10979011 (10brouberol) 05In progress→03Resolved [13:53:32] 06Data-Engineering: Sqoop block and block_target as replacement to the ipblocks table - https://phabricator.wikimedia.org/T398828 (10xcollazo) 03NEW [14:02:47] (03CR) 10Milimetric: [C:03+2] Enable arabic language in Wikimedia Statistics [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/1112849 (https://phabricator.wikimedia.org/T384265) (owner: 10Mr.Ibrahem) [14:03:15] (03CR) 10Milimetric: [V:03+2 C:03+2] Enable arabic language in Wikimedia Statistics [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/1112849 (https://phabricator.wikimedia.org/T384265) (owner: 10Mr.Ibrahem) [14:17:37] (03CR) 10Milimetric: [V:03+2 C:03+2] "Unfortunately numbro.js doesn't define what arabic numbers should look like. So I'm going to make it think it's english for now, and if a" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/1112849 (https://phabricator.wikimedia.org/T384265) (owner: 10Mr.Ibrahem) [14:18:18] 06Data-Engineering, 10Data-Engineering-Wikistats, 13Patch-For-Review: Enable arabic language in Wikimedia Statistics - https://phabricator.wikimedia.org/T384265#10979201 (10Milimetric) I'm merging this, and deploying it. Unfortunately numbro.js doesn't define what arabic numbers should look like. So I'm goi... [14:19:04] !log running a rolling-restart of kafka-jumbo - T398826 [14:19:07] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:19:07] T398826: Bring kafka-jumbo101[6-8] into service - https://phabricator.wikimedia.org/T398826 [14:24:35] 06Data-Engineering, 06DBA, 10MediaWiki-Page-derived-data: Normalize categorylinks table - https://phabricator.wikimedia.org/T299951#10979252 (10Milimetric) retroactive +1 added to the review. Thanks for the patch. [14:26:38] (03PS1) 10Milimetric: Release 2.10.5 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/1166845 [14:26:54] (03CR) 10Milimetric: [C:03+2] Release 2.10.5 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/1166845 (owner: 10Milimetric) [14:28:21] (03Merged) 10jenkins-bot: Release 2.10.5 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/1166845 (owner: 10Milimetric) [14:38:16] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10EventStreams, 13Patch-For-Review: Figure out how Eventstreams connected client metrics went negative - https://phabricator.wikimedia.org/T398325#10979368 (10tchin) Was digging through the logs and found a bunch of [[ https://logstash.wikimedia.org/goto/0e... [14:41:11] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10EventStreams, 13Patch-For-Review: Figure out how Eventstreams connected client metrics went negative - https://phabricator.wikimedia.org/T398325#10979395 (10tchin) I should also mention that metrics eventually recovered, probably due to T383977 having uni... [15:18:33] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [15:44:09] 06Data-Engineering, 10Data-Platform-SRE (2025.07.05 - 2025.07.25): Errors from refinery-sqoop-whole-mediawiki.service - July 2025 - https://phabricator.wikimedia.org/T398602#10979877 (10BTullis) 05Open→03Resolved This has now finished, so I think that means the whole sqoop process has finished successf... [15:47:07] 06Data-Engineering, 10Dumps-Generation, 10Wikidata, 10Data-Platform-SRE (2025.07.05 - 2025.07.25): No wikidata dumps this week (20250630) - https://phabricator.wikimedia.org/T398756#10979889 (10BTullis) 05In progress→03Resolved The [[https://airflow-test-k8s.wikimedia.org/dags/sync_wikibase_wikidat... [16:02:00] 06Data-Engineering, 10Data-Engineering-Wikistats: Enable arabic language in Wikimedia Statistics - https://phabricator.wikimedia.org/T384265#10979950 (10Milimetric) This is now deployed, Arabic can be selected, but it's using english numbers for the reasons above. [16:09:10] 06Data-Engineering, 10Dumps-Generation, 10Wikidata, 10Data-Platform-SRE (2025.07.05 - 2025.07.25): No wikidata dumps this week (20250630) - https://phabricator.wikimedia.org/T398756#10979977 (10Lydia_Pintscher) Thank you for fixing it! [16:19:44] 06Data-Engineering, 06Diffusion-Repository-Administrators, 10Dumps-Generation, 10Projects-Cleanup, and 2 others: Undeploy and archive ActiveAbstract - https://phabricator.wikimedia.org/T382069#10980018 (10Jdforrester-WMF) [16:26:38] 06Data-Engineering, 06Diffusion-Repository-Administrators, 10Dumps-Generation, 10Projects-Cleanup, and 2 others: Undeploy and archive ActiveAbstract - https://phabricator.wikimedia.org/T382069#10980041 (10Jdforrester-WMF) [16:27:40] 06Data-Engineering, 06Diffusion-Repository-Administrators, 10Dumps-Generation, 10Projects-Cleanup, and 2 others: Undeploy and archive ActiveAbstract - https://phabricator.wikimedia.org/T382069#10980045 (10Jdforrester-WMF) >>! In T382069#10897001, @Novem_Linguae wrote: > I think the last 3 unmarked items in... [18:14:41] 06Data-Engineering, 06cloud-services-team, 10Data-Services, 13Patch-For-Review: Add global_edit_count to wikireplicas - https://phabricator.wikimedia.org/T344108#10980310 (10SD0001) This will be useful for T355594 too. My script for generating voter lists for global elections can run quite faster if it can... [18:33:29] 06Data-Engineering, 10Dumps-Generation, 10Wikidata, 10Data-Platform-SRE (2025.07.05 - 2025.07.25): No wikidata dumps this week (20250630) - https://phabricator.wikimedia.org/T398756#10980364 (10Melderick) @BTullis Thank you for the quick fix. I will keep an eye on this week dumps. They usually start sh... [18:51:42] 06Data-Engineering, 06DBA, 10MediaWiki-Page-derived-data: Normalize categorylinks table - https://phabricator.wikimedia.org/T299951#10980487 (10Zabe) [18:53:42] 06Data-Engineering, 06DBA, 10MediaWiki-Page-derived-data: Normalize categorylinks table - https://phabricator.wikimedia.org/T299951#10980507 (10Zabe) [19:18:33] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [20:45:11] 06Data-Engineering, 06Diffusion-Repository-Administrators, 10Dumps-Generation, 10Projects-Cleanup, and 2 others: Undeploy and archive ActiveAbstract - https://phabricator.wikimedia.org/T382069#10980979 (10Novem_Linguae) @taavi, would you be willing to do "reparent on All-Archived-Project" using your Gerrit... [23:18:33] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem