[03:18:34] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [06:18:00] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add cl_timestamp_id index to categorylinks table - https://phabricator.wikimedia.org/T399249#10994110 (10Marostegui) a:03Marostegui [06:18:17] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add cl_timestamp_id index to categorylinks table - https://phabricator.wikimedia.org/T399249#10994112 (10Marostegui) p:05Triage→03Medium [06:18:22] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add cl_timestamp_id index to categorylinks table - https://phabricator.wikimedia.org/T399249#10994113 (10Marostegui) p:05Medium→03High [07:18:42] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [08:56:25] hey folks! [08:56:39] trying to ping again :) I'd need to go ahead with a new eventgate stream: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1167438 [08:56:49] who's the best poc for these code changes? [09:08:14] I'd say aqu or ottomata [09:09:45] I agree, but otto is off today. j.oal is also out on leave. Maybe x.collazo when he is online later. [09:10:10] elukey: Is it urgent to get it out today? [09:18:41] thanks! nono I wouldn't really deploy today, it was just a ping :) [09:44:00] 06Data-Engineering, 06DBA, 13Patch-For-Review, 07Schema-change-in-production: Add cl_timestamp_id index to categorylinks table - https://phabricator.wikimedia.org/T399249#10994709 (10Ladsgroup) One second. I think we should bundle the change of PK so we can stop writing to the old columns without needing t... [09:46:02] 06Data-Engineering, 06DBA, 13Patch-For-Review, 07Schema-change-in-production: Add cl_timestamp_id index to categorylinks table - https://phabricator.wikimedia.org/T399249#10994714 (10Ladsgroup) How bad it would be to add this alter table? `alter table categorylinks drop primary key, add primary key(cl_from... [10:18:42] 06Data-Engineering, 10Event-Platform: mediawiki.content_history: flink applications experiencing frequent restarts - https://phabricator.wikimedia.org/T397330#10994777 (10gmodena) >>! In T397330#10948116, @Ahoelzl wrote: > related to spikes observed here https://phabricator.wikimedia.org/T397854 ? We see thes... [11:02:07] 06Data-Engineering, 06DBA, 13Patch-For-Review, 07Schema-change-in-production: Add cl_timestamp_id index to categorylinks table - https://phabricator.wikimedia.org/T399249#10994940 (10Marostegui) Fine from my side if you both are happy with that. [11:18:34] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [13:19:42] 06Data-Engineering, 06DBA, 13Patch-For-Review, 07Schema-change-in-production: Add cl_timestamp_id index to categorylinks table - https://phabricator.wikimedia.org/T399249#10995375 (10Zabe) Sounds good to me [13:21:49] 06Data-Engineering, 06DBA, 13Patch-For-Review, 07Schema-change-in-production: Add cl_timestamp_id index to categorylinks table - https://phabricator.wikimedia.org/T399249#10995386 (10Marostegui) I will amend the patch. [13:21:50] 06Data-Engineering, 10service-utils: Migrate and re-deploy eventgate using new service-utils - https://phabricator.wikimedia.org/T361768#10995389 (10akosiaris) >>! In T361768#10459087, @Ahoelzl wrote: > This includes upgrade to Nodejs 20. Hi! Has this happened? Looking at the images currently deployed per dep... [13:25:53] 10Data-Engineering (Q4 2025 April 1st - June 30th): Investigate mw-page-content-change memory alerts - https://phabricator.wikimedia.org/T397336#10995414 (10gmodena) Re-opening to investigate memory alloc patterns / leaks that we encountered with `mw-page-content-change-enrich` post {T347282} deployment. [14:18:29] hi folks. not sure if known but we have a bunch of Puppet failures on an-worker* hosts in eqiad [14:18:32] https://puppetboard.wikimedia.org/failures [14:18:48] pointing it out since we will go into the weekend soon and might be worth a look [14:18:51] btullis: ^ [14:20:20] sukhe: Thanks. This is known, it's my fault for being lazy and not finding a way to silence these alerts ahead of time. T397160 - I'll add a specific silence for this puppetfailure alert. [14:20:48] T397160: Set 52 hadoop workers into decommissioning status - https://phabricator.wikimedia.org/T397160 [14:20:50] btullis: ah thanks, and no worries. as long as it is known. [14:20:56] thank you [15:18:34] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [15:31:00] 06Data-Engineering, 10service-utils: Migrate and re-deploy eventgate using new service-utils - https://phabricator.wikimedia.org/T361768#10995844 (10tchin) >>! In T361768#10995389, @akosiaris wrote: >>>! In T361768#10459087, @Ahoelzl wrote: >> This includes upgrade to Nodejs 20. > > Hi! Has this happened? Loo... [15:39:02] 06Data-Engineering, 10Commons-Impact-Metrics, 10Commons-Impact-Metrics-Requests: Update Commons Impact Metrics allow-list June 2025 - https://phabricator.wikimedia.org/T398149#10995889 (10GFontenelle_WMF) 05Open→03Resolved [15:39:33] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Commons-Impact-Metrics, 10Commons-Impact-Metrics-Requests: Update Commons Impact Metrics allow-list May 2025 - https://phabricator.wikimedia.org/T395639#10995890 (10GFontenelle_WMF) 05Open→03Resolved [17:58:08] 06Data-Engineering, 06Data-Engineering-Radar, 06Experimentation Lab, 10MediaWiki-extensions-EventLogging, 07Technical-Debt: Remove deprecated $logger parameter from EventLogging::submit() - https://phabricator.wikimedia.org/T376410#10996372 (10phuedx) [18:00:51] 06Data-Engineering, 06Data-Engineering-Radar, 10MediaWiki-extensions-EventLogging, 10Experimentation Lab (Experiment Platform Sprint 9), and 2 others: Remove deprecated $logger parameter from EventLogging::submit() - https://phabricator.wikimedia.org/T376410#10996377 (10phuedx) [19:18:34] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [23:18:34] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem