[01:22:15] 06serviceops, 10envoy, 06SRE: Upgrade Envoy to v1.35.6 - https://phabricator.wikimedia.org/T410975 (10RLazarus) 03NEW [03:59:16] 06serviceops, 07Epic, 06MediaWiki-Platform-Team (Kanban Board): Migrate Wikimedia production from PHP 8.1 to PHP 8.3 - https://phabricator.wikimedia.org/T360995#11404005 (10Jdforrester-WMF) [08:37:05] 06serviceops, 06Content-Transform-Team, 10Wikifeeds, 06Wikipedia-Android-App-Backlog: Significant increase in wikifeeds latency and mobileapps error rate since 2025/11/13 - https://phabricator.wikimedia.org/T410296#11404236 (10Jgiannelos) From a quick research I agree that this the memory management introd... [09:37:04] hnowlan: https://grafana.wikimedia.org/goto/25673emDg?orgId=1 looks like you got the 2s chance in place right on time? [09:41:22] *change [10:26:15] vgutierrez: not yet! https://gerrit.wikimedia.org/r/q/topic:%22hnowlan/thumbor_queue_time%22 [10:26:52] looks like that was a fairly legit queuing incident [10:53:48] 06serviceops, 10CXServer, 10LPL Projects (Other), 07Unplanned-Sprint-Work: cxserver: Remove Yandex MT key from production - https://phabricator.wikimedia.org/T408138#11404861 (10jijiki) @KartikMistry I will have a look, sorry for that [13:17:40] hnowlan: any reason to not go straightforward to 2s? [14:51:44] vgutierrez: I'd be curious to see if there's any dramatic change at 10s, but I'd be okay to just test that for a few minutes [15:07:13] hnowlan: cool, thx [15:10:01] 06serviceops, 06Connection-Team: MediaWiki periodic job campaignevents-aggregateanswers-metawiki failed - https://phabricator.wikimedia.org/T410748#11405819 (10Clement_Goubert) The commands should be run on `deployment.eqiad.wmnet`, these are in the task just for ease of copy/pasting, a more complete troublesh... [15:11:39] 06serviceops, 06MW-Interfaces-Team, 07OKR-Work: Capacity plan for REST gateway - https://phabricator.wikimedia.org/T400490#11405835 (10Clement_Goubert) 05Open→03Resolved a:03Clement_Goubert [15:23:23] 06serviceops, 06Data-Engineering, 06Data-Engineering-Radar, 10Observability-Logging, and 2 others: Fix Kafka replicas skew - https://phabricator.wikimedia.org/T407185#11405911 (10Clement_Goubert) Waiting until {T405950} is done with moving the `kafka-main` nodes so we don't run into a network blip if the r... [15:46:28] 06serviceops, 10CXServer, 10LPL Projects (Other), 07Unplanned-Sprint-Work: cxserver: Remove Yandex MT key from production - https://phabricator.wikimedia.org/T408138#11406066 (10jijiki) 05Open→03Resolved Key has been removed from puppet, please reopen if something is not right [16:29:21] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: eqiad row C/D Service Ops host migrations - https://phabricator.wikimedia.org/T405950#11406316 (10RobH) Both kafka-main100[89] moved, last one to move is wikikube-ctrl1003 [16:47:07] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: eqiad row C/D Service Ops host migrations - https://phabricator.wikimedia.org/T405950#11406436 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by cgoubert@cumin1003 depool for host wikikube-ctrl1003.eqiad.wmnet completed: - wikikube-ctr... [16:50:44] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: eqiad row C/D Service Ops host migrations - https://phabricator.wikimedia.org/T405950#11406451 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by cgoubert@cumin1003 pool for host wikikube-ctrl1003.eqiad.wmnet completed: - wikikube-ctrl1... [16:53:00] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: eqiad row C/D Service Ops host migrations - https://phabricator.wikimedia.org/T405950#11406473 (10Clement_Goubert) 05In progress→03Resolved All ServiceOps hosts have been migrated to the new switch. [17:12:00] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: eqiad row C/D Service Ops host migrations - https://phabricator.wikimedia.org/T405950#11406575 (10Jclark-ctr) a:05brouberol→03Jclark-ctr [18:32:02] 06serviceops, 13Patch-For-Review: MediaWiki on PHP 8.3 production workload migration - https://phabricator.wikimedia.org/T405955#11406934 (10Scott_French) [18:37:20] 06serviceops, 13Patch-For-Review: MediaWiki on PHP 8.3 production workload migration - https://phabricator.wikimedia.org/T405955#11406968 (10Scott_French) At this point, there are no active (or potential, as in the case of `mwscript-k8s`) consumers of the `publish-81` image flavour, so we should be unblocked t... [18:40:38] 06serviceops, 06Connection-Team: MediaWiki periodic job campaignevents-aggregateanswers-metawiki failed - https://phabricator.wikimedia.org/T410748#11406975 (10Daimona) >>! In T410748#11405819, @Clement_Goubert wrote: > The commands should be run on `deployment.eqiad.wmnet`, these are in the task just for ease... [18:48:27] 06serviceops, 13Patch-For-Review: MediaWiki on PHP 8.3 production workload migration - https://phabricator.wikimedia.org/T405955#11406998 (10Scott_French) Alright, https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/220 is live, and scap's `build-and-push-container-images` phase no longer builds... [18:49:49] 06serviceops, 13Patch-For-Review: MediaWiki on PHP 8.3 production workload migration - https://phabricator.wikimedia.org/T405955#11407001 (10Scott_French) [19:21:02] 06serviceops, 07Datacenter-Switchover: Simplify switchover of deployment server - https://phabricator.wikimedia.org/T348990#11407091 (10Raine) p:05Triage→03Low a:05Raine→03None [19:22:03] 06serviceops, 07Datacenter-Switchover, 07good first task: Automate the "make sure email works" step of the DC switchover - https://phabricator.wikimedia.org/T348985#11407103 (10Raine) Thank you for tagging this task with #good_first_task for Wikimedia newcomers! Newcomers often may not be aware of things th... [20:02:50] 06serviceops, 07Datacenter-Switchover: Automate the "make sure email works" step of the DC switchover - https://phabricator.wikimedia.org/T348985#11407240 (10taavi) [23:12:00] 06serviceops, 10MinT, 10Prod-Kubernetes, 06SRE: machinetranslation eqiad pods in state ContainerStatusUnknown - https://phabricator.wikimedia.org/T411058 (10RLazarus) 03NEW p:05Triage→03High