[08:09:44] 06serviceops, 10Deployments, 13Patch-For-Review, 10Release-Engineering-Team (Radar), 07Wikimedia-production-error: httpb sometimes fails upon deployment with a HTTP 503 - https://phabricator.wikimedia.org/T380958#10898302 (10akosiaris) I 've run a battery of tests in the previous days against mwdebug2001... [08:14:50] 06serviceops, 06Abstract Wikipedia team, 10function-evaluator, 10function-orchestrator, and 2 others: Unable to deploy wikifunctions services in production: Pool wf-codfw has no failover servers list, route /local/wf - https://phabricator.wikimedia.org/T396033#10898326 (10akosiaris) Indeed, I commented... [08:15:41] 06serviceops, 06Abstract Wikipedia team, 10function-orchestrator, 07Essential-Work: Unable to deploy wikifunctions services in production: Helm timeout for prod push of memcached access - https://phabricator.wikimedia.org/T396074#10898329 (10akosiaris) Tried deploying this, see https://sal.toolforge.org/lo... [10:18:50] 06serviceops, 10MW-on-K8s, 10Data-Platform-SRE (2025.05.24 - 2025.06.13), 10Discovery-Search (2025.05.24 - 2025.06.13), 10MW-1.45-notes (1.45.0-wmf.4; 2025-06-03): Investigate EQIAD daily completion suggester rebuild failure - https://phabricator.wikimedia.org/T395465#10898988 (10Clement_Goubert) [10:20:04] 06serviceops, 10MW-on-K8s: Limit CPU usage for mw-on-k8s cli deployments - https://phabricator.wikimedia.org/T395436#10898991 (10Clement_Goubert) 05Resolved→03Open Re-opened for disabling limits until the spiky memory behavior identified in T395465 is fixed. [11:04:25] 06serviceops, 10MW-on-K8s, 10Data-Platform-SRE (2025.05.24 - 2025.06.13), 10Discovery-Search (2025.05.24 - 2025.06.13), and 2 others: Investigate EQIAD daily completion suggester rebuild failure - https://phabricator.wikimedia.org/T395465#10899122 (10Clement_Goubert) Memory limit disabled. [13:01:11] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Fix dependencies between admin_ng deployments - https://phabricator.wikimedia.org/T389080#10899530 (10JMeybohm) 05Open→03Resolved Dependencies are fixed now for wikikube clusters. Other cluster maintainer... [13:52:24] 06serviceops, 06Abstract Wikipedia team, 10function-orchestrator, 07Essential-Work: Unable to deploy wikifunctions services in production: Helm timeout for prod push of memcached access - https://phabricator.wikimedia.org/T396074#10899800 (10jijiki) 05Open→03Resolved a:03jijiki There was a typo i... [14:46:29] 06serviceops, 10MediaWiki-extensions-ReadingLists, 06MW-Interfaces-Team, 10RESTBase Sunsetting, and 3 others: Switchover plan from RESTbase to REST Gateway for Reading Lists endpoints - https://phabricator.wikimedia.org/T384891#10900073 (10hnowlan) Given that the migration of reading lists is blocking the... [15:36:18] 06serviceops, 10Prod-Kubernetes, 06Traffic, 07Kubernetes, 13Patch-For-Review: Handling inbound IPIP traffic on low traffic LVS k8s based realservers - https://phabricator.wikimedia.org/T352956#10900403 (10cmooney) @akosiaris a quick question about this: > meaning that ICMP traffic to e.g. coredns gets d... [15:43:50] 06serviceops, 10Prod-Kubernetes, 06Traffic, 07Kubernetes, 13Patch-For-Review: Handling inbound IPIP traffic on low traffic LVS k8s based realservers - https://phabricator.wikimedia.org/T352956#10900449 (10akosiaris) >>! In T352956#10900403, @cmooney wrote: > @akosiaris a quick question about this: > >>... [16:31:01] 06serviceops, 10Prod-Kubernetes, 06Traffic, 07Kubernetes, 13Patch-For-Review: Handling inbound IPIP traffic on low traffic LVS k8s based realservers - https://phabricator.wikimedia.org/T352956#10900719 (10cmooney) @akosiaris thanks for confirming. So overall my thinking is: * Path MTU discovery should... [16:32:20] 06serviceops, 10Prod-Kubernetes, 06Traffic, 07Kubernetes, 13Patch-For-Review: Handling inbound IPIP traffic on low traffic LVS k8s based realservers - https://phabricator.wikimedia.org/T352956#10900737 (10cmooney) >>! In T352956#10900449, @akosiaris wrote: >>>! In T352956#10900403, @cmooney wrote: >> @ak... [22:12:04] 06serviceops, 10MediaWiki-extensions-ReadingLists, 06MW-Interfaces-Team, 10RESTBase Sunsetting, and 3 others: Switchover plan from RESTbase to REST Gateway for Reading Lists endpoints - https://phabricator.wikimedia.org/T384891#10902171 (10HCoplin-WMF) Just confirming here that we spoke in Slack, and every... [22:34:11] 06serviceops, 10GrowthExperiments, 10MW-on-K8s, 10Growth-Team (Current Sprint), and 2 others: growthexperiments-fixlinkrecommendationdata-dryrun 'link-recommendation' is not a link recommendation task type error - https://phabricator.wikimedia.org/T395249#10902203 (10Etonkovidova) To monitor - `k8s_event...