[10:39:10] <wikibugs>	 06serviceops: OOM livelock stalls - https://phabricator.wikimedia.org/T358634#11499103 (10fgiunchedi) I'm untagging cloud VPS since this doesn't seem to be cloud vps specific, feel free to tweak as needed though
[11:01:03] <wikibugs>	 06serviceops, 10Horizon, 06SRE, 10Striker, 06cloud-services-team (FY2025/2026-Q3-Q4): Move cloudweb to Ganeti VMs and repurpose the servers as wikikube nodes - https://phabricator.wikimedia.org/T392478#11499214 (10fgiunchedi)
[11:06:29] <wikibugs>	 06serviceops, 10Horizon, 06SRE, 10Striker, 06cloud-services-team (FY2025/2026-Q3-Q4): Move cloudweb to Ganeti VMs and repurpose the servers as wikikube nodes - https://phabricator.wikimedia.org/T392478#11499233 (10taavi) See also: {T411783}
[12:58:34] <wikibugs>	 06serviceops, 07Datacenter-Switchover: Northward Datacenter Switchover (Mar. 2026) - https://phabricator.wikimedia.org/T413974 (10Blake) 03NEW
[12:59:01] <wikibugs>	 06serviceops, 07Datacenter-Switchover: Northward Datacenter Switchover (Mar. 2026) - https://phabricator.wikimedia.org/T413974#11499504 (10Blake)
[12:59:40] <wikibugs>	 06serviceops, 07Datacenter-Switchover: Northward Datacenter Switchover (Mar. 2026) - https://phabricator.wikimedia.org/T413974#11499507 (10Blake)
[13:03:10] <wikibugs>	 06serviceops, 07Datacenter-Switchover: Northward Datacenter Switchover (Mar. 2026) - https://phabricator.wikimedia.org/T413974#11499524 (10Blake)
[13:25:04] <Amir1>	 Hii, I have a "fun" problem. I need to run a script on all wikis that would go through all edits in 2004 and check them for utf-8 issues so I can mark them as corrupted. The fun thing is that no matter how I slice it, it OOMs. The only option left is that I run it per day (edits done in Jan 1, 2004 then edits done in Jan 2, 2004, ...) That means I'm going to run 356 mw-k8s-script (serial but still). Is that okay? Context: 
[13:25:04] <Amir1>	 https://phabricator.wikimedia.org/T351953
[14:32:39] <wikibugs>	 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker refresh - https://phabricator.wikimedia.org/T408760#11499905 (10Jclark-ctr) >>! In T408760#11482291, @VRiley-WMF wrote:  > wikikube-worker1371 > E8 > U32 > CableID 230304500167 > Port 5 >  > wikikube-worker1372 > F8 > U18 > C...
[14:42:55] <wikibugs>	 06serviceops, 06MediaWiki-Engineering: Migrate Wikimedia production from PHP 8.3 to PHP 8.4 or PHP 8.5 - https://phabricator.wikimedia.org/T413223#11499934 (10MSantos) I say we should aim for PHP 8.5. The reason for that is that with PHP 8.5, we are catching up with active support from PHP.  I understand that...
[14:46:20] <Raine>	 Amir1: do you know how much memory you'd need to not have to slice it like that?
[14:46:50] <Raine>	 but also, in principle it shouldn't be a problem, especially serially, assuming the job runs for a while 
[14:49:08] <Raine>	 is that the findBadBlobs.php script that you were fighting at the beginning of Dec?
[14:51:33] <Raine>	 in principle, we have lots of memory (https://grafana.wikimedia.org/d/WG4NjDISk/cluster-status-and-capacity ), so I can just bump the limits for you if it's not completely insane
[15:08:39] <claime>	 Amir1: Raine: +1 on just removing the memory limit for that script it should be fine
[15:26:08] <Amir1>	 "do you know how much memory you'd need to not have to slice it like that?" <- Yes
[15:26:59] <Amir1>	 I don't know how to run it with extra memory :(
[15:27:29] <Amir1>	 I can't say exactly how much more memory it'll need but it shouldn't be too much
[15:33:25] <claime>	 Amir1: We need to push a small values patch to mw-script, that will remove limits for everyone temporarily, then you launch your script and we revert
[15:33:38] <claime>	 Basically just reverting the mw-script part of this patch https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1151689
[15:41:32] <Amir1>	 Oh that's clever 
[15:41:36] <wikibugs>	 06serviceops, 10envoy, 06SRE: Upgrade Envoy to v1.35.7 - https://phabricator.wikimedia.org/T410975#11500242 (10Clement_Goubert) For the record, [[https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1217611|121761]] updates the rest-gateway to 1.35.7
[15:41:39] <Amir1>	 I need to be afk for a bit
[15:41:46] <Amir1>	 Let's do it tomorrow 
[15:43:03] <claime>	 Amir1: sure
[16:29:16] <wikibugs>	 06serviceops, 06MediaWiki-Platform-Team (Kanban Board): Create namespace and support files for redioscope - https://phabricator.wikimedia.org/T413999 (10Clement_Goubert) 03NEW
[16:29:38] <wikibugs>	 06serviceops, 06MediaWiki-Platform-Team (Kanban Board): Create namespace and support files for redioscope - https://phabricator.wikimedia.org/T413999#11500405 (10Clement_Goubert) p:05Triage→03Medium
[16:37:51] <wikibugs>	 06serviceops, 10RESTBase Sunsetting, 07Essential-Work, 06MW-Interfaces-Team (MWI-Sprint-24 (2025-12-02 to 2025-12-24)), 13Patch-For-Review: Reroute /api/rest_v1 documentation to REST Sandbox - https://phabricator.wikimedia.org/T396807#11500423 (10Clement_Goubert) rest-gateway patch merged and deployed, I...
[16:43:57] <wikibugs>	 06serviceops, 06MediaWiki-Platform-Team (Kanban Board), 07OKR-Work: Switch ratelimit backend redis to instance 6380 - https://phabricator.wikimedia.org/T413876#11500445 (10Clement_Goubert) 05In progress→03Resolved
[16:57:01] <wikibugs>	 06serviceops, 05FY2025-26 KR 5.1, 06MediaWiki-Platform-Team (Roadmap), 07OKR-Work: Upgrade ratelimit service to latest master - https://phabricator.wikimedia.org/T414002 (10Clement_Goubert) 03NEW
[16:57:23] <wikibugs>	 06serviceops, 05FY2025-26 KR 5.1, 06MediaWiki-Platform-Team (Roadmap), 07OKR-Work: Upgrade ratelimit service to latest master - https://phabricator.wikimedia.org/T414002#11500497 (10Clement_Goubert) 05Open→03In progress p:05Triage→03Medium
[16:57:45] <wikibugs>	 06serviceops, 05FY2025-26 KR 5.1, 06MediaWiki-Platform-Team (Roadmap), 07OKR-Work: Upgrade ratelimit service to latest main - https://phabricator.wikimedia.org/T414002#11500501 (10Clement_Goubert)
[18:32:02] <rzl>	 open to adding an arg to mw-script to override or remove resource limits too, wouldn't be too hard
[18:32:57] <rzl>	 adding an arg to mwscript-k8s, that is
[18:33:54] <rzl>	 a little bit concerned about making that *too* easy for obvious reasons, but since the script doesn't give anyone access to resources they can't use already, I think it's worth considering
[21:13:53] <wikibugs>	 06serviceops, 06SRE, 10Wikimedia-Site-requests, 13Patch-For-Review: Change $wgMaxArticleSize limit from byte-based to character-based - https://phabricator.wikimedia.org/T275319#11501360 (10Jeff_G) See also https://commons.wikimedia.org/w/index.php?title=Commons%3AAdministrators%27_noticeboard&diff=1143860...