[04:33:43] (03PS1) 10Hslater: Zuul: [mediawiki/extensions/BlueSpiceWikiFarm] Add BlueSpiceExtendedSearch dep [integration/config] - 10https://gerrit.wikimedia.org/r/1196553 [04:37:53] (03PS2) 10Hslater: Zuul: [mediawiki/extensions/BlueSpiceWikiFarm] Add dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/1196553 [04:43:39] 10GitLab (Project Migration), 06Community-Tech, 10SVG Translate Tool: Migrate SVG Translate from GitHub to GitLab - https://phabricator.wikimedia.org/T402505#11279833 (10Samwilson) [06:16:39] 10GitLab (Project Migration), 06Community-Tech, 10SVG Translate Tool: Migrate SVG Translate from GitHub to GitLab - https://phabricator.wikimedia.org/T402505#11279998 (10Samwilson) [06:17:55] 10GitLab (Project Migration), 06Community-Tech, 10SVG Translate Tool: Migrate SVG Translate from GitHub to GitLab - https://phabricator.wikimedia.org/T402505#11280001 (10Samwilson) 05Open→03Resolved This is all done now I think. [07:15:28] paladox: I think the issue is jgleeson|away was throttled by Apache mod_qos [07:15:32] arnaudb: ^^ [07:15:56] that looks like similar to the issue we had last week which prompted me to disable mod_qos entirely [07:46:25] yep makes sense hashar paladox, I'll adjust the knobs thresholds a bit higher then. Yesterday we were scrapped to death regularly so we had to enforce those limits, unfortunately they are hard to adjust to be both meaningful and not degrade UX. Sorry for the disturbance jgleeson|away! [08:02:22] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.45.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T405679#11280168 (10hashar) [08:03:41] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.45.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T405679#11280172 (10hashar) There are some PHP warning with the parser TOC code which I have filed as T407323. I have reached out to Isab... [08:05:31] fwiw: I posted a blurb about it on slack yesterday (https://wikimedia.slack.com/archives/C01R06P8D1B/p1760515715381959)! Here's the change: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1196568 [08:16:39] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.45.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T405679#11280233 (10hashar) >>! In T405679#11280168, @hashar wrote: > There are some PHP warning with the parser TOC code which I have fil... [08:16:49] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.45.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T405679#11280234 (10hashar) [09:06:07] np arnaudb thanks for digging into it! [09:27:13] 06Release-Engineering-Team, 10ChangeProp, 10Citoid, 06Data-Engineering, and 19 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#11280488 (10Mvolz) [11:18:37] 10Gerrit, 06collaboration-services, 13Patch-For-Review: gerrit2003 is trying to backup incrementally 3.5 million files every hour, clogging backus and filling in available disk space - https://phabricator.wikimedia.org/T406762#11280808 (10LSobanski) p:05High→03Medium [11:44:33] (03PS1) 10Dreamy Jazz: Zuul: [mediawiki/extensions/ConfirmEdit] Add VisualEditor and MobileFrontend deps [integration/config] - 10https://gerrit.wikimedia.org/r/1196662 (https://phabricator.wikimedia.org/T407377) [11:46:13] (03PS2) 10Dreamy Jazz: Zuul: [mediawiki/extensions/ConfirmEdit] Add VisualEditor and MobileFrontend deps [integration/config] - 10https://gerrit.wikimedia.org/r/1196662 (https://phabricator.wikimedia.org/T407377) [11:47:23] (03PS3) 10Dreamy Jazz: Zuul: [mediawiki/extensions/ConfirmEdit] Add VisualEditor and MobileFrontend deps [integration/config] - 10https://gerrit.wikimedia.org/r/1196662 (https://phabricator.wikimedia.org/T407377) [11:48:58] (03CR) 10CI reject: [V:04-1] Zuul: [mediawiki/extensions/ConfirmEdit] Add VisualEditor and MobileFrontend deps [integration/config] - 10https://gerrit.wikimedia.org/r/1196662 (https://phabricator.wikimedia.org/T407377) (owner: 10Dreamy Jazz) [11:54:59] (03PS4) 10Dreamy Jazz: Zuul: ConfirmEdit extension: Add VisualEditor and MobileFrontend deps [integration/config] - 10https://gerrit.wikimedia.org/r/1196662 (https://phabricator.wikimedia.org/T407377) [12:33:05] (03CR) 10Harroyo-wmf: [C:03+1] Zuul: ConfirmEdit extension: Add VisualEditor and MobileFrontend deps [integration/config] - 10https://gerrit.wikimedia.org/r/1196662 (https://phabricator.wikimedia.org/T407377) (owner: 10Dreamy Jazz) [13:28:46] 10Gerrit: Reviewer-bot does not add reviewers when leaving initial WIP mode - https://phabricator.wikimedia.org/T267787#11281157 (10Daimona) The same thing happens if you do `git review --work-in-progress` or similar. As someone who heavily relies on gerrit notifications as opposed to dashboards, this bug is ind... [13:28:51] Dreamy_Jazz: hi, I am going to deploy your CI config change :] [13:31:16] (03CR) 10Hashar: [C:03+2] "Perfect and thank you for having taken the time to write a long rationale in the commit message 😎" [integration/config] - 10https://gerrit.wikimedia.org/r/1196662 (https://phabricator.wikimedia.org/T407377) (owner: 10Dreamy Jazz) [13:32:57] (03Merged) 10jenkins-bot: Zuul: ConfirmEdit extension: Add VisualEditor and MobileFrontend deps [integration/config] - 10https://gerrit.wikimedia.org/r/1196662 (https://phabricator.wikimedia.org/T407377) (owner: 10Dreamy Jazz) [13:34:52] !log Zuul: ConfirmEdit extension: Add VisualEditor and MobileFrontend deps [13:34:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:36:10] hashar: Thanks! [13:36:26] Dreamy_Jazz: I did the recheck and phan runs at https://integration.wikimedia.org/ci/job/mwext-php81-phan/41326/console [13:37:14] it is still not happy :/ [13:37:16] Yeah, I'll need to update the .phan/config.php file. I'll get on that now [13:37:23] ahhh [13:37:34] one day we would automatize that part somehow ! :b [13:37:59] Definitely. One less step in adding/removing dependencies [13:39:16] * hashar watches [13:39:36] I think umherider once filed a task about having .phan/config.php to generate the config based on CI variables [13:40:30] success! https://integration.wikimedia.org/ci/job/mwext-php81-phan/41328/console [13:42:37] :D [13:42:43] Thanks for the quick review and merge [13:43:30] your welcome! [14:43:27] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/BlueSpiceWikiFarm] Add dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/1196553 (owner: 10Hslater) [14:44:52] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/BlueSpiceWikiFarm] Add dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/1196553 (owner: 10Hslater) [14:46:02] (03CR) 10Hashar: [C:03+2] "Deployed!" [integration/config] - 10https://gerrit.wikimedia.org/r/1196553 (owner: 10Hslater) [15:48:23] 10GitLab (CI & Job Runners), 06collaboration-services, 10Toolforge, 06cloud-services-team (FY2025/26-Q1), 13Patch-For-Review: tofu-provisioning: Failed to install provider - https://phabricator.wikimedia.org/T405742#11281753 (10fnegri) Not all Cloud VPS VMs have the MTU set to 1450. From a quick search,... [16:17:58] (03open) 10dancy: Fix git info cache for master branch [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1013 (https://phabricator.wikimedia.org/T405738) [16:18:01] (03update) 10dancy: Fix git info cache for master branch [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1013 (https://phabricator.wikimedia.org/T405738) [16:54:50] (03update) 10dancy: Fix git info cache for master branch [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1013 (https://phabricator.wikimedia.org/T405738) [16:57:18] (03merge) 10dancy: Fix git info cache for master branch [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1013 (https://phabricator.wikimedia.org/T405738) [16:59:38] (03update) 10dancy: Release 4.214.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1014 [16:59:39] (03open) 10dancy: Release 4.214.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1014 [17:01:58] (03merge) 10dancy: Release 4.214.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1014 [17:08:22] !log Updating scap to 4.214.0 in beta cluster. [17:08:23] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:47:02] 10Diffusion, 10Phabricator, 06collaboration-services, 13Patch-For-Review: Disable IO for diffusion repositories - https://phabricator.wikimedia.org/T405596#11282320 (10Aklapper) As a next step I'm tempted to mass-set all http (no s at the end) URIs which neither observe nor mirror to `io=none && display=ne... [17:47:42] 10Diffusion, 10Phabricator, 06Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services, 13Patch-For-Review: Disable IO for diffusion repositories - https://phabricator.wikimedia.org/T405596#11282323 (10Aklapper) [18:28:49] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 13Patch-For-Review: puppetize setup of new zuul VMs - https://phabricator.wikimedia.org/T395938#11282548 (10Dzahn) Could not change the jute.maxbuffer size using the `/etc/zookeeper/conf/environment` config or the `/etc/zookeep... [18:41:52] PROBLEM - gerrit process on gerrit2003 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/lib/jvm/java-17-openjdk-amd64/bin/java .*-jar /var/lib/gerrit/review_site/bin/gerrit.war daemon -d /var/lib/gerrit/review_site https://wikitech.wikimedia.org/wiki/Gerrit [18:51:41] ^ checked and it's disabled by admin [18:51:48] extended downtime for a few days [21:50:55] 10Diffusion, 10Phabricator, 06Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services, 13Patch-For-Review: Disable IO for diffusion repositories - https://phabricator.wikimedia.org/T405596#11283483 (10bd808) >>! In T405596#11282717, @A_smart_kitten wrote: >>>! In T405596#11244081, @Aklapp... [23:10:44] 10GitLab (CI & Job Runners), 13Patch-For-Review: kokkuri cannot publish "public" images from WMCS runners due to a lack of a local registry - https://phabricator.wikimedia.org/T396924#11283619 (10bd808) >>! In T396924#11153735, @bd808 wrote: > At the moment a job will still need to set a `KOKKURI_REGISTRY_PUBL...