[00:32:53] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T413805#11585687 (10brennen) Looks like @Reedy's handling backports for T416456, which should unblock. Given time of day, I think train is... [07:58:55] I need to report someone who.goes by makaylahippo [08:00:57] I made a post trying to find out about my family origins my last broughman so far I've found that it's viking and from orkney islands where it held a family seat but I was trying find out more and this person threatened to have banned for vandalizing wikipedia and did nothing of the sort [09:39:42] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 13Patch-For-Review, 05Release, 05Train Deployments: 1.46.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T413805#11586367 (10hashar) > I think train is probably safest to stay parked until @hashar's morning. Indeed :]... [09:55:37] 10Gerrit, 06collaboration-services: Gerrit failover process - https://phabricator.wikimedia.org/T412779#11586429 (10ABran-WMF) neat, that will simplify things! I've added an item with a draft document to our team meeting so we can address missing bits to make handling a gerrit failover manageable in an emergency [10:15:28] 06Release-Engineering-Team (Radar), 10Ceph, 06ServiceOps new, 10SRE-swift-storage, and 3 others: Move the docker registry's /restricted prefix to Docker Distribution backed up by Ceph - https://phabricator.wikimedia.org/T412951#11586516 (10elukey) @hashar Hi! Would you be available Mon/Tue next week, durin... [10:29:53] 06Release-Engineering-Team, 10Citoid, 07Essential-Work, 13Patch-For-Review, 07Technical-Debt: update nodejs22-slim image to 22.21.0 to support proxy env variables for outbound requests - https://phabricator.wikimedia.org/T416471#11586554 (10Mvolz) Hi @Reeti, I think you've misunderstood the task. This... [10:30:25] 06Release-Engineering-Team, 10Citoid, 07Essential-Work, 07Technical-Debt: update nodejs22-slim image to 22.21.0 to support proxy env variables for outbound requests - https://phabricator.wikimedia.org/T416471#11586557 (10Mvolz) [10:52:04] 06Release-Engineering-Team, 10Citoid, 07Essential-Work, 07Technical-Debt: update nodejs22-slim image to 22.21.0 to support proxy env variables for outbound requests - https://phabricator.wikimedia.org/T416471#11586705 (10Mvolz) [10:57:29] (03CR) 10Reedy: [C:03+1] "Probably... not a bad thing. https://gerrit.wikimedia.org/r/c/mediawiki/extensions/OATHAuth/+/1236294 is adding usages and would save the " [integration/config] - 10https://gerrit.wikimedia.org/r/1236606 (owner: 10Jforrester) [12:15:20] 06Release-Engineering-Team, 10Citoid, 07Essential-Work, 07Technical-Debt: update nodejs22-slim image to 22.21.0 to support proxy env variables for outbound requests - https://phabricator.wikimedia.org/T416471#11587123 (10Reeti) Thanks @Mvolz for the clarification! I misunderstood and thought I needed to en... [12:51:16] 10Continuous-Integration-Infrastructure, 07Jenkins, 06collaboration-services: APT error when installing Jenkins package in releases instances - https://phabricator.wikimedia.org/T405352#11587274 (10jnuche) 05In progress→03Resolved a:03Dzahn A Jenkins upgrade to a new version finally happened as par... [13:13:27] 10Phabricator: Gerrit patches are easy to miss in Phabricator history stream - https://phabricator.wikimedia.org/T416510#11587363 (10Esanders) > It would be neat to have some styling that is more prominent than other timeline entries, but less prominent than human comments. I think this proposal is still much... [13:34:24] 06Release-Engineering-Team (Doing 😎), 10Catalyst, 07Essential-Work, 13Patch-For-Review: Update cluster's GitLab agent - https://phabricator.wikimedia.org/T416464#11587436 (10jnuche) [14:40:03] maintenance-disconnect-full-disks build 778448 integration-agent-docker-1042 (/: 36%, /srv: 95%, /var/lib/docker: 31%): OFFLINE due to disk space [14:45:03] maintenance-disconnect-full-disks build 778449 integration-agent-docker-1042 (/: 36%, /srv: 81%, /var/lib/docker: 31%): RECOVERY disk space OK [14:51:48] (03CR) 10Jforrester: [C:03+2] Zuul: […/OATHAuth] Add phan dependency on CentralAuth [integration/config] - 10https://gerrit.wikimedia.org/r/1236606 (owner: 10Jforrester) [15:09:39] (03approved) 10jforrester: Bump Codex to 2.3.4 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/117 (https://phabricator.wikimedia.org/T415727) (owner: 10volker-e) [15:09:52] (03merge) 10jforrester: Bump Codex to 2.3.4 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/117 (https://phabricator.wikimedia.org/T415727) (owner: 10volker-e) [15:15:49] (03CR) 10Jforrester: [C:03+2] Zuul: […/OATHAuth] Add dependency on CentralAuth [integration/config] - 10https://gerrit.wikimedia.org/r/1236605 (owner: 10Jforrester) [15:17:19] (03Merged) 10jenkins-bot: Zuul: […/OATHAuth] Add dependency on CentralAuth [integration/config] - 10https://gerrit.wikimedia.org/r/1236605 (owner: 10Jforrester) [15:17:35] (03Merged) 10jenkins-bot: Zuul: […/OATHAuth] Add phan dependency on CentralAuth [integration/config] - 10https://gerrit.wikimedia.org/r/1236606 (owner: 10Jforrester) [15:18:32] !log Zuul: […/OATHAuth] Add dependency and phan dependency on CentralAuth [15:18:33] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:24:38] James_F: [15:24:39] 15:23:50 Error: A dependency error was encountered while installing the extension "CentralAuth": Could not find the registration file for the extension "AntiSpoof" [15:24:39] 15:23:50 <<< Finish: Install MediaWiki, db=, in 0.284 s [15:24:46] from OATHAuth [15:24:56] I guess *another* recursive dependancy needed [15:30:37] (03PS1) 10Reedy: Zuul: Add AntiSpoof as dependancy for OATHAuth [integration/config] - 10https://gerrit.wikimedia.org/r/1237254 [15:32:51] (03CR) 10Reedy: [C:03+2] Zuul: Add AntiSpoof as dependancy for OATHAuth [integration/config] - 10https://gerrit.wikimedia.org/r/1237254 (owner: 10Reedy) [15:34:40] (03Merged) 10jenkins-bot: Zuul: Add AntiSpoof as dependancy for OATHAuth [integration/config] - 10https://gerrit.wikimedia.org/r/1237254 (owner: 10Reedy) [15:35:29] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1237254 [15:35:30] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:01:03] 06Project-Admins, 10MediaWiki-Core-Tests, 06MediaWiki-Engineering: The api-testing library has no specific phab tag and its docs link to an archived one - https://phabricator.wikimedia.org/T415836#11588119 (10Jdforrester-WMF) OK, I've created #api-testing-tool. [16:01:20] 06Project-Admins, 10api-testing-tool, 10MediaWiki-Core-Tests, 06MediaWiki-Engineering: The api-testing library has no specific phab tag and its docs link to an archived one - https://phabricator.wikimedia.org/T415836#11588120 (10Jdforrester-WMF) 05Open→03Resolved [16:02:12] 06Release-Engineering-Team, 10api-testing-tool, 10MediaWiki-Core-Tests, 06MediaWiki-Engineering: api-testing: avoid processing the entire job queue when changing user rights - https://phabricator.wikimedia.org/T389898#11588125 (10Jdforrester-WMF) [16:03:24] 10Continuous-Integration-Config, 10api-testing-tool, 10MediaWiki-Core-Tests, 07Code-Health, and 2 others: Run api-testing tests in parallel - https://phabricator.wikimedia.org/T298735#11588131 (10Jdforrester-WMF) [16:08:58] Reedy: Yeah. :-( Thanks for fixing. [16:11:08] Easy enough :) [16:17:19] (03CR) 10Hashar: "Something to watch is that if any test in `AntiSpoof` requires an extension that is not a dependency of `OATHAuth`, that test will fail. " [integration/config] - 10https://gerrit.wikimedia.org/r/1237254 (owner: 10Reedy) [16:18:03] (03CR) 10Reedy: [C:03+2] "Adding AntiSpoof because CentralAuth needs it..." [integration/config] - 10https://gerrit.wikimedia.org/r/1237254 (owner: 10Reedy) [16:28:59] 10GitLab (CI & Job Runners), 06collaboration-services: registry.cloud.releng.team returning 503s - https://phabricator.wikimedia.org/T406733#11588259 (10dancy) 05Open→03Resolved a:03dancy I believe that the work done in T408740 has resolved this issue. [16:30:04] Project mediawiki-core-doxygen build #17541: 04FAILURE in 12 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/17541/ [16:50:50] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T413805#11588449 (10hashar) [16:51:22] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T413805#11588452 (10hashar) I have noticed a TypeError on ruwiki emanating from CentralAuth / skin and filed T416619. I don't know about... [17:18:45] 06Release-Engineering-Team, 10LDAP-Access-Requests, 06Security-Team, 06SRE, and 2 others: DannyS712 "offboarding" - https://phabricator.wikimedia.org/T413634#11588560 (10bd808) >>! In T413634#11580566, @A_smart_kitten wrote: >>>! In T413634#11580561, @sbassett wrote: >> Anything else left to do here? > >... [17:25:49] (03PS1) 10Gergő Tisza: Zuul: […/WikimediaCustomizations] Add six new dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/1237276 (https://phabricator.wikimedia.org/T404334) [17:33:06] Yippee, build fixed! [17:33:06] Project mediawiki-core-doxygen build #17542: 09FIXED in 15 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/17542/ [17:55:12] (03CR) 10Jforrester: [C:03+2] Zuul: […/WikimediaCustomizations] Add six new dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/1237276 (https://phabricator.wikimedia.org/T404334) (owner: 10Gergő Tisza) [17:56:43] (03Merged) 10jenkins-bot: Zuul: […/WikimediaCustomizations] Add six new dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/1237276 (https://phabricator.wikimedia.org/T404334) (owner: 10Gergő Tisza) [17:58:01] !log Zuul: […/WikimediaCustomizations] Add six new dependencies for T404334 [17:58:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:00:19] (03CR) 10Jforrester: [C:03+2] "00:00:36.475 Error: A dependency error was encountered while installing the extension "WikimediaEvents": Could not find the registration f" [integration/config] - 10https://gerrit.wikimedia.org/r/1237276 (https://phabricator.wikimedia.org/T404334) (owner: 10Gergő Tisza) [18:02:55] (03PS1) 10Jforrester: Zuul: […/WikimediaCustomizations] Also depend on Echo & EventLogging [integration/config] - 10https://gerrit.wikimedia.org/r/1237284 (https://phabricator.wikimedia.org/T404334) [18:03:05] (03CR) 10Jforrester: [C:03+2] Zuul: […/WikimediaCustomizations] Also depend on Echo & EventLogging [integration/config] - 10https://gerrit.wikimedia.org/r/1237284 (https://phabricator.wikimedia.org/T404334) (owner: 10Jforrester) [18:04:33] (03Merged) 10jenkins-bot: Zuul: […/WikimediaCustomizations] Also depend on Echo & EventLogging [integration/config] - 10https://gerrit.wikimedia.org/r/1237284 (https://phabricator.wikimedia.org/T404334) (owner: 10Jforrester) [18:09:04] (03PS1) 10Jforrester: Zuul: […/WikimediaCustomizations] Also depend on EventBus [integration/config] - 10https://gerrit.wikimedia.org/r/1237285 (https://phabricator.wikimedia.org/T404334) [18:10:18] (03CR) 10Jforrester: [C:03+2] Zuul: […/WikimediaCustomizations] Also depend on EventBus [integration/config] - 10https://gerrit.wikimedia.org/r/1237285 (https://phabricator.wikimedia.org/T404334) (owner: 10Jforrester) [18:11:45] (03Merged) 10jenkins-bot: Zuul: […/WikimediaCustomizations] Also depend on EventBus [integration/config] - 10https://gerrit.wikimedia.org/r/1237285 (https://phabricator.wikimedia.org/T404334) (owner: 10Jforrester) [19:03:23] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T413805#11588995 (10brennen) [19:06:49] (03CR) 10Gergő Tisza: "T416628" [integration/config] - 10https://gerrit.wikimedia.org/r/1237276 (https://phabricator.wikimedia.org/T404334) (owner: 10Gergő Tisza) [19:53:29] (03update) 10mstyles: Scap: Add Security Deploy Command [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1085 (https://phabricator.wikimedia.org/T361709) [19:53:49] (03update) 10mstyles: Scap: Add Security Deploy Command [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1085 (https://phabricator.wikimedia.org/T361709) [20:43:32] I'm seeing a patch to a repo that runs quibble-apitests that's consistently failing with "internal_api_error_DBQueryError". Is there a testing machine somewhere that needs to be prodded? [20:43:41] The log: https://integration.wikimedia.org/ci/job/quibble-apitests-only-vendor-php83/7763/consoleFull [20:48:13] I apparently can't view https://integration.wikimedia.org/ci/computer/integration%2Dagent%2Ddocker%2D1061/builds [20:51:37] Reedy: https://gerrit.wikimedia.org/g/operations/puppet/+/production/modules/contint/templates/apache/proxy_jenkins.erb#53 [20:51:58] Kemayo: clicking around reveals https://integration.wikimedia.org/ci/job/quibble-apitests-only-vendor-php83/7763/artifact/log/mw-error.log as the actual error [20:52:27] taavi: Still looks like some sort of DB error completely unrelated to the patch itself, though. [20:53:17] which seems very likely related to https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/1237222 [20:53:51] can you file a #wmf-deployed-build-failure task? [20:54:46] FWIW I made this to see host/job/whatever info: https://releng-data.wmcloud.org/-/dashboards/jenkins/recent-builds?host_name=integration-agent-docker-1061&date_start=2026-02-05&repo_name=&job_name=&date_end= [20:58:34] taavi: I have filed T416637. [20:58:34] T416637: quibble-apitests failing on unrelated patches - https://phabricator.wikimedia.org/T416637 [20:59:34] thansk, put up a revert at https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/1237336 [22:11:54] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T413805#11589773 (10brennen) 05Open→03Resolved Currently stable on all wikis.