[01:02:17] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for https://g.dev/nocluemicheal - https://phabricator.wikimedia.org/T414294 (10Nocluemicheal) 03NEW [01:22:05] 06Project-Admins: Create project tag for English WP:RSP Restructuring - https://phabricator.wikimedia.org/T413709#11510795 (10aliu) @Peachey88 Apologies if I wasn't supposed to ping you. Since you created many other requested project tags, could you take a look at this when you have the time? Thanks in advance! [01:29:58] Project mediawiki-core-doxygen build #16806: 04FAILURE in 11 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/16806/ [02:32:43] Yippee, build fixed! [02:32:43] Project mediawiki-core-doxygen build #16807: 09FIXED in 14 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/16807/ [04:45:30] 10Continuous-Integration-Infrastructure, 10MediaWiki-Codesniffer: Update mw-tools-codesniffer-mwcore-testrun to run with php8.2+ - https://phabricator.wikimedia.org/T414278#11510863 (10Novem_Linguae) Possibly relevant lines of the integration/config/layout.yaml file: ` - name: mediawiki/tools/codesniffer... [07:27:42] (03PS1) 10Hashar: dockerfiles: [mediawiki-phan-testrun] upgrade to PHP 8.3 [integration/config] - 10https://gerrit.wikimedia.org/r/1225296 (https://phabricator.wikimedia.org/T414278) [07:29:06] (03CR) 10CI reject: [V:04-1] dockerfiles: [mediawiki-phan-testrun] upgrade to PHP 8.3 [integration/config] - 10https://gerrit.wikimedia.org/r/1225296 (https://phabricator.wikimedia.org/T414278) (owner: 10Hashar) [07:30:34] (03PS2) 10Hashar: dockerfiles: [mediawiki-phan-testrun] upgrade to PHP 8.3 [integration/config] - 10https://gerrit.wikimedia.org/r/1225296 (https://phabricator.wikimedia.org/T414278) [07:35:01] (03PS1) 10Hashar: jjb: update mediawiki-phan-testrun to PHP 8.3 [integration/config] - 10https://gerrit.wikimedia.org/r/1225297 (https://phabricator.wikimedia.org/T414278) [07:35:45] (03CR) 10Hashar: [C:03+2] dockerfiles: [mediawiki-phan-testrun] upgrade to PHP 8.3 [integration/config] - 10https://gerrit.wikimedia.org/r/1225296 (https://phabricator.wikimedia.org/T414278) (owner: 10Hashar) [07:35:53] 10Continuous-Integration-Infrastructure, 10MediaWiki-Codesniffer, 13Patch-For-Review: Update mw-tools-codesniffer-mwcore-testrun to run with php8.2+ - https://phabricator.wikimedia.org/T414278#11511036 (10hashar) ` lang=yaml,name=zuul/layout.yaml - name: mediawiki/tools/codesniffer template: # (T410941... [07:37:27] (03Merged) 10jenkins-bot: dockerfiles: [mediawiki-phan-testrun] upgrade to PHP 8.3 [integration/config] - 10https://gerrit.wikimedia.org/r/1225296 (https://phabricator.wikimedia.org/T414278) (owner: 10Hashar) [07:40:55] (03PS2) 10Hashar: jjb: update mw-tools-phan-mwcore-testrun to PHP 8.3 [integration/config] - 10https://gerrit.wikimedia.org/r/1225297 (https://phabricator.wikimedia.org/T414278) [07:46:18] (03PS3) 10Hashar: jjb: update Phan testrun jobs to PHP 8.3 [integration/config] - 10https://gerrit.wikimedia.org/r/1225297 (https://phabricator.wikimedia.org/T414278) [07:47:55] (03CR) 10Hashar: [C:03+2] jjb: update Phan testrun jobs to PHP 8.3 [integration/config] - 10https://gerrit.wikimedia.org/r/1225297 (https://phabricator.wikimedia.org/T414278) (owner: 10Hashar) [07:49:39] (03Merged) 10jenkins-bot: jjb: update Phan testrun jobs to PHP 8.3 [integration/config] - 10https://gerrit.wikimedia.org/r/1225297 (https://phabricator.wikimedia.org/T414278) (owner: 10Hashar) [07:55:58] 10Continuous-Integration-Infrastructure, 10MediaWiki-Codesniffer, 13Patch-For-Review: Update mw-tools-codesniffer-mwcore-testrun to run with php8.2+ - https://phabricator.wikimedia.org/T414278#11511083 (10hashar) I have built the image and switched the two jobs to php 8.3. I have rebuilt the job and it pass... [07:56:15] 10Continuous-Integration-Infrastructure, 10MediaWiki-Codesniffer: Update mw-tools-codesniffer-mwcore-testrun to run with php8.2+ - https://phabricator.wikimedia.org/T414278#11511084 (10hashar) 05Open→03Resolved a:03hashar [07:57:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance deployment-shellbox01 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:57:33] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-shellbox01 on project deployment-prep - https://phabricator.wikimedia.org/T414304 (10wmcs-alerts) 03NEW [08:07:04] (03CR) 10Hslater: [V:03+2 C:03+2] Review access change [extensions/WikiRAG] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1224588 (owner: 10Hslater) [08:07:28] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance deployment-mediawiki13 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [08:17:28] FIRING: [5x] PuppetAgentNoResources: No Puppet resources found on instance deployment-mediawiki13 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [08:22:28] FIRING: [7x] PuppetAgentNoResources: No Puppet resources found on instance deployment-deploy04 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [08:27:28] FIRING: [8x] PuppetAgentNoResources: No Puppet resources found on instance deployment-deploy04 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [08:59:13] (03PS1) 10Hashar: Zuul: [DiscordRCFeed] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1225480 (https://phabricator.wikimedia.org/T389998) [09:03:25] (03CR) 10Hashar: [C:03+2] Zuul: [DiscordRCFeed] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1225480 (https://phabricator.wikimedia.org/T389998) (owner: 10Hashar) [09:04:08] (03PS1) 10Hashar: Zuul: [CheckLists] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1225482 (https://phabricator.wikimedia.org/T389998) [09:04:59] (03Merged) 10jenkins-bot: Zuul: [DiscordRCFeed] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1225480 (https://phabricator.wikimedia.org/T389998) (owner: 10Hashar) [09:05:03] (03CR) 10Hashar: [C:03+2] Zuul: [CheckLists] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1225482 (https://phabricator.wikimedia.org/T389998) (owner: 10Hashar) [09:06:31] (03Merged) 10jenkins-bot: Zuul: [CheckLists] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1225482 (https://phabricator.wikimedia.org/T389998) (owner: 10Hashar) [09:21:46] (03PS1) 10Hashar: Zuul: [DiscussionTools] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1225484 [09:27:25] (03PS1) 10Hashar: Zuul: remove FileAnnotations deps after archival [integration/config] - 10https://gerrit.wikimedia.org/r/1225487 [09:28:39] (03CR) 10Hashar: [C:03+2] Zuul: remove FileAnnotations deps after archival [integration/config] - 10https://gerrit.wikimedia.org/r/1225487 (owner: 10Hashar) [09:28:44] (03CR) 10Hashar: [C:03+2] Zuul: [DiscussionTools] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1225484 (owner: 10Hashar) [09:30:13] (03Merged) 10jenkins-bot: Zuul: [DiscussionTools] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1225484 (owner: 10Hashar) [09:30:23] (03Merged) 10jenkins-bot: Zuul: remove FileAnnotations deps after archival [integration/config] - 10https://gerrit.wikimedia.org/r/1225487 (owner: 10Hashar) [09:31:53] 10Scap: SecurityPatchBot: Only change a task's status to 'open' if its current status is a 'closed status' - https://phabricator.wikimedia.org/T414273#11511229 (10SomeRandomDeveloper) From what I've seen, "in progress" usually means that somebody is actively working on the task. However, if there's a problem wit... [09:38:43] 10Scap: SecurityPatchBot: Only change a task's status to 'open' if its current status is a 'closed status' - https://phabricator.wikimedia.org/T414273#11511276 (10A_smart_kitten) Hmm, that's a good point. Thank you for bringing that up, @somerandomdeveloper :) In that case, I wonder if the only change here shou... [09:48:02] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for https://g.dev/nocluemicheal - https://phabricator.wikimedia.org/T414294#11511353 (10Aklapper) Hi, thanks for your request. Please fill in the form in this task by answering its question. Also, which project(s) in... [09:57:42] 10Scap: SecurityPatchBot: Only change a task's status to 'open' if its current status is a 'closed status' - https://phabricator.wikimedia.org/T414273#11511413 (10Aklapper) `@SecurityPatchBot` comments as there is some work to do, an issue to fix: Rebasing a patch. Which problem is solved by changing `return bas... [10:19:45] 06Project-Admins, 06Release-Engineering-Team (Doing 😎): Create project tag for English WP:RSP Restructuring - https://phabricator.wikimedia.org/T413709#11511499 (10Aklapper) 05Open→03Resolved a:03Aklapper Sorry, just slow here trying to catch up... Requested public project #English-Wikipedia-RSP-rest... [10:26:21] 10Scap: SecurityPatchBot: Only change a task's status to 'open' if its current status is a 'closed status' - https://phabricator.wikimedia.org/T414273#11511538 (10A_smart_kitten) >>! In T414273#11511413, @Aklapper wrote: > `@SecurityPatchBot` comments as there is some work to do, an issue to fix: Rebasing a patc... [10:41:14] is there something weird happening with Phabricator? I'm getting a 403 when I curl for https://phabricator.wikimedia.org/diffusion/NLSP/new-lexeme-special-page.git/ (Request served via cp3067 cp3067, Varnish XID 606190832
Upstream caches: cp3067 int
Error: 403, 5fc3f46ec3c3afc352d999a4f0 (315d9a2) (cdb8beb) at Mon, 12 Jan 2026 10:40:22 GMT). I've sometimes seen 429 when I've been rate limited, but 403 seems new [10:45:24] codders: T409519 [10:45:25] T409519: WikibaseLexeme submodule is not available - https://phabricator.wikimedia.org/T409519 [10:45:47] 10Scap: SecurityPatchBot: Only change a task's status to 'open' if its current status is a 'closed status' - https://phabricator.wikimedia.org/T414273#11511634 (10Aklapper) Ah, I see, thanks for explaining! Feels like a neglectable edge case to me though :) [10:45:54] ah - thanks taavi [10:55:45] (03PS1) 10Hashar: Zuul: disable recursion on Tuleap repos [integration/config] - 10https://gerrit.wikimedia.org/r/1225512 [10:55:57] (03CR) 10Hashar: [C:03+2] Zuul: disable recursion on Tuleap repos [integration/config] - 10https://gerrit.wikimedia.org/r/1225512 (owner: 10Hashar) [10:58:46] (03Merged) 10jenkins-bot: Zuul: disable recursion on Tuleap repos [integration/config] - 10https://gerrit.wikimedia.org/r/1225512 (owner: 10Hashar) [11:04:58] 10Diffusion, 10Phabricator, 06collaboration-services, 10Wikidata, and 3 others: WikibaseLexeme submodule is not available - https://phabricator.wikimedia.org/T409519#11511749 (10ArthurTaylor) I'm seeing this again today when I try and run Quibble locally: ` fatal: unable to access 'https://phabricator.wik... [11:08:00] codders: https://phabricator.wikimedia.org/T409519#11400225 ) [11:08:16] codders: there are a few libs under https://github.com/wmde/ which would need to be moved to Gerrit [11:08:22] (or alternatively gitlab) [11:08:49] we will eventually decommission the git service on Phabricator [11:09:33] I run Quibble inside of docker - would be slightly tricky to get a .gitconfig in there. [11:09:55] (because I try and run the same docker images and commands that run on CI) [11:11:19] I'll try passing it in over the environment [11:11:29] 10Continuous-Integration-Config, 07Epic, 07PHP 8.5 support: Make PHP 8.5 voting on development (master) branch of MW ecosystem (core, vendor, extensions, skins, libraries) in CI - https://phabricator.wikimedia.org/T411814#11511769 (10A_smart_kitten) [11:16:11] (03update) 10aklapper: PHPMailer: Remove use of class.phpmailer-lite.php [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/89 [11:16:40] (03update) 10aklapper: PHPMailer: Remove use of class.phpmailer-lite.php [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/89 [11:17:01] (03update) 10aklapper: PHPMailer: Remove use of class.phpmailer-lite.php [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/89 [11:19:33] 10GitLab, 10GitLab-Application-Security-Pipeline, 10gitlab-settings: Requesting GitLab account activation for https://www.g.dev/nocluemicheal - https://phabricator.wikimedia.org/T414294#11511815 (10Nocluemicheal) [11:20:03] maintenance-disconnect-full-disks build 771515 integration-agent-docker-1056 (/: 35%, /srv: 97%, /var/lib/docker: 31%): OFFLINE due to disk space [11:20:45] 10GitLab (Account Approval): Requesting GitLab account activation for https://www.g.dev/nocluemicheal - https://phabricator.wikimedia.org/T414294#11511820 (10Aklapper) [11:21:53] 10GitLab (Account Approval): Requesting GitLab account activation for https://www.g.dev/nocluemicheal - https://phabricator.wikimedia.org/T414294#11511825 (10Aklapper) 05Open→03Declined Given the editing of random task parameters, I assume that this is not a genuine request. [11:25:03] maintenance-disconnect-full-disks build 771516 integration-agent-docker-1056 (/: 35%, /srv: 65%, /var/lib/docker: 31%): RECOVERY disk space OK [11:44:30] !log ssh integration-castor05.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mediawiki-node20 # fix failure seen in mediawiki-node20 46331, 46344 [11:44:31] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:03:50] 06Release-Engineering-Team, 06SRE, 10SRE-Access-Requests: Add yubikey ssh key for dancy - https://phabricator.wikimedia.org/T414032#11511963 (10JMeybohm) a:03dancy [12:09:14] (03PS1) 10Hashar: Zuul: [AutoModerator] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1225532 (https://phabricator.wikimedia.org/T389998) [12:09:16] (03PS1) 10Hashar: Zuul: [Babel] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1225533 (https://phabricator.wikimedia.org/T389998) [12:09:41] (03CR) 10Hashar: [C:03+2] Zuul: [AutoModerator] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1225532 (https://phabricator.wikimedia.org/T389998) (owner: 10Hashar) [12:09:46] (03CR) 10Hashar: [C:03+2] Zuul: [Babel] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1225533 (https://phabricator.wikimedia.org/T389998) (owner: 10Hashar) [12:11:51] (03Merged) 10jenkins-bot: Zuul: [AutoModerator] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1225532 (https://phabricator.wikimedia.org/T389998) (owner: 10Hashar) [12:11:53] (03Merged) 10jenkins-bot: Zuul: [Babel] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1225533 (https://phabricator.wikimedia.org/T389998) (owner: 10Hashar) [12:23:32] (03PS1) 10Hslater: Zuul: [mediawiki/extensions/WikiRAG] Add OOJSPlus as dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1225540 [12:24:25] (03PS1) 10Hashar: Zuul: [AchievementsBadges] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1225541 [12:25:00] (03CR) 10Hashar: [C:03+2] Zuul: [AchievementsBadges] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1225541 (owner: 10Hashar) [12:26:38] (03Merged) 10jenkins-bot: Zuul: [AchievementsBadges] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1225541 (owner: 10Hashar) [12:47:43] 10Continuous-Integration-Config, 10Wikidata, 07ci-test-error, 05MW-1.44-release: Wikibase REL1_44 gate-and-submit always times out - https://phabricator.wikimedia.org/T393869#11512098 (10A_smart_kitten) 05Open→03Resolved >>! In T393869#10823548, @hashar wrote: > For this task T393869, enabling para... [12:48:55] 06Project-Admins, 06Release-Engineering-Team (Doing 😎): Create project tag for English WP:RSP Restructuring - https://phabricator.wikimedia.org/T413709#11512103 (10Audiodude) Thank you so much! [12:56:28] 10Scap, 06serviceops, 06SRE, 07Datacenter-Switchover: Add scap lock/unlock steps to sre.switchdc.mediawiki cookbook - https://phabricator.wikimedia.org/T330996#11512111 (10Clement_Goubert) a:03Blake [13:25:03] maintenance-disconnect-full-disks build 771540 integration-agent-docker-1051 (/: 36%, /srv: 98%, /var/lib/docker: 27%): OFFLINE due to disk space [13:30:03] maintenance-disconnect-full-disks build 771541 integration-agent-docker-1051 (/: 36%, /srv: 70%, /var/lib/docker: 26%): RECOVERY disk space OK [14:32:30] 10Phabricator (Upstream), 07JavaScript, 07Upstream: Meme audio doesn't stop with preview regenerate - https://phabricator.wikimedia.org/T99759#11512444 (10Aklapper) [14:32:34] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance deployment-mx03 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [14:33:08] 10Phabricator (Upstream), 07JavaScript, 07Upstream: Meme audio doesn't stop with preview regenerate - https://phabricator.wikimedia.org/T99759#11512446 (10Aklapper) Well, Firefox 146 tells me in the Console that > * Autoplay is only allowed when approved by the user, the site is activated by the user, or med... [15:10:57] (03update) 10jforrester: Provide initial trixie images [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/81 [15:31:43] (03PS1) 10Hashar: Zuul: [ContentStabilization] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1225590 (https://phabricator.wikimedia.org/T389998) [15:32:37] (03PS1) 10Hashar: Zuul: [ContentDroplets] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1225591 (https://phabricator.wikimedia.org/T389998) [15:34:45] (03PS1) 10Hashar: Zuul: disable recursion on broken repos [integration/config] - 10https://gerrit.wikimedia.org/r/1225592 (https://phabricator.wikimedia.org/T389998) [15:35:09] (03CR) 10Hashar: [C:03+2] Zuul: disable recursion on broken repos [integration/config] - 10https://gerrit.wikimedia.org/r/1225592 (https://phabricator.wikimedia.org/T389998) (owner: 10Hashar) [15:35:11] (03CR) 10Hashar: [C:03+2] Zuul: [ContentStabilization] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1225590 (https://phabricator.wikimedia.org/T389998) (owner: 10Hashar) [15:35:12] (03CR) 10Hashar: [C:03+2] Zuul: [ContentDroplets] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1225591 (https://phabricator.wikimedia.org/T389998) (owner: 10Hashar) [15:36:43] (03Merged) 10jenkins-bot: Zuul: [ContentStabilization] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1225590 (https://phabricator.wikimedia.org/T389998) (owner: 10Hashar) [15:36:45] (03Merged) 10jenkins-bot: Zuul: [ContentDroplets] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1225591 (https://phabricator.wikimedia.org/T389998) (owner: 10Hashar) [15:36:46] (03Merged) 10jenkins-bot: Zuul: disable recursion on broken repos [integration/config] - 10https://gerrit.wikimedia.org/r/1225592 (https://phabricator.wikimedia.org/T389998) (owner: 10Hashar) [15:43:44] (03open) 10dancy: k8s-pvc-cleaner-values.yaml.tftpl: Use image.tag v1.0.8 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/546 [15:43:47] (03update) 10dancy: k8s-pvc-cleaner-values.yaml.tftpl: Use image.tag v1.0.8 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/546 [15:44:59] (03merge) 10dancy: k8s-pvc-cleaner-values.yaml.tftpl: Use image.tag v1.0.8 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/546 [15:57:31] 10Continuous-Integration-Infrastructure, 10Browser Test Platform, 10QS-Test-Automation, 07ci-test-error (WMF-deployed Build Failure): Selenium timeouts can cause the job to remain stuck until the build times out - https://phabricator.wikimedia.org/T389536#11512900 (10Daimona) As for the question of whether... [16:01:03] nudge on https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/231, would be nice to get that merged before the train cut [16:20:05] (03merge) 10aklapper: PHPMailer: Remove use of class.phpmailer-lite.php [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/89 [16:21:11] (03update) 10aklapper: Revert "Restrict project workboard access to logged in users only" [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/111 (https://phabricator.wikimedia.org/T403385) [16:21:59] 10Phabricator (Upstream), 07JavaScript, 07Upstream: Meme audio doesn't stop with preview regenerate - https://phabricator.wikimedia.org/T99759#11513010 (10Aklapper) 05Open→03Stalled a:03Aklapper [16:22:19] 10Phabricator (Upstream), 06Release-Engineering-Team (Doing 😎), 07JavaScript, 07Upstream: Meme audio doesn't stop with preview regenerate - https://phabricator.wikimedia.org/T99759#11513015 (10Aklapper) [16:22:30] (03merge) 10aklapper: Revert "Restrict project workboard access to logged in users only" [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/111 (https://phabricator.wikimedia.org/T403385) [16:23:33] good morning! pinging again to request review on https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/231 to hopefully make it into the release train that starts today. thcipriani perhaps? [16:45:03] maintenance-disconnect-full-disks build 771580 integration-agent-docker-1056 (/: 36%, /srv: 95%, /var/lib/docker: 32%): OFFLINE due to disk space [16:50:03] maintenance-disconnect-full-disks build 771581 integration-agent-docker-1056 (/: 36%, /srv: 65%, /var/lib/docker: 31%): RECOVERY disk space OK [17:03:56] tbodt: morning! I saw the ping on Sunday on GitLab, lemme catch up on the task [17:04:04] thanks! [17:04:19] 10Beta-Cluster-Infrastructure, 06collaboration-services, 13Patch-For-Review: Replace deployment-mx03 with a bookworm-based instance (was Puppet failure: "Unable to locate package spamd") - https://phabricator.wikimedia.org/T412975#11513241 (10dancy) >>! In T412975#11498683, @Dzahn wrote: > Should I abandon o... [17:07:38] 10Beta-Cluster-Infrastructure, 06collaboration-services, 13Patch-For-Review: Replace deployment-mx03 with a bookworm-based instance (was Puppet failure: "Unable to locate package spamd") - https://phabricator.wikimedia.org/T412975#11513242 (10dancy) p:05Triage→03Medium a:05thcipriani→03dancy [17:09:25] (03merge) 10thcipriani: make-release: Start branching MultiTitle [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/231 (https://phabricator.wikimedia.org/T404461) (owner: 10tbodt) [17:09:45] thankss [17:09:59] tbodt: merged, it should branch at 2am UTC [17:10:08] tonight [17:10:16] yay [17:13:29] 10Continuous-Integration-Infrastructure, 10Browser Test Platform, 10QS-Test-Automation, 07ci-test-error (WMF-deployed Build Failure): Selenium timeouts can cause the job to remain stuck until the build times out - https://phabricator.wikimedia.org/T389536#11513263 (10SLong-WMF) Thanks - this helps. Do you... [17:25:33] 10Scap: SecurityPatchBot: Only change a task's status to 'open' if its current status is a 'closed status' - https://phabricator.wikimedia.org/T414273#11513324 (10SomeRandomDeveloper) >>! In T414273#11511276, @A_smart_kitten wrote: > Hmm, that's a good point. Thank you for bringing that up, @somerandomdeveloper... [17:35:01] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 06Abstract Wikipedia team, 06collaboration-services, and 4 others: Wikifunctions function orchestrator and evaluator test suites failing on GitLab CI with OOM errors - https://phabricator.wikimedia.org/T399348#11513384 (10vaughnwalters) >>! In T3... [17:37:34] RESOLVED: PuppetAgentFailure: Puppet agent failure detected on instance deployment-mx03 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [17:38:28] FIRING: InstanceDown: Project deployment-prep instance deployment-mx03 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [17:38:35] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-mx03 is down - https://phabricator.wikimedia.org/T414366 (10wmcs-alerts) 03NEW [17:41:07] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 06Abstract Wikipedia team, 06collaboration-services, and 4 others: Wikifunctions function orchestrator and evaluator test suites failing on GitLab CI with OOM errors - https://phabricator.wikimedia.org/T399348#11513416 (10dancy) >>! In T399348#11... [17:42:11] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-mx03 is down - https://phabricator.wikimedia.org/T414366#11513428 (10dancy) I turned off this instance as part of T412975 [17:42:26] !log Turned off instance deployment-prep.deployment-mx03 [17:42:27] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:43:28] RESOLVED: InstanceDown: Project deployment-prep instance deployment-mx03 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [18:16:01] 06Release-Engineering-Team, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add yubikey ssh key for dancy - https://phabricator.wikimedia.org/T414032#11513565 (10Dzahn) [18:16:40] 06Release-Engineering-Team, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add yubikey ssh key for dancy - https://phabricator.wikimedia.org/T414032#11513566 (10dancy) 05Open→03Resolved [18:19:00] 10Beta-Cluster-Infrastructure, 06collaboration-services, 13Patch-For-Review: Replace deployment-mx03 with a bookworm-based instance (was Puppet failure: "Unable to locate package spamd") - https://phabricator.wikimedia.org/T412975#11513575 (10bd808) [18:19:04] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-mx03 is down - https://phabricator.wikimedia.org/T414366#11513577 (10bd808) →14Duplicate dup:03T412975 [18:19:47] bd808: consider acking the alert in alertmanager so you don't have to close a duplicate every day :P [18:25:08] 10Beta-Cluster-Infrastructure, 06collaboration-services, 13Patch-For-Review: Replace deployment-mx03 with a bookworm-based instance (was Puppet failure: "Unable to locate package spamd") - https://phabricator.wikimedia.org/T412975#11513606 (10bd808) [18:25:09] 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-mx03 in project deployment-prep - https://phabricator.wikimedia.org/T414262#11513608 (10bd808) →14Duplicate dup:03T412975 [18:26:39] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-shellbox01 on project deployment-prep - https://phabricator.wikimedia.org/T414304#11513613 (10bd808) `lang=shell-session,counterexample bd808@deployment-shellbox01:~$ sudo -i puppet agent -tv Info: Using environment 'production' Info... [18:45:30] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-shellbox01 on project deployment-prep - https://phabricator.wikimedia.org/T414304#11513679 (10bd808) Failure appears to be caused by https://gerrit.wikimedia.org/r/c/operations/puppet/+/1219770 where @ABran-WMF introduced new Hera se... [18:49:04] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-shellbox01 on project deployment-prep - https://phabricator.wikimedia.org/T414304#11513688 (10bd808) 05Open→03Resolved a:03bd808 A local fix for Beta Cluster is copying the new hiera settings into the project global hiera:... [18:52:28] FIRING: [8x] PuppetAgentNoResources: No Puppet resources found on instance deployment-deploy04 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:02:28] FIRING: [8x] PuppetAgentNoResources: No Puppet resources found on instance deployment-deploy04 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:03:58] FIRING: [8x] PuppetAgentNoResources: No Puppet resources found on instance deployment-deploy04 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:07:28] FIRING: [8x] PuppetAgentNoResources: No Puppet resources found on instance deployment-deploy04 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:12:28] FIRING: [8x] PuppetAgentNoResources: No Puppet resources found on instance deployment-deploy04 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:14:28] RESOLVED: [8x] PuppetAgentNoResources: No Puppet resources found on instance deployment-deploy04 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [20:59:00] (03PS1) 10Pppery: Zuul: Add SARTHAK SINGH to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1225676 [21:06:09] Project beta-scap-sync-world build #240744: 04FAILURE in 1 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/240744/ [21:06:47] !log Unblock 66.81.168.0/21 (T414303) [21:06:48] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:07:04] "Warning: Undefined array key "mpic" in /srv/mediawiki-staging/wmf-config/CommonSettings.php on line 5069" [21:08:41] got reverted: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1225677 [21:14:20] Yippee, build fixed! [21:14:20] Project beta-scap-sync-world build #240745: 09FIXED in 2 min 22 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/240745/ [21:30:31] 06Release-Engineering-Team, 10Projects-Cleanup, 06Reader Growth Team (Sprint 1 (Jan 6 - 26) Q3 25/26), 03Readers Essential Work 2025 (Sunsetting): Archive and remove VueTest repositories (permissions-blocked follow-up) - https://phabricator.wikimedia.org/T413828#11514404 (10KSarabia-WMF) p:05Low→03High [21:33:26] 06Release-Engineering-Team, 10Projects-Cleanup, 06Reader Growth Team (Sprint 1 (Jan 6 - 26) Q3 25/26), 03Readers Essential Work 2025 (Sunsetting): Archive and remove VueTest repositories (permissions-blocked follow-up) - https://phabricator.wikimedia.org/T413828#11514421 (10Pppery) It's unfortunately very... [21:33:45] !log zabe@deployment-mwmaint03:~$ foreachwiki migrateLinksTable.php --table imagelinks # T413668 [21:33:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:33:47] T413668: Run the data migration of imagelinks - https://phabricator.wikimedia.org/T413668 [21:34:07] 10Gerrit, 06Release-Engineering-Team, 10Projects-Cleanup, 10Wikimedia-GitHub, and 2 others: Archive and remove VueTest repositories (permissions-blocked follow-up) - https://phabricator.wikimedia.org/T413828#11514423 (10KSarabia-WMF) [21:34:15] 10Gerrit, 06Release-Engineering-Team, 10Projects-Cleanup, 10Wikimedia-GitHub, and 2 others: Archive and remove VueTest repositories (permissions-blocked follow-up) - https://phabricator.wikimedia.org/T413828#11514437 (10KSarabia-WMF) a:05KSarabia-WMF→03None [21:38:11] 10Continuous-Integration-Infrastructure, 10Browser Test Platform, 10QS-Test-Automation, 07ci-test-error (WMF-deployed Build Failure): Selenium timeouts can cause the job to remain stuck until the build times out - https://phabricator.wikimedia.org/T389536#11514440 (10Daimona) >>! In T389536#11513262, @SLon... [22:06:08] 10Gerrit, 06Release-Engineering-Team, 10Projects-Cleanup, 10Wikimedia-GitHub, and 2 others: Archive and remove VueTest repositories (permissions-blocked follow-up) - https://phabricator.wikimedia.org/T413828#11514518 (10thcipriani) [22:07:32] 10Gerrit, 06Release-Engineering-Team, 10Projects-Cleanup, 10Wikimedia-GitHub, and 2 others: Archive and remove VueTest repositories (permissions-blocked follow-up) - https://phabricator.wikimedia.org/T413828#11514530 (10thcipriani) Archived [[https://gerrit.wikimedia.org/g/mediawiki/extensions/VueTest|gerr... [23:23:27] 10Gerrit, 06Release-Engineering-Team (Doing 😎), 10Projects-Cleanup, 10Wikimedia-GitHub, and 4 others: Archive and remove VueTest repositories (permissions-blocked follow-up) - https://phabricator.wikimedia.org/T413828#11514708 (10thcipriani) 05Open→03Resolved a:03thcipriani [23:23:43] 10Gerrit: Request Gerrit access for Developer Account codeurluce - https://phabricator.wikimedia.org/T414390#11514713 (10A_smart_kitten) [23:37:43] 10Gerrit: Request Gerrit access for Developer Account codeurluce - https://phabricator.wikimedia.org/T414390#11514756 (10thcipriani) I see a couple of these in the Gerrit logs: ` 'codeurluce' failed to sign in: Incorrect username or password ` I can see you have a [[https://ldap.toolforge.org/user/codeurluce|d... [23:49:11] 10Gerrit: Request Gerrit access for Developer Account codeurluce - https://phabricator.wikimedia.org/T414390#11514771 (10Codeurluce) I reset my password via Wikitech and confirmed that I can log in to both Wikitech and Phabricator with the same credentials. However, Gerrit still rejects the login with an "incorr...