[00:08:13] (03CR) 10Drii10291: "hi" [integration/config] - 10https://gerrit.wikimedia.org/r/1279397 (https://phabricator.wikimedia.org/T395924) (owner: 10Krinkle) [00:11:45] 06Release-Engineering-Team (Radar), 10Scap, 10MediaWiki-Internationalization, 13Patch-For-Review, and 2 others: Use static php array files for l10n cache at WMF (instead of CDB) - https://phabricator.wikimedia.org/T99740#11885055 (10Peachey88) [09:06:18] (03CR) 10Hashar: [C:04-1] "`integration/docroot` still requires 7.4. It is used for https://integration.wikimedia.org/ which runs on contint machines which are on Bu" [integration/docroot] - 10https://gerrit.wikimedia.org/r/1281273 (owner: 10Jforrester) [09:46:03] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [extensions/RecentPageStats] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1282289 [09:46:03] (03CR) 10QChris: [V:03+2 C:03+2] Allow “Gerrit Managers” to import history [extensions/RecentPageStats] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1282289 (owner: 10QChris) [09:46:07] (03PS1) 10QChris: Import done. Revoke import grants [extensions/RecentPageStats] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1282290 [09:46:07] (03CR) 10QChris: [V:03+2 C:03+2] Import done. Revoke import grants [extensions/RecentPageStats] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1282290 (owner: 10QChris) [10:10:29] 10Continuous-Integration-Infrastructure, 10Castor, 07ci-test-error (WMF-deployed Build Failure): integration-castor06 is basically out of disk space and blocking CI - https://phabricator.wikimedia.org/T424769#11886134 (10hashar) Space free went from ~0% to ~ 40% Inodes free from ~70% to ~80% There is th... [10:11:30] 10Beta-Cluster-Infrastructure, 06Traffic, 13Patch-For-Review: Puppet agent failure detected on instance deployment-cache-text08 in project deployment-prep - https://phabricator.wikimedia.org/T424549#11886136 (10elukey) 05Open→03Resolved Fixed! [10:12:20] (03PS1) 10Hslater: Zuul: Provide non-voting PHP 8.4/8.5 Quibble jobs for bluespice template [integration/config] - 10https://gerrit.wikimedia.org/r/1282296 [10:16:02] (03CR) 10Jforrester: Zuul: Provide non-voting PHP 8.4/8.5 Quibble jobs for bluespice template (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1282296 (owner: 10Hslater) [10:25:07] (03CR) 10Hslater: Zuul: Provide non-voting PHP 8.4/8.5 Quibble jobs for bluespice template (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1282296 (owner: 10Hslater) [10:35:13] (03CR) 10Jforrester: Zuul: Provide non-voting PHP 8.4/8.5 Quibble jobs for bluespice template (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1282296 (owner: 10Hslater) [10:35:16] (03CR) 10Jforrester: [C:03+2] Zuul: Provide non-voting PHP 8.4/8.5 Quibble jobs for bluespice template [integration/config] - 10https://gerrit.wikimedia.org/r/1282296 (owner: 10Hslater) [10:37:00] (03Merged) 10jenkins-bot: Zuul: Provide non-voting PHP 8.4/8.5 Quibble jobs for bluespice template [integration/config] - 10https://gerrit.wikimedia.org/r/1282296 (owner: 10Hslater) [10:40:58] !log Zuul: Provide non-voting PHP 8.4/8.5 Quibble jobs for bluespice template [10:40:58] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:24:46] 10Phabricator, 07CSS, 07Mobile, 13Patch-For-Review: Simplified patch status comments have inconsistent margin on mobile - https://phabricator.wikimedia.org/T416964#11886331 (10matmarex) Yep, the changes I proposed fix all of these problems. [13:37:10] (03PS2) 10Hashar: Add test plans for sqlite and postgres [integration/quibble] - 10https://gerrit.wikimedia.org/r/1268433 (https://phabricator.wikimedia.org/T422461) (owner: 10Phedenskog) [13:47:02] 10Continuous-Integration-Infrastructure, 07Jenkins, 10Castor, 07Spike, 06Test Platform (Plovdiv 25): Realistic measure castor wait times and plan if there's a way to make it faster - https://phabricator.wikimedia.org/T423557#11886730 (10Peter) [13:51:29] (03Merged) 10jenkins-bot: Add test plans for sqlite and postgres [integration/quibble] - 10https://gerrit.wikimedia.org/r/1268433 (https://phabricator.wikimedia.org/T422461) (owner: 10Phedenskog) [14:04:36] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance deployment-eventgate-4 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [14:10:04] 06Release-Engineering-Team (Priority Backlog 📥), 10Scap, 10MW-on-K8s, 06ServiceOps new, 07Sustainability (Incident Followup): Add helm rollback functionality to scap - https://phabricator.wikimedia.org/T371069#11886837 (10MLechvien-WMF) Adding the as incident followup tag as it was identified in [[ https... [14:14:39] (03PS1) 10Hashar: tests: fix indentation in yaml test plans [integration/quibble] - 10https://gerrit.wikimedia.org/r/1282371 [14:23:41] 10Phabricator: Disable Phabricator account jhsoby-WMNO - https://phabricator.wikimedia.org/T424900#11886856 (10Dzahn) I note that the offboarding script does not consider these groups to be something a user has to be removed from. So it seems either there is nothing that needs to be done or there are some gr... [14:36:09] (03PS1) 10Hashar: tests: avoid repeatings common args in test plans [integration/quibble] - 10https://gerrit.wikimedia.org/r/1282375 [14:36:43] 06Release-Engineering-Team (Priority Backlog 📥), 10dev-images, 06Infrastructure-Foundations, 06SRE: Rebuild dev-images using a base image without mirrors.wikimedia.org in the apt sources - https://phabricator.wikimedia.org/T423972#11886917 (10MoritzMuehlenhoff) p:05Triage→03Medium [14:51:29] !log Upgrading gitlab cloud runners (staging) from 1.35.1-do.3 to 1.35.1-do.5 [14:51:30] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:14:19] (03open) 10dancy: staging/digitalocean.tfvars: kubernetes_version = "1.35.1-do.5" [auto-deploy] [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/590 (https://phabricator.wikimedia.org/T423726) [15:14:20] (03update) 10dancy: staging/digitalocean.tfvars: kubernetes_version = "1.35.1-do.5" [auto-deploy] [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/590 (https://phabricator.wikimedia.org/T423726) [15:15:30] (03merge) 10dancy: staging/digitalocean.tfvars: kubernetes_version = "1.35.1-do.5" [auto-deploy] [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/590 (https://phabricator.wikimedia.org/T423726) [15:19:43] !log Upgrading gitlab cloud runners (prod) from 1.35.1-do.3 to 1.35.1-do.5 [15:19:43] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:28:51] (03open) 10dancy: production/digitalocean.tfvars: kubernetes_version = "1.35.1-do.5" [auto-deploy] [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/591 (https://phabricator.wikimedia.org/T423726) [15:28:53] (03update) 10dancy: production/digitalocean.tfvars: kubernetes_version = "1.35.1-do.5" [auto-deploy] [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/591 (https://phabricator.wikimedia.org/T423726) [15:29:47] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 13Patch-For-Review: Update gitlab-cloud-runners kubernetes - https://phabricator.wikimedia.org/T423726#11887120 (10dancy) Updated to 1.35.1-do.5 today to address CVE-2026-31431 [15:30:28] (03merge) 10dancy: production/digitalocean.tfvars: kubernetes_version = "1.35.1-do.5" [auto-deploy] [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/591 (https://phabricator.wikimedia.org/T423726) [15:39:50] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 06collaboration-services, 06Traffic: Purge frontend cache when publish new coverage report under https://doc.wikimedia.org/cover - https://phabricator.wikimedia.org/T423951#11887159 (10Dzahn) 05Open→03Resolved [15:40:20] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 06collaboration-services, 06Traffic: Purge frontend cache when publish new coverage report under https://doc.wikimedia.org/cover - https://phabricator.wikimedia.org/T423951#11887160 (10Dzahn) a:05Dzahn→03None [15:41:07] (03open) 10dancy: main.py: Announce rollbacks [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1162 [15:41:11] (03update) 10dancy: main.py: Announce rollbacks [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1162 [15:43:12] (03update) 10dancy: Scap: Add Security Deploy Command [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1085 (https://phabricator.wikimedia.org/T361709) (owner: 10mstyles) [15:43:20] (03update) 10dancy: Scap: Add Security Deploy Command [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1085 (https://phabricator.wikimedia.org/T361709) (owner: 10mstyles) [15:43:44] (03merge) 10dancy: main.py: Announce rollbacks [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1162 [15:44:27] (03update) 10dancy: Release 4.260.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1163 [15:44:29] (03open) 10dancy: Release 4.260.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1163 [15:47:38] (03merge) 10dancy: Release 4.260.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1163 [15:53:33] 10Phabricator: Disable Phabricator account jhsoby-WMNO - https://phabricator.wikimedia.org/T424900#11887227 (10A_smart_kitten) I can't speak to the intentions behind the folks who developed the offboarding script, but I guess one possibility might be that maybe those groups aren't considered sensitive //enou... [16:28:47] 10Phabricator: Disable Phabricator account jhsoby-WMNO - https://phabricator.wikimedia.org/T424900#11887345 (10Dzahn) The script itself is in the public repo but the config file it uses for Phabricator is not it seems. [16:36:24] 10Phabricator: Disable Phabricator account jhsoby-WMNO - https://phabricator.wikimedia.org/T424900#11887360 (10Dzahn) re: "Trusted Contributors" - I don't think just because someone is offboard from a paid position with an affiliate makes them not a trusted user anymore? re: "acl*Project-Admins" - I think... [16:44:43] 10Phabricator: Disable Phabricator account jhsoby-WMNO - https://phabricator.wikimedia.org/T424900#11887374 (10A_smart_kitten) >>! In T424900#11887360, @Dzahn wrote: > re: "Trusted Contributors" - I don't think just because someone is offboarded from a paid position with an affiliate makes them not a truste... [16:46:10] 10Phabricator: Disable Phabricator account jhsoby-WMNO - https://phabricator.wikimedia.org/T424900#11887375 (10Aklapper) Accounts have roles. If roles don't exist anymore, such accounts ideally get disabled. [19:08:14] (03open) 10dancy: scap/patches.py: Remove hard-coded /srv/patches references [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1164 [19:08:17] (03update) 10dancy: scap/patches.py: Remove hard-coded /srv/patches references [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1164 [19:10:44] (03merge) 10dancy: scap/patches.py: Remove hard-coded /srv/patches references [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1164 [19:56:56] (03open) 10dancy: main.py: Remove a completed FIXME comment [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1165 [19:57:02] (03update) 10dancy: main.py: Remove a completed FIXME comment [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1165 [19:58:23] (03merge) 10dancy: main.py: Remove a completed FIXME comment [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1165 [20:03:51] (03PS6) 10Jeena Huneidi: WikiLambda selenium: Use ci-client image [integration/config] - 10https://gerrit.wikimedia.org/r/1277759 (https://phabricator.wikimedia.org/T423330) [20:04:24] (03CR) 10CI reject: [V:04-1] WikiLambda selenium: Use ci-client image [integration/config] - 10https://gerrit.wikimedia.org/r/1277759 (https://phabricator.wikimedia.org/T423330) (owner: 10Jeena Huneidi) [20:04:53] (03CR) 10Jeena Huneidi: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/1277759 (https://phabricator.wikimedia.org/T423330) (owner: 10Jeena Huneidi) [20:45:21] (03CR) 10Jeena Huneidi: [C:03+2] WikiLambda selenium: Use ci-client image [integration/config] - 10https://gerrit.wikimedia.org/r/1277759 (https://phabricator.wikimedia.org/T423330) (owner: 10Jeena Huneidi) [20:46:56] (03Merged) 10jenkins-bot: WikiLambda selenium: Use ci-client image [integration/config] - 10https://gerrit.wikimedia.org/r/1277759 (https://phabricator.wikimedia.org/T423330) (owner: 10Jeena Huneidi) [22:07:33] what would you say are the best mailing lists to cover all wikimedia gerrit users? wikitech-l I would think. oh, maybe if I have to notify all users for tech or maintenance reasons we can actually make gerrit itself mail all the user emails? [22:23:38] when we send train blocker mails, we hit wikitech-l, engineering-all, and ops [22:23:59] i don't know if gerrit has a _feature_ for that but i guess it would be possible to get the list of user mails, maybe [22:38:36] thanks brennen, happy to follow the standards for train blocker mails, or at least definitely include those plus more [22:38:56] i'd guess that catches a whole lot of the set of gerrit users, at least [22:39:00] yea, just feels to me like it should be gerrit itself [22:39:15] yeah, there might be something i'm missing there too [22:39:22] in this case it will be about "Fingerprint changes when you ssh to gerrit" [22:39:32] so that's really all of them [22:41:13] oh, heh. " Administrators can use the gerrit announce command via SSH to send a message to all connected users.", if true [22:41:29] nice (although i wonder what "connected" means) [22:41:39] but I mean.. I need to tell them before they connect :) [22:42:37] could be shell script since the backend is git and not mysql like back in the dayss [23:19:33] FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance deployment-eventgate-4 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun