[00:05:42] (03CR) 10Bartosz Dziewoński: [C:03+1] "I think it's okay to add people to the "trusted users" section after 1-2 commits, but this seems fine too. I'm not sure if anyone is going" [integration/config] - 10https://gerrit.wikimedia.org/r/1201769 (owner: 10Gergő Tisza) [00:38:07] 10GitLab, 06Release-Engineering-Team: GitLab Private Repository Request for: user-scripts-data-tooling - https://phabricator.wikimedia.org/T409219#11342958 (10brennen) 05Open→03Resolved a:03brennen Created as: https://gitlab.wikimedia.org/repos/projects/user-scripts-data-tooling Added mentioned memb... [05:43:09] 10GitLab, 06collaboration-services: Continuous breakages of apt-staging - https://phabricator.wikimedia.org/T409253 (10Joe) 03NEW [05:43:41] 10GitLab, 06collaboration-services: Continuous breakages of apt-staging - https://phabricator.wikimedia.org/T409253#11343311 (10Joe) p:05Triage→03Unbreak! Triaging to UBN! as this is blocking activity for at least two hypotheses. [05:59:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance deployment-docker-cxserver02 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [05:59:38] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-docker-cxserver02 on project deployment-prep - https://phabricator.wikimedia.org/T409254 (10wmcs-alerts) 03NEW [06:29:05] 10GitLab, 06collaboration-services, 06Infrastructure-Foundations: Continuous breakages of apt-staging - https://phabricator.wikimedia.org/T409253#11343342 (10LSobanski) [07:52:54] (03PS3) 10Hashar: zuul: sort dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/1199459 [07:56:44] (03CR) 10Hashar: [C:03+2] "The list has a lot of accounts that had no activity in years, then we don't automatically deactivate accounts nor do we have any system to" [integration/config] - 10https://gerrit.wikimedia.org/r/1201769 (owner: 10Gergő Tisza) [07:57:03] (03CR) 10Hashar: [C:03+2] zuul: sort dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/1199459 (owner: 10Hashar) [07:58:19] (03Merged) 10jenkins-bot: Add camilojdiaz to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1201769 (owner: 10Gergő Tisza) [07:58:31] (03Merged) 10jenkins-bot: zuul: sort dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/1199459 (owner: 10Hashar) [08:31:32] 10GitLab, 06collaboration-services, 06Infrastructure-Foundations: Continuous breakages of apt-staging - https://phabricator.wikimedia.org/T409253#11343607 (10MatthewVernon) It might still be caching, but `https://apt-staging.wikimedia.org/wikimedia-staging/dists/trixie-wikimedia/main/binary-amd64/` is saying... [08:39:54] 06Project-Admins, 10Projects-Cleanup: Clarify the status of Structured Content / Structured Data team - https://phabricator.wikimedia.org/T404286#11343615 (10Bugreporter) >>! In T404286#11342319, @Aklapper wrote: > I archived #Structured_Data_Engineering and #structured-data-backlog. > Likely there will be man... [08:52:02] 10Phabricator (2025-11-04), 06Release-Engineering-Team (Doing 😎): Restrict project workboard access to logged in users only - https://phabricator.wikimedia.org/T403385#11343636 (10Aklapper) Hi, please do provide clear use cases, and why the person cannot register an account. Thanks. [09:10:21] 10GitLab, 06collaboration-services, 06Infrastructure-Foundations: Continuous breakages of apt-staging - https://phabricator.wikimedia.org/T409253#11343726 (10MoritzMuehlenhoff) I came to the same conclusion, I've moved the botched wmfmariadbpy changes file away and now we have at least up-to-date Packages fi... [09:20:03] maintenance-disconnect-full-disks build 751907 integration-agent-docker-1065 (/: 25%, /srv: 99%, /var/lib/docker: 42%): OFFLINE due to disk space [09:25:03] maintenance-disconnect-full-disks build 751908 integration-agent-docker-1065 (/: 25%, /srv: 42%, /var/lib/docker: 41%): RECOVERY disk space OK [09:38:16] (03CR) 10Lucas Werkmeister (WMDE): "I think this caused T409269? (Though I’m not sure how the image and PHP versions correspond to one another. But the timing matches up.)" [integration/config] - 10https://gerrit.wikimedia.org/r/1201680 (https://phabricator.wikimedia.org/T408653) (owner: 10Hashar) [09:55:55] (03CR) 10Hashar: [C:03+2] "Probably! I will follow up on the task with my investigation." [integration/config] - 10https://gerrit.wikimedia.org/r/1201680 (https://phabricator.wikimedia.org/T408653) (owner: 10Hashar) [09:59:24] 10GitLab, 06collaboration-services, 06Infrastructure-Foundations, 13Patch-For-Review: Continuous breakages of apt-staging - https://phabricator.wikimedia.org/T409253#11343878 (10LSobanski) p:05Unbreak!→03High As an aside, this task doesn''t meet Unbreak Now criteria as defined in https://www.mediawiki.... [10:04:11] 10Beta-Cluster-Infrastructure: [betalabs-minor] Delete local overwrite of MediaWiki:File-exists-duplicate on commons.wikimedia.beta.wmflabs.org - https://phabricator.wikimedia.org/T269191#11343893 (10A_smart_kitten) …in lieu of a more appropriate tag that I'm aware of for requesting admin actions on the beta clu... [10:13:45] Lucas_WMDE: at a quick glance the date `+92000000200-05-31T00:00:00Z` is rejected by PHP 8.3.25 because the year is above `INT32_MAX - 1` [10:14:17] I don't know how PHP beahved before the bound check [10:14:31] maybe that caused some integer overflow error or the year just rolled modulo INT32_MAX [10:14:51] yes, I’ve been looking into it already (testing the previous behavior in mw-debug-repl) [10:15:01] (https://phabricator.wikimedia.org/T409269 and attached change) [10:16:05] I guess the code needs to be adjusted to catch the ValueError and interpret that as an invalid date [10:17:13] (and possibly the code dealing with date in Wikibase can be upstreamed to a library, maybe wikimedia/timestamp [10:18:43] oh you get a patch already, kudos! [10:25:10] I just hope it’s also correct ^^ [10:25:13] let’s see what code review says [10:25:22] (03PS1) 10Hashar: Zuul: use 2 spaces indent in YAML deps files [integration/config] - 10https://gerrit.wikimedia.org/r/1202058 [10:25:24] (03CR) 10Hashar: [C:03+2] Zuul: use 2 spaces indent in YAML deps files [integration/config] - 10https://gerrit.wikimedia.org/r/1202058 (owner: 10Hashar) [10:27:03] (03Merged) 10jenkins-bot: Zuul: use 2 spaces indent in YAML deps files [integration/config] - 10https://gerrit.wikimedia.org/r/1202058 (owner: 10Hashar) [10:28:27] hashar: that year <= INT32_MAX - 1 condition reminds me of “ The fix started with if (size > INT_MAX) return NULL; and went downhill from there.” :D [10:28:41] (yes, yes, I know, comparing a zend_long to INT*32*_MAX is sensible enough, at least on 64-bit platforms) [11:49:31] 06Project-Admins, 07Tracking-Neverending: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706#11344205 (10Ladsgroup) >>! In T706#11342796, @SherryYang-WMF wrote: > Hello! I'm onboarding as Reader Growth PM--could you add me to acl*Project-Admins so I ca... [12:43:52] (03approved) 10jnuche: backport.py: Improve dependency cycle detection [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1031 (https://phabricator.wikimedia.org/T408675) (owner: 10dancy) [12:50:14] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Sike), 07Essential-Work: Don't hard fail wiki creation on importDump.php failures - https://phabricator.wikimedia.org/T408828#11344329 (10jnuche) [12:51:00] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Sike), 07Essential-Work: OpenTofu: Clean up older local path provisioner addons - https://phabricator.wikimedia.org/T408379#11344331 (10jnuche) [13:45:44] 06Release-Engineering-Team, 10MediaWiki-REST-API, 05MW-1.45-notes, 05MW-1.45-release, 06MW-Interfaces-Team (MWI-Sprint-21 (2025-10-21 to 2025-11-04)): Remove SwaggerUI from MW-1.45 release - https://phabricator.wikimedia.org/T397473#11344590 (10Reedy) 05Open→03Resolved [14:09:12] 06Release-Engineering-Team, 10MediaWiki-REST-API, 07Epic, 05MW-1.43-notes, and 2 others: [REST Sandbox] Remove SwaggerUI from MediaWiki Releases - https://phabricator.wikimedia.org/T397470#11344735 (10Reedy) [14:17:31] 06Release-Engineering-Team (Doing 😎), 10Scap, 07Essential-Work: scap not complaining about dependencies only partially deployed with the train - https://phabricator.wikimedia.org/T397931#11344759 (10Lucas_Werkmeister_WMDE) Seems to be working in https://spiderpig.wikimedia.org/jobs/859 ^^ [14:27:08] 06Release-Engineering-Team (Doing 😎), 10Scap, 07Essential-Work: scap not complaining about dependencies only partially deployed with the train - https://phabricator.wikimedia.org/T397931#11344804 (10Lucas_Werkmeister_WMDE) Actually, that SpiderPig deployment is still confusing to me. The change it’s comp... [15:39:36] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 07Essential-Work: Buildkit v0.25.2 released - https://phabricator.wikimedia.org/T409313 (10dancy) 03NEW [15:39:52] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 07Essential-Work: Buildkit v0.23.2 released - https://phabricator.wikimedia.org/T397204#11345269 (10dancy) 05Open→03Declined [15:41:19] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 07Essential-Work: Buildkit v0.25.2 released - https://phabricator.wikimedia.org/T409313#11345272 (10dancy) [15:57:55] 06Release-Engineering-Team (Doing 😎), 10Scap, 07Essential-Work, 13Patch-For-Review: Proposal for refactoring of backport dependency handling - https://phabricator.wikimedia.org/T362987#11345342 (10dancy) 05Resolved→03Open In T397931#11344804 (https://spiderpig.wikimedia.org/jobs/859) we have someone be... [15:59:43] 10Diffusion, 10Phabricator, 06Release-Engineering-Team (Doing 😎), 06cloud-services-team, 10Striker: Striker: Always set http IO to none when creating Diffusion repositories - https://phabricator.wikimedia.org/T407705#11345349 (10Aklapper) This has been merged but does not seem to be deployed yet, looking... [15:59:49] 06Release-Engineering-Team (Doing 😎), 10Scap, 07Essential-Work: scap not complaining about dependencies only partially deployed with the train - https://phabricator.wikimedia.org/T397931#11345352 (10dancy) >>! In T397931#11344804, @Lucas_Werkmeister_WMDE wrote: > Actually, that SpiderPig deployment is st... [15:59:57] 10Diffusion, 10Phabricator, 06Release-Engineering-Team (Doing 😎), 06cloud-services-team, 10Striker: Striker: Always expose observed external URI when creating Diffusion repositories - https://phabricator.wikimedia.org/T407624#11345353 (10Aklapper) This has been merged but does not seem to be deployed yet... [16:01:09] 06Release-Engineering-Team (Doing 😎), 10Scap, 07Essential-Work: scap not complaining about dependencies only partially deployed with the train - https://phabricator.wikimedia.org/T397931#11345364 (10dancy) >>! In T397931#11344804, @Lucas_Werkmeister_WMDE wrote: > Is it “deployable” because it hasn’t bee... [16:10:12] 06Release-Engineering-Team (Doing 😎), 10Scap, 07Essential-Work, 13Patch-For-Review: Proposal for refactoring of backport dependency handling - https://phabricator.wikimedia.org/T362987#11345387 (10jnuche) > In T397931#11344804 (https://spiderpig.wikimedia.org/jobs/859) we have someone being blocked from ba... [16:10:44] (03open) 10dancy: Use buildkit v0.25.2 in staging and prod [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/529 (https://phabricator.wikimedia.org/T409313) [16:10:45] (03update) 10dancy: Use buildkit v0.25.2 in staging and prod [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/529 (https://phabricator.wikimedia.org/T409313) [16:12:45] (03merge) 10dancy: Use buildkit v0.25.2 in staging and prod [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/529 (https://phabricator.wikimedia.org/T409313) [16:20:12] !log Updated buildkitd to v0.25.2 in gitlab-cloud-runners (staging and prod) (T409313) [16:20:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:20:18] T409313: Buildkit v0.25.2 released - https://phabricator.wikimedia.org/T409313 [16:20:44] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 07Essential-Work, 13Patch-For-Review: Buildkit v0.25.2 released - https://phabricator.wikimedia.org/T409313#11345473 (10dancy) p:05Triage→03High a:03dancy [16:20:47] (03CR) 10Jforrester: "Thank you!" [integration/config] - 10https://gerrit.wikimedia.org/r/1199529 (https://phabricator.wikimedia.org/T408653) (owner: 10Daimona Eaytoy) [16:26:04] AFAICT gate-and-submit is currently basically blocked by T409303 [16:26:05] T409303: ChangeWikiConfigTest fails in gate and submit, blocking Wikibase CI - https://phabricator.wikimedia.org/T409303 [16:26:11] it’s just slowly failing one build at a time [16:26:22] and triggering the rest of the increasingly long queue to rebuild again and again :S [16:29:13] it's funny how many examples we keep getting of why adding more and more things to the gate is a bad idea... [16:30:11] 10Continuous-Integration-Config, 10Math, 06MediaWiki-Platform-Team (Radar), 05MW-1.45-notes (1.45.0-wmf.24; 2025-10-21), and 2 others: Allow control over which extra extensions are installed (Math REL1_43 jobs exceed 60min timeout) - https://phabricator.wikimedia.org/T389998#11345527 (10hashar) [16:30:36] Lucas_WMDE: It's weird that nothing has really changed in CommunityConfiguration in basically a month [16:31:08] 06Release-Engineering-Team (Doing 😎), 10Scap, 07Essential-Work, 13Patch-For-Review: Proposal for refactoring of backport dependency handling - https://phabricator.wikimedia.org/T362987#11345531 (10Lucas_Werkmeister_WMDE) To be honest, I would prefer a warning prompt either way. In the specific case mention... [16:31:33] yeah, I’m also very confused so far :( [16:31:50] urbanecm: Around by any chance? [16:32:08] it seems to be some kind of Captcha thing but nothing in ConfirmEdit looks like a smoking gun either [16:34:03] Lucas_WMDE: I can help you work around the backport issue you had in https://spiderpig.wikimedia.org/jobs/859 [16:34:21] I've just flagged it in the hCaptcha slack channel [16:34:57] thanks [16:35:18] dancy: I’m fine right now (or rather, busy with other errors :D) [16:35:26] ok! [16:35:29] it turned out we had to revert that config change anyway [16:35:59] Thanks for you input on T362987 as well [16:36:00] T362987: Proposal for refactoring of backport dependency handling - https://phabricator.wikimedia.org/T362987 [16:41:42] (03update) 10dancy: backport.py: Improve dependency cycle detection [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1031 (https://phabricator.wikimedia.org/T408675) [16:43:53] (03merge) 10dancy: backport.py: Improve dependency cycle detection [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1031 (https://phabricator.wikimedia.org/T408675) [16:45:33] (03open) 10dancy: Release 4.223.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1034 [16:46:27] 06Release-Engineering-Team, 10Scap, 07Essential-Work, 13Patch-For-Review: Improve dependency cycle detection during a backport - https://phabricator.wikimedia.org/T408675#11345595 (10dancy) p:05Triage→03Medium a:03dancy [16:47:41] (03merge) 10dancy: Release 4.223.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1034 [16:56:14] thcipriani: I don't see that user profile setting listed in our GitLab instance; https://docs.gitlab.com/administration/settings/account_and_limit_settings/#prevent-users-from-making-their-profiles-private probably implies that we need to pay [16:59:03] 06Release-Engineering-Team, 10Scap, 07Essential-Work, 13Patch-For-Review: Improve dependency cycle detection during a backport - https://phabricator.wikimedia.org/T408675#11345659 (10dancy) 05Open→03Resolved Deployed via scap 4.223.0. [17:15:47] Reedy: I see you got some people’s attention ;) https://phabricator.wikimedia.org/T409303#11345687 [17:17:31] heh [17:17:38] I mean, it is getting close to UBN territory [17:18:21] I thought it qualified for UBN anyway [17:18:28] if it breaks CI for more than one repo [17:18:33] (but I didn’t bump it yet) [17:18:43] I guess it doesn’t 100% break CI, but it’s pretty close [17:18:53] yeah [17:19:01] I don't think anyone would blame you for marking as such [17:20:07] eh, with the latest message I think it has enough attention for now [17:20:29] see if your confirmedit patch passes... [17:21:08] I’m also gonna sign off pretty soon [17:21:15] if the patch seems promising but turns out to need changes, feel free to amend it [17:21:42] the backlog in gate-and-submit isn't exactly helpful either [17:30:32] (03CR) 10Hashar: [C:03+2] "The `archive-log-allow-empty` makes Jenkins to retrieve artifacts stored in the `log` directory within the build workspace. This happens " [integration/config] - 10https://gerrit.wikimedia.org/r/1199868 (owner: 10Pwangai) [17:30:40] 10MediaWiki-Releasing, 05MW-1.45-notes (1.45.0-wmf.25; 2025-10-28), 05MW-1.45-release, 06MW-Interfaces-Team (MWI-Sprint-21 (2025-10-21 to 2025-11-04)), 13Patch-For-Review: Update CREDITS in MW master - https://phabricator.wikimedia.org/T408401#11345791 (10Aklapper) Sorry, I should have elaborated - t... [17:31:59] (03Merged) 10jenkins-bot: jjb: Storing wikilambda selenium artifacts from catalyst builds [integration/config] - 10https://gerrit.wikimedia.org/r/1199868 (owner: 10Pwangai) [17:32:42] * Lucas_WMDE off [17:32:52] hoping someone else will have figured out the CI issue by the time I’m back tomorrow 🤞 [17:36:34] 10MediaWiki-Releasing, 05MW-1.45-notes (1.45.0-wmf.25; 2025-10-28), 05MW-1.45-release, 06MW-Interfaces-Team (MWI-Sprint-21 (2025-10-21 to 2025-11-04)), 13Patch-For-Review: Update CREDITS in MW master - https://phabricator.wikimedia.org/T408401#11345817 (10A_smart_kitten) >>! In T408401#11345791, @Akl... [17:44:10] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 07Essential-Work: Buildkit v0.25.2 released - https://phabricator.wikimedia.org/T409313#11345841 (10dancy) 05Open→03Resolved [17:47:09] sometimes I wish we could put patches to the top of the CI queue... [17:48:06] sure it could be abused, but.... [17:51:43] Reedy: Zuul has that feature but IIRC it is broken [17:51:57] 10Continuous-Integration-Infrastructure, 07Zuul: Force a patch to the top of the CI gate and submit queue - https://phabricator.wikimedia.org/T409326 (10Reedy) 03NEW [17:52:05] hashar: heh [17:52:18] on contint.wikimedia.org [17:52:53] zuul promote --pipeline gate-and-submit --changes 12345 12366 [17:53:09] which would reshuffle the queue with those two changes at the top [17:53:22] and of course cancel every single jobs that were running [17:53:27] Definitely happens often enough that would be really useful [17:53:34] If we know a large % of them are going to fail anyway... [17:53:42] but last time I tried I think it was broken due to some python str/int mismatch, I think due to a change in Gerrit [17:53:59] well, I've filed a task about it :P [17:54:09] then if a job fail, the change is marked as failing and it is moved [17:54:53] I can try the promote command and see what happens [17:55:13] Reedy: which change should be moved on top ? [17:55:21] 1202210 [17:55:38] Some of the ones before it may pass... But many probably won't [17:56:26] it does not work :) [17:56:30] booo [17:56:38] can we blame paladox? ;P [17:57:09] Project beta-scap-sync-world build #231137: 04FAILURE in 1 min 58 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/231137/ [17:57:28] >17:57:05 17:57:05 Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'ConnectTimeoutError(, 'Connection to logging-logstash-02.logging.eqiad1.wikimedia.cloud timed out. (connect timeout=10)')': /logstash-*/_search [17:57:33] heh [18:00:47] 10Continuous-Integration-Infrastructure, 07Zuul: Force a patch to the top of the CI gate and submit queue - https://phabricator.wikimedia.org/T409326#11345919 (10hashar) 05Open→03Declined Zuul has a `zuul promote` command to do that: ` usage: zuul promote [-h] --pipeline PIPELINE --changes CHANGES [CHA... [18:01:23] hashar: Shouldn't that be a "stalled" or something pending an upgrade of Zuul? [18:01:37] Because it really would be a useful thing... even if it is doable via CLI [18:04:31] possibly, then my life is easier with less task open/stalled :] [18:05:17] regardless, I agree promoting change would be a good thing [18:05:46] but you just declined it :P [18:06:56] Project beta-scap-sync-world build #231138: 04STILL FAILING in 1 min 48 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/231138/ [18:08:31] * Reedy continues to abort some long running jobs after quicker jobs already failed [18:17:02] Project beta-scap-sync-world build #231139: 04STILL FAILING in 1 min 51 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/231139/ [18:24:03] next patch through CI should stop most of these failures we're seeing... [18:26:58] Project beta-scap-sync-world build #231140: 04STILL FAILING in 1 min 47 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/231140/ [18:31:11] 10GitLab, 06Release-Engineering-Team: GitLab Private Repository Request for: user-scripts-data-tooling - https://phabricator.wikimedia.org/T409219#11346030 (10sbassett) Hey @brennen - Thanks! Looks like we're all good here. [18:36:51] Project beta-scap-sync-world build #231141: 04STILL FAILING in 1 min 40 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/231141/ [18:37:57] No route to host')': /logstash-*/_search [18:37:59] poor beta [18:46:54] Project beta-scap-sync-world build #231142: 04STILL FAILING in 1 min 42 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/231142/ [18:54:11] hrm, wonder if the beta logstash hostname changed? [18:56:51] https://beta-logs.wmcloud.org/ still works [18:56:58] Project beta-scap-sync-world build #231143: 04STILL FAILING in 1 min 44 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/231143/ [19:01:10] sure enough, it's 03 now [19:03:05] hrm. Looks like a cascading error happening too [19:03:42] just noticed this is hard coded in scap.cfg in puppet. I wonder how hard this would be to template... [19:04:19] that is, in modules/scap/templates/scap.cfg.erb in puppet [19:06:30] and, yeah, message is a lil noisy [19:06:42] I'll fix that bug [19:06:48] Project beta-scap-sync-world build #231144: 04STILL FAILING in 1 min 41 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/231144/ [19:07:17] And I can make the scap.cfg.erb change. [19:07:24] (The simple change) [19:07:43] ack, thanks [19:17:02] Project beta-scap-sync-world build #231145: 04STILL FAILING in 1 min 49 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/231145/ [19:17:23] Hmm. I can't `curl logging-logstash-03.logging.eqiad1.wikimedia.cloud:9200` from deployment-deploy04. [19:17:31] I'll file a bug [19:21:22] 10Beta-Cluster-Infrastructure, 10Scap: Scap can't connect to logging-logstash-02.logging.eqiad1.wikimedia.cloud in beta - https://phabricator.wikimedia.org/T409339 (10dancy) 03NEW [19:23:04] 10Beta-Cluster-Infrastructure, 10Scap: Scap can't connect to logging-logstash-02.logging.eqiad1.wikimedia.cloud in beta - https://phabricator.wikimedia.org/T409339#11346319 (10dancy) p:05Triage→03High [19:24:28] 10Beta-Cluster-Infrastructure, 10Scap: Scap can't connect to logging-logstash-02.logging.eqiad1.wikimedia.cloud in beta - https://phabricator.wikimedia.org/T409339#11346326 (10dancy) [19:25:31] 10Beta-Cluster-Infrastructure, 10Scap, 10Observability-Logging: Scap can't connect to logging-logstash-02.logging.eqiad1.wikimedia.cloud in beta - https://phabricator.wikimedia.org/T409339#11346329 (10dancy) [19:26:28] 10Beta-Cluster-Infrastructure, 10Scap, 10Observability-Logging: Scap can't connect to logging-logstash-02.logging.eqiad1.wikimedia.cloud in beta - https://phabricator.wikimedia.org/T409339#11346332 (10dancy) [19:26:58] Project beta-scap-sync-world build #231146: 04STILL FAILING in 1 min 49 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/231146/ [19:28:26] !log Disabled beta-scap-sync-world job (T409339) [19:28:28] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:28:29] T409339: Scap can't connect to logging-logstash-02.logging.eqiad1.wikimedia.cloud in beta - https://phabricator.wikimedia.org/T409339 [19:29:11] 10Diffusion, 10Phabricator, 06Release-Engineering-Team (Doing 😎), 06cloud-services-team, 10Striker: Striker: Always set ssh IO to none when creating Diffusion repositories - https://phabricator.wikimedia.org/T409341 (10Aklapper) 03NEW p:05Triage→03Low [19:46:49] 10Diffusion, 10Phabricator, 06Release-Engineering-Team (Doing 😎), 06cloud-services-team, 10Striker: Striker: Always set all IO to none and hidden for internal URIs when creating Diffusion repositories - https://phabricator.wikimedia.org/T409341#11346429 (10Aklapper) [19:48:45] (03open) 10dancy: logstash_checker.py: Reexport CheckServiceError from logstash package [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1035 [19:48:48] (03update) 10dancy: logstash_checker.py: Reexport CheckServiceError from logstash package [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1035 [19:52:44] (03update) 10dancy: logstash_checker.py: Reexport CheckServiceError from logstash package [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1035 [19:53:55] (03update) 10dancy: logstash_checker.py: Reexport CheckServiceError from logstash package [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1035 [19:58:12] (03update) 10dancy: logstash_checker.py: Reexport CheckServiceError from logstash package [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1035 [19:58:17] (03update) 10dancy: logstash_checker.py: Reexport CheckServiceError from logstash package [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1035 [20:00:27] (03merge) 10dancy: logstash_checker.py: Reexport CheckServiceError from logstash package [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1035 [20:01:17] (03update) 10dancy: Release 4.224.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1036 [20:01:17] (03open) 10dancy: Release 4.224.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1036 [20:02:34] (03merge) 10dancy: Release 4.224.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1036 [20:10:11] !log Upgrading scap to 4.224.0 in beta cluster [20:10:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:20:11] (03open) 10jforrester: releases: Bump jsdoc from 4.0.4 to 4.0.5, as a low-priority fix [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/97 [20:20:15] (03merge) 10jforrester: releases: Bump jsdoc from 4.0.4 to 4.0.5, as a low-priority fix [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/97 [20:31:41] Project mediawiki-core-doxygen build #14904: 04FAILURE in 13 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/14904/ [20:32:07] 06Release-Engineering-Team (Doing 😎), 10Scap, 07Essential-Work, 13Patch-For-Review: Proposal for refactoring of backport dependency handling - https://phabricator.wikimedia.org/T362987#11346637 (10dancy) >>! In T362987#11345387, @jnuche wrote: > Then we can adapt the current logic to make a distinction for... [20:50:38] 10GitLab, 06Release-Engineering-Team: GitLab Private Repository Request for: user-scripts-data-tooling - https://phabricator.wikimedia.org/T409219#11346717 (10Aklapper) [20:54:18] 10Beta-Cluster-Infrastructure, 10Scap, 10Observability-Logging: Scap can't connect to logging-logstash-02.logging.eqiad1.wikimedia.cloud in beta - https://phabricator.wikimedia.org/T409339#11346755 (10bd808) https://openstack-browser.toolforge.org/server/logging-logstash-02.logging.eqiad1.wikimedia.cloud rep... [20:58:23] 10Beta-Cluster-Infrastructure, 10Scap, 10Observability-Logging: Scap can't connect to logging-logstash-02.logging.eqiad1.wikimedia.cloud in beta - https://phabricator.wikimedia.org/T409339#11346759 (10bd808) > As an alternate I tried connecting to logging-logstash-03.logging.eqiad1.wikimedia.cloud:9200 but t... [20:59:46] cwhite: we could use your help figuring out what to do to fix T409339. [20:59:47] T409339: Scap can't connect to logging-logstash-02.logging.eqiad1.wikimedia.cloud in beta - https://phabricator.wikimedia.org/T409339 [21:00:31] 10Beta-Cluster-Infrastructure, 10Scap, 10Observability-Logging: Beta Cluster MediaWiki updates blocked because scap can't connect to logging-logstash-02.logging.eqiad1.wikimedia.cloud in beta - https://phabricator.wikimedia.org/T409339#11346777 (10bd808) [21:04:00] bd808: I powereed it back on to unblock. Let's look into pointing scap at a newer host. :) [21:04:30] add the needed network ACL to another host and we can do that :) [21:05:41] It might be nice to have a service name to point Beta Cluster at rather than a single instance too. [21:07:17] !log Manually triggered beta-scap-sync-world to test T409339 [21:07:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:07:20] T409339: Beta Cluster MediaWiki updates blocked because scap can't connect to logging-logstash-02.logging.eqiad1.wikimedia.cloud in beta - https://phabricator.wikimedia.org/T409339 [21:08:16] 10Beta-Cluster-Infrastructure, 10Scap, 10Observability-Logging: Beta Cluster MediaWiki updates blocked because scap can't connect to logging-logstash-02.logging.eqiad1.wikimedia.cloud in beta - https://phabricator.wikimedia.org/T409339#11346806 (10bd808) `lang=irc [21:03] < cwhite> bd808: I powereed it ba... [21:08:48] Yippee, build fixed! [21:08:48] Project beta-scap-sync-world build #231147: 09FIXED in 2 min 3 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/231147/ [21:16:29] 10Beta-Cluster-Infrastructure, 10Scap, 10Observability-Logging: Beta Cluster MediaWiki updates blocked because scap can't connect to logging-logstash-02.logging.eqiad1.wikimedia.cloud in beta - https://phabricator.wikimedia.org/T409339#11346822 (10bd808) Leaving this as high for now as the current fix is tem... [21:18:44] 10Beta-Cluster-Infrastructure, 10Scap, 10Observability-Logging: Beta Cluster MediaWiki updates require logging-logstash-02.logging.eqiad1.wikimedia.cloud to allow access to port 9200 by `scap` - https://phabricator.wikimedia.org/T409339#11346826 (10bd808) [21:22:55] bd808: the service name is a good idea. How might I go about doing that? The project->dns->domains path from the wikitech docs looks missing. [21:24:35] cwhite: In Horizon there is a DNS > Zones screen where you would typically be able to create custom records. Interestingly I am not seeing any delegated zones in the logging project's Zones screen at the moment. [21:25:07] I'm not sure if that is a bug in my admin access or missing delegation for the project. [21:25:25] Yeah, I see that too. Only a text box to filter and no buttons to modify anything. [21:25:30] 06Release-Engineering-Team (Radar), 06collaboration-services, 06SRE, 06Traffic, 13Patch-For-Review: Deploy a TCP proxy across all DCs - https://phabricator.wikimedia.org/T408532#11346868 (10Dzahn) We are debugging why things work from SOME of the VMs but not from others..in this pattern: {P84954} [21:25:54] cwhite: ok. that's good confirmation that we need to give you some zones :) [21:27:19] Is that difficult to do? I've opened up logging-logstash-04 port 9200 as well so scap can use that if it's a pain. [21:27:47] https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/DNS/Designate#svc_zone_in_.wikimedia.cloud_domain is a runbook for the zone creation and delegation, but it may be outdated. I can't remember if this is done with tofu these days or not. [21:29:14] Yippee, build fixed! [21:29:14] Project mediawiki-core-doxygen build #14905: 09FIXED in 11 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/14905/ [21:29:55] cwhite: we should at least open a task about getting the zones straightened out. I would expect logging.wmcloud.org, logging.eqiad1.wmcloud.org, and svc.logging.eqiad1.wmcloud.org to be under project control. [21:30:10] * bd808 does that task creation [21:32:26] Thank you! In the mean time, I'll upload a patch for scap. [21:52:31] 10Beta-Cluster-Infrastructure, 10Scap, 10Observability-Logging, 13Patch-For-Review: Beta Cluster MediaWiki updates require logging-logstash-02.logging.eqiad1.wikimedia.cloud to allow access to port 9200 by `scap` - https://phabricator.wikimedia.org/T409339#11347028 (10colewhite) The network ACLs and the sc... [21:53:08] !log Forced puppet run on deployment-deploy04.deployment-prep.eqiad1.wikimedia.cloud to pick up new scap config (T409339) [21:53:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:53:10] T409339: Beta Cluster MediaWiki updates require logging-logstash-02.logging.eqiad1.wikimedia.cloud to allow access to port 9200 by `scap` - https://phabricator.wikimedia.org/T409339 [21:54:56] 10Beta-Cluster-Infrastructure, 10Scap, 10Observability-Logging, 13Patch-For-Review: Beta Cluster MediaWiki updates require logging-logstash-02.logging.eqiad1.wikimedia.cloud to allow access to port 9200 by `scap` - https://phabricator.wikimedia.org/T409339#11347032 (10bd808) 05Open→03In progress a:03c... [21:55:03] maintenance-disconnect-full-disks build 752058 integration-agent-docker-1040 (/: 36%, /srv: 96%, /var/lib/docker: 35%): OFFLINE due to disk space [22:00:03] maintenance-disconnect-full-disks build 752059 integration-agent-docker-1040 (/: 36%, /srv: 23%, /var/lib/docker: 34%): RECOVERY disk space OK [22:01:56] 10Beta-Cluster-Infrastructure, 10Scap, 10Observability-Logging: Setup service name for Beta Cluster access to logstash service in logging project - https://phabricator.wikimedia.org/T409363 (10bd808) 03NEW [22:02:17] 10Beta-Cluster-Infrastructure, 10Scap, 10Observability-Logging: Setup service name for Beta Cluster access to logstash service in logging project - https://phabricator.wikimedia.org/T409363#11347066 (10bd808) [22:02:40] 10Beta-Cluster-Infrastructure, 10Scap, 10Observability-Logging: Setup service name for Beta Cluster access to logstash service in logging project - https://phabricator.wikimedia.org/T409363#11347068 (10bd808) 05Open→03Stalled Blocked on {T409361} [22:08:38] 10MediaWiki-Releasing, 05MW-1.45-release, 06MW-Interfaces-Team (MWI-Sprint-21 (2025-10-21 to 2025-11-04)), 13Patch-For-Review: Update CREDITS in MW master - https://phabricator.wikimedia.org/T408401#11347096 (10Aklapper) >>! In T408401#11345817, @A_smart_kitten wrote: > It does though (at least, IMO) -... [22:10:42] 10Continuous-Integration-Infrastructure (Zuul upgrade), 13Patch-For-Review: Provision Kubernetes cluster and bastion using OpenTofu and Magnum - https://phabricator.wikimedia.org/T396936#11347119 (10bd808) [22:20:14] 10MediaWiki-Releasing, 05MW-1.45-release, 06MW-Interfaces-Team (MWI-Sprint-21 (2025-10-21 to 2025-11-04)), 13Patch-For-Review: Update CREDITS in MW master - https://phabricator.wikimedia.org/T408401#11347163 (10A_smart_kitten) Yeah, but - if I'm being slightly [[https://en.wiktionary.org/wiki/tongue-in... [22:30:08] 10Beta-Cluster-Infrastructure: [betalabs-minor] Delete local overwrite of MediaWiki:File-exists-duplicate on commons.wikimedia.beta.wmflabs.org - https://phabricator.wikimedia.org/T269191#11347177 (10bd808) 05Open→03Resolved p:05Triage→03Medium a:03bd808 https://commons.wikimedia.beta.wmcloud.org/w... [23:10:03] maintenance-disconnect-full-disks build 752073 integration-agent-docker-1041 (/: 37%, /srv: 95%, /var/lib/docker: 34%): OFFLINE due to disk space [23:15:03] maintenance-disconnect-full-disks build 752074 integration-agent-docker-1041 (/: 37%, /srv: 54%, /var/lib/docker: 34%): RECOVERY disk space OK [23:56:51] Project beta-scap-sync-world build #231164: 04FAILURE in 1 min 41 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/231164/