[00:15:34] <_Gerges> Hi [00:17:58] <_Gerges> Task T399139, is it missing something? Can someone start it? [00:17:59] T399139: Run tests for mediawiki/tools/ParseWiki - https://phabricator.wikimedia.org/T399139 [00:26:01] <_Gerges> +voiced [00:58:48] _Gerges: integration/config.git, check the zuul/layout.yaml file. [00:58:58] every repo lists there which jobs to run. [00:59:19] We encourage standard entry points, so you can probably use the same composer-test jobs as other repos. https://www.mediawiki.org/wiki/Continuous_integration/Entry_points [01:02:21] <_Gerges> Will just having a test script in composer.json make ci test work? [01:04:55] <_Gerges> @Krinkle: ^ [01:06:03] If you repo is listed in integration/config.git:/zuul/layout.yaml, and if the job listed there is composer-test, then it will automatically run on any patch uploaded by trusted users. [01:07:29] https://gerrit.wikimedia.org/r/c/mediawiki/tools/code-utils/+/1124506 > composer-php74 > https://gerrit.wikimedia.org/g/integration/config/+/b1d989bdbd2b5b5a36ff58ee86753b59e3db5721/zuul/layout.yaml#2536 [01:07:33] For example [01:07:53] That one line is all it takes. Add it to integration/config if it is not there already. [01:08:09] Then someone will deploy it for you to activate it (one time only). Aftter that it will work forever :) [01:11:03] <_Gerges> Thanks [01:31:31] (03PS1) 10GergesShamon: dockerfiles: add ParseWiki project with composer-php83 template [integration/config] - 10https://gerrit.wikimedia.org/r/1168655 (https://phabricator.wikimedia.org/T399139) [01:33:06] (03PS2) 10GergesShamon: Zuul: add ParseWiki project with composer-php83 template [integration/config] - 10https://gerrit.wikimedia.org/r/1168655 (https://phabricator.wikimedia.org/T399139) [01:35:25] (03CR) 10CI reject: [V:04-1] Zuul: add ParseWiki project with composer-php83 template [integration/config] - 10https://gerrit.wikimedia.org/r/1168655 (https://phabricator.wikimedia.org/T399139) (owner: 10GergesShamon) [03:13:43] (03open) 10bd808: kubernetes: create templates from variable [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/29 [03:36:35] (03update) 10bd808: kubernetes: create templates from variable [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/29 [08:00:23] 10Beta-Cluster-Infrastructure, 10Pywikibot, 07Pywikibot-tests: Unable to generate family for wpbeta:zh with github action (ClientError: (403) Request forbidden) - https://phabricator.wikimedia.org/T399415#10999020 (10Xqt) >>! In T399415#10998648, @bd808 wrote: > I guess my first question is if these tests co... [08:03:20] 10Beta-Cluster-Infrastructure, 10Pywikibot, 07Pywikibot-tests, 07Upstream: CI tests fails with TimeoutError when userinfo is retrieved - https://phabricator.wikimedia.org/T399367#10999027 (10Xqt) My comment to this problem: T399415#10999020 [08:40:23] 10GitLab (Infrastructure), 06collaboration-services: upgrade gitlab hosts to bookworm - https://phabricator.wikimedia.org/T399306#10999160 (10Jelto) p:05Triageβ†’03High a:03Jelto [09:06:50] 10GitLab (Infrastructure), 06collaboration-services: upgrade gitlab hosts to bookworm - https://phabricator.wikimedia.org/T399306#10999234 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1003 for host gitlab2003.wikimedia.org with OS bookworm [09:29:46] 10GitLab (Project Migration), 06cloud-services-team, 13Patch-For-Review, 10Toolforge (Toolforge iteration 21): Migrate misctools package to GitLab - https://phabricator.wikimedia.org/T398202#10999330 (10dcaro) [09:35:05] 10GitLab (Project Migration), 06cloud-services-team, 13Patch-For-Review, 10Toolforge (Toolforge iteration 21): Migrate misctools package to GitLab - https://phabricator.wikimedia.org/T398202#10999387 (10dcaro) a:03dcaro [09:35:09] 10GitLab (Project Migration), 06cloud-services-team, 13Patch-For-Review, 10Toolforge (Toolforge iteration 21): Migrate misctools package to GitLab - https://phabricator.wikimedia.org/T398202#10999390 (10dcaro) 05Openβ†’03Resolved [09:46:16] 10GitLab (Infrastructure), 06collaboration-services: upgrade gitlab hosts to bookworm - https://phabricator.wikimedia.org/T399306#10999476 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1003 for host gitlab2003.wikimedia.org with OS bookworm completed: - gitlab2003 (**PASS**)... [09:48:05] 10GitLab (Infrastructure), 06collaboration-services: upgrade gitlab hosts to bookworm - https://phabricator.wikimedia.org/T399306#10999503 (10Jelto) [12:04:07] 10Gerrit, 07Upstream: Unable to push commit whose parent is not the latest commit for the parent patch - https://phabricator.wikimedia.org/T399241#11000038 (10hashar) From a thread on upstream discord https://discord.com/channels/775374026587373568/1019975766756311130/1394287901197734033: The feature was crea... [12:17:06] 10GitLab (Infrastructure), 06collaboration-services, 13Patch-For-Review: upgrade gitlab hosts to bookworm - https://phabricator.wikimedia.org/T399306#11000072 (10Jelto) [12:17:35] 10GitLab (Infrastructure), 06collaboration-services, 13Patch-For-Review: upgrade gitlab hosts to bookworm - https://phabricator.wikimedia.org/T399306#11000074 (10Jelto) The new test instance `gitlab-1002` is running `bookworm`. I'll do a few more tests but it looks promising so far. [12:29:41] 10Gerrit, 07Upstream: Unable to push commit whose parent is not the latest commit for the parent patch - https://phabricator.wikimedia.org/T399241#11000097 (10Daimona) Thanks for investigating and fixing! >>! In T399241#10998344, @hashar wrote: > @Daimona you were right that it started happening when I have c... [13:21:02] (03CR) 10Jforrester: "This isn't a template. What are you looking to test? PHP 8.3 only? PHP 8.3+ only? PHP 8.1+ including 8.3?" [integration/config] - 10https://gerrit.wikimedia.org/r/1168655 (https://phabricator.wikimedia.org/T399139) (owner: 10GergesShamon) [13:24:42] 10Scap (SpiderPig πŸ•ΈοΈ): SpiderPig live job log view (terminal / console) sometimes freezes - https://phabricator.wikimedia.org/T399462 (10Lucas_Werkmeister_WMDE) 03NEW [13:26:41] 10Scap (SpiderPig πŸ•ΈοΈ): SpiderPig live job log view (terminal / console) sometimes freezes - https://phabricator.wikimedia.org/T399462#11000254 (10Lucas_Werkmeister_WMDE) There’s an error in the console: > populateTerminal unhandled error: {} index-QhE3UUx_.js:5:303 > l https://spiderpig.wikimedia.org/assets/... [13:30:35] 10Scap (SpiderPig πŸ•ΈοΈ): SpiderPig live job log view (terminal / console) sometimes freezes - https://phabricator.wikimedia.org/T399462#11000301 (10Lucas_Werkmeister_WMDE) >>! In T399462#11000254, @Lucas_Werkmeister_WMDE wrote: > I’ll see if it seems to match the time when the console stops updating. Seems to mat... [13:53:46] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Code-Stewardship-Reviews, 07Documentation: Document checklist steps to undeploy / sunset a codebase on WMF servers (not: archiving) - https://phabricator.wikimedia.org/T294329#11000386 (10Jdforrester-WMF) >>! In T294329#10997437, @A_smart_kitten wrote: >>>!... [13:58:28] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [13:58:37] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T399463 (10wmcs-alerts) 03NEW [14:02:13] 06Project-Admins, 07Tracking-Neverending: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706#11000434 (10pfischer) Hi! Could you add me to acl*Project-Admins, please? I am proxying for @Gehel (see T398015) while he's OoO and need permission to create... [14:07:04] hashar: Did you want to meet? [14:07:35] 10Phabricator, 10Release-Engineering-Team (Doing 😎), 13Patch-For-Review: Update to Phorge upstream 2024.35 release - https://phabricator.wikimedia.org/T370266#11000484 (10LSobanski) @jcrespo who offered support from the backup perspective. [14:25:13] (03open) 10lucaswerkmeister-wmde: JobLog.vue: Log error with more details [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/892 (https://phabricator.wikimedia.org/T399462) [14:25:22] (03update) 10lucaswerkmeister-wmde: JobLog.vue: Log error with more details [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/892 (https://phabricator.wikimedia.org/T399462) [14:25:58] 10Scap (SpiderPig πŸ•ΈοΈ), 13Patch-For-Review: SpiderPig live job log view (terminal / console) sometimes freezes - https://phabricator.wikimedia.org/T399462#11000628 (10Lucas_Werkmeister_WMDE) The above MR isn’t going to fix the error, but it should at least provide more information about what’s wrong. [14:51:45] James_F: I note that it ie le 14 juillet (Bastille day) so hash.ar is unlikely to be lurking [14:52:02] thcipriani: Ah, yes, that explains. :-) [14:53:46] * andre seriously considers renaming Phab's Global Search to "Dumb Search" (or at least "Simple Search") [15:01:29] (03PS3) 10Krinkle: Zuul: add ParseWiki project with composer-php83 template [integration/config] - 10https://gerrit.wikimedia.org/r/1168655 (https://phabricator.wikimedia.org/T399139) (owner: 10GergesShamon) [15:03:02] (03CR) 10Krinkle: "I've moved the entry to nearby other mw-tools repos, and changed it to a template. Per James, indeed, if you want only composer-test php 8" [integration/config] - 10https://gerrit.wikimedia.org/r/1168655 (https://phabricator.wikimedia.org/T399139) (owner: 10GergesShamon) [15:14:28] andre: I just burst out laughing at that but it's true [15:15:07] well I think I'll cook up some upstream patches soon when it comes to UX at least [15:16:43] andre: we would also accept "slow and probably not what you were looking for" search as a new label [15:17:52] the width of the search scope dropdown to show menu item names is unfortunately hardcoded [15:18:03] #design [15:18:56] 10Phabricator, 06collaboration-services: Update Phabricator 429 message - https://phabricator.wikimedia.org/T398316#11000900 (10LSobanski) a:03Jelto [15:19:01] 10Phabricator, 06collaboration-services: Update Phabricator 429 message - https://phabricator.wikimedia.org/T398316#11000901 (10LSobanski) 05Openβ†’03Resolved [15:20:44] I can say that my occasional use of the Phabrictor global search go much better after somebody told me to change the scope to "current application" with the little dropdown menu on the search field. [15:20:51] *got [15:28:48] I...have never used the current application search, but it does seem to behave as I'd expect on the home page and the search page (e.g., defaults to the "not the search you're looking for" search) [15:29:06] if only someone knew about the scope dropdown... I wrote a patch in Istanbul to change its appearance: https://phabricator.wikimedia.org/F64381021 but that's probably not the best approach [15:29:23] (03merge) 10dancy: utils.py: Make select_latest_patches treat /srv/patches/next as latest [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/888 (https://phabricator.wikimedia.org/T295925 https://phabricator.wikimedia.org/T398873) [15:29:45] anyway, I got some ideas at least after a day of playing with the UI part only [15:32:05] honestly, that joke isn't too bad. If it was words instead of an icon people would probably think to change it more (e.g., if instead of the 🌎 it said "All the things πŸ”»" I'd guess folks might click it) [15:36:28] (03update) 10dancy: spiderpig: Annotate optional fields in TrainStatus [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/890 (owner: 10dduvall) [15:40:29] 10Scap: apt-get update does not work in docker-registry.wikimedia.org/wikimedia-buster:20250706 image - https://phabricator.wikimedia.org/T399469 (10dancy) 03NEW [15:40:57] !log ssh integration-castor05.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mwgate-node20 # fix failure seen in mwgate-node20 77048 [15:40:58] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:41:29] 10Scap: apt-get update does not work in docker-registry.wikimedia.org/wikimedia-buster:20250706 image - https://phabricator.wikimedia.org/T399469#11001044 (10dancy) p:05Triageβ†’03High [15:47:12] 10Scap: apt-get update does not work in docker-registry.wikimedia.org/wikimedia-buster:20250706 image - https://phabricator.wikimedia.org/T399469#11001082 (10MoritzMuehlenhoff) FWIW, Buster got archived on the Debian servers over the weekend. https://phabricator.wikimedia.org/T397209 deals with the changes neede... [15:54:03] 10Scap, 06serviceops: Drop support for Debian Buster - https://phabricator.wikimedia.org/T392693#11001122 (10dancy) 05Openβ†’03In progress p:05Lowβ†’03High a:03dancy [15:58:52] (03close) 10dduvall: spiderpig: Support MediaWiki train deployments [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/883 (https://phabricator.wikimedia.org/T392610) [16:01:16] (03update) 10dancy: Drop support for Debian Buster [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/893 (https://phabricator.wikimedia.org/T392693 https://phabricator.wikimedia.org/T399469) [16:01:18] (03open) 10dancy: Drop support for Debian Buster [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/893 (https://phabricator.wikimedia.org/T392693 https://phabricator.wikimedia.org/T399469) [16:03:02] (03merge) 10dancy: Drop support for Debian Buster [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/893 (https://phabricator.wikimedia.org/T392693 https://phabricator.wikimedia.org/T399469) [16:03:54] 10Scap, 06serviceops, 13Patch-For-Review: Drop support for Debian Buster - https://phabricator.wikimedia.org/T392693#11001205 (10dancy) 05In progressβ†’03Resolved [16:04:21] 10Scap, 13Patch-For-Review: apt-get update does not work in docker-registry.wikimedia.org/wikimedia-buster:20250706 image - https://phabricator.wikimedia.org/T399469#11001211 (10dancy) 05Openβ†’03Resolved a:03dancy [16:04:41] (03update) 10dancy: spiderpig: Annotate optional fields in TrainStatus [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/890 (owner: 10dduvall) [16:05:46] (03merge) 10dancy: spiderpig: Annotate optional fields in TrainStatus [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/890 (owner: 10dduvall) [16:08:38] (03update) 10dancy: spiderpig: Display errors and warnings in their own row [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/891 (owner: 10dduvall) [16:09:52] (03merge) 10dancy: spiderpig: Display errors and warnings in their own row [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/891 (owner: 10dduvall) [16:10:34] (03update) 10dancy: JobLog.vue: Log error with more details [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/892 (https://phabricator.wikimedia.org/T399462) (owner: 10lucaswerkmeister-wmde) [16:13:10] (03merge) 10dancy: JobLog.vue: Log error with more details [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/892 (https://phabricator.wikimedia.org/T399462) (owner: 10lucaswerkmeister-wmde) [16:13:56] 10Phabricator, 10Release-Engineering-Team (Doing 😎), 13Patch-For-Review: Update to Phorge upstream 2024.35 release - https://phabricator.wikimedia.org/T370266#11001276 (10Aklapper) (Do we still redirect to https://www.mediawiki.org/wiki/Phabricator/Maintenance for that time? I am clueless how "standardized"... [16:20:22] (03open) 10dancy: Release 4.188.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/894 [16:21:31] (03merge) 10dancy: Release 4.188.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/894 [16:24:15] 10Beta-Cluster-Infrastructure, 10Pywikibot, 07Pywikibot-tests: Unable to generate family for wpbeta:zh with github action (ClientError: (403) Request forbidden) - https://phabricator.wikimedia.org/T399415#11001343 (10bd808) >>! In T399415#10999020, @Xqt wrote: >>>! In T399415#10998648, @bd808 wrote: >> I gue... [16:29:41] thcipriani: yeah a name next to the icon would be great but all that CSS has hardcoded widths and it's a PITA to do it properly :( [16:34:20] 10Scap: Can't update scap on phab1005.eqiad.wmnet - https://phabricator.wikimedia.org/T399480 (10dancy) 03NEW [16:41:37] (03open) 10dancy: api.py: Add route for /mediawiki/train [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/895 (https://phabricator.wikimedia.org/T392610) [16:41:40] (03update) 10dancy: api.py: Add route for /mediawiki/train [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/895 (https://phabricator.wikimedia.org/T392610) [16:41:52] (03update) 10dancy: api.py: Add route for /mediawiki/train [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/895 (https://phabricator.wikimedia.org/T392610) [16:42:53] (03merge) 10dancy: api.py: Add route for /mediawiki/train [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/895 (https://phabricator.wikimedia.org/T392610) [16:43:30] (03open) 10dancy: Release 4.188.1 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/896 [16:44:46] (03merge) 10dancy: Release 4.188.1 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/896 [16:54:49] 10Scap: Can't update scap on phab1005.eqiad.wmnet - https://phabricator.wikimedia.org/T399480#11001534 (10Aklapper) cf P78239 / T377889 [17:06:56] 10Scap: Can't update scap on phab1005.eqiad.wmnet - https://phabricator.wikimedia.org/T399480#11001596 (10Dzahn) The scap user is not allowed to move that directory around.. but it's owned by the scap user. Feels like something was done by root previously that should have been done by the scap user? ` sudo -u... [17:13:49] 10Scap: Can't update scap on phab1005.eqiad.wmnet - https://phabricator.wikimedia.org/T399480#11001619 (10Dzahn) Yes, that's it. Look at the difference between phab1005 and phab1004: Under /var/lib/scap: 4.0K drwxr-xr-x 5 root root 4.0K May 28 20:41 scap vs 4.0K drwxr-xr-x 6 scap scap 4.0K Jul 14 16:30 sc... [17:14:41] 10Phabricator, 10Scap, 06collaboration-services: Can't update scap on phab1005.eqiad.wmnet - https://phabricator.wikimedia.org/T399480#11001624 (10Dzahn) [17:14:58] 10Phabricator, 10Scap, 06collaboration-services: Can't update scap on phab1005.eqiad.wmnet - https://phabricator.wikimedia.org/T399480#11001625 (10Dzahn) [17:15:01] 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: install a service on phab1005 - https://phabricator.wikimedia.org/T377889#11001626 (10Dzahn) [17:16:05] mutante: if it's okay, do you mind if i PM you about something? (not urgent) [17:19:42] A_smart_kitten: yea, PM me. no guarantees I have answers, but please go ahead [17:27:28] 10Beta-Cluster-Infrastructure, 10Pywikibot, 07Pywikibot-tests: Find a way for pywikibot GitHub Actions to avoid IP range blocks of Microsoft Azure hosted runners - https://phabricator.wikimedia.org/T399485 (10bd808) 03NEW [17:27:54] 10Beta-Cluster-Infrastructure, 10Pywikibot, 07Pywikibot-tests: Find a way for pywikibot GitHub Actions to avoid IP range blocks of Microsoft Azure hosted runners - https://phabricator.wikimedia.org/T399485#11001704 (10bd808) >>! In T399415#10999020, @Xqt wrote: >>>! In T399415#10998648, @bd808 wrote: >> I gu... [17:28:03] 10Beta-Cluster-Infrastructure, 10Pywikibot, 07Pywikibot-tests: Find a way for pywikibot GitHub Actions to avoid IP range blocks of Microsoft Azure hosted runners - https://phabricator.wikimedia.org/T399485#11001722 (10bd808) >>! In T399415#11001343, @bd808 wrote: >>>! In T399415#10999020, @Xqt wrote: >>>>! I... [17:36:20] 10Beta-Cluster-Infrastructure, 10Pywikibot, 07Pywikibot-tests: Find a way for pywikibot GitHub Actions to avoid IP range blocks of Microsoft Azure hosted runners - https://phabricator.wikimedia.org/T399485#11001784 (10bd808) The proxy idea started with @thcipriani while discussing the self-hosted runner conc... [17:37:41] 10Beta-Cluster-Infrastructure, 10Pywikibot, 07Pywikibot-tests: Unable to generate family for wpbeta:zh with github action (ClientError: (403) Request forbidden) - https://phabricator.wikimedia.org/T399415#11001788 (10bd808) Let's take the conversations about what we might be able to do to {T399485}. I think... [17:37:59] 10Beta-Cluster-Infrastructure, 10Pywikibot, 07Pywikibot-tests: Unable to generate family for wpbeta:zh with github action (ClientError: (403) Request forbidden) - https://phabricator.wikimedia.org/T399415#11001792 (10bd808) [17:38:00] 10Beta-Cluster-Infrastructure, 10Pywikibot, 07Pywikibot-tests: Find a way for pywikibot GitHub Actions to avoid IP range blocks of Microsoft Azure hosted runners - https://phabricator.wikimedia.org/T399485#11001793 (10bd808) [17:38:30] 10Beta-Cluster-Infrastructure, 10Pywikibot, 07Pywikibot-tests: Find a way for pywikibot GitHub Actions to avoid IP range blocks of Microsoft Azure hosted runners - https://phabricator.wikimedia.org/T399485#11001795 (10bd808) [17:38:31] 10Beta-Cluster-Infrastructure, 10Pywikibot, 07Pywikibot-tests, 07Upstream: CI tests fails with TimeoutError when userinfo is retrieved - https://phabricator.wikimedia.org/T399367#11001794 (10bd808) [17:52:31] 10Beta-Cluster-Infrastructure, 10MW-1.45-notes (1.45.0-wmf.9; 2025-07-08), 13Patch-For-Review: Move *.beta.wmflabs.org to *.beta.wmcloud.org - https://phabricator.wikimedia.org/T289318#11001811 (10Ottomata) Hi, [[ https://wikimedia.slack.com/archives/CSV483812/p1752510526368429 | reported in Slack ]]: https... [18:04:23] dancy: I was about to finally come back to your offer to hold my hand deploying a newer Phatality version but realize that c.white already did so in the meantime. Thanks to both of you, I guess :) [18:04:43] 10Release-Engineering-Team (Doing 😎), 10Phatality: Query Maniphest Advanced Search instead of Phabricator Primary Global Search - https://phabricator.wikimedia.org/T398305#11001827 (10Aklapper) 05Openβ†’03Resolved This is deployed. [18:09:15] andre: excellent! [18:10:28] scap on phab1005 also could work now [18:10:41] fixed root-owned scap file [18:41:04] 10Beta-Cluster-Infrastructure, 10MW-1.45-notes (1.45.0-wmf.9; 2025-07-08), 13Patch-For-Review: Move *.beta.wmflabs.org to *.beta.wmcloud.org - https://phabricator.wikimedia.org/T289318#11001900 (10Krinkle) >>! In T289318#11001811, @Ottomata wrote: > Hi, reported in Slack: https://stream.wikimedia.beta.wmclo... [19:28:18] (03CR) 10GergesShamon: "Thanks @krinkle@fastmail.com, Who can review this patch?" [integration/config] - 10https://gerrit.wikimedia.org/r/1168655 (https://phabricator.wikimedia.org/T399139) (owner: 10GergesShamon) [19:36:39] (03CR) 10Dzahn: "looks to me like you already have the right people on here as reviwers" [integration/config] - 10https://gerrit.wikimedia.org/r/1168655 (https://phabricator.wikimedia.org/T399139) (owner: 10GergesShamon) [19:57:46] mutante: Thanks! [19:58:34] mutante: I still get the same error when I re-test. [20:09:00] 10Phabricator, 10Scap, 06collaboration-services: Can't update scap on phab1005.eqiad.wmnet - https://phabricator.wikimedia.org/T399480#11002161 (10dancy) Thanks @Dzahn. Here's a retry: ` dancy@deploy1003:~$ scap install-world --version 4.188.0 --limit phab1005.eqiad.wmnet Scap version "4.188.0" will be inst... [20:17:24] 10Scap (SpiderPig πŸ•ΈοΈ): SpiderPig live job log view (terminal / console) sometimes freezes - https://phabricator.wikimedia.org/T399462#11002181 (10dancy) Chrome: ` index-CyYBfYfR.js:5 populateTerminal unhandled error: TypeError: network error ` Firefox: ` populateTerminal unhandled error: TypeError: Error in inp... [20:23:49] dancy: /var/lib/scap/scap is root:root owned again.. hmm.. looks more like an actual scap bug now. but then.. how is it not an issue on phab1004. the only difference is the distro version [20:26:00] did this same thing happen on other phab servers already? [20:26:15] No. Just phab1005. [20:26:30] just catching up on scrollback; phab1005 state is definitely a result of hackery i (and thcipriani) did to get a working scap state for testing. [20:27:00] regardless of previous hackery though.. I did a "chown scap:scap scap" [20:27:07] and now it's root:root again as before [20:29:03] on phab1004 I see scap is already 4.188.0 and no such issue [20:30:21] the user ID and the sudo rules also look the same to me [20:31:38] maybe error is because /tmp/scap.6wW already exists, owned by scap:scap? [20:31:54] * Install target host (DEFAULT): Wheels for the host distro must be already available in the user's HOME [20:32:45] brennen: well, that I can test [20:33:10] root@phab1005:/var/lib/scap# chown scap:scap scap/ [20:33:10] root@phab1005:/var/lib/scap# sudo -u scap mv /var/lib/scap/scap /tmp/scap.6wW/scap [20:33:23] no error.. did not touch /tmp .. so no [20:34:22] checking if puppet has anything to do with it [20:35:30] yes, it's puppet :/ [20:35:31] Notice: /Stage[main]/Main/File[/var/lib/scap/scap]/ensure: created (corrective) [20:35:41] and: Provider scap3 is not functional on this host [20:36:13] working on a fix [20:37:17] [20:37:47] except I dont even see the reason yet, haha [20:39:44] the code says the owner is supposed to be scap ... hrmmm [20:40:12] lets just use rsync for phab deploys:) [20:44:20] suspects it might be about wmflib::dir::mkdir_p .. will try something in a few [20:45:03] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T399463#11002262 (10bd808) 05Openβ†’03Resolved p:05Triageβ†’03Medium a:03bd808 https://gerrit.wikimedia.org/r... [20:49:03] (03open) 10dancy: JobLog.vue: Trap errors in logRecordsProcessor [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/897 (https://phabricator.wikimedia.org/T399462) [20:49:07] (03update) 10dancy: JobLog.vue: Trap errors in logRecordsProcessor [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/897 (https://phabricator.wikimedia.org/T399462) [20:50:09] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T399463#11002295 (10bd808) Attempting to test on deployment-cache-text08 now shows: `COUNTEREXAMPLE Error: Could no... [20:50:19] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T399463#11002296 (10bd808) 05Resolvedβ†’03In progress [20:51:31] (03merge) 10dancy: JobLog.vue: Trap errors in logRecordsProcessor [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/897 (https://phabricator.wikimedia.org/T399462) [20:52:17] (03open) 10dancy: Release 4.188.2 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/898 [20:53:23] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T399463#11002303 (10bd808) >>! In T399463#11002295, @bd808 wrote: > Attempting to test on deployment-cache-text08 now s... [20:53:28] (03merge) 10dancy: Release 4.188.2 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/898 [20:53:28] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [20:53:34] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T399501 (10wmcs-alerts) 03NEW [20:54:06] bd808: looks like https://gerrit.wikimedia.org/r/c/operations/puppet/+/1167695 ? [20:54:43] mutante: yup. working on it. :) [20:54:53] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T399463#11002335 (10Dzahn) https://gerrit.wikimedia.org/r/c/operations/puppet/+/1167695 looks like it it removed the... [20:55:07] bd808: probably in project Hiera.. eh [20:55:31] re: phab deploys - we talked about this in releng team meeting the other day, it feels like just running it as a container under systemd might be comparatively straightforward. [20:55:50] planning to think about that once tomorrow's upgrade is out of the way. [20:56:50] (totally open to other simpler approaches too, but this definitely feels like a thing where we should stop fighting scap + puppet.) [20:57:36] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T399463#11002353 (10bd808) [20:57:38] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T399501#11002355 (10bd808) β†’14Duplicate dup:03T399463 [21:00:13] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T399463#11002359 (10bd808) https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/cd03967b5c4c4e9228d10... [21:00:27] 10Scap (SpiderPig πŸ•ΈοΈ), 13Patch-For-Review: SpiderPig live job log view (terminal / console) sometimes freezes - https://phabricator.wikimedia.org/T399462#11002360 (10dancy) 05Openβ†’03Resolved a:03dancy Fix deployed [21:02:00] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T399463#11002364 (10bd808) `lang=shell-session,lines=10 bd808@deployment-cache-text08.deployment-prep.eqiad1:~$ sudo -i... [21:06:23] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T399463#11002379 (10bd808) 05In progressβ†’03Resolved [21:11:25] 10Phabricator, 10Scap, 06collaboration-services, 13Patch-For-Review: Can't update scap on phab1005.eqiad.wmnet - https://phabricator.wikimedia.org/T399480#11002427 (10Dzahn) Puppet was handling `/var/lib/scap/scap/bin` and ensured it was scap-owned, but it did not handle `/var/lib/scap/scap`, which meant i... [21:13:49] dancy: brennen: fixed! (maybe this was the last thing we had to fix :) [21:13:54] Notice: /Stage[main]/Main/File[/var/lib/scap/scap]/owner: owner changed 'root' to 'scap' [21:14:24] fingers crossed. :) [21:14:55] :) [21:15:01] 10Phabricator, 10Scap, 06collaboration-services, 13Patch-For-Review: Can't update scap on phab1005.eqiad.wmnet - https://phabricator.wikimedia.org/T399480#11002432 (10Dzahn) @dancy @brennen It should work now for real. puppet: ` Notice: /Stage[main]/Main/File[/var/lib/scap/scap]/owner: owner changed 'ro... [21:16:06] !log Unblocked 94.31.116.0/22 (T399497) [21:16:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:16:16] so if you use wmflib_mkdir_p to say "make sure that /var/lib/scap/scap/bin is owned by scap" but don't mention /var/lib/scap/scap ... that is still root-owned [21:17:13] and it's not an issue on phab1004 because phab1005 is still in the "migration" role .. for migrating phab to new hosts [21:19:49] 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Beta cluster IP block page should not point to noc@wikimedia.org - https://phabricator.wikimedia.org/T393404#11002448 (10bd808) Cherry-pick caused {T399463}. The patch has been updated so that it applies cleanly again. [21:37:30] !log Unblock 113.22.48.0/24 (T399461) [21:37:31] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:48:08] 10GitLab (Administration, Settings & Policy), 06MediaWiki-Gerrit-Group-Requests: Remove +2 rights from SD0001 for SecurePoll - https://phabricator.wikimedia.org/T399508 (10Snaevar) 03NEW [21:50:27] 10Beta-Cluster-Infrastructure, 10MW-1.45-notes (1.45.0-wmf.9; 2025-07-08), 13Patch-For-Review: Move *.beta.wmflabs.org to *.beta.wmcloud.org - https://phabricator.wikimedia.org/T289318#11002542 (10bd808) >>! In T289318#11001900, @Krinkle wrote: > You can edit the CDN configuration for the EventStreams domain... [21:55:00] (03update) 10brennen: Merge upstream 2024.35 into wmf/stable [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/91 (https://phabricator.wikimedia.org/T370266) [21:55:42] !log Forced puppet run and `systemctl reload varnish-frontend` on deployment-cache-text08 for stream.wikimedia.beta.wmcloud.org (T289318) [21:55:44] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:55:44] T289318: Move *.beta.wmflabs.org to *.beta.wmcloud.org - https://phabricator.wikimedia.org/T289318 [21:59:59] (03merge) 10bd808: kubernetes: create templates from variable [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/29 [22:02:21] (03update) 10brennen: update submodules for upstream 2024.35 merge [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/69 (https://phabricator.wikimedia.org/T370266) [22:27:17] 06Release-Engineering-Team, 06Abstract Wikipedia team, 10function-evaluator, 10function-orchestrator, 13Patch-For-Review: gitlab CI mem issue - https://phabricator.wikimedia.org/T399348#11002621 (10ecarg) [22:31:04] 10GitLab (Administration, Settings & Policy), 06MediaWiki-Gerrit-Group-Requests: Remove +2 rights from SD0001 for SecurePoll - https://phabricator.wikimedia.org/T399508#11002637 (10Zabe) > Violation 1: > Creation of a patch in T396441, while the underlying discussion at https://en.wikipedia.org/wiki/Wikipedia:... [22:32:54] 06Release-Engineering-Team, 06Abstract Wikipedia team, 10function-evaluator, 10function-orchestrator, 13Patch-For-Review: gitlab CI mem issue - https://phabricator.wikimedia.org/T399348#11002643 (10ecarg) Hello! We're running into some OOM stuff for the Evaluator in gitlab CI: so I created a [[ https://... [22:33:01] 06Release-Engineering-Team, 06Abstract Wikipedia team, 10function-evaluator, 10function-orchestrator, 13Patch-For-Review: gitlab CI mem issue - https://phabricator.wikimedia.org/T399348#11002644 (10ecarg) p:05Triageβ†’03High [23:01:45] (03open) 10dduvall: spiderpig: Better support days leading up to train [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/899 [23:02:20] (03update) 10dduvall: spiderpig: Better support days leading up to train [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/899 [23:02:22] (03update) 10dduvall: spiderpig: Better support days leading up to train [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/899 [23:11:21] (03update) 10dduvall: scap: Add date to fake train blocker JSON [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/148 [23:11:27] (03open) 10dduvall: scap: Add date to fake train blocker JSON [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/148 [23:11:28] (03update) 10dduvall: scap: Add date to fake train blocker JSON [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/148 [23:11:28] (03update) 10dduvall: scap: Add date to fake train blocker JSON [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/148