[00:04:16] (03open) 10dduvall: cli: Support log level via `--log-level` and `KOKKURI_LOG_LEVEL` [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/131 [00:09:02] (03update) 10dduvall: cli: Support log level via `--log-level` and `KOKKURI_LOG_LEVEL` [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/131 [00:09:04] (03update) 10dduvall: cli: Support log level via `--log-level` and `KOKKURI_LOG_LEVEL` [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/131 [00:15:33] (03PS1) 10Hslater: Use bluespice template for ImageMapEdit [integration/config] - 10https://gerrit.wikimedia.org/r/1160340 [00:16:44] (03open) 10bd808: tofu: add a bastion and a web proxy [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/7 (https://phabricator.wikimedia.org/T396936) [00:18:59] (03merge) 10bd808: tofu: add a bastion and a web proxy [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/7 (https://phabricator.wikimedia.org/T396936) [00:22:08] (03open) 10dduvall: image: Sanitize export variable names [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/132 [00:22:13] (03update) 10dduvall: image: Sanitize export variable names [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/132 [00:22:14] (03update) 10dduvall: image: Sanitize export variable names [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/132 [00:23:58] 10Continuous-Integration-Infrastructure (Zuul upgrade), 13Patch-For-Review: Provision Kubernetes cluster and bastion using OpenTofu and Magnum - https://phabricator.wikimedia.org/T396936#10926187 (10bd808) The project now has: * 1 Kubernetes control plane instance * 1 Kubernetes worker instance * 1 web proxy e... [00:23:58] (03update) 10dduvall: image: Sanitize export variable names [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/132 [00:23:59] (03update) 10dduvall: image: Sanitize export variable names [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/132 [00:24:05] (03update) 10dduvall: image: Sanitize export variable names [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/132 [00:51:48] !!log Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/80 (T372005) [00:51:48] T372005: Create WMF standalone development build - https://phabricator.wikimedia.org/T372005 [00:55:55] (03approved) 10dancy: image: Sanitize export variable names [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/132 (owner: 10dduvall) [00:56:28] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance deployment-cache-upload08 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [00:56:37] 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-cache-upload08 in project deployment-prep - https://phabricator.wikimedia.org/T397263#10926208 (10wmcs-alerts) [00:57:01] (03approved) 10dancy: cli: Support log level via `--log-level` and `KOKKURI_LOG_LEVEL` [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/131 (owner: 10dduvall) [01:41:28] RESOLVED: PuppetAgentFailure: Puppet agent failure detected on instance deployment-cache-upload08 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [01:41:39] 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-cache-upload08 in project deployment-prep - https://phabricator.wikimedia.org/T397263#10926229 (10wmcs-alerts) [04:10:20] !log Change shortener_domain in deployment-cache-text prefix from `w-beta.wmflabs.org` to `w.beta.wmcloud.org`, to apply VCL normalization for w.wiki in Beta, ref T289318, T396012 [04:10:23] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [04:10:24] T289318: Move *.beta.wmflabs.org to *.beta.wmcloud.org - https://phabricator.wikimedia.org/T289318 [04:10:24] T396012: Beta cluster URL shortener (w-beta.wmflabs.org) broken - https://phabricator.wikimedia.org/T396012 [04:13:54] !log Fix profile::trafficserver::backend::mapping_rules in deployment-cache-upload to include `rb-mw-mangling-beta.lua` as otherwise w.beta.wmcloud.org serves 404 Domain Not Configured, ref T289318, T396012 [04:13:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [04:13:58] !log Fix profile::trafficserver::backend::mapping_rules in deployment-cache-text to include `rb-mw-mangling-beta.lua` as otherwise w.beta.wmcloud.org serves 404 Domain Not Configured, ref T289318, T396012 [04:14:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [04:25:14] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-UrlShortener, 13Patch-For-Review: Beta cluster URL shortener (w-beta.wmflabs.org) broken - https://phabricator.wikimedia.org/T396012#10926452 (10Krinkle) >>! In T396012#10887761, @gerritbot wrote: > Change #1153632 **merged** by Ladsgroup: > %%%[opera... [07:27:47] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06cloud-services-team, 10Cloud-VPS (Quota-requests): Increase volume storage quota for zuul project - https://phabricator.wikimedia.org/T397098#10926707 (10dcaro) Oh, I got confused by the tofu patches, lately I'm not sure if it's tofu or cookbooks... [07:58:00] 06Project-Admins, 06Release-Engineering-Team: Consider archiving #Data³ - https://phabricator.wikimedia.org/T396891#10926841 (10A_smart_kitten) Thanks @thcipriani :D [10:08:28] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-UrlShortener, 13Patch-For-Review: Beta cluster URL shortener (w-beta.wmflabs.org) broken - https://phabricator.wikimedia.org/T396012#10927245 (10Ladsgroup) I personally to keep both in case the first layer breaks or removed accidentally. But no strong... [10:13:28] 10Phabricator (Upstream), 10Release-Engineering-Team (Doing 😎), 07Upstream: Query for global Feed Transaction Logs: Exception: ConpherenceTransactionQuery overheated - https://phabricator.wikimedia.org/T344232#10927288 (10Aklapper) 05Open→03Stalled a:03Aklapper [10:13:35] 10Phabricator (Upstream), 10Release-Engineering-Team (Doing 😎), 07Upstream: Query for global Feed Transaction Logs: Exception: ConpherenceTransactionQuery overheated - https://phabricator.wikimedia.org/T344232#10927292 (10Aklapper) [10:13:38] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥): Update to Phorge upstream 202x.xx release (the one after 2025.18) - https://phabricator.wikimedia.org/T393840#10927293 (10Aklapper) [11:41:27] 10Gerrit, 06Release-Engineering-Team: Push to Gerrit refs/for/ is slow - https://phabricator.wikimedia.org/T397317 (10hashar) 03NEW [11:55:01] 10Phabricator: "Other assignee" icon disappears briefly after loading a workboard - https://phabricator.wikimedia.org/T396878#10927810 (10Aklapper) I can't make much out of that, sorry. :-/ The JS redraw code is in https://phabricator.wikimedia.org/source/phabricator/browse/wmf%252Fstable/webroot/rsrc/js/applica... [12:24:07] (03open) 10jnuche: Release 4.178.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/843 [12:24:35] 10Phabricator: "Other assignee" icon disappears briefly after loading a workboard - https://phabricator.wikimedia.org/T396878#10927919 (10Daimona) I'm not familiar with the phab internals, but from a quick read at the code, it's redrawing cards based on the response of an API request. So I reopened the board to... [12:26:16] (03merge) 10jnuche: Release 4.178.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/843 [12:52:34] (03open) 10jnuche: Release 4.178.1 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/844 [13:00:24] (03update) 10jnuche: Release 4.178.1 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/844 [13:02:30] (03merge) 10jnuche: Release 4.178.1 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/844 [13:03:10] (03PS2) 10Jforrester: Zuul: [mediawiki/extensions/ImageMapEdit] Use bluespice template [integration/config] - 10https://gerrit.wikimedia.org/r/1160340 (owner: 10Hslater) [13:05:33] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/ImageMapEdit] Use bluespice template [integration/config] - 10https://gerrit.wikimedia.org/r/1160340 (owner: 10Hslater) [13:06:45] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/ImageMapEdit] Use bluespice template [integration/config] - 10https://gerrit.wikimedia.org/r/1160340 (owner: 10Hslater) [13:06:57] !log Zuul: [mediawiki/extensions/ImageMapEdit] Use bluespice template [13:06:58] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:07:14] (03CR) 10Jforrester: [C:03+2] Zuul: add James E. Blair to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1159933 (owner: 10Hashar) [13:08:01] (03CR) 10Jforrester: [C:03+2] zuul: Add EggRoll97 to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1159506 (owner: 10Dreamrimmer) [13:08:24] (03Merged) 10jenkins-bot: Zuul: add James E. Blair to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1159933 (owner: 10Hashar) [13:08:29] !log Zuul: Add James E. Blair to CI allowlist [13:08:30] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:09:38] (03Merged) 10jenkins-bot: zuul: Add EggRoll97 to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1159506 (owner: 10Dreamrimmer) [13:10:29] !log Zuul: Add EggRoll97 to CI allowlist [13:10:30] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:25:26] (03open) 10jnuche: Release 4.178.2 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/845 [13:27:32] (03merge) 10jnuche: Release 4.178.2 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/845 [14:04:23] (03open) 10jnuche: Release 4.178.3 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/846 [14:05:37] (03merge) 10jnuche: Release 4.178.3 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/846 [15:11:06] 10Scap, 13Patch-For-Review: Are `php_fpm`/`php_version` inside `scap.cfg` used anymore? - https://phabricator.wikimedia.org/T396166#10928651 (10dancy) The second test used `scap sync-world --force`. [15:11:22] 10Scap, 13Patch-For-Review: Are `php_fpm`/`php_version` inside `scap.cfg` used anymore? - https://phabricator.wikimedia.org/T396166#10928652 (10dancy) 05Open→03In progress p:05Triage→03Low a:03dancy [15:12:08] (03merge) 10dduvall: cli: Support log level via `--log-level` and `KOKKURI_LOG_LEVEL` [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/131 [15:12:20] (03update) 10dduvall: image: Sanitize export variable names [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/132 [15:18:19] (03open) 10dduvall: caches: Fix formatting of `caches` examples in schema [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/133 [15:18:33] (03update) 10dduvall: caches: Fix formatting of `caches` examples in schema [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/133 [15:20:42] (03merge) 10dduvall: image: Sanitize export variable names [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/132 [15:24:33] (03merge) 10dduvall: caches: Fix formatting of `caches` examples in schema [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/133 [15:31:21] (03update) 10dduvall: wmf: Optimize .pipeline/blubber.yaml [repos/releng/zuul/zuul] (wmf/12.0.0) - 10https://gitlab.wikimedia.org/repos/releng/zuul/zuul/-/merge_requests/12 (https://phabricator.wikimedia.org/T396245) [15:34:16] (03open) 10dduvall: version: v2.6.0 [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/133 [15:38:00] (03merge) 10dduvall: version: v2.6.0 [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/133 [15:41:21] 10Gerrit, 10Release-Engineering-Team (Radar), 06collaboration-services, 06Traffic, 13Patch-For-Review: Separate Gerrit https and ssh/git hostnames - https://phabricator.wikimedia.org/T394271#10928782 (10Jelto) 05Open→03Stalled > Discuss with RelEng if a change of hostname is reasonable (for voluntee... [15:41:38] 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-cache-upload08 in project deployment-prep - https://phabricator.wikimedia.org/T397263#10928791 (10bd808) 05Open→03Invalid Fixed upstream apparently. ` $ sudo -i puppet agent -tv Info: Using environment 'production' Info:... [15:44:03] (03update) 10dduvall: wmf: Optimize .pipeline/blubber.yaml [repos/releng/zuul/zuul] (wmf/12.0.0) - 10https://gitlab.wikimedia.org/repos/releng/zuul/zuul/-/merge_requests/12 (https://phabricator.wikimedia.org/T396245) [15:57:54] (03merge) 10dduvall: wmf: Optimize .pipeline/blubber.yaml [repos/releng/zuul/zuul] (wmf/12.0.0) - 10https://gitlab.wikimedia.org/repos/releng/zuul/zuul/-/merge_requests/12 (https://phabricator.wikimedia.org/T396245) [16:19:35] 10Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services: Upgrade phab (phorge) hosts to bookworm - https://phabricator.wikimedia.org/T372619#10928910 (10Dzahn) In T377889 we now managed to deploy phabricator with scap now..and installed PHP 8. This let's us confirm if Phabricator works on bo... [16:23:27] (03CR) 10Hashar: [C:03+2] "Sorry I had back to back meeting for the last two days :] Lets doooo it!" [integration/config] - 10https://gerrit.wikimedia.org/r/1159377 (https://phabricator.wikimedia.org/T393865) (owner: 10Bartosz Wójtowicz) [16:24:38] (03Merged) 10jenkins-bot: inference-services: Remove unnecessary CI jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1159377 (https://phabricator.wikimedia.org/T393865) (owner: 10Bartosz Wójtowicz) [16:24:57] 10GitLab (CI & Job Runners): kokkuri doesn't seem to work from WMCS runners - https://phabricator.wikimedia.org/T396924#10928939 (10dancy) Only trusted runners are allowed to publish to the production docker registry. The `wmcs` runners are not trusted. [16:25:55] (03CR) 10Hashar: [C:03+2] "Deployed!" [integration/config] - 10https://gerrit.wikimedia.org/r/1159377 (https://phabricator.wikimedia.org/T393865) (owner: 10Bartosz Wójtowicz) [16:26:18] 10GitLab (CI & Job Runners): kokkuri doesn't seem to work from WMCS runners - https://phabricator.wikimedia.org/T396924#10928942 (10dancy) To clarify, the gitlab-cloud-runners do have a registry available for hosting temporary images. There is no corresponding thing in wmcs. [16:41:42] 10GitLab (CI & Job Runners): kokkuri cannot publish "public" images from WMCS runners due to a lack of a local registry - https://phabricator.wikimedia.org/T396924#10929014 (10bd808) [16:41:59] dduvall: I have asked GPT-4o mini to "build and publish a container image for project mediawiki/core" [16:41:59] >>> https://gist.github.com/hashar/7463cf7e3a993736b2d3f55f351f29bd [16:42:49] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services: puppetize setup of new zuul VMs - https://phabricator.wikimedia.org/T395938#10929023 (10Corvus) Recommendations for setting up containers: All zuul components (excluding nodepool): * Bind mount a directory with zuul.conf and ten... [16:43:03] it even seems to more or less support Blubber syntax somehow [16:43:15] lol PHP 8 [16:43:20] hashar: i have never felt less threatened by an llm than i do atm :) [16:51:33] wait til our quaterly goals are to make them more accurate [17:08:15] 10Phabricator, 06Security-Team, 07SecTeam-Processed: Audit Phabricator security policies and groups membership - https://phabricator.wikimedia.org/T391150#10929085 (10sbassett) [17:09:35] 10Phabricator, 06Security-Team, 07SecTeam-Processed: Audit Phabricator security policies and groups membership - https://phabricator.wikimedia.org/T391150#10929088 (10sbassett) [17:09:37] 10Phabricator, 06Security-Team, 07SecTeam-Processed, 07Security: Audit members of acl*security for more than 12 months of no activity (May 2025) - https://phabricator.wikimedia.org/T368224#10929089 (10sbassett) [17:22:35] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Cloud-Services, 07ContentSecurityPolicy: Object storage web service CSP does not allow inline images - https://phabricator.wikimedia.org/T397351 (10hashar) 03NEW The #Cloud-Services project tag is not intended to have any tasks. Please check the lis... [17:24:55] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06cloud-services-team, 10Data-Services, 07ContentSecurityPolicy: Object storage web service CSP does not allow inline images - https://phabricator.wikimedia.org/T397351#10929174 (10hashar) [17:30:59] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06cloud-services-team, 10Cloud-VPS, 07ContentSecurityPolicy: Object storage web service CSP does not allow inline images - https://phabricator.wikimedia.org/T397351#10929198 (10taavi) a:03taavi [17:58:03] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06cloud-services-team, 10Cloud-VPS, 07ContentSecurityPolicy, 13Patch-For-Review: Object storage web service CSP does not allow inline images - https://phabricator.wikimedia.org/T397351#10929296 (10taavi) 05Open→03Resolved a:05taavi→03hash... [17:58:04] 10WikimediaDebug, 10Arc-Lamp, 10observability, 07Regression: https://performance.wikimedia.org/php-profiling/ leads to 404 for all listed sources - https://phabricator.wikimedia.org/T391516#10929299 (10Krinkle) 05Resolved→03Open >>! In T391516#10760915, @gerritbot wrote: > Change #1118209 **merged** by... [18:07:01] (03open) 10bd808: Draft: Provision files on bastion via ssh publisher [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/8 [18:09:46] (03update) 10bd808: Draft: Provision files on bastion via ssh publisher [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/8 [18:19:18] !log `docker system prune --all` on runner-1023.gitlab-runners.eqiad1.wikimedia.cloud [18:19:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:27:00] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06cloud-services-team, 10Cloud-VPS, 07ContentSecurityPolicy, 13Patch-For-Review: Object storage web service CSP does not allow inline images - https://phabricator.wikimedia.org/T397351#10929381 (10hashar) That is wonderful thank you. [18:28:16] (03update) 10swfrench: make-container-image: introduce webserver-bookworm flavour [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/184 (https://phabricator.wikimedia.org/T378128) [18:41:28] FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [18:41:40] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-sessionstore06 is down - https://phabricator.wikimedia.org/T397360 (10wmcs-alerts) 03NEW [18:46:28] RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [18:46:35] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-sessionstore06 is down - https://phabricator.wikimedia.org/T397360#10929419 (10wmcs-alerts) [19:31:21] (03open) 10jforrester: [WIP] Provide initial trixie images [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/81 [19:31:23] (03update) 10jforrester: Draft: [WIP] Provide initial trixie images [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/81 [19:45:10] (03open) 10dancy: spiderpig: Debugging setup for T397097 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/847 (https://phabricator.wikimedia.org/T397097) [19:45:12] (03update) 10dancy: spiderpig: Debugging setup for T397097 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/847 (https://phabricator.wikimedia.org/T397097) [19:47:01] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-sessionstore06 is down - https://phabricator.wikimedia.org/T397360#10929578 (10bd808) 05Open→03Invalid `lang=shell-session bd808@deployment-sessionstore06:~$ hostname -f deployment-sessionstore06.deployment-prep.eqiad1.wikimedia.c... [19:47:18] (03merge) 10dancy: spiderpig: Debugging setup for T397097 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/847 (https://phabricator.wikimedia.org/T397097) [19:47:53] (03open) 10dancy: Release 4.179.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/848 [19:50:10] (03merge) 10dancy: Release 4.179.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/848 [19:59:53] (03open) 10dancy: spiderpig: Debugging setup for T397097 (followup) [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/849 (https://phabricator.wikimedia.org/T397097) [19:59:55] (03update) 10dancy: spiderpig: Debugging setup for T397097 (followup) [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/849 (https://phabricator.wikimedia.org/T397097) [20:00:19] (03update) 10dancy: spiderpig: Debugging setup for T397097 (followup) [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/849 (https://phabricator.wikimedia.org/T397097) [20:02:10] (03merge) 10dancy: spiderpig: Debugging setup for T397097 (followup) [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/849 (https://phabricator.wikimedia.org/T397097) [20:03:04] (03open) 10dancy: Release 4.179.1 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/850 [20:04:29] (03merge) 10dancy: Release 4.179.1 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/850 [20:10:33] 10Continuous-Integration-Infrastructure (Zuul upgrade), 07ci-test-error: MediaWiki\Tests\Maintenance\WikiBirthdayTest test failures in new Zuul - https://phabricator.wikimedia.org/T397368 (10hashar) 03NEW [20:23:36] well [20:23:44] I have no idea what it is [20:23:45] :/ [20:24:51] 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: install a service on phab1005 - https://phabricator.wikimedia.org/T377889#10929709 (10Dzahn) [20:29:21] 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: install a service on phab1005 - https://phabricator.wikimedia.org/T377889#10929728 (10Dzahn) A scap phab deployment has been working now! :) I re-enabled puppet after we had applied some manual hacks. (like commenting out apache rest... [20:30:59] 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: install a service on phab1005 - https://phabricator.wikimedia.org/T377889#10929733 (10Dzahn) Also: PHP8.2 is installed.. and if this works then Phorge is just fine with bookworm default version. [20:39:47] Is beta down? [20:40:44] Okay maybe not down but, like, REALLY slow [20:48:33] (03update) 10dancy: Revert "spiderpig: Debugging setup for T397097" [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/851 (https://phabricator.wikimedia.org/T397097) [20:48:34] (03update) 10dancy: spiderpig/api.py: Include Content-Type header in job log response (v2) [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/852 (https://phabricator.wikimedia.org/T397097) [20:48:34] (03open) 10dancy: Revert "spiderpig: Debugging setup for T397097" [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/851 (https://phabricator.wikimedia.org/T397097) [20:48:34] (03open) 10dancy: spiderpig/api.py: Include Content-Type header in job log response (v2) [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/852 (https://phabricator.wikimedia.org/T397097) [20:48:42] (03update) 10dancy: Revert "spiderpig: Debugging setup for T397097" [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/851 (https://phabricator.wikimedia.org/T397097) [20:48:50] (03update) 10dancy: spiderpig/api.py: Include Content-Type header in job log response (v2) [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/852 (https://phabricator.wikimedia.org/T397097) [20:50:18] (03update) 10dancy: spiderpig/api.py: Include Content-Type header in job log response (v2) [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/852 (https://phabricator.wikimedia.org/T397097) [20:50:22] (03update) 10dancy: spiderpig/api.py: Include Content-Type header in job log response (v2) [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/852 (https://phabricator.wikimedia.org/T397097) [20:50:48] (03merge) 10dancy: Revert "spiderpig: Debugging setup for T397097" [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/851 (https://phabricator.wikimedia.org/T397097) [20:50:50] (03update) 10dancy: spiderpig/api.py: Include Content-Type header in job log response (v2) [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/852 (https://phabricator.wikimedia.org/T397097) [20:51:58] (03update) 10dancy: spiderpig/api.py: Include Content-Type header in job log response (v2) [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/852 (https://phabricator.wikimedia.org/T397097) [20:52:03] (03update) 10dancy: spiderpig/api.py: Include Content-Type header in job log response (v2) [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/852 (https://phabricator.wikimedia.org/T397097) [20:53:00] (03merge) 10dancy: spiderpig/api.py: Include Content-Type header in job log response (v2) [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/852 (https://phabricator.wikimedia.org/T397097) [20:53:35] (03update) 10dancy: Release 4.180.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/853 [20:53:37] (03open) 10dancy: Release 4.180.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/853 [20:55:41] (03merge) 10dancy: Release 4.180.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/853 [21:08:47] 10Scap (SpiderPig 🕸️), 13Patch-For-Review: SpiderPig live job log view is broken - https://phabricator.wikimedia.org/T397097#10929892 (10dancy) 05Open→03Resolved a:03dancy Using a content type of `application/octet-stream` was the workaround. [21:18:56] Daimona: yeah, it's odd. when I use curl locally on mediawiki13 or 14, it's quite snappy. [21:19:04] so php/mysql is the fast part :P [21:19:27] Huh [21:24:39] 200 0.314938krinkle@deployment-mediawiki13:~$ time curl https://en.wikinews.beta.wmflabs.org/wiki/Main_Page --connect-to ::deployment-mediawiki13 -s -o /dev/null -w '%{response_code} %{time_total}s' [21:24:39] curl: 200 0.244036s [21:24:39] time: real 0m0.263s user 0m0.021s sys 0m0.017s [21:25:36] The same externally, once warmed up, `real 0m0.810s` [21:25:46] not too bad now. it seems to come and go. [21:26:49] https://grafana.wmcloud.org/d/0g9N-7pVz/cloud-vps-project-board?var-project=deployment-prep&orgId=1&from=now-24h&to=now&timezone=browser&var-instance=deployment-cache-text08&var-instance=deployment-db11&var-instance=deployment-db14&var-instance=deployment-mediawiki13&var-instance=deployment-mediawiki14 [21:27:01] mediawiki14 is pretty hammered. I guess that makes sense. that's the main instance publicly [21:27:03] I was curling on the idle host [21:27:19] still it ocasionally is fast. [21:27:53] It's AI scrapers time? [21:40:29] 10Beta-Cluster-Infrastructure, 07Epic, 13Patch-For-Review: 2025 tracking task for Beta Cluster (deployment-prep) traffic overload protection (blocking unwanted crawlers) - https://phabricator.wikimedia.org/T393487#10930017 (10bd808) [22:02:18] !log Blocked 146.174.160.0/19 "Huawei-Cloud-SG" network. (T397378) [22:02:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:03:08] 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: install a service on phab1005 - https://phabricator.wikimedia.org/T377889#10930093 (10Dzahn) >>! In T377889#10929728, @Dzahn wrote: > The symlink is now puppetized; apache can be restarted, puppet can be enabled and disabled; just one... [22:10:17] !log Blocked 202.76.160.0/20 "Huawei-Cloud-SG" network. (T397378) [22:10:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:19:20] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services, 06DBA, 13Patch-For-Review: Prepare a database test for m3 - https://phabricator.wikimedia.org/T390034#10930115 (10brennen) Ok, this is finally in a state where we can test the migration, but I get: ` Applying patch "phabricator:202... [22:25:58] Krinkle: I think things should be a bit better now. Load on mw14 is trending down with those networks blocked. [22:35:38] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services, 06DBA: Prepare a database test for m3 - https://phabricator.wikimedia.org/T390034#10930145 (10brennen) 05Resolved→03Open @Marostegui could we get this in a read/write state? We need to be able to delete a bunch of rows (and I think p... [22:38:03] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services, 06DBA: Prepare a database test for m3 - https://phabricator.wikimedia.org/T390034#10930153 (10Marostegui) Done [22:41:26] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services, 06DBA: Prepare a database test for m3 - https://phabricator.wikimedia.org/T390034#10930156 (10brennen) Thanks! Running test migration now. [22:51:26] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 10Data-Persistence-Backup: Backups for new m1 database: zuul - https://phabricator.wikimedia.org/T396322#10930167 (10Dzahn) @jcrespo We are fine with whatever the default is for other m1 databases. We don't need anything specia... [22:53:30] (03open) 10dduvall: wmf: Build nodepool-launcher from upstream repo [repos/releng/zuul/zuul] (wmf/12.0.0) - 10https://gitlab.wikimedia.org/repos/releng/zuul/zuul/-/merge_requests/13 (https://phabricator.wikimedia.org/T396245) [23:12:08] (03merge) 10dduvall: wmf: Build nodepool-launcher from upstream repo [repos/releng/zuul/zuul] (wmf/12.0.0) - 10https://gitlab.wikimedia.org/repos/releng/zuul/zuul/-/merge_requests/13 (https://phabricator.wikimedia.org/T396245) [23:22:19] (03update) 10dduvall: image: Simplify remote context handling for bake [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/134 [23:22:24] (03update) 10dduvall: image: Simplify remote context handling for bake [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/134 [23:22:24] (03update) 10dduvall: image: Simplify remote context handling for bake [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/134 [23:22:30] (03open) 10dduvall: image: Simplify remote context handling for bake [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/134 [23:26:38] !log Blocked 128.241.0.0/16 "NTT America" network. (T397378) [23:26:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:36:13] (03update) 10dduvall: image: Simplify remote context handling for bake [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/134 [23:36:13] (03update) 10dduvall: image: Simplify remote context handling for bake [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/134 [23:36:16] (03update) 10dduvall: image: Simplify remote context handling for bake [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/134 [23:44:19] (03update) 10bd808: Draft: Provision files on bastion via ssh publisher [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/8 (https://phabricator.wikimedia.org/T396936) [23:44:50] (03update) 10bd808: Provision files on bastion via ssh publisher [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/8 (https://phabricator.wikimedia.org/T396936) [23:46:36] (03update) 10bd808: Provision files on bastion via ssh publisher [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/8 (https://phabricator.wikimedia.org/T396936) [23:47:46] (03merge) 10bd808: Provision files on bastion via ssh publisher [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/8 (https://phabricator.wikimedia.org/T396936) [23:59:19] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services: puppetize setup of new zuul VMs - https://phabricator.wikimedia.org/T395938#10930251 (10Dzahn) I created a new SSH key pair and put it into the private puppet repository under `modules/secret/secrets/zuul`. This is the "global s... [23:59:40] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services: puppetize setup of new zuul VMs - https://phabricator.wikimedia.org/T395938#10930252 (10dduvall) @Dzahn the WMF based production images for Zuul and Nodepool have been built and published to our registry. I'll post a summary abou...