[00:06:56] FIRING: SystemdUnitDown: The service unit logrotate.service is in failed status on host cloudgw1004. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudgw1004 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [01:01:56] RESOLVED: SystemdUnitDown: The service unit logrotate.service is in failed status on host cloudgw1004. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudgw1004 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [07:23:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-67 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [08:07:35] (03update) 10dcaro: Draft: dcaro test [repos/cloud/toolforge/jobs-api] (fix_diff_bug) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/182 [08:22:51] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "auditlogging" project Buster deprecation - https://phabricator.wikimedia.org/T367522#10999101 (10Aklapper) @Andrew / @Southparkfan: Can this ticket be resolved by now, or is there more to do? [08:34:47] (03approved) 10dcaro: package: add toolforge- prefix to more files [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/5 (https://phabricator.wikimedia.org/T399238) (owner: 10lucaswerkmeister) [08:34:51] (03merge) 10dcaro: package: add toolforge- prefix to more files [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/5 (https://phabricator.wikimedia.org/T399238) (owner: 10lucaswerkmeister) [08:36:54] (03open) 10dcaro: d/changelog: bump to 1.49.3 [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/6 (https://phabricator.wikimedia.org/T399238) [08:38:14] 06cloud-services-team, 10Toolforge, 07Kubernetes: Unable to load Toolforge job: ERROR: TjfCliError: Unknown error (403 Client Error: Forbidden for url - https://phabricator.wikimedia.org/T399417#10999154 (10dcaro) I don't see this on my tools, so might be related to the specific situation of this tool, looking [08:52:29] (03PS1) 10Muehlenhoff: Remove dummy keytab for sretest1001 (decommed) [labs/private] - 10https://gerrit.wikimedia.org/r/1169040 [09:03:28] 06cloud-services-team, 10PAWS: [2025-07-08] PAWS down - https://phabricator.wikimedia.org/T398912#10999225 (10dcaro) It was rebooted by @Andrew during the weekend: https://sal.toolforge.org/log/BQrWAZgBvg159pQryhXW I don't think it's the same issue, as this one would not have gotten fixed by a reboot, wil... [09:21:40] 06cloud-services-team, 10Toolforge: What happenned to /shared symlink? - https://phabricator.wikimedia.org/T399369#10999278 (10dcaro) I added a note in the wiki: https://wikitech.wikimedia.org/wiki/Help:Shared_storage#/data/project/shared/mediawiki This has been the status quo for a while now since we moved o... [09:24:39] (03approved) 10dcaro: d/changelog: bump to 1.49.3 [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/6 (https://phabricator.wikimedia.org/T399238) [09:24:42] (03merge) 10dcaro: d/changelog: bump to 1.49.3 [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/6 (https://phabricator.wikimedia.org/T399238) [09:26:35] 06cloud-services-team, 10Toolforge (Toolforge iteration 21), 13Patch-For-Review: Missing bash completion for `become` - https://phabricator.wikimedia.org/T399238#10999298 (10dcaro) p:05Triage→03Medium [09:26:43] 06cloud-services-team, 10Toolforge (Toolforge iteration 21), 13Patch-For-Review: Missing bash completion for `become` - https://phabricator.wikimedia.org/T399238#10999301 (10dcaro) [09:26:50] 06cloud-services-team, 10Toolforge (Toolforge iteration 21), 13Patch-For-Review: Missing bash completion for `become` - https://phabricator.wikimedia.org/T399238#10999302 (10dcaro) a:03dcaro [09:26:59] 06cloud-services-team, 10Toolforge (Toolforge iteration 21), 13Patch-For-Review: Missing bash completion for `become` - https://phabricator.wikimedia.org/T399238#10999304 (10dcaro) 05Open→03Resolved [09:27:47] 06cloud-services-team, 10Toolforge (Toolforge iteration 21), 13Patch-For-Review: Missing bash completion for `become` - https://phabricator.wikimedia.org/T399238#10999307 (10dcaro) Deployed and working :), thanks @LucasWerkmeister and @bd808 for the fix! [09:28:01] 06cloud-services-team, 10Toolforge, 07Kubernetes: Unable to load Toolforge job: ERROR: TjfCliError: Unknown error (403 Client Error: Forbidden for url - https://phabricator.wikimedia.org/T399417#10999312 (10dcaro) p:05Triage→03High [09:28:22] 06cloud-services-team, 10Toolforge: Add automatic linting for toolforge API openapi specs - https://phabricator.wikimedia.org/T399363#10999314 (10dcaro) p:05Triage→03Medium [09:28:51] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: Duplicate operationId `cancel_tool_deployment` in components API spec - https://phabricator.wikimedia.org/T399362#10999317 (10dcaro) p:05Triage→03High [09:29:20] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge (Toolforge iteration 21): Decision request - Reuse toolforge user tools central logging for toolforge infrastructure logging - https://phabricator.wikimedia.org/T398285#10999327 (10dcaro) [09:29:35] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS, 10Toolforge (Toolforge iteration 21): 2025-07-11 Ceph issues causing Toolforge and Cloud VPS failures - https://phabricator.wikimedia.org/T399281#10999328 (10dcaro) p:05Unbreak!→03High [09:29:46] 06cloud-services-team, 10Toolforge (Toolforge iteration 21), 10GitLab (Project Migration), 13Patch-For-Review: Migrate misctools package to GitLab - https://phabricator.wikimedia.org/T398202#10999330 (10dcaro) [09:31:06] 06cloud-services-team, 10Toolforge (Toolforge iteration 21), 13Patch-For-Review: [components-api] Provide a standalone version of tool config schema - https://phabricator.wikimedia.org/T397724#10999342 (10dcaro) 05In progress→03Resolved [09:31:11] 06cloud-services-team, 10Toolforge (Toolforge iteration 21), 13Patch-For-Review: [components-api] Provide a standalone version of tool config schema - https://phabricator.wikimedia.org/T397724#10999345 (10dcaro) 05Resolved→03In progress [09:32:10] 06cloud-services-team, 10Toolforge (Toolforge iteration 21), 13Patch-For-Review: [components-api] Provide a standalone version of tool config schema - https://phabricator.wikimedia.org/T397724#10999364 (10dcaro) I'll close this, but let's upload the schema to schemastore once the beta ends [09:32:54] 10Cloud Services Proposals, 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge (Toolforge iteration 21), 05Cloud-Services-Origin-Team, and 3 others: [builds-api,components-api,webservice,jobs-api] Make Toolforge a proper platform as a service with push... - https://phabricator.wikimedia.org/T194332#10999371 [09:33:05] 06cloud-services-team, 10Toolforge (Toolforge iteration 21), 13Patch-For-Review: [components-api] Provide a standalone version of tool config schema - https://phabricator.wikimedia.org/T397724#10999373 (10dcaro) 05In progress→03Resolved [09:33:10] 10Toolforge (Toolforge iteration 21): [components-api,api-gateway] allow getting a deployment status using the deployment token - https://phabricator.wikimedia.org/T398623#10999377 (10dcaro) 05In progress→03Resolved [09:33:59] 10Toolforge (Toolforge iteration 21): [k8s,infra] use the new docker-registry.svc.toolforge.org host everywhere - https://phabricator.wikimedia.org/T394902#10999379 (10dcaro) a:05dcaro→03None [09:34:21] 06cloud-services-team, 10Toolforge: [k8s,infra] use the new docker-registry.svc.toolforge.org host everywhere - https://phabricator.wikimedia.org/T394902#10999381 (10dcaro) 05In progress→03Open [09:35:04] 06cloud-services-team, 10Toolforge (Toolforge iteration 21), 10GitLab (Project Migration), 13Patch-For-Review: Migrate misctools package to GitLab - https://phabricator.wikimedia.org/T398202#10999387 (10dcaro) a:03dcaro [09:35:09] 06cloud-services-team, 10Toolforge (Toolforge iteration 21), 10GitLab (Project Migration), 13Patch-For-Review: Migrate misctools package to GitLab - https://phabricator.wikimedia.org/T398202#10999390 (10dcaro) 05Open→03Resolved [09:35:33] 10Toolforge (Toolforge iteration 21), 13Patch-For-Review: [lima-kilo] docker-compose not found - https://phabricator.wikimedia.org/T398322#10999399 (10dcaro) p:05Triage→03Low [09:35:58] 10Toolforge (Toolforge iteration 21), 13Patch-For-Review: [lima-kilo] support range syntax for ansible tags - https://phabricator.wikimedia.org/T398306#10999401 (10dcaro) p:05Triage→03Medium [09:36:05] 10Toolforge (Toolforge iteration 21), 13Patch-For-Review: [lima-kilo] support range syntax for ansible tags - https://phabricator.wikimedia.org/T398306#10999403 (10dcaro) 05Open→03Resolved [09:36:37] 10Toolforge (Toolforge iteration 21), 13Patch-For-Review: [lima-kilo] support range syntax for ansible tags - https://phabricator.wikimedia.org/T398306#10999406 (10dcaro) 05Resolved→03In progress [09:36:55] 06cloud-services-team, 10Toolforge: [components-cli,toolforge-cli] add shortcuts to top-level cli for deploy/config - https://phabricator.wikimedia.org/T397725#10999410 (10dcaro) [09:37:33] 10Toolforge (Toolforge iteration 21): [infra] 2025-06-21 Several correlated potentially network issues during the night - https://phabricator.wikimedia.org/T397566#10999414 (10dcaro) I'll close this, it has not happened again, and we have no time to investigate now. [09:38:19] 10Toolforge: What happened to /shared symlink? - https://phabricator.wikimedia.org/T399369#10999415 (10Aklapper) [09:38:22] 10Toolforge: What happened to /shared symlink? - https://phabricator.wikimedia.org/T399369#10999416 (10Aklapper) [09:38:24] 10Toolforge (Toolforge iteration 21): [infra] 2025-06-21 tools-prometheus-8 stopped responding for a bit - https://phabricator.wikimedia.org/T397563#10999417 (10dcaro) I'll close this as it has not happened again, the query log is on as it's not using a lot of space, so next time if it happens we'll have some ex... [09:39:11] 10Toolforge (Toolforge iteration 21): [infra] 2025-06-21 Several correlated potentially network issues during the night - https://phabricator.wikimedia.org/T397566#10999423 (10dcaro) 05Open→03Resolved a:03dcaro [09:39:14] 10Toolforge (Toolforge iteration 21): [infra] 2025-06-21 tools-prometheus-8 stopped responding for a bit - https://phabricator.wikimedia.org/T397563#10999426 (10dcaro) 05Open→03Resolved a:03dcaro [09:39:27] 10Toolforge (Toolforge iteration 21): [maintain-harbor,ci] the version number does not get bumped on every release - https://phabricator.wikimedia.org/T396504#10999432 (10dcaro) 05Open→03Resolved [09:39:49] 06cloud-services-team, 10Toolforge: [infra] move toolsbeta to `toolsbeta.org` domain - https://phabricator.wikimedia.org/T394997#10999434 (10dcaro) [09:40:19] 06cloud-services-team, 10Toolforge: [functional-tests,builds-builder] create a test suite to run builds for all the sample tools we have - https://phabricator.wikimedia.org/T394927#10999436 (10dcaro) [09:40:24] 06cloud-services-team, 10Toolforge: [envvars] show the 'global' envvars when running `toolforge envvars list` - https://phabricator.wikimedia.org/T394408#10999438 (10dcaro) [09:40:41] 06cloud-services-team, 10Toolforge: [builds-api] define a policy to update runtimes - https://phabricator.wikimedia.org/T393937#10999443 (10dcaro) [09:41:19] 06cloud-services-team, 10Toolforge: [jobs-api] Indicate when a job is too big to be scheduled - https://phabricator.wikimedia.org/T383515#10999445 (10dcaro) [09:41:42] 06cloud-services-team, 10Toolforge: [toolforge,jobs] "toolforge jobs logs" fails when job has not started yet - https://phabricator.wikimedia.org/T349775#10999447 (10dcaro) [09:43:27] 06cloud-services-team, 10Toolforge (Toolforge iteration 21), 10Sustainability (Incident Followup): [docs,envvars-api,jobs-api,builds-api] create docs on how to operate the cluster and core components - https://phabricator.wikimedia.org/T380959#10999450 (10dcaro) [09:43:39] 10Toolforge (Toolforge iteration 21): [components-cli] Allow reading tool configuration from stdin - https://phabricator.wikimedia.org/T398424#10999452 (10dcaro) a:03dcaro [09:43:48] 10Toolforge (Toolforge iteration 21): [docs] enable docs linter in one of the repos - https://phabricator.wikimedia.org/T397949#10999453 (10dcaro) a:03dcaro [09:44:10] 06cloud-services-team, 10Toolforge (Toolforge iteration 21), 10Sustainability (Incident Followup): [docs,envvars-api,jobs-api,builds-api] create docs on how to operate the cluster and core components - https://phabricator.wikimedia.org/T380959#10999454 (10dcaro) a:03dcaro [09:44:19] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge (Toolforge iteration 21): Decision request - Reuse toolforge user tools central logging for toolforge infrastructure logging - https://phabricator.wikimedia.org/T398285#10999455 (10dcaro) a:03taavi [09:44:36] 06cloud-services-team, 10Toolforge (Toolforge iteration 21): [components-api] optionally log deployments to SAL automatically - https://phabricator.wikimedia.org/T393169#10999456 (10dcaro) a:03dcaro [09:47:41] 10Toolforge (Toolforge iteration 22), 13Patch-For-Review: [jobs-api] refactor models - https://phabricator.wikimedia.org/T389118#10999488 (10dcaro) [09:47:42] 06cloud-services-team, 10Toolforge (Toolforge iteration 22), 13Patch-For-Review: toolsbeta harbor disk full - https://phabricator.wikimedia.org/T398715#10999484 (10dcaro) [09:47:43] 06cloud-services-team, 10Toolforge (Toolforge iteration 22), 13Patch-For-Review: [jobs-api] Split the `*Job` API models into three - https://phabricator.wikimedia.org/T390136#10999486 (10dcaro) [09:47:44] 06cloud-services-team, 10Toolforge (Toolforge iteration 22), 13Patch-For-Review: [harbor,infra] Find a way to manage toolforge project policies with code - https://phabricator.wikimedia.org/T360509#10999490 (10dcaro) [09:47:46] 10Toolforge (Toolforge iteration 22): [jobs-api] bug in runtime diff_with_running_job function - https://phabricator.wikimedia.org/T394734#10999498 (10dcaro) [09:47:49] 10Toolforge (Toolforge iteration 22), 13Patch-For-Review: [jobs-api] check for diff in services when running diff_with_running_job - https://phabricator.wikimedia.org/T392717#10999496 (10dcaro) [09:47:51] 06cloud-services-team, 10Toolforge (Toolforge iteration 22), 13Patch-For-Review: Move Kubernetes log source multi-pod handling from jobs-api to toolforge-weld - https://phabricator.wikimedia.org/T398647#10999494 (10dcaro) [09:47:55] 06cloud-services-team, 10Toolforge (Toolforge iteration 22), 13Patch-For-Review: [jobs-api] Periodically refresh image-config data - https://phabricator.wikimedia.org/T357112#10999492 (10dcaro) [09:48:13] 10Toolforge (Toolforge iteration 22): [toolforge] simplify calling the different toolforge apis from within the containers - https://phabricator.wikimedia.org/T356377#10999504 (10dcaro) [09:48:17] 10Toolforge (Toolforge iteration 22), 07Upstream: [builds-builder] golang based images get infinite nested loops for procfile entries - https://phabricator.wikimedia.org/T363417#10999506 (10dcaro) [09:48:26] 10Toolforge (Toolforge iteration 22), 07Upstream: [builds-builder,jobs-api,upstream] Calling nontrivial Procfile commands with arguments results in confusing error (“no such file or directory”) - https://phabricator.wikimedia.org/T356016#10999508 (10dcaro) [09:48:34] 06cloud-services-team, 10Toolforge (Toolforge iteration 22), 05Goal: [harbor] Move harbor data to object storage service - https://phabricator.wikimedia.org/T350687#10999515 (10dcaro) [09:48:38] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge (Toolforge iteration 22): Intermittent redis connection timeouts in Toolforge - https://phabricator.wikimedia.org/T318479#10999513 (10dcaro) [09:48:42] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge (Toolforge iteration 22), 05Goal, 13Patch-For-Review: [infra] Decommission the Grid Engine infrastructure - https://phabricator.wikimedia.org/T314664#10999511 (10dcaro) [09:48:54] 10Toolforge (Toolforge iteration 22), 13Patch-For-Review: [jobs-api] Create storage layer, and save business models in persistent storage - https://phabricator.wikimedia.org/T359650#10999518 (10dcaro) [09:49:14] 10Toolforge (Toolforge iteration 22): [clis] standardize the package names - https://phabricator.wikimedia.org/T399080#10999540 (10dcaro) [09:49:18] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS, 10Toolforge (Toolforge iteration 22): 2025-07-11 Ceph issues causing Toolforge and Cloud VPS failures - https://phabricator.wikimedia.org/T399281#10999538 (10dcaro) [09:49:22] 10Toolforge (Toolforge iteration 22), 13Patch-For-Review: [lima-kilo] support range syntax for ansible tags - https://phabricator.wikimedia.org/T398306#10999542 (10dcaro) [09:49:26] 06cloud-services-team, 10Toolforge (Toolforge iteration 22), 13Patch-For-Review: [k8s,infra] Upgrade Toolforge to Uwubernetes (1.30) - https://phabricator.wikimedia.org/T362869#10999546 (10dcaro) [09:49:30] 06cloud-services-team, 10Toolforge (Toolforge iteration 22): [builds-builder] Upgrade python buildpack to v0.17.0 or newer for Poetry support - https://phabricator.wikimedia.org/T374056#10999548 (10dcaro) [09:49:34] 10Cloud Services Proposals, 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge (Toolforge iteration 22), 05Cloud-Services-Origin-Team, and 3 others: [builds-api,components-api,webservice,jobs-api] Make Toolforge a proper platform as a service with push... - https://phabricator.wikimedia.org/T194332#10999544 [09:49:38] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge (Toolforge iteration 22), 07Epic: [KR] WE6.3 Introduce a sustainability scoring system for the Toolforge platform - https://phabricator.wikimedia.org/T368600#10999550 (10dcaro) [09:49:46] 06cloud-services-team, 10Toolforge (Toolforge iteration 22), 07Epic: [jobs-api] expose jobs-api continuous jobs to the internet via `toolname.toolforge.org`, just like webservice - https://phabricator.wikimedia.org/T388092#10999552 (10dcaro) [09:49:50] 10Toolforge (Toolforge iteration 22), 07Epic: [cicd] Streamline toolforge cli deployment and external contributor ci flows - https://phabricator.wikimedia.org/T392524#10999558 (10dcaro) [09:49:54] 06cloud-services-team, 10Toolforge (Toolforge iteration 22), 07Epic: [jobs-api,webservice] Run webservices via the jobs framework - https://phabricator.wikimedia.org/T348755#10999556 (10dcaro) [09:49:59] 06cloud-services-team, 10Toolforge (Toolforge iteration 22), 13Patch-For-Review: Toolforge: Replace all bastion with grid-less bookworm based bastion hosts - https://phabricator.wikimedia.org/T314665#10999554 (10dcaro) [09:50:03] 10Toolforge (Toolforge iteration 22), 13Patch-For-Review: Persist important toolforge k8s components logs - https://phabricator.wikimedia.org/T383081#10999563 (10dcaro) [09:50:07] 06cloud-services-team, 10Toolforge (Toolforge iteration 22), 13Patch-For-Review: [jobs-api,infra] upgrade all the existing toolforge jobs to the latest job version - https://phabricator.wikimedia.org/T359649#10999561 (10dcaro) [09:50:11] 06cloud-services-team, 10Toolforge (Toolforge iteration 22), 13Patch-For-Review: [jobs-api] when running a command with wrong quoting, no logs nor useful feedback is given to the user - https://phabricator.wikimedia.org/T356267#10999565 (10dcaro) [09:50:15] 10Toolforge (Toolforge iteration 22), 13Patch-For-Review: [cicd] create cicd flow for non repo owners - https://phabricator.wikimedia.org/T394595#10999567 (10dcaro) [09:50:19] 06cloud-services-team, 10Toolforge (Toolforge iteration 22): [kyverno] Upgrade to `3.3.9` chart (`1.13` app) for k8s 1.30 support - https://phabricator.wikimedia.org/T394787#10999569 (10dcaro) [09:50:23] 06cloud-services-team, 10Toolforge (Toolforge iteration 22), 13Patch-For-Review: [builds-builder] Add support for Heroku's "24" builder stack based on Ubuntu 2024.04 noble - https://phabricator.wikimedia.org/T380127#10999571 (10dcaro) [09:50:29] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge (Toolforge iteration 22): Decision request - Reuse toolforge user tools central logging for toolforge infrastructure logging - https://phabricator.wikimedia.org/T398285#10999573 (10dcaro) [09:50:33] 06cloud-services-team, 10Toolforge (Toolforge iteration 22): [components-api,beta] CI pipelines should wait until Toolforge deployment is 100% successful - https://phabricator.wikimedia.org/T398485#10999575 (10dcaro) [09:50:37] 10Toolforge (Toolforge iteration 22): [components-cli] Allow reading tool configuration from stdin - https://phabricator.wikimedia.org/T398424#10999576 (10dcaro) [09:50:41] 06cloud-services-team, 10Toolforge (Toolforge iteration 22), 13Patch-For-Review: [jobs-api] Jobs API should query logs from Loki - https://phabricator.wikimedia.org/T398645#10999574 (10dcaro) [09:50:45] 10Toolforge (Toolforge iteration 22): [docs] enable docs linter in one of the repos - https://phabricator.wikimedia.org/T397949#10999578 (10dcaro) [09:50:49] 06cloud-services-team, 10Toolforge (Toolforge iteration 22): [tools-static,infra] NFS issues should not bring tools-static down - https://phabricator.wikimedia.org/T397634#10999579 (10dcaro) [09:50:57] 10Toolforge (Toolforge iteration 22), 13Patch-For-Review: [lima-kilo] docker-compose not found - https://phabricator.wikimedia.org/T398322#10999577 (10dcaro) [09:51:01] 10Toolforge (Toolforge iteration 22): [builds-cli] add resolved reference when showing a build - https://phabricator.wikimedia.org/T394300#10999581 (10dcaro) [09:51:05] 10Toolforge (Toolforge iteration 22), 07good first task: [components-api] use the `build.params.image_name` to compare with the `component` - https://phabricator.wikimedia.org/T395076#10999580 (10dcaro) [09:51:09] 06cloud-services-team, 10Toolforge (Toolforge iteration 22): [components-api] optionally log deployments to SAL automatically - https://phabricator.wikimedia.org/T393169#10999582 (10dcaro) [09:51:17] 06cloud-services-team, 10Toolforge (Toolforge iteration 22), 10Sustainability (Incident Followup): [docs,envvars-api,jobs-api,builds-api] create docs on how to operate the cluster and core components - https://phabricator.wikimedia.org/T380959#10999583 (10dcaro) [09:51:21] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS (Debian Buster Deprecation), 10Toolforge (Toolforge iteration 22), 07Epic, 05Goal: [infra] Toolforge: migrate to Debian Bullseye or later - https://phabricator.wikimedia.org/T311897#10999585 (10dcaro) [09:51:29] 10Toolforge: What happened to /shared symlink? - https://phabricator.wikimedia.org/T399369#10999595 (10dcaro) 05Open→03Declined p:05Triage→03Low [10:10:10] 10Cloud-VPS (Project-requests): Request creation of voterlists VPS project - https://phabricator.wikimedia.org/T399418#10999639 (10Raymond_Ndibe) a:03Raymond_Ndibe [10:23:28] 10Toolforge (Quota-requests): Request increased build quota for toc Toolforge tool - https://phabricator.wikimedia.org/T398780#10999676 (10Raymond_Ndibe) >>! In T398780#10981423, @Kanashimi wrote: >> signature-check is a different tool? Can you open a task for it if so? > > Yes, they are different tools. also I... [10:37:05] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge (Toolforge iteration 22): Intermittent redis connection timeouts in Toolforge - https://phabricator.wikimedia.org/T318479#10999722 (10SD0001) >>! In T318479#10670161, @fnegri wrote: > This one is just "stalled" but not blocked. I'm a bit out of ideas on ho... [10:41:28] 06cloud-services-team, 10Toolforge (Toolforge iteration 22), 10Sustainability (Incident Followup): [docs,envvars-api,jobs-api,builds-api] create docs on how to operate the cluster and core components - https://phabricator.wikimedia.org/T380959#10999755 (10dcaro) [10:41:59] 06cloud-services-team, 10Toolforge (Toolforge iteration 22), 10Sustainability (Incident Followup): [docs,envvars-api,jobs-api,builds-api] create docs on how to operate the cluster and core components - https://phabricator.wikimedia.org/T380959#10999759 (10dcaro) Updated the docs and the dashboards in grafana... [10:42:18] 06cloud-services-team, 10Toolforge (Toolforge iteration 22), 10Sustainability (Incident Followup): [docs,envvars-api,jobs-api,builds-api] create docs on how to operate the cluster and core components - https://phabricator.wikimedia.org/T380959#10999761 (10dcaro) 05Open→03Resolved [10:44:36] 06cloud-services-team, 10Toolforge (Toolforge iteration 22): [components-api,beta] CI pipelines should wait until Toolforge deployment is 100% successful - https://phabricator.wikimedia.org/T398485#10999769 (10dcaro) 05Open→03In progress [10:45:28] (03close) 10raymond-ndibe: docker-compose] use `docker compose` instead of docker-compose [repos/cloud/toolforge/lima-kilo] (ensure_use_of_amd64_arch) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/252 (https://phabricator.wikimedia.org/T398322) [10:46:33] 10Toolforge (Toolforge iteration 22), 13Patch-For-Review: [lima-kilo] docker-compose not found - https://phabricator.wikimedia.org/T398322#10999775 (10Raymond_Ndibe) the underlying problem was solved in a different that no longer created this follow up problem, so closing [10:47:15] 10Toolforge (Toolforge iteration 22), 13Patch-For-Review: [lima-kilo] docker-compose not found - https://phabricator.wikimedia.org/T398322#10999779 (10Raymond_Ndibe) 05Open→03Declined [11:15:07] FIRING: HarborDown: Harbor is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborDown [11:21:51] (03open) 10raymond-ndibe: [toolsbeta-harbor] expand registry volume size [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/63 (https://phabricator.wikimedia.org/T398715) [11:38:03] FIRING: [2x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-67 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcess [12:00:29] 10Toolforge (Toolforge iteration 22): [lima-kilo] docker-compose not found - https://phabricator.wikimedia.org/T398322#11000026 (10Raymond_Ndibe) 05Declined→03Resolved [12:03:51] 10Cloud-VPS (Project-requests): Request creation of voterlists VPS project - https://phabricator.wikimedia.org/T399418#11000035 (10dcaro) +1, this could be an interesting project to port to toolforge once we have persistent volumes available (for valkey, not in the short term though) [12:04:16] 10Cloud-VPS (Quota-requests): Change quota for mobileappsperformance account - https://phabricator.wikimedia.org/T398638#11000040 (10dcaro) +1 [12:04:45] (03update) 10raymond-ndibe: runtimes.k8s.images: use config for image refresh interval [repos/cloud/toolforge/jobs-api] (refresh_image_config_data) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/165 (owner: 10dcaro) [12:13:28] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance toolsbeta-harbor-1 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [12:16:48] 06cloud-services-team, 14Toolforge (Toolforge iteration 21): Missing bash completion for `become` - https://phabricator.wikimedia.org/T399238#11000071 (10LucasWerkmeister) Can confirm it’s working again, thank you! [12:28:03] FIRING: [2x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-67 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcess [12:33:28] RESOLVED: PuppetAgentFailure: Puppet agent failure detected on instance toolsbeta-harbor-1 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [12:39:11] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-78 [12:45:08] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-78 [13:17:26] 06cloud-services-team: CephSlowOps Ceph cluster in eqiad has 1574 slow ops - https://phabricator.wikimedia.org/T399328#11000220 (10Andrew) →14Duplicate dup:03T399281 [13:17:33] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS, 10Toolforge (Toolforge iteration 22): 2025-07-11 Ceph issues causing Toolforge and Cloud VPS failures - https://phabricator.wikimedia.org/T399281#11000222 (10Andrew) [13:23:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-78 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [13:53:10] FIRING: [2x] ProjectProxyMainProxyCertificateExpiry: Certificate for proxy on proxy-5 is about to expire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProjectProxyMainProxyCertificateExpiry [13:56:01] (03approved) 10dcaro: [toolsbeta-harbor] expand registry volume size [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/63 (https://phabricator.wikimedia.org/T398715) (owner: 10raymond-ndibe) [13:56:12] (03merge) 10raymond-ndibe: [toolsbeta-harbor] expand registry volume size [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/63 (https://phabricator.wikimedia.org/T398715) [13:58:53] 06cloud-services-team, 10Cloud-VPS: wp1-db-server trove DB instance in error - https://phabricator.wikimedia.org/T399464 (10Benoit74) 03NEW [13:59:21] 06cloud-services-team, 10Cloud-VPS: wp1-db-server trove DB instance in error - https://phabricator.wikimedia.org/T399464#11000423 (10Benoit74) @Andrew this is probably for you ^^ [14:03:11] 06cloud-services-team, 10Cloud-VPS: wp1-db-server trove DB instance in error - https://phabricator.wikimedia.org/T399464#11000442 (10fnegri) p:05Triage→03High a:03fnegri Andrew is out today, I'm having a look. :) [14:06:57] 06cloud-services-team, 10Toolforge, 07Epic: [cicd] Streamline toolforge cli deployment and external contributor ci flows - https://phabricator.wikimedia.org/T392524#11000461 (10dcaro) [14:10:11] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [cicd] create cicd flow for non repo owners - https://phabricator.wikimedia.org/T394595#11000500 (10dcaro) [14:13:00] 06cloud-services-team, 10Cloud-VPS: wp1-db-server trove DB instance in error - https://phabricator.wikimedia.org/T399464#11000516 (10fnegri) 05Open→03Resolved I could not ssh to the instance, so I tried a hard reboot. After that, I could ssh and the database looks healthy: ` 2025-07-14 14:05:44 0 [Not... [14:14:01] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS: wp1-db-server trove DB instance in error - https://phabricator.wikimedia.org/T399464#11000522 (10fnegri) [14:16:04] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS: wp1-db-server trove DB instance in error - https://phabricator.wikimedia.org/T399464#11000540 (10Benoit74) I confirm it is back online, thank you very much ; let's hope it will not crash again. Looking at some data, at least there is some good data r... [14:25:07] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.openstack.quota_increase (T398715) [14:25:12] T398715: toolsbeta harbor disk full - https://phabricator.wikimedia.org/T398715 [14:25:13] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) (T398715) [14:37:07] RESOLVED: HarborDown: Harbor is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborDown [14:39:27] (03update) 10raymond-ndibe: [maintain-harbor] reduce toolforge project quota [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/875 (https://phabricator.wikimedia.org/T398715) [14:40:33] 06cloud-services-team, 06Infrastructure-Foundations, 07IPv6, 07LDAP: Make ldap-ro service available over IPv6 - https://phabricator.wikimedia.org/T397149#11000682 (10joanna_borun) p:05Triage→03Low [14:42:52] (03update) 10raymond-ndibe: [maintain-harbor] reduce toolforge project quota [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/875 (https://phabricator.wikimedia.org/T398715) [14:43:11] (03update) 10raymond-ndibe: [maintain-harbor] reduce toolforge project quota [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/875 (https://phabricator.wikimedia.org/T398715) [15:04:54] (03approved) 10dcaro: [maintain-harbor] reduce toolforge project quota [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/875 (https://phabricator.wikimedia.org/T398715) (owner: 10raymond-ndibe) [15:15:09] 06cloud-services-team: KernelErrors Server cloudcephosd1013 logged kernel errors - https://phabricator.wikimedia.org/T399366#11000888 (10dcaro) This was due to a reboot, after sdj disappeared. [15:45:19] 06cloud-services-team: KernelErrors Server cloudcephosd1013 logged kernel errors - https://phabricator.wikimedia.org/T399366#11001074 (10fnegri) > This was due to a reboot, after sdj disappeared. The alert was not due to the reboot. The metric was reporting 318 kernel errors starting on Jul 11 at 14:17 UTC (bef... [16:00:51] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS, 10Toolforge (Toolforge iteration 22): 2025-07-11 Ceph issues causing Toolforge and Cloud VPS failures - https://phabricator.wikimedia.org/T399281#11001181 (10fnegri) [16:01:43] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS, 10Toolforge (Toolforge iteration 22): 2025-07-11 Ceph issues causing Toolforge and Cloud VPS failures - https://phabricator.wikimedia.org/T399281#11001186 (10fnegri) 05In progress→03Resolved This incident is resolved. Separate follow-up tasks wi... [16:07:36] (03approved) 10raymond-ndibe: [maintain-harbor] reduce toolforge project quota [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/875 (https://phabricator.wikimedia.org/T398715) [16:07:36] (03update) 10raymond-ndibe: [maintain-harbor] reduce toolforge project quota [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/875 (https://phabricator.wikimedia.org/T398715) [16:07:45] (03merge) 10raymond-ndibe: [maintain-harbor] reduce toolforge project quota [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/875 (https://phabricator.wikimedia.org/T398715) [16:24:48] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudgw1004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [16:29:48] RESOLVED: PuppetZeroResources: Puppet has failed generate resources on cloudgw1004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [16:39:44] (03open) 10dcaro: toolforge-cd: add a reusable toolforge deploy task [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/67 [16:44:25] 06cloud-services-team, 10Cloud-VPS, 10Moderator-Tools-Team (Kanban): Swift container endpoints are unavailable - https://phabricator.wikimedia.org/T399481 (10Kgraessle) 03NEW [16:44:43] 06cloud-services-team, 10Cloud-VPS, 06Moderator-Tools-Team: Swift container endpoints are unavailable - https://phabricator.wikimedia.org/T399481#11001465 (10Kgraessle) [16:45:08] (03update) 10dcaro: toolforge-cd: add a reusable toolforge deploy task [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/67 [16:46:05] 06cloud-services-team, 10Cloud-VPS, 06Moderator-Tools-Team: Swift container endpoints are unavailable - https://phabricator.wikimedia.org/T399481#11001476 (10fnegri) p:05Triage→03High [16:47:01] 06cloud-services-team, 10Cloud-VPS, 06Moderator-Tools-Team: Swift container endpoints are unavailable - https://phabricator.wikimedia.org/T399481#11001483 (10fnegri) @Andrew this apparently stopped working during the outage on Friday and is still broken. [16:48:02] 06cloud-services-team, 10Cloud-VPS, 06Moderator-Tools-Team: Swift container endpoints are unavailable - https://phabricator.wikimedia.org/T399481#11001494 (10Aklapper) [16:48:14] (03update) 10dcaro: toolforge-cd: add a reusable toolforge deploy task [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/67 [16:57:35] (03update) 10dcaro: toolforge-cd: add a reusable toolforge deploy task [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/67 [17:09:14] (03approved) 10dcaro: toolforge-cd: add a reusable toolforge deploy task [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/67 [17:09:17] (03merge) 10dcaro: toolforge-cd: add a reusable toolforge deploy task [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/67 [17:10:37] 06cloud-services-team, 10Toolforge, 07Privacy: tools-static.wmflabs.org/cdnjs may return redirects to speedcf.cloudflareaccess.com, violating user privacy - https://phabricator.wikimedia.org/T399483 (10LucasWerkmeister) 03NEW [17:17:44] 06cloud-services-team, 10Toolforge, 07Privacy: tools-static.wmflabs.org/cdnjs may return redirects to speedcf.cloudflareaccess.com, violating user privacy - https://phabricator.wikimedia.org/T399483#11001630 (10LucasWerkmeister) Hm, the [Puppet / nginx config](https://gerrit.wikimedia.org/g/operations/puppet... [17:19:59] 06cloud-services-team, 10Toolforge (Toolforge iteration 22): [components-api,beta] CI pipelines should wait until Toolforge deployment is 100% successful - https://phabricator.wikimedia.org/T398485#11001640 (10dcaro) I just added a reusable gitlab script to deploy tools that will wait for the success of the de... [17:20:21] 06cloud-services-team, 10Toolforge (Toolforge iteration 22): [components-api,beta] CI pipelines should wait until Toolforge deployment is 100% successful - https://phabricator.wikimedia.org/T398485#11001642 (10dcaro) 05In progress→03Resolved [17:20:55] 06cloud-services-team, 10Toolforge, 13Patch-For-Review, 07Privacy: tools-static.wmflabs.org/cdnjs may return redirects to speedcf.cloudflareaccess.com, violating user privacy - https://phabricator.wikimedia.org/T399483#11001646 (10LucasWerkmeister) ^ I haven’t the faintest idea if this will help or not, bu... [17:22:01] (03update) 10raymond-ndibe: [start-devenv.sh] enable use of range syntax for ansible tags [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/251 (https://phabricator.wikimedia.org/T398306) [17:24:11] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [17:24:57] 06cloud-services-team, 10Toolforge, 13Patch-For-Review, 07Privacy: tools-static.wmflabs.org/cdnjs may return redirects to speedcf.cloudflareaccess.com, violating user privacy - https://phabricator.wikimedia.org/T399483#11001671 (10LucasWerkmeister) [17:26:17] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor [17:26:52] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [17:27:57] 06cloud-services-team, 10Toolforge (Toolforge iteration 22), 13Patch-For-Review: toolsbeta harbor disk full - https://phabricator.wikimedia.org/T398715#11001701 (10Raymond_Ndibe) 05In progress→03Resolved [17:29:50] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor [17:34:31] !log raymond-ndibe@cloudcumin1001 mobileappsperformance START - Cookbook wmcs.openstack.quota_increase (T398638) [17:34:36] T398638: Change quota for mobileappsperformance account - https://phabricator.wikimedia.org/T398638 [17:34:38] !log raymond-ndibe@cloudcumin1001 mobileappsperformance END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) (T398638) [17:35:24] !log raymond-ndibe@cloudcumin1001 mobileappsperformance START - Cookbook wmcs.openstack.quota_increase (T398638) [17:35:31] !log raymond-ndibe@cloudcumin1001 mobileappsperformance END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) (T398638) [17:36:29] !log raymond-ndibe@cloudcumin1001 mobileappsperformance START - Cookbook wmcs.openstack.quota_increase (T398638) [17:36:36] !log raymond-ndibe@cloudcumin1001 mobileappsperformance END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) (T398638) [17:39:26] 10Cloud-VPS (Quota-requests): Change quota for mobileappsperformance account - https://phabricator.wikimedia.org/T398638#11001797 (10Raymond_Ndibe) Before: | Resource | Limit | | cores | 8 | | ram | 16384 | | gigabytes | 80 | [17:39:39] 10Cloud-VPS (Quota-requests): Change quota for mobileappsperformance account - https://phabricator.wikimedia.org/T398638#11001798 (10Raymond_Ndibe) **After:** | Resource | Limit | | cores | 16 | | ram | 32768 | | gigabytes | 100 | [17:41:02] 10Cloud-VPS (Quota-requests): Change quota for mobileappsperformance account - https://phabricator.wikimedia.org/T398638#11001802 (10Raymond_Ndibe) closing the task as resolved @Jgiannelos . You can re-open if you notice any issue. [17:41:37] 10Cloud-VPS (Quota-requests): Change quota for mobileappsperformance account - https://phabricator.wikimedia.org/T398638#11001803 (10Raymond_Ndibe) 05Open→03Resolved [17:58:10] !log raymond-ndibe@cloudcumin1001 voterlists START - Cookbook wmcs.vps.create_project for project voterlists in eqiad1 (T399418) [17:58:10] raymond-ndibe@cloudcumin1001: Unknown project "voterlists" [17:58:11] T399418: Request creation of voterlists VPS project - https://phabricator.wikimedia.org/T399418 [17:58:50] (03open) 10group_199_bot_333a6c67971a471aeb1cf0b14ccf9f49: projects: added project voterlists [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/257 (https://phabricator.wikimedia.org/T399418) [18:02:21] 10VPS-project-Phabricator, 06collaboration-services: Phabricator test project requires email verification but can't send email - https://phabricator.wikimedia.org/T388022#11001822 (10A_smart_kitten) p:05Low→03Medium Gently bumping this task, as I feel like it might be a major reason for more people not usi... [18:02:22] raymond-ndibe@cloudcumin1001 create_project (PID 2723356) is awaiting input [18:16:38] 06cloud-services-team, 10Cloud-VPS: Add k8s_admin, k8s_developer, and k8s_viewer roles expected by default Magnum config for Kubernetes auth using Keystone auth - https://phabricator.wikimedia.org/T399488 (10bd808) 03NEW [18:22:17] 06cloud-services-team, 10Cloud-VPS: Add k8s_admin, k8s_developer, and k8s_viewer roles expected by default Magnum config for Kubernetes auth using Keystone auth - https://phabricator.wikimedia.org/T399488#11001868 (10bd808) I asked @Andrew about adding these roles a week or so ago in IRC and he seemed to think... [18:37:30] (03update) 10raymond-ndibe: [start-devenv.sh] enable use of range syntax for ansible tags [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/251 (https://phabricator.wikimedia.org/T398306) [18:38:03] (03update) 10raymond-ndibe: [start-devenv.sh] enable use of range syntax for ansible tags [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/251 (https://phabricator.wikimedia.org/T398306) [18:38:12] (03update) 10raymond-ndibe: [start-devenv.sh] enable use of range syntax for ansible tags [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/251 (https://phabricator.wikimedia.org/T398306) [18:38:23] (03approved) 10raymond-ndibe: [start-devenv.sh] enable use of range syntax for ansible tags [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/251 (https://phabricator.wikimedia.org/T398306) [18:39:01] (03merge) 10raymond-ndibe: [start-devenv.sh] enable use of range syntax for ansible tags [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/251 (https://phabricator.wikimedia.org/T398306) [18:54:57] 10Toolforge (Toolforge iteration 22), 13Patch-For-Review: [lima-kilo] support range syntax for ansible tags - https://phabricator.wikimedia.org/T398306#11001926 (10Raymond_Ndibe) 05In progress→03Resolved [19:16:58] 10Cloud-Services: openstack-browser.toolforge.org - internal server error for certain URL - https://phabricator.wikimedia.org/T399492 (10Dzahn) 03NEW The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ and replace it... [19:31:18] 10Tool-openstack-browser: openstack-browser.toolforge.org - internal server error for certain URL - https://phabricator.wikimedia.org/T399492#11002079 (10JJMC89) [21:48:07] 10Cloud-VPS (Project-requests), 13Patch-For-Review: Request creation of voterlists VPS project - https://phabricator.wikimedia.org/T399418#11002517 (10Snaevar) -1, it is forbidden to put private sql tables to Wikimedia Cloud. This task does not have a reason to move it. I also find the reasoning in T355594 ver... [22:27:11] 10Toolforge (Quota-requests): Request increased build quota for toc Toolforge tool - https://phabricator.wikimedia.org/T398780#11002620 (10Kanashimi) @Raymond_Ndibe Thank you for the response. Although I didn't specify the number of CPUs, I checked the quota. It seems that the quota is not enough because of the...