[00:10:18] (03update) 10raymond-ndibe: logs-api: add more logs tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1140 (https://phabricator.wikimedia.org/T418326) [00:24:17] (03update) 10raymond-ndibe: logs-api: add more logs tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1140 (https://phabricator.wikimedia.org/T418326) [00:37:40] (03update) 10raymond-ndibe: logs-api: add more logs tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1140 (https://phabricator.wikimedia.org/T418326) [01:01:20] (03update) 10raymond-ndibe: logs-api: add more logs tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1140 (https://phabricator.wikimedia.org/T418326) [01:27:29] FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance extdist-06 in project extdist - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [01:30:43] (03open) 10davidcoronel: Draft: initial commit for pipeline [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/182 [01:33:01] (03update) 10davidcoronel: Draft: initial commit for pipeline [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/182 [01:53:04] 10Cloud-VPS (Quota-requests): Quota increases for gitlab-runners - https://phabricator.wikimedia.org/T418813#11676279 (10thcipriani) Thanks for this answer @Andrew! We should talk more when you're not sick (shoo! :)) I'm interested in figuring out the right path here with the new Magnum and the vision of a puppe... [03:09:13] 06cloud-services-team, 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: [harbor,toolsbeta] for some reason maintain_harbor seems to not be cleaning up toolforge/* images - https://phabricator.wikimedia.org/T417894#11676333 (10Raymond_Ndibe) described the issue and solution here https://gitlab.wi... [03:09:26] 06cloud-services-team, 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: [harbor,toolsbeta] for some reason maintain_harbor seems to not be cleaning up toolforge/* images - https://phabricator.wikimedia.org/T417894#11676335 (10Raymond_Ndibe) 05Open→03Resolved [03:09:33] (03close) 10raymond-ndibe: jobs: fix toolsbeta stale images cleanup bug [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/62 (https://phabricator.wikimedia.org/T417894) [04:26:03] 10Tool-campwiz-nxt: CampWiz Nxt Redesign: Root Path - https://phabricator.wikimedia.org/T415408#11676367 (10Henryl746) Hi there! I heard that the contribution period began a couple of days ago - I went ahead and submitted my first PR (https://github.com/Open-Knowledge-Impact-Foundation/campwiz-nxt-frontend2/pull... [09:19:29] (03PS1) 10Majavah: t2: Fix lack of spaces in message [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1248406 [09:21:01] (03CR) 10Majavah: [C:03+2] t2: Fix lack of spaces in message [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1248406 (owner: 10Majavah) [09:23:00] (03Merged) 10jenkins-bot: t2: Fix lack of spaces in message [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1248406 (owner: 10Majavah) [09:35:23] 10Toolforge (Toolforge iteration 26), 13Patch-For-Review: add more logs tests to toolforge-deploy - https://phabricator.wikimedia.org/T418326#11676776 (10dcaro) [09:35:25] 10Toolforge (Toolforge iteration 26), 13Patch-For-Review: [jobs-api] omitting filelog field from jobs creation payload completely breaks jobs-api for the tool - https://phabricator.wikimedia.org/T417518#11676778 (10dcaro) [09:35:31] 10Toolforge (Toolforge iteration 26), 13Patch-For-Review: Replace job image variants with webservice image variants - https://phabricator.wikimedia.org/T415322#11676780 (10dcaro) [09:35:36] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge (Toolforge iteration 26), 13Patch-For-Review: [jobs-api] make job status an enum, with clearly defined states - https://phabricator.wikimedia.org/T401172#11676784 (10dcaro) [09:35:38] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge (Toolforge iteration 26), 07Epic, 13Patch-For-Review: [jobs-api] expose jobs-api continuous jobs to the internet via `toolname.toolforge.org`, just like webservice - https://phabricator.wikimedia.org/T388092#11676782 (10dcaro) [09:35:41] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge (Toolforge iteration 26), 13Patch-For-Review: [jobs-api] Allow customizing time to request Loki logs for - https://phabricator.wikimedia.org/T400917#11676786 (10dcaro) [09:35:49] 10Toolforge (Toolforge iteration 26), 07Upstream: [builds-builder] golang based images get infinite nested loops for procfile entries - https://phabricator.wikimedia.org/T363417#11676803 (10dcaro) [09:35:50] 10Toolforge (Toolforge iteration 26), 07Upstream: [builds-builder,jobs-api,upstream] Calling nontrivial Procfile commands with arguments results in confusing error (“no such file or directory”) - https://phabricator.wikimedia.org/T356016#11676801 (10dcaro) [09:35:55] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge (Toolforge iteration 26), 13Patch-For-Review: [builds-api,harbor,image-config] Move pre-built images to harbor - https://phabricator.wikimedia.org/T409727#11676805 (10dcaro) [09:35:57] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge (Toolforge iteration 26): [jobs-api] Create storage layer, and save business models in persistent storage - https://phabricator.wikimedia.org/T359650#11676809 (10dcaro) [09:35:58] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge (Toolforge iteration 26), 07Epic, 13Patch-For-Review: [jobs-api,webservice] Run webservices via the jobs framework - https://phabricator.wikimedia.org/T348755#11676807 (10dcaro) [09:36:01] 06cloud-services-team, 10Toolforge (Toolforge iteration 26), 13Patch-For-Review: [jobs-api] Refactor before webservice support - https://phabricator.wikimedia.org/T359804#11676811 (10dcaro) [09:36:17] 06cloud-services-team, 10Toolforge (Toolforge iteration 26): Replace ingress-nginx before upstream EOL date - https://phabricator.wikimedia.org/T392356#11676817 (10dcaro) [09:36:18] 10Toolforge (Toolforge iteration 26): [harbor,tools] Harbor object usage in S3 is steadily increasing - https://phabricator.wikimedia.org/T418528#11676821 (10dcaro) [09:36:19] 10Toolforge (Toolforge iteration 26), 13Patch-For-Review: [components-api] Queue builds when the build queue is full - https://phabricator.wikimedia.org/T402568#11676815 (10dcaro) [09:36:21] 06cloud-services-team, 10Toolforge (Toolforge iteration 26), 13Patch-For-Review: [fourohfour] general unavailability / overload - https://phabricator.wikimedia.org/T418829#11676819 (10dcaro) [09:36:29] 10Toolforge (Toolforge iteration 26), 13Patch-For-Review: [docs] update all readmes with the same deployment docs - https://phabricator.wikimedia.org/T407477#11676823 (10dcaro) [09:36:31] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge (Toolforge iteration 26), 07Epic: [KR] WE6.3 Introduce a sustainability scoring system for the Toolforge platform - https://phabricator.wikimedia.org/T368600#11676829 (10dcaro) [09:36:33] 10Toolforge (Toolforge iteration 26), 13Patch-For-Review: [builds-api, maintain-harbor] fix build/image cleanup - https://phabricator.wikimedia.org/T404157#11676827 (10dcaro) [09:36:35] 10Cloud Services Proposals, 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge (Toolforge iteration 26), 05Cloud-Services-Origin-Team, and 3 others: [builds-api,components-api,webservice,jobs-api] Make Toolforge a proper platform as a service with push... - https://phabricator.wikimedia.org/T194332#11676825 [09:36:45] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge (Toolforge iteration 26), 13Patch-For-Review: [builds-builder] Add support for Heroku's "24" builder stack based on Ubuntu 2024.04 noble - https://phabricator.wikimedia.org/T380127#11676831 (10dcaro) [09:36:47] 06cloud-services-team, 10Toolforge (Toolforge iteration 26): [harbor] Update HarborDown runbook with the incident debugging details - https://phabricator.wikimedia.org/T354739#11676835 (10dcaro) [09:36:49] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge (Toolforge iteration 26): [infra,k8s] Upgrade Toolforge Kubernetes to version 1.32 - https://phabricator.wikimedia.org/T379047#11676834 (10dcaro) [09:36:52] 10Toolforge (Toolforge iteration 26): [docs] enable docs linter in one of the repos - https://phabricator.wikimedia.org/T397949#11676837 (10dcaro) [09:36:57] 06cloud-services-team, 10Toolforge (Toolforge iteration 26), 13Patch-For-Review: [components-api] allow specifying `source_repo`+`ref` for the config - https://phabricator.wikimedia.org/T402764#11676836 (10dcaro) [09:41:16] (03update) 10dcaro: [jobs-api] save business models in a DB [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/114 (owner: 10raymond-ndibe) [09:41:17] (03approved) 10dcaro: [jobs-api] save business models in a DB [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/114 (owner: 10raymond-ndibe) [09:41:22] (03merge) 10dcaro: [jobs-api] save business models in a DB [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/114 (owner: 10raymond-ndibe) [09:41:25] (03update) 10dcaro: core: add prometheus counter for jobs synced from runtime [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/253 [09:44:47] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.474-20260305094134-618549fd [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1169 (https://phabricator.wikimedia.org/T359650) [09:44:52] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.474-20260305094134-618549fd [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1169 (https://phabricator.wikimedia.org/T359650) [10:17:12] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [10:18:58] (03approved) 10dcaro: jobs-api: bump to 0.0.474-20260305094134-618549fd [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1169 (https://phabricator.wikimedia.org/T359650) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [10:19:03] (03merge) 10dcaro: jobs-api: bump to 0.0.474-20260305094134-618549fd [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1169 (https://phabricator.wikimedia.org/T359650) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [10:33:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [10:38:28] 06cloud-services-team, 10Tools: [superyetkin] Add primary key to visited_pages_agg - https://phabricator.wikimedia.org/T419115 (10fnegri) 03NEW [10:38:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [10:40:51] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge (Toolforge iteration 25): [toolsdb] ToolsToolsDBReplicationLagIsTooHigh - 2026-03-04 - https://phabricator.wikimedia.org/T419045#11677035 (10fnegri) The table `visited_pages_agg` does not have a primary key, and that's what is causing replication to be... [11:09:55] (03merge) 10dcaro: core: add prometheus counter for jobs synced from runtime [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/253 [11:11:28] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [11:11:30] !log dcaro@cloudcumin1001 toolsbeta END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api [11:13:29] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [11:13:36] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.475-20260305111008-ede9b4bf [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1170 (https://phabricator.wikimedia.org/T359650) [11:13:40] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.475-20260305111008-ede9b4bf [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1170 (https://phabricator.wikimedia.org/T359650) [11:25:12] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [11:32:15] (03update) 10taavi: Deploy istio and a Gateway [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1143 (https://phabricator.wikimedia.org/T418274) [11:58:16] (03PS1) 10Mvolz: Remove deprecated citoid variable [labs/countervandalism/cvn-infrastructure] - 10https://gerrit.wikimedia.org/r/1248441 (https://phabricator.wikimedia.org/T361576) [12:24:46] (03update) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/34 [12:50:54] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge (Toolforge iteration 26), 13Patch-For-Review: [jobs-api] Create storage layer, and save business models in persistent storage - https://phabricator.wikimedia.org/T359650#11677353 (10dcaro) [13:56:21] 06cloud-services-team, 10Toolforge (Toolforge iteration 25): [harbor,toolsbeta] for some reason maintain_harbor seems to not be cleaning up toolforge/* images - https://phabricator.wikimedia.org/T417894#11677544 (10dcaro) 🎉 [13:59:31] 10Toolforge (Toolforge iteration 26): [docs] enable docs linter in one of the repos - https://phabricator.wikimedia.org/T397949#11677554 (10dcaro) 05Open→03In progress [14:05:46] 06cloud-services-team, 10Toolforge (Toolforge iteration 26): [harbor] Update HarborDown runbook with the incident debugging details - https://phabricator.wikimedia.org/T354739#11677584 (10dcaro) 05Open→03Resolved [14:17:36] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [14:30:37] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [14:37:53] (03approved) 10dcaro: jobs-api: bump to 0.0.475-20260305111008-ede9b4bf [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1170 (https://phabricator.wikimedia.org/T359650) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:37:58] (03merge) 10dcaro: jobs-api: bump to 0.0.475-20260305111008-ede9b4bf [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1170 (https://phabricator.wikimedia.org/T359650) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:44:23] !log root@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [14:44:58] !log root@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) [14:45:17] (03approved) 10taavi: toolforge_get_versions: use component for td version [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1167 (owner: 10dcaro) [14:50:18] (03update) 10dcaro: toolforge_get_versions: use component for td version [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1167 [14:50:29] (03open) 10nurahwakili: Add missing information for author and work page [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/183 [14:51:08] (03merge) 10dcaro: toolforge_get_versions: use component for td version [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1167 [14:52:55] !log root@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [14:54:24] (03approved) 10dcaro: Deploy istio and a Gateway [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1143 (https://phabricator.wikimedia.org/T418274) (owner: 10taavi) [14:56:37] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge (Toolforge iteration 26), 13Patch-For-Review: [jobs-api] Create storage layer, and save business models in persistent storage - https://phabricator.wikimedia.org/T359650#11677773 (10dcaro) The counter of jobs not found storage but found in runtime can be... [14:56:52] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge (Toolforge iteration 26), 13Patch-For-Review: [jobs-api] Create storage layer, and save business models in persistent storage - https://phabricator.wikimedia.org/T359650#11677776 (10dcaro) 05Stalled→03In progress [14:58:27] (03update) 10taavi: Deploy istio and a Gateway [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1143 (https://phabricator.wikimedia.org/T418274) [14:58:51] (03update) 10taavi: Deploy istio and a Gateway [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1143 (https://phabricator.wikimedia.org/T418274) [15:01:19] (03update) 10taavi: Deploy istio and a Gateway [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1143 (https://phabricator.wikimedia.org/T418274) [15:02:13] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component istio-system [15:02:20] !log root@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) [15:05:22] (03open) 10taavi: k8s: Install Istio components [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/313 (https://phabricator.wikimedia.org/T418274) [15:05:25] (03update) 10taavi: k8s: Install Istio components [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/313 (https://phabricator.wikimedia.org/T418274) [15:05:35] (03update) 10taavi: k8s: Install Istio components [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/313 (https://phabricator.wikimedia.org/T418274) [15:12:45] (03open) 10taavi: kubernetes: Provision HTTPRoute resources [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/97 (https://phabricator.wikimedia.org/T418349) [15:12:51] (03update) 10taavi: kubernetes: Provision HTTPRoute resources [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/97 (https://phabricator.wikimedia.org/T418349) [15:12:58] RESOLVED: PuppetAgentFailure: Puppet agent failure detected on instance toolsbeta-test-k8s-etcd-34 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [15:14:19] (03update) 10taavi: kubernetes: Provision HTTPRoute resources [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/97 (https://phabricator.wikimedia.org/T418349) [15:16:20] (03update) 10taavi: app: cache the suggested tools in memory [toolforge-repos/fourohfour] - 10https://gitlab.wikimedia.org/toolforge-repos/fourohfour/-/merge_requests/16 (owner: 10dcaro) [15:16:21] (03update) 10taavi: app: cache the suggested tools in memory [toolforge-repos/fourohfour] - 10https://gitlab.wikimedia.org/toolforge-repos/fourohfour/-/merge_requests/16 (owner: 10dcaro) [15:16:22] (03approved) 10taavi: app: cache the suggested tools in memory [toolforge-repos/fourohfour] - 10https://gitlab.wikimedia.org/toolforge-repos/fourohfour/-/merge_requests/16 (owner: 10dcaro) [15:18:01] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-system [15:18:57] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway [15:19:09] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway [15:20:11] 06cloud-services-team, 10Toolforge (Toolforge iteration 26): [harbor,toolsbeta] for some reason maintain_harbor seems to not be cleaning up toolforge/* images - https://phabricator.wikimedia.org/T417894#11677874 (10Raymond_Ndibe) [16:38:55] FIRING: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [17:01:37] (03update) 10dcaro: app: cache the suggested tools in memory [toolforge-repos/fourohfour] - 10https://gitlab.wikimedia.org/toolforge-repos/fourohfour/-/merge_requests/16 [17:01:42] (03merge) 10dcaro: app: cache the suggested tools in memory [toolforge-repos/fourohfour] - 10https://gitlab.wikimedia.org/toolforge-repos/fourohfour/-/merge_requests/16 [17:02:44] 06cloud-services-team, 10Toolforge (Toolforge iteration 26): [fourohfour] general unavailability / overload - https://phabricator.wikimedia.org/T418829#11678372 (10dcaro) This seems to have been stable for some time, the improvement patches have been merged, I'll reopen if it crashes again. [17:03:04] 06cloud-services-team, 10Toolforge (Toolforge iteration 26): [fourohfour] general unavailability / overload - https://phabricator.wikimedia.org/T418829#11678375 (10dcaro) 05In progress→03Resolved [17:07:31] (03update) 10dcaro: fix filelog bug [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/275 (https://phabricator.wikimedia.org/T417518) (owner: 10raymond-ndibe) [17:13:55] RESOLVED: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [18:08:36] (03update) 10raymond-ndibe: fix filelog bug [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/275 (https://phabricator.wikimedia.org/T417518) [18:12:18] (03update) 10raymond-ndibe: fix filelog bug [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/275 (https://phabricator.wikimedia.org/T417518) [19:47:32] (03PS1) 10Krinkle: js-env: Fix missing slash in apiPath [labs/tools/intuition-web] - 10https://gerrit.wikimedia.org/r/1248582 [19:47:59] (03PS2) 10Krinkle: js-env: Fix missing slash in apiPath [labs/tools/intuition-web] - 10https://gerrit.wikimedia.org/r/1248582 [19:49:56] (03CR) 10Krinkle: [C:03+2] js-env: Fix missing slash in apiPath [labs/tools/intuition-web] - 10https://gerrit.wikimedia.org/r/1248582 (owner: 10Krinkle) [19:50:35] (03Merged) 10jenkins-bot: js-env: Fix missing slash in apiPath [labs/tools/intuition-web] - 10https://gerrit.wikimedia.org/r/1248582 (owner: 10Krinkle) [20:03:59] 10Tool-wiktlexbot: Treat P1343 as a reference - https://phabricator.wikimedia.org/T419159 (10Redmin) 03NEW [20:12:46] (03PS1) 10Krinkle: js-env: Add file header with self-URL to ease discovery and updates [labs/tools/intuition-web] - 10https://gerrit.wikimedia.org/r/1248595 [20:13:21] (03PS2) 10Krinkle: js-env: Add file header with self-URL to ease discovery and updates [labs/tools/intuition-web] - 10https://gerrit.wikimedia.org/r/1248595 [20:13:25] (03CR) 10Krinkle: [C:03+2] js-env: Add file header with self-URL to ease discovery and updates [labs/tools/intuition-web] - 10https://gerrit.wikimedia.org/r/1248595 (owner: 10Krinkle) [20:13:59] (03Merged) 10jenkins-bot: js-env: Add file header with self-URL to ease discovery and updates [labs/tools/intuition-web] - 10https://gerrit.wikimedia.org/r/1248595 (owner: 10Krinkle) [20:19:11] (03update) 10raymond-ndibe: logs-api: add more logs tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1140 (https://phabricator.wikimedia.org/T418326) [20:19:38] (03update) 10raymond-ndibe: logs-api: add more logs tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1140 (https://phabricator.wikimedia.org/T418326) [20:41:18] 06cloud-services-team, 10PAWS, 06tools-platform-team: New upstream release for OpenRefine - https://phabricator.wikimedia.org/T418629#11679123 (10LibUp-bot) A new upstream version of OpenRefine is now available: 3.10.1. * https://github.com/OpenRefine/OpenRefine/releases/tag/3.10.1 [23:13:51] 10Cloud-VPS (Project-requests): Request creation of azwikimedia VPS project - https://phabricator.wikimedia.org/T417736#11679749 (10Nemoralis) Can I get a status update until the weekend, please? I want to work on this during weekend [23:20:42] 10Cloud-VPS (Project-requests): Request creation of VPS project - https://phabricator.wikimedia.org/T419182 (10Yug) 03NEW [23:22:22] 10Cloud-VPS (Project-requests): Request creation of VPS project - https://phabricator.wikimedia.org/T419182#11679793 (10Yug) [23:22:44] 10Cloud-VPS (Project-requests): Request creation of lingualibre VPS project - https://phabricator.wikimedia.org/T419182#11679795 (10JJMC89) [23:26:50] 10Cloud-VPS (Project-requests): Request creation of lingualibre VPS project - https://phabricator.wikimedia.org/T419182#11679799 (10Yug) [23:29:42] 10Cloud-VPS (Project-requests): Request creation of lingualibre VPS project - https://phabricator.wikimedia.org/T419182#11679813 (10Yug) [23:30:39] 10Cloud-VPS (Project-requests): Request creation of lingualibre VPS project - https://phabricator.wikimedia.org/T419182#11679817 (10Yug) [23:33:06] 10Cloud-VPS (Project-requests): Request creation of lingualibre VPS project - https://phabricator.wikimedia.org/T419182#11679823 (10Yug) [23:35:01] 10Cloud-VPS (Project-requests): Request creation of lingualibre VPS project - https://phabricator.wikimedia.org/T419182#11679825 (10bd808) [23:37:37] 10Cloud-VPS (Project-requests): Request creation of lingualibre VPS project - https://phabricator.wikimedia.org/T419182#11679837 (10Yug) [23:40:03] 10Cloud-VPS (Project-requests): Request creation of lingualibre VPS project - https://phabricator.wikimedia.org/T419182#11679842 (10Yug)