[01:27:30] FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance extdist-06 in project extdist - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [01:49:48] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudgw1003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [01:59:48] RESOLVED: PuppetZeroResources: Puppet has failed generate resources on cloudgw1003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [07:37:02] (03CR) 10Thiemo Kreuz (WMDE): [C:03+1] Remove deprecated citoid variable [labs/countervandalism/cvn-infrastructure] - 10https://gerrit.wikimedia.org/r/1248441 (https://phabricator.wikimedia.org/T361576) (owner: 10Mvolz) [08:59:49] (03update) 10dcaro: sql: show the default of --cluster in help text [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/10 (owner: 10krinkle) [09:08:00] (03approved) 10dcaro: sql: show the default of --cluster in help text [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/10 (owner: 10krinkle) [09:08:07] (03merge) 10dcaro: sql: show the default of --cluster in help text [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/10 (owner: 10krinkle) [09:09:47] (03open) 10dcaro: bash-completion: fix missing ;; [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/11 [09:11:00] (03approved) 10dcaro: bash-completion: fix missing ;; [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/11 [09:11:04] (03merge) 10dcaro: bash-completion: fix missing ;; [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/11 [09:22:20] (03open) 10dcaro: gitlab: move to trixie images [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/12 [09:22:30] (03close) 10dcaro: gitlab: move to bookworm images [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/8 [09:23:26] (03update) 10dcaro: gitlab: move to trixie images [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/12 [09:39:52] (03update) 10dcaro: gitlab: move to trixie images [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/12 [10:25:28] (03PS1) 10Brouberol: idp: provision the airflow_fr_tech placeholder client secret [labs/private] - 10https://gerrit.wikimedia.org/r/1249225 [10:25:37] (03CR) 10Brouberol: [C:03+2] idp: provision the airflow_fr_tech placeholder client secret [labs/private] - 10https://gerrit.wikimedia.org/r/1249225 (owner: 10Brouberol) [10:25:41] (03CR) 10Brouberol: [V:03+2 C:03+2] idp: provision the airflow_fr_tech placeholder client secret [labs/private] - 10https://gerrit.wikimedia.org/r/1249225 (owner: 10Brouberol) [10:29:29] 10Cloud-VPS (Project-requests), 03Hackathon-Northwestern-Europe-2026: Request creation of lingualibre VPS project - https://phabricator.wikimedia.org/T419182#11686840 (10Yug) [10:30:41] 10Cloud-VPS (Project-requests), 10Lingua-Libre, 03Hackathon-Northwestern-Europe-2026: Request creation of lingualibre VPS project - https://phabricator.wikimedia.org/T419182#11686841 (10Yug) [10:32:26] (03PS1) 10Majavah: t5: Handle unblock title being in uppercase [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1249228 [10:34:32] 10VPS-project-Phabricator, 06collaboration-services: phabricator.wmcloud.org account verification request: DSantamaria - https://phabricator.wikimedia.org/T419396 (10DSantamaria) 03NEW [10:38:03] (03CR) 10Majavah: [C:03+2] t5: Handle unblock title being in uppercase [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1249228 (owner: 10Majavah) [10:40:51] (03Merged) 10jenkins-bot: t5: Handle unblock title being in uppercase [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1249228 (owner: 10Majavah) [11:26:00] (03CR) 10Filippo Giunchedi: [C:03+1] wmcs_libs: k8s: Add new 'gateway' worker type [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1248798 (https://phabricator.wikimedia.org/T419231) (owner: 10Majavah) [11:28:00] (03CR) 10Majavah: [C:03+2] wmcs_libs: k8s: Add new 'gateway' worker type [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1248798 (https://phabricator.wikimedia.org/T419231) (owner: 10Majavah) [11:31:23] (03Merged) 10jenkins-bot: wmcs_libs: k8s: Add new 'gateway' worker type [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1248798 (https://phabricator.wikimedia.org/T419231) (owner: 10Majavah) [11:34:46] 06cloud-services-team, 10Tools: [superyetkin] Add primary key to visited_pages_agg - https://phabricator.wikimedia.org/T419115#11687316 (10fnegri) @Superyetkin thank you! [11:35:19] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.add_k8s_node for a gateway role in the toolsbeta cluster [11:37:34] 10Cloud-VPS (Project-requests), 10Lingua-Libre, 03Hackathon-Northwestern-Europe-2026: Request creation of lingualibre VPS project - https://phabricator.wikimedia.org/T419182#11687320 (10Yug) [11:38:12] 10VPS-project-Codesearch, 06RoadToWiki, 07good first task: "Phabricator" format in Codesearch should have a way to load all matches (beyond default 20 per repo) - https://phabricator.wikimedia.org/T348921#11687322 (10A_smart_kitten) >>! In T348921#11685944, @Krinkle wrote: > Do you know of a case where y... [11:41:53] 10VPS-project-Codesearch, 06RoadToWiki, 07good first task: "Phabricator" format in Codesearch should indicate when there are more results for a repo than are displayed (was: Codesearch limits Phabricator format to 20 results per repo) - https://phabricator.wikimedia.org/T348921#11687329 (10A_smart_kitten) [11:44:55] !log taavi@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a gateway role in the toolsbeta cluster [12:09:11] 10Tools, 06All-and-every-Wikisource, 06Community-Tech, 10Wikimedia OCR: Wikisource OCR UI supplies a non-standard thumbnail size to the OCR tool hosted on the cloud - https://phabricator.wikimedia.org/T419246#11687449 (10zdev) [12:11:02] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-gateway-1 [12:11:55] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-gateway-1 [12:12:22] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.add_k8s_node for a gateway role in the toolsbeta cluster [12:20:55] RESOLVED: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [12:22:37] !log taavi@cloudcumin1001 toolsbeta Added a new k8s gateway toolsbeta-test-k8s-gateway-1.toolsbeta.eqiad1.wikimedia.cloud to the cluster [12:22:37] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a gateway role in the toolsbeta cluster [12:25:33] (03merge) 10taavi: Deploy istio and a Gateway [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1143 (https://phabricator.wikimedia.org/T418274) [12:29:32] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.add_k8s_node for a gateway role in the toolsbeta cluster [12:29:51] !log taavi@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a gateway role in the toolsbeta cluster [12:32:32] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.openstack.quota_increase by 65536 ram [12:32:39] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) by 65536 ram [12:32:42] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.add_k8s_node for a gateway role in the toolsbeta cluster [12:34:14] (03open) 10taavi: istio-gateway: Configure preference to gateway nodes [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1171 (https://phabricator.wikimedia.org/T419231) [12:34:19] (03update) 10taavi: istio-gateway: Configure preference to gateway nodes [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1171 (https://phabricator.wikimedia.org/T419231) [12:36:26] (03update) 10taavi: istio-gateway: Configure preference to gateway nodes [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1171 (https://phabricator.wikimedia.org/T419231) [12:39:23] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/60 [12:39:24] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/32 [12:41:40] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/massmailer] - 10https://gerrit.wikimedia.org/r/1249275 (owner: 10L10n-bot) [12:42:53] !log taavi@cloudcumin1001 toolsbeta Added a new k8s gateway toolsbeta-test-k8s-gateway-2.toolsbeta.eqiad1.wikimedia.cloud to the cluster [12:42:53] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a gateway role in the toolsbeta cluster [12:42:58] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.add_k8s_node for a gateway role in the toolsbeta cluster [12:43:21] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway [12:43:33] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway [12:46:14] (03update) 10taavi: istio-gateway: Configure preference to gateway nodes [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1171 (https://phabricator.wikimedia.org/T419231) [12:46:26] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway [12:46:38] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway [12:47:41] (03open) 10dcaro: debian-builder-trixie: add arch-specific libc/gcc [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/80 [12:49:20] (03update) 10taavi: istio-gateway: Configure pod sizing and scheduling [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1171 (https://phabricator.wikimedia.org/T419231) [12:49:25] (03update) 10taavi: istio-gateway: Configure pod sizing and scheduling [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1171 (https://phabricator.wikimedia.org/T419231) [12:50:55] FIRING: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [12:53:20] !log taavi@cloudcumin1001 toolsbeta Added a new k8s gateway toolsbeta-test-k8s-gateway-3.toolsbeta.eqiad1.wikimedia.cloud to the cluster [12:53:20] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a gateway role in the toolsbeta cluster [12:53:25] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway [12:53:37] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway [12:55:46] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway [12:55:51] (03update) 10taavi: istio-gateway: Configure pod sizing and scheduling [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1171 (https://phabricator.wikimedia.org/T419231) [12:55:58] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway [12:56:56] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway [12:57:01] (03update) 10taavi: istio-gateway: Configure pod sizing and scheduling [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1171 (https://phabricator.wikimedia.org/T419231) [12:57:08] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway [12:57:08] (03open) 10dcaro: debian-builder-trixie: added a test target [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/81 [12:57:43] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.add_k8s_node for a gateway role in the tools cluster [13:01:57] 10Tool-campwiz-nxt, 10Google-Summer-of-Code (Google Summer of Code (2026)): GSoC 2026: CampWiz NxT Redesign - https://phabricator.wikimedia.org/T414269#11687601 (10Codeurluce) Hi @Nokib_Sarkar and @Tiven2240 👋 Following up on LGoto's message about the contribution period starting March 2: I'm Lucien (coderlu... [13:05:48] (03approved) 10filippo: debian-builder-trixie: added a test target [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/81 (owner: 10dcaro) [13:06:10] (03update) 10taavi: istio-gateway: Configure pod sizing and scheduling [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1171 (https://phabricator.wikimedia.org/T419231) [13:06:40] (03approved) 10dcaro: debian-builder-trixie: added a test target [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/81 [13:06:45] (03merge) 10dcaro: debian-builder-trixie: added a test target [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/81 [13:07:31] (03update) 10dcaro: gitlab: move to trixie images [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/12 [13:07:54] !log taavi@cloudcumin1001 tools Added a new k8s gateway tools-k8s-gateway-1.tools.eqiad1.wikimedia.cloud to the cluster [13:07:54] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a gateway role in the tools cluster [13:08:34] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.add_k8s_node for a gateway role in the tools cluster [13:08:38] (03open) 10taavi: Move deployment to gateway nodes [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/88 (https://phabricator.wikimedia.org/T419231) [13:08:40] (03update) 10taavi: Move deployment to gateway nodes [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/88 (https://phabricator.wikimedia.org/T419231) [13:12:07] 06cloud-services-team, 10Tools: [superyetkin] Add primary key to visited_pages_agg - https://phabricator.wikimedia.org/T419115#11687664 (10fnegri) [13:13:20] (03open) 10dcaro: toolforge_get_versions: show component [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1172 [13:13:49] (03update) 10dcaro: debian-builder-trixie: add arch-specific libc/gcc [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/80 [13:14:10] (03update) 10dcaro: debian-builder-trixie: add arch-specific libc/gcc [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/80 [13:15:55] (03update) 10dcaro: debian-builder-trixie: add arch-specific libc/gcc [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/80 [13:16:28] FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance tools-k8s-gateway-2 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [13:17:34] (03update) 10dcaro: toolforge_get_versions: show component [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1172 [13:18:40] (03approved) 10dcaro: toolforge_get_versions: show component [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1172 [13:18:43] (03merge) 10dcaro: toolforge_get_versions: show component [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1172 [13:19:17] !log taavi@cloudcumin1001 tools Added a new k8s gateway tools-k8s-gateway-2.tools.eqiad1.wikimedia.cloud to the cluster [13:19:17] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a gateway role in the tools cluster [13:19:53] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.add_k8s_node for a gateway role in the tools cluster [13:20:09] (03update) 10taavi: Move deployment to gateway nodes [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/88 (https://phabricator.wikimedia.org/T419231) [13:20:20] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/60 (owner: 10l10n-bot) [13:20:24] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/60 (owner: 10l10n-bot) [13:21:06] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/32 (owner: 10l10n-bot) [13:21:09] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/32 (owner: 10l10n-bot) [13:21:28] RESOLVED: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance tools-k8s-gateway-2 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [13:24:30] 06cloud-services-team, 10Tools: [superyetkin] Add primary key to visited_pages_agg - https://phabricator.wikimedia.org/T419115#11687694 (10fnegri) @Superyetkin I noticed that you also removed the old index `KEY idx_project (vpa_year,vpa_month,vpa_day)`. If you plan to run other queries like the one in T41... [13:25:01] !log dcaro@cloudcumin1001 development-metrics START - Cookbook wmcs.vps.create_project for project development-metrics in eqiad1 (T419296) [13:25:02] dcaro@cloudcumin1001: Unknown project "development-metrics" [13:25:16] (03update) 10nurahwakili: Add missing information for author and work page [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/183 [13:25:35] !log dcaro@cloudcumin1001 development-metrics END (FAIL) - Cookbook wmcs.vps.create_project (exit_code=99) for project development-metrics in eqiad1 (T419296) [13:25:35] dcaro@cloudcumin1001: Unknown project "development-metrics" [13:26:54] (03approved) 10filippo: debian-builder-trixie: add arch-specific libc/gcc [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/80 (owner: 10dcaro) [13:27:09] 10Cloud-VPS (Project-requests): Request creation of azwikimedia VPS project - https://phabricator.wikimedia.org/T417736#11687706 (10dcaro) >>! In T417736#11682500, @Andrew wrote: > So -- I reiterate that I don't think a cloud-vps project is the right way to handle things like this. If I were solving the issues y... [13:27:40] (03approved) 10dcaro: debian-builder-trixie: add arch-specific libc/gcc [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/80 [13:27:44] (03merge) 10dcaro: debian-builder-trixie: add arch-specific libc/gcc [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/80 [13:28:58] !log dcaro@cloudcumin1001 development-metrics START - Cookbook wmcs.vps.create_project for project development-metrics in eqiad1 (T419296) [13:28:59] dcaro@cloudcumin1001: Unknown project "development-metrics" [13:29:39] (03open) 10group_199_bot_f98be072172e323ae6d1441939d3e461: projects: added project development-metrics [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/298 (https://phabricator.wikimedia.org/T419296) [13:30:05] !log taavi@cloudcumin1001 tools Added a new k8s gateway tools-k8s-gateway-3.tools.eqiad1.wikimedia.cloud to the cluster [13:30:05] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a gateway role in the tools cluster [13:30:15] (03update) 10dcaro: gitlab: move to trixie images [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/12 [13:30:55] RESOLVED: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [13:32:39] (03open) 10taavi: kubernetes: Adapt Ingress scheduling alerts for Istio Gateway [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/57 (https://phabricator.wikimedia.org/T419231) [13:32:42] (03update) 10taavi: kubernetes: Adapt Ingress scheduling alerts for Istio Gateway [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/57 (https://phabricator.wikimedia.org/T419231) [13:32:54] dcaro@cloudcumin1001 create_project (PID 4055991) is awaiting input [13:32:55] (03approved) 10dcaro: projects: added project development-metrics [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/298 (https://phabricator.wikimedia.org/T419296) (owner: 10group_199_bot_f98be072172e323ae6d1441939d3e461) [13:33:13] (03merge) 10dcaro: projects: added project development-metrics [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/298 (https://phabricator.wikimedia.org/T419296) (owner: 10group_199_bot_f98be072172e323ae6d1441939d3e461) [13:34:40] !log dcaro@cloudcumin1001 development-metrics END (PASS) - Cookbook wmcs.vps.create_project (exit_code=0) for project development-metrics in eqiad1 (T419296) [13:34:40] dcaro@cloudcumin1001: Unknown project "development-metrics" [13:48:04] 10Tool-paulina, 10Outreachy (Round 31): Outreachy 31: Features to edit author and work data on Wikidata directly from Paulina - https://phabricator.wikimedia.org/T392429#11687825 (10Nurah_Wakili) **Weekly Internship Report** Week 12: February 23 – February 27 **Work Completed:** - Completed development of... [13:50:32] (03approved) 10dcaro: kubernetes: Provision HTTPRoute resources [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/97 (https://phabricator.wikimedia.org/T418349) (owner: 10taavi) [13:52:16] (03update) 10dcaro: gitlab: move to trixie images [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/12 [14:04:23] (03approved) 10dcaro: fix filelog bug [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/275 (https://phabricator.wikimedia.org/T417518) (owner: 10raymond-ndibe) [14:04:25] (03update) 10dcaro: fix filelog bug [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/275 (https://phabricator.wikimedia.org/T417518) (owner: 10raymond-ndibe) [14:04:34] (03CR) 10Krinkle: "Declining per comment thread in T414466. TLDR: This is a standalone tool at cvn.wmflabs.org that uses an old copy of wmf-config for a very" [labs/countervandalism/cvn-infrastructure] - 10https://gerrit.wikimedia.org/r/1248441 (https://phabricator.wikimedia.org/T361576) (owner: 10Mvolz) [14:04:40] (03PS2) 10Krinkle: Remove deprecated citoid variable [labs/countervandalism/cvn-infrastructure] - 10https://gerrit.wikimedia.org/r/1248441 (owner: 10Mvolz) [14:04:44] (03Abandoned) 10Krinkle: Remove deprecated citoid variable [labs/countervandalism/cvn-infrastructure] - 10https://gerrit.wikimedia.org/r/1248441 (owner: 10Mvolz) [14:25:38] 06cloud-services-team, 10Cloud-VPS, 07Upstream: openstack flamingo: "'enabled' is a required property" for LDAP-managed users - https://phabricator.wikimedia.org/T416483#11688014 (10Andrew) p:05Medium→03Low This is resolved in codfw1dev. Task is still open because I'm trying to track (and fix?) the issue... [14:27:01] 10Cloud-VPS (Project-requests), 10Lingua-Libre, 03Hackathon-Northwestern-Europe-2026: Request creation of lingualibre VPS project - https://phabricator.wikimedia.org/T419182#11688022 (10Andrew) +1 [14:30:32] 10Cloud-VPS (Project-requests), 10Lingua-Libre, 03Hackathon-Northwestern-Europe-2026: Request creation of lingualibre VPS project - https://phabricator.wikimedia.org/T419182#11688032 (10dcaro) [14:31:11] 10Cloud-VPS (Project-requests), 10Lingua-Libre, 03Hackathon-Northwestern-Europe-2026: Request creation of lingualibre VPS project - https://phabricator.wikimedia.org/T419182#11688035 (10dcaro) 05Open→03In progress [14:31:19] 10Cloud-VPS (Project-requests), 10Lingua-Libre, 03Hackathon-Northwestern-Europe-2026: Request creation of lingualibre VPS project - https://phabricator.wikimedia.org/T419182#11688037 (10dcaro) a:03dcaro [14:33:38] !log dcaro@cloudcumin1001 lingualibre START - Cookbook wmcs.vps.create_project for project lingualibre in eqiad1 (T419182) [14:33:40] dcaro@cloudcumin1001: Unknown project "lingualibre" [14:33:40] T419182: Request creation of lingualibre VPS project - https://phabricator.wikimedia.org/T419182 [14:34:24] (03update) 10group_199_bot_f98be072172e323ae6d1441939d3e461: projects: added project lingualibre [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/299 (https://phabricator.wikimedia.org/T419182) [14:34:30] (03open) 10group_199_bot_f98be072172e323ae6d1441939d3e461: projects: added project lingualibre [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/299 (https://phabricator.wikimedia.org/T419182) [14:34:56] (03update) 10raymond-ndibe: [status] make job status an enum, with clearly defined states [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/208 (https://phabricator.wikimedia.org/T401172) [14:35:55] (03approved) 10dcaro: projects: added project lingualibre [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/299 (https://phabricator.wikimedia.org/T419182) (owner: 10group_199_bot_f98be072172e323ae6d1441939d3e461) [14:36:04] (03merge) 10dcaro: projects: added project lingualibre [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/299 (https://phabricator.wikimedia.org/T419182) (owner: 10group_199_bot_f98be072172e323ae6d1441939d3e461) [14:37:14] !log dcaro@cloudcumin1001 lingualibre END (PASS) - Cookbook wmcs.vps.create_project (exit_code=0) for project lingualibre in eqiad1 (T419182) [14:37:14] dcaro@cloudcumin1001: Unknown project "lingualibre" [14:38:21] (03update) 10raymond-ndibe: [status] make job status an enum, with clearly defined states [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/208 (https://phabricator.wikimedia.org/T401172) [14:40:09] 10Cloud-VPS (Project-requests), 10Lingua-Libre, 03Hackathon-Northwestern-Europe-2026, 13Patch-For-Review: Request creation of lingualibre VPS project - https://phabricator.wikimedia.org/T419182#11688089 (10dcaro) 05In progress→03Resolved @Yug I'm not certain what the 50GB you requested is of (Ram/v... [14:51:31] 10Cloud-VPS (Project-requests): Request creation of azwikimedia VPS project - https://phabricator.wikimedia.org/T417736#11688139 (10Nemoralis) @dcaro, yes please. I still want to go with building my own infra from scratch, I am OK with that [14:56:46] 06cloud-services-team, 10Tools: [superyetkin] Add primary key to visited_pages_agg - https://phabricator.wikimedia.org/T419115#11688158 (10Superyetkin) I am planning to manually partition the table by year and drop it. Thanks for the update. [14:57:42] 06cloud-services-team, 10Tools: [superyetkin] Add primary key to visited_pages_agg - https://phabricator.wikimedia.org/T419115#11688163 (10fnegri) Excellent, thank you! [14:58:57] (03update) 10taavi: kubernetes: Adapt Ingress scheduling alerts for Istio Gateway [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/57 (https://phabricator.wikimedia.org/T419231) [15:05:28] (03approved) 10filippo: istio-gateway: Configure pod sizing and scheduling [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1171 (https://phabricator.wikimedia.org/T419231) (owner: 10taavi) [15:08:12] (03update) 10taavi: istio-gateway: Configure pod sizing and scheduling [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1171 (https://phabricator.wikimedia.org/T419231) [15:08:38] (03update) 10raymond-ndibe: [status] make job status an enum, with clearly defined states [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/208 (https://phabricator.wikimedia.org/T401172) [15:08:58] (03merge) 10taavi: istio-gateway: Configure pod sizing and scheduling [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1171 (https://phabricator.wikimedia.org/T419231) [15:09:00] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway [15:09:00] !log taavi@cloudcumin1001 toolsbeta END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component istio-gateway [15:09:22] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway [15:09:35] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway [15:13:39] (03merge) 10taavi: kubernetes: Adapt Ingress scheduling alerts for Istio Gateway [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/57 (https://phabricator.wikimedia.org/T419231) [15:18:23] (03merge) 10taavi: kubernetes: Provision HTTPRoute resources [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/97 (https://phabricator.wikimedia.org/T418349) [15:22:06] (03update) 10dcaro: gitlab: move to trixie images [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/12 [15:22:17] (03open) 10taavi: d/changelog: bump to 0.103.19 [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/98 [15:22:21] (03update) 10taavi: d/changelog: bump to 0.103.19 [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/98 [15:25:10] (03merge) 10taavi: d/changelog: bump to 0.103.19 [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/98 [15:29:49] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component toolforge-webservice-cli [15:29:57] !log taavi@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-webservice-cli [15:30:07] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli [15:31:28] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli [15:31:41] 10Toolforge (Toolforge iteration 26): [harbor,tools] Harbor object usage in S3 is steadily increasing - https://phabricator.wikimedia.org/T418528#11688329 (10dcaro) [15:31:53] 10Toolforge (Toolforge iteration 26): [harbor,tools] Harbor object usage in S3 is steadily increasing - https://phabricator.wikimedia.org/T418528#11688332 (10dcaro) [15:32:40] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli [15:34:10] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli [15:35:59] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: Make webservice provision HTTPRoutes for each tool - https://phabricator.wikimedia.org/T418349#11688360 (10taavi) 05Open→03Resolved [15:39:55] FIRING: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [16:21:59] (03approved) 10dcaro: Move deployment to gateway nodes [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/88 (https://phabricator.wikimedia.org/T419231) (owner: 10taavi) [16:34:20] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Data-Services: [wikireplicas] Create views for new wiki kaiwiki - https://phabricator.wikimedia.org/T414240#11688741 (10fnegri) 05Stalled→03Open The views can now be created, I will do it later this week. [16:38:04] RESOLVED: ToolsNfsAlmostFull: Toolforge NFS is almost full - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsNfsAlmostFull - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsNfsAlmostFull [16:41:24] 10VPS-project-Phabricator, 06collaboration-services: phabricator.wmcloud.org account verification request: DSantamaria - https://phabricator.wikimedia.org/T419396#11688810 (10Dzahn) 05Open→03Resolved a:03Dzahn Done! I ran the "verify" command on dsantamaria@ [16:57:14] (03merge) 10taavi: Move deployment to gateway nodes [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/88 (https://phabricator.wikimedia.org/T419231) [16:59:39] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: api-gateway: bump to 0.0.89-20260309165737-26083d23 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1173 (https://phabricator.wikimedia.org/T419231) [16:59:48] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: api-gateway: bump to 0.0.89-20260309165737-26083d23 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1173 (https://phabricator.wikimedia.org/T419231) [16:59:59] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [17:04:41] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [17:08:13] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [17:13:01] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [17:15:52] (03merge) 10taavi: api-gateway: bump to 0.0.89-20260309165737-26083d23 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1173 (https://phabricator.wikimedia.org/T419231) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:16:14] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: Scale up Toolforge Istio deployment for production traffic levels - https://phabricator.wikimedia.org/T419231#11689010 (10taavi) 05Open→03Resolved [17:30:09] 10VPS-project-Extdist, 10CirrusSearch, 06Discovery-Search: latest Elastica release does not include Elastica\Client? - https://phabricator.wikimedia.org/T418469#11689071 (10Reedy) Nothing has changed in that sense; I imagine the extdist vm is doing something weird. Encountering an error or something building... [17:34:00] 10Cloud-VPS (Project-requests): Request creation of azwikimedia VPS project - https://phabricator.wikimedia.org/T417736#11689095 (10dcaro) a:03dcaro [17:34:07] 10Cloud-VPS (Project-requests): Request creation of azwikimedia VPS project - https://phabricator.wikimedia.org/T417736#11689096 (10dcaro) [17:34:10] 10Cloud-VPS (Project-requests): Request creation of azwikimedia VPS project - https://phabricator.wikimedia.org/T417736#11689098 (10dcaro) 05Open→03In progress [17:36:07] !log dcaro@cloudcumin1001 azwikimedia START - Cookbook wmcs.vps.create_project for project azwikimedia in eqiad1 (T417736) [17:36:09] dcaro@cloudcumin1001: Unknown project "azwikimedia" [17:36:10] T417736: Request creation of azwikimedia VPS project - https://phabricator.wikimedia.org/T417736 [17:36:50] (03open) 10group_199_bot_f98be072172e323ae6d1441939d3e461: projects: added project azwikimedia [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/300 (https://phabricator.wikimedia.org/T417736) [17:38:07] (03approved) 10dcaro: projects: added project azwikimedia [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/300 (https://phabricator.wikimedia.org/T417736) (owner: 10group_199_bot_f98be072172e323ae6d1441939d3e461) [17:38:14] (03merge) 10dcaro: projects: added project azwikimedia [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/300 (https://phabricator.wikimedia.org/T417736) (owner: 10group_199_bot_f98be072172e323ae6d1441939d3e461) [17:38:59] !log dcaro@cloudcumin1001 azwikimedia END (PASS) - Cookbook wmcs.vps.create_project (exit_code=0) for project azwikimedia in eqiad1 (T417736) [17:38:59] dcaro@cloudcumin1001: Unknown project "azwikimedia" [17:41:00] 10Cloud-VPS (Project-requests), 13Patch-For-Review: Request creation of azwikimedia VPS project - https://phabricator.wikimedia.org/T417736#11689132 (10dcaro) 05In progress→03Resolved Done :), don't forget to subscribe to the email list https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.w... [17:43:31] (03update) 10dcaro: k8s: Install Istio components [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/313 (https://phabricator.wikimedia.org/T418274) (owner: 10taavi) [17:44:15] (03approved) 10dcaro: k8s: Install Istio components [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/313 (https://phabricator.wikimedia.org/T418274) (owner: 10taavi) [18:06:54] (03update) 10taavi: k8s: Install Istio components [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/313 (https://phabricator.wikimedia.org/T418274) [18:07:00] (03merge) 10taavi: k8s: Install Istio components [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/313 (https://phabricator.wikimedia.org/T418274) [18:10:45] 06cloud-services-team, 10Toolforge, 06Privacy Engineering: Add Content-Security-Policy header enforcing 3rd party web interaction restrictions to proxy responses - https://phabricator.wikimedia.org/T130748#11689308 (10dcaro) [18:50:03] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Persistence, and 4 others: Set up x1 replication to an-redacteddb1001 - https://phabricator.wikimedia.org/T407485#11689496 (10taavi) This is causing some issues with the wiki replicas account maintenance tooling given that the x1 port is no... [19:53:37] 10Cloud-VPS (Quota-requests): Quota increases for gitlab-runners - https://phabricator.wikimedia.org/T418813#11689829 (10dduvall) Thanks, @Andrew >>! In T418813#11675877, @Andrew wrote: > Hey folks, sorry about the not-very-coherent response on this. The bottom line is that compute+storage resources are not an... [20:13:32] 10Cloud-VPS (Quota-requests): Quota increases for gitlab-runners - https://phabricator.wikimedia.org/T418813#11689880 (10Andrew) >>! In T418813#11689829, @dduvall wrote: > Thanks, @Andrew > >>>! In T418813#11675877, @Andrew wrote: >> Hey folks, sorry about the not-very-coherent response on this. The bottom lin... [20:24:25] 10VPS-project-Extdist, 10CirrusSearch, 06Discovery-Search: latest Elastica release does not include Elastica\Client? - https://phabricator.wikimedia.org/T418469#11689935 (10Nimworking) >>! In T418469#11688882, @EBernhardson wrote: > @Reedy It looks like in 1.42 the vendor/ directory was included, but in 43-... [20:44:39] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: cloudgw2004-dev service implementation - https://phabricator.wikimedia.org/T418765#11690019 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin2002 for host cloudgw2004-dev.codfw.wmnet with OS trixie [21:22:16] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: cloudgw2004-dev service implementation - https://phabricator.wikimedia.org/T418765#11690105 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin2002 for host cloudgw2004-dev.codfw.wmnet with OS trixie executed with errors:... [21:22:39] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: cloudgw2004-dev service implementation - https://phabricator.wikimedia.org/T418765#11690106 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin2002 for host cloudgw2004-dev.codfw.wmnet with OS trixie [21:28:03] 10VPS-project-Extdist, 10CirrusSearch, 06Discovery-Search: latest Elastica release does not include Elastica\Client? - https://phabricator.wikimedia.org/T418469#11690145 (10Reedy) [21:32:32] 10VPS-project-Extdist, 10CirrusSearch, 06Discovery-Search: latest Elastica release does not include Elastica\Client? - https://phabricator.wikimedia.org/T418469#11690156 (10Reedy) I imagine this explains it: {F72783300 size=full} [21:36:40] 10VPS-project-Extdist, 10CirrusSearch, 06Discovery-Search: latest Elastica release does not include Elastica\Client? - https://phabricator.wikimedia.org/T418469#11690170 (10Reedy) `lang=bash reedy@extdist-06:~$ df -h Filesystem Size Used Avail Use% Mounted on udev 2.0G 0 2.0G 0% /dev... [21:38:09] 10VPS-project-Extdist, 10CirrusSearch, 06Discovery-Search: latest Elastica release does not include Elastica\Client? - https://phabricator.wikimedia.org/T418469#11690184 (10Reedy) Another instance of {T253588} ` root@extdist-06:/# ls -al /var/log/extdist -rw-r--r-- 1 extdist extdist 12295168000 Mar 9 21:14... [21:46:58] 10VPS-project-Extdist, 10CirrusSearch, 06Discovery-Search: latest Elastica release does not include Elastica\Client? - https://phabricator.wikimedia.org/T418469#11690213 (10Reedy) This *should* fix itself overnight when(ever) the nightly jobs run to build the latest tarballs... [21:47:30] RESOLVED: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance extdist-06 in project extdist - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [22:03:12] 06cloud-services-team, 10Cloud-VPS: cloudgw2004-dev service implementation - https://phabricator.wikimedia.org/T418765#11690241 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin2002 for host cloudgw2004-dev.codfw.wmnet with OS trixie completed: - cloudgw2004-dev (**WARN**) -... [22:05:11] 10VPS-project-Extdist, 13Patch-For-Review: extdist is not rotating logs - https://phabricator.wikimedia.org/T253588#11690262 (10taavi) a:03taavi [23:01:02] 10VPS-project-Codesearch: not getting matches in modules/admin/data/data.yaml - https://phabricator.wikimedia.org/T419484 (10jeremyb) 03NEW