[00:31:55] <wmcs-alerts>	 FIRING: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity
[01:27:29] <wmcs-alerts>	 FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance extdist-06 in project extdist   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun
[02:16:57] <wikibugs>	 (03update) 10raymond-ndibe: logs-api,logs: test since, until params for logs-api and jobs-api log endpoint [repos/cloud/toolforge/toolforge-deploy] (add_logs_api_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1141 (https://phabricator.wikimedia.org/T400917)
[02:17:56] <wikibugs>	 (03update) 10raymond-ndibe: logs-api,logs: test since, until params for logs-api and jobs-api log endpoint [repos/cloud/toolforge/toolforge-deploy] (add_logs_api_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1141 (https://phabricator.wikimedia.org/T400917)
[02:19:32] <wikibugs>	 (03update) 10raymond-ndibe: logs: test since, until params for jobs-api logs [repos/cloud/toolforge/toolforge-deploy] (add_logs_api_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1141 (https://phabricator.wikimedia.org/T400917)
[02:20:17] <wikibugs>	 (03update) 10raymond-ndibe: logs-api: add more logs tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1140 (https://phabricator.wikimedia.org/T418326)
[02:21:26] <wikibugs>	 (03update) 10raymond-ndibe: [cli] support since and until params in logs [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/145 (https://phabricator.wikimedia.org/T400917)
[02:21:52] <wikibugs>	 (03update) 10raymond-ndibe: logs: test since, until params for jobs-api logs [repos/cloud/toolforge/toolforge-deploy] (add_logs_api_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1141 (https://phabricator.wikimedia.org/T400917)
[02:22:18] <wikibugs>	 (03update) 10raymond-ndibe: logs: test since, until params for jobs-api logs [repos/cloud/toolforge/toolforge-deploy] (add_logs_api_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1141 (https://phabricator.wikimedia.org/T400917)
[02:22:36] <wikibugs>	 (03update) 10raymond-ndibe: [cli] support since and until params in logs [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/145 (https://phabricator.wikimedia.org/T400917)
[02:49:08] <wikibugs>	 (03update) 10raymond-ndibe: logs: test since, until params for jobs-api logs [repos/cloud/toolforge/toolforge-deploy] (add_logs_api_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1141 (https://phabricator.wikimedia.org/T400917)
[05:41:55] <wmcs-alerts>	 RESOLVED: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity
[05:44:17] <wikibugs>	 (03update) 10legoktm: Require OAuth login to use the tool [toolforge-repos/otd-helper] - 10https://gitlab.wikimedia.org/toolforge-repos/otd-helper/-/merge_requests/4
[06:02:36] <wikibugs>	 (03merge) 10legoktm: Require OAuth login to use the tool [toolforge-repos/otd-helper] - 10https://gitlab.wikimedia.org/toolforge-repos/otd-helper/-/merge_requests/4
[06:29:55] <wmcs-alerts>	 FIRING: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity
[08:25:34] <jinxer-wm>	 FIRING: DiskSpace: Disk space cloudbackup1004:9100:/srv 6.972% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[09:08:07] <wm-bot2>	 !log tools.cluebotng-review Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22756671151 (https://github.com/cluebotng/component-configs/commits/397fc33968a3c4795b97b1791a0b991ebeb81430)
[09:08:09] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL
[11:21:32] <wikibugs>	 10Tool-curator: Curator: Allow the upload of failed files from image sequences uploaded by other users - https://phabricator.wikimedia.org/T419217 (10PantheraLeo1359531) 03NEW
[11:22:50] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component istio-system
[11:23:27] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-system
[11:25:24] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway
[11:25:36] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway
[11:27:40] <wikibugs>	 (03open) 10taavi: d/changelog: bump to 1.6.13 [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/90 (https://phabricator.wikimedia.org/T408783 https://phabricator.wikimedia.org/T418349)
[11:27:42] <wikibugs>	 (03update) 10taavi: d/changelog: bump to 1.6.13 [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/90 (https://phabricator.wikimedia.org/T408783 https://phabricator.wikimedia.org/T418349)
[11:30:12] <wikibugs>	 (03merge) 10taavi: d/changelog: bump to 1.6.13 [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/90 (https://phabricator.wikimedia.org/T408783 https://phabricator.wikimedia.org/T418349)
[11:32:47] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
[11:32:49] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
[11:33:28] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
[11:45:56] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
[11:46:27] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
[11:46:52] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
[12:18:03] <wikibugs>	 06cloud-services-team, 10Toolforge, 13Patch-For-Review: Pick and install a Gateway API implementation for Toolforge - https://phabricator.wikimedia.org/T418274#11681378 (10taavi) 05Open→03Resolved
[12:25:49] <jinxer-wm>	 FIRING: DiskSpace: Disk space cloudbackup1004:9100:/srv 6.44% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[12:28:33] <wikibugs>	 06cloud-services-team, 10Toolforge: Scale up Toolforge Istio deployment for production traffic levels - https://phabricator.wikimedia.org/T419231 (10taavi) 03NEW
[12:29:11] <wikibugs>	 (03PS1) 10Majavah: wmcs_libs: k8s: Add new 'gateway' worker type [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1248798 (https://phabricator.wikimedia.org/T419231)
[12:31:52] <wikibugs>	 06cloud-services-team, 10Toolforge, 13Patch-For-Review: Scale up Toolforge Istio deployment for production traffic levels - https://phabricator.wikimedia.org/T419231#11681450 (10taavi)
[12:31:54] <wikibugs>	 06cloud-services-team, 10Toolforge, 13Patch-For-Review: Pick and install a Gateway API implementation for Toolforge - https://phabricator.wikimedia.org/T418274#11681451 (10taavi)
[12:32:05] <wikibugs>	 06cloud-services-team, 10Toolforge, 13Patch-For-Review: Scale up Toolforge Istio deployment for production traffic levels - https://phabricator.wikimedia.org/T419231#11681452 (10taavi) p:05Triage→03High
[12:39:04] <wikibugs>	 10VPS-project-Wikistats: Add urwikisource to wikistats - https://phabricator.wikimedia.org/T415979#11681479 (10Dzahn) 05Open→03Resolved a:03Dzahn ` MariaDB [wikistats]> insert into wikisources (prefix, lang, loclang, method) select prefix,lang,loclang,method from wikipedias where prefix="ur"; Query OK,...
[12:42:37] <wikibugs>	 10VPS-project-Wikistats: Add kaiwiki to wikistats - https://phabricator.wikimedia.org/T414242#11681496 (10Dzahn) 05Stalled→03Resolved a:03Dzahn ` MariaDB [wikistats]> insert into wikipedias (prefix, lang, loclang, method) values ('kai','Karai-Karai','&#x42;&ograve;&#x6f; Karai-Karai',8); Query OK, 1 ro...
[13:51:03] <wikibugs>	 06cloud-services-team, 10Toolforge, 06All-and-every-Wikisource: Page Game script/util hosted at toolforge does not recieve( or cannot access) page scan thumbnails - https://phabricator.wikimedia.org/T419239 (10ShakespeareFan00) 03NEW
[13:54:27] <wikibugs>	 06cloud-services-team, 10Toolforge, 06All-and-every-Wikisource: Page Game script/util hosted at toolforge does not recieve( or cannot access) page scan thumbnails - https://phabricator.wikimedia.org/T419239#11681762 (10ShakespeareFan00) I have my suspicions that this is down to a non-standard thumbnail size...
[13:55:17] <wikibugs>	 10Tools, 06All-and-every-Wikisource: Page Game script/util hosted at toolforge does not recieve( or cannot access) page scan thumbnails - https://phabricator.wikimedia.org/T419239#11681771 (10taavi)
[14:00:34] <jinxer-wm>	 RESOLVED: DiskSpace: Disk space cloudbackup1004:9100:/srv 6.987% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[14:02:05] <wikibugs>	 10Cloud Services Proposals, 06cloud-services-team (FY2025/2026-Q3-Q4), 10Data-Services, 06Data-Persistence, and 2 others: Decision request - Who runs wikireplicas cookbooks - https://phabricator.wikimedia.org/T382607#11681882 (10Gehel)
[14:02:26] <wikibugs>	 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Engineering-Radar, and 3 others: Create wiki replicas views for globaljsonlinks tables - https://phabricator.wikimedia.org/T387419#11681884 (10Gehel)
[14:03:10] <wikibugs>	 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Persistence, and 3 others: Set up x1 replication to Wiki Replicas - https://phabricator.wikimedia.org/T395881#11681904 (10Gehel)
[14:13:35] <wikibugs>	 10Tools, 06All-and-every-Wikisource, 06Community-Tech, 10Wikimedia OCR: Wikisource OCR UI supplies a non-standard thumbnail size to the OCR tool hosted on the cloud, - https://phabricator.wikimedia.org/T419246 (10ShakespeareFan00) 03NEW
[14:13:47] <wikibugs>	 10Tools, 06All-and-every-Wikisource, 06Community-Tech, 10Wikimedia OCR: Wikisource OCR UI supplies a non-standard thumbnail size to the OCR tool hosted on the cloud, - https://phabricator.wikimedia.org/T419246#11682141 (10ShakespeareFan00)
[14:14:05] <wikibugs>	 10Tools, 06All-and-every-Wikisource, 06Community-Tech, 10Wikimedia OCR: Wikisource OCR UI supplies a non-standard thumbnail size to the OCR tool hosted on the cloud, - https://phabricator.wikimedia.org/T419246#11682144 (10ShakespeareFan00)
[14:28:36] <wikibugs>	 10Tools, 06All-and-every-Wikisource, 06Community-Tech, 10Wikimedia OCR: Wikisource OCR UI supplies a non-standard thumbnail size to the OCR tool hosted on the cloud, - https://phabricator.wikimedia.org/T419246#11682214 (10TheDJ)
[14:28:58] <wikibugs>	 10Tools, 06All-and-every-Wikisource: Page Game script/util hosted at toolforge does not recieve( or cannot access) page scan thumbnails - https://phabricator.wikimedia.org/T419239#11682218 (10TheDJ)
[14:42:25] <wikibugs>	 (03update) 10volans: Allow to force ARM kubernetes stack [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/303
[15:07:39] <wikibugs>	 10Cloud-VPS (Project-requests): Request creation of azwikimedia VPS project - https://phabricator.wikimedia.org/T417736#11682500 (10Andrew) So -- I reiterate that I don't think a cloud-vps project is the right way to handle things like this. If I were solving the issues you're solving, I would definitely look fo...
[15:43:25] <wikibugs>	 10Tools, 06All-and-every-Wikisource: ws-page-game.toolforge.org does not receive page scan thumbnails - https://phabricator.wikimedia.org/T419239#11682597 (10Aklapper)
[15:50:53] <wikibugs>	 10Tools, 07ContentSecurityPolicy: Adopt process (or migrations) to ensure tools and scripts acessing "external" resources not unduly impacted by CSP changes. - https://phabricator.wikimedia.org/T419237#11682642 (10ShakespeareFan00)
[16:41:26] <wikibugs>	 (03merge) 10volans: Allow to force ARM kubernetes stack [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/303
[17:11:36] <wikibugs>	 10Tool-inteGraality: Support explicit groupings - https://phabricator.wikimedia.org/T419059#11682950 (10JeanFred) Works, see example at https://www.wikidata.org/wiki/Wikidata:Tenth_Birthday/DACH_Culture_Contest/Lists/Culture/2
[17:14:34] <wikibugs>	 10Tool-inteGraality: Support explicit groupings - https://phabricator.wikimedia.org/T419059#11682959 (10JeanFred) Parameter documented at https://www.wikidata.org/wiki/Special:Diff/2469907080
[17:16:43] <wikibugs>	 10Tool-inteGraality: Support explicit groupings - https://phabricator.wikimedia.org/T419059#11682963 (10JeanFred) 05Open→03Resolved
[17:21:29] <wikibugs>	 (03update) 10pepepiton: Add missing information for author and work page [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/183 (owner: 10nurahwakili)
[17:27:07] <wikibugs>	 10Cloud-VPS (Project-requests): Request creation of lingualibre VPS project - https://phabricator.wikimedia.org/T419182#11682975 (10Yug)
[18:14:52] <wikibugs>	 10Cloud-VPS (Project-requests): Request creation of lingualibre VPS project - https://phabricator.wikimedia.org/T419182#11683081 (10Yug)
[18:44:55] <wmcs-alerts>	 RESOLVED: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity
[19:11:55] <wmcs-alerts>	 FIRING: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity
[19:36:55] <wmcs-alerts>	 RESOLVED: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity
[20:25:55] <wmcs-alerts>	 FIRING: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity
[20:46:10] <wmcs-alerts>	 RESOLVED: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity
[20:47:22] <wikibugs>	 10Tools, 06All-and-every-Wikisource: ws-page-game.toolforge.org does not receive page scan thumbnails (HTTP 429) - https://phabricator.wikimedia.org/T419239#11683654 (10Aklapper)
[20:47:40] <wmcs-alerts>	 FIRING: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity
[21:30:14] <wikibugs>	 (03open) 10ttaylor: Adding search functionality [toolforge-repos/wikipedia-tree-browser] - 10https://gitlab.wikimedia.org/toolforge-repos/wikipedia-tree-browser/-/merge_requests/1
[21:35:08] <wikibugs>	 (03update) 10ttaylor: Adding search functionality [toolforge-repos/wikipedia-tree-browser] - 10https://gitlab.wikimedia.org/toolforge-repos/wikipedia-tree-browser/-/merge_requests/1
[22:20:55] <wmcs-alerts>	 RESOLVED: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity
[22:51:55] <wmcs-alerts>	 FIRING: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity
[23:50:28] <wikibugs>	 10Tools, 06All-and-every-Wikisource, 06Community-Tech, 10Wikimedia OCR: Wikisource OCR UI supplies a non-standard thumbnail size to the OCR tool hosted on the cloud, - https://phabricator.wikimedia.org/T419246#11684058 (10Samwilson) This is happening because the Wikisource extension is using OpenSeadragon'...