[01:18:18] <wikibugs>	 10Tool-wsindex, 10Wikisource Reader App: Display extra information from Wikidata in Book Details Screen - https://phabricator.wikimedia.org/T406245#11498446 (10Muguro) I have opened another branch for this with the mentioned areas of concern, kindly check   https://github.com/cis-india/Wikisource-Reader/pull/72
[01:58:30] <wikibugs>	 06cloud-services-team, 10Striker, 10Phabricator: Ensure that auto-created Phabricator projects for Toolforge tools are always editable by that tool's maintainers - https://phabricator.wikimedia.org/T412607#11498489 (10bd808) >>! In T412607#11462318, @A_smart_kitten wrote: > What about in a hypothetical case...
[02:59:59] <wikibugs>	 06cloud-services-team, 10Striker, 10Phabricator: Ensure that auto-created Phabricator projects for Toolforge tools are always editable by that tool's maintainers - https://phabricator.wikimedia.org/T412607#11498533 (10Pppery) Ancient but https://wikitech.wikimedia.org/wiki/User_talk:Dispenser#Blocked seems t...
[08:13:49] <wikibugs>	 (03PS1) 10Dzahn: remove phan-taint-check-plugin from config, indexed twice [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1224014 (https://phabricator.wikimedia.org/T413879)
[09:00:03] <wikibugs>	 (03CR) 10A smart kitten: "dupe of I915383af74a2bb9e493c3dba2725221d10f00947?" [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1224014 (https://phabricator.wikimedia.org/T413879) (owner: 10Dzahn)
[09:09:19] <wikibugs>	 (03Abandoned) 10Dzahn: remove phan-taint-check-plugin from config, indexed twice [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1224014 (https://phabricator.wikimedia.org/T413879) (owner: 10Dzahn)
[09:33:41] <wikibugs>	 06cloud-services-team (FY2025/2026-Q1-Q2): [ceph] 2025-08-27 ceph outage when bringing in a big osd host all at once (cloudcephosd1048) - https://phabricator.wikimedia.org/T403043#11498861 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi After a bit more experience with ceph I'm confident this happened...
[09:38:34] <wikibugs>	 10VPS-project-Codesearch: codesearch vm running out of inode again - https://phabricator.wikimedia.org/T413739#11498867 (10Dzahn) We already added logrotate for this, but since that is not enough, just turned it off:  ` sudo /usr/sbin/accton off Turning off process accounting. `
[09:39:34] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 06SRE: ceph: test and decide 1 network interface setup - https://phabricator.wikimedia.org/T325531#11498871 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi Will be done as part of {T399180}
[09:41:57] <wikibugs>	 10VPS-project-Codesearch: codesearch vm running out of inode again - https://phabricator.wikimedia.org/T413739#11498877 (10Dzahn) weird file names though. maybe the logrotate config had a problem?  I think we can just keep it off though.
[09:42:26] <wikibugs>	 10VPS-project-Codesearch: codesearch vm running out of inode again - https://phabricator.wikimedia.org/T413739#11498888 (10Dzahn) ` cat /etc/logrotate.d/pacct  # SPDX-License-Identifier: Apache-2.0 /var/log/account/* {     daily     missingok     compress     delaycompress     notifempty     copytruncate }  `
[10:00:21] <logmsgbot_cloud>	 !log fnegri@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
[10:00:42] <logmsgbot_cloud>	 !log fnegri@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch
[10:03:04] <wikibugs>	 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS: [tofu-infra] "tofu plan" failing in codfw - https://phabricator.wikimedia.org/T410265#11498975 (10fnegri) p:05Medium→03High Raising to "High" as this is preventing us from using tofu in codfw, and is also causing an error when running the cookbook `wm...
[10:17:18] <wikibugs>	 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS: [tofu-infra] "tofu plan" failing in codfw - https://phabricator.wikimedia.org/T410265#11499019 (10fnegri)
[10:17:42] <wikibugs>	 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS: [tofu-infra] "tofu plan" failing in codfw - https://phabricator.wikimedia.org/T410265#11499022 (10fnegri)
[10:23:48] <wikibugs>	 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge, 13Patch-For-Review: [toolforge.infra] run and monitor our own sample tools - https://phabricator.wikimedia.org/T357977#11499037 (10fgiunchedi)
[10:23:59] <wikibugs>	 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge: Toolforge on bare metal POC - https://phabricator.wikimedia.org/T407296#11499041 (10fgiunchedi)
[10:24:08] <wikibugs>	 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge: Start collecting uptime metrics for toolforge - https://phabricator.wikimedia.org/T412349#11499042 (10fgiunchedi)
[10:24:18] <wikibugs>	 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS: metricsinfra-prometheus-3 using more space than metricsinfra-prometheus-2 - https://phabricator.wikimedia.org/T412926#11499044 (10fgiunchedi)
[10:24:29] <wikibugs>	 06cloud-services-team (FY2025/2026-Q3-Q4): SystemdUnitDown and SystemdUnitDownForLong - ditch the latter (?) - https://phabricator.wikimedia.org/T411193#11499045 (10fgiunchedi)
[10:24:51] <wikibugs>	 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS: Increase signal of Tofuinfratest alerts - https://phabricator.wikimedia.org/T413185#11499046 (10fgiunchedi)
[10:25:07] <wikibugs>	 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS, 10Data-Services: Plan to make clouddumps more resilient and easier to operate - https://phabricator.wikimedia.org/T411248#11499047 (10fgiunchedi)
[10:25:47] <wikibugs>	 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS, 13Patch-For-Review: Support nftables in Cloud VPS - https://phabricator.wikimedia.org/T411089#11499049 (10fgiunchedi)
[10:25:57] <wikibugs>	 06cloud-services-team (FY2025/2026-Q3-Q4), 06Infrastructure-Foundations, 10Puppet-Core, 13Patch-For-Review: Puppet types causing issues around nftables::service - https://phabricator.wikimedia.org/T411102#11499052 (10fgiunchedi)
[10:26:09] <wikibugs>	 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS (Debian Bullseye Deprecation): Migrate WMCS-managed NFS servers off of Bullseye - https://phabricator.wikimedia.org/T401812#11499056 (10fgiunchedi)
[10:26:20] <wikibugs>	 06cloud-services-team (FY2025/2026-Q3-Q4), 10Data-Services: maintain-dbusers: Use cloud-private to talk to NFS servers instead of proxies - https://phabricator.wikimedia.org/T392794#11499059 (10fgiunchedi)
[10:26:44] <wikibugs>	 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge: Improve detection of failing ssh to toolforge bastions - https://phabricator.wikimedia.org/T404054#11499060 (10fgiunchedi)
[10:27:11] <wikibugs>	 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge: [toolforge.infra] Replace Toolschecker alerts with Prometheus based ones - https://phabricator.wikimedia.org/T313030#11499064 (10fgiunchedi)
[10:27:55] <wikibugs>	 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS, 10Toolforge, 10Observability-Alerting, and 2 others: Move WMCS off of Icinga and introduce alertmanager - https://phabricator.wikimedia.org/T328502#11499067 (10fgiunchedi)
[10:27:56] <wikibugs>	 06cloud-services-team (FY2025/2026-Q3-Q4): Remove Icinga checks for Cloud VPS projects (not: infrastructure) - https://phabricator.wikimedia.org/T345983#11499068 (10fgiunchedi)
[10:28:17] <wikibugs>	 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS, 06SRE-OnFire, 13Patch-For-Review, 07Sustainability (Incident Followup): Add external meta-monitoring for metricsinfra - https://phabricator.wikimedia.org/T288053#11499069 (10fgiunchedi)
[10:39:32] <wikibugs>	 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS: Move Cloud VPS control plane alerting to alertmanager - https://phabricator.wikimedia.org/T345294#11499105 (10fgiunchedi)
[10:40:18] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 10Toolforge: Create alerts for bastion hosts - Usage and latency - https://phabricator.wikimedia.org/T186552#11499107 (10fgiunchedi) 05Open→03Declined Declining in favor of {T404054}
[10:41:01] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: nova-api seems to die after a while, complains of a full listen queue - https://phabricator.wikimedia.org/T354483#11499114 (10fgiunchedi) 05Open→03Invalid I don't think we've seen this recently, optimistically resolving
[10:42:15] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project: cookbook.ceph: allow adding hosts in batches - https://phabricator.wikimedia.org/T315663#11499119 (10fgiunchedi) 05Open→03Declined Given the higher density cloudcephosd hosts I don't think we can ad...
[10:43:28] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 10Puppet-Core: Investigate use of Puppet "environments" for per-project Puppet manifests - https://phabricator.wikimedia.org/T170370#11499126 (10fgiunchedi) 05Open→03Declined I don't think we'll be using puppet environments in cloud...
[10:52:33] <wikibugs>	 10Tool-curator: Integrate with Edit Groups - https://phabricator.wikimedia.org/T413962 (10DaxServer) 03NEW
[10:52:54] <wikibugs>	 10Tool-curator: Integrate with Edit Groups - https://phabricator.wikimedia.org/T413962#11499184 (10DaxServer) p:05Triage→03Low
[10:53:24] <wikibugs>	 10Tool-curator: Add geocoding info to Mapillary images - https://phabricator.wikimedia.org/T412708#11499188 (10DaxServer) p:05Medium→03Low
[10:53:28] <wikibugs>	 10Tool-curator: Integrate with Edit Groups - https://phabricator.wikimedia.org/T413962#11499189 (10DaxServer) p:05Low→03Medium
[10:55:40] <wikibugs>	 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS: VM metadata service slow response - https://phabricator.wikimedia.org/T410983#11499205 (10fgiunchedi)
[10:56:59] <wikibugs>	 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS: Fix pacct rotation properly everywhere - https://phabricator.wikimedia.org/T410410#11499207 (10fgiunchedi)
[10:58:35] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 10Ceph: cloudceph: automate nic names in profile::cloudceph::osd::hosts - https://phabricator.wikimedia.org/T405374#11499209 (10fgiunchedi) Once we have single-nic everywhere, and old cloudcephosd hosts are decom'd, we can ditch having to hardcode nic names
[10:58:46] <wikibugs>	 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS: Add fqdn input to instance-related wmcs cookbooks - https://phabricator.wikimedia.org/T404052#11499210 (10fgiunchedi)
[11:01:02] <wikibugs>	 06cloud-services-team (FY2025/2026-Q3-Q4), 10Horizon, 10Striker, 06serviceops, 06SRE: Move cloudweb to Ganeti VMs and repurpose the servers as wikikube nodes - https://phabricator.wikimedia.org/T392478#11499214 (10fgiunchedi)
[11:02:46] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: neutron: investigate using IRC conntrack helpers to improve IRC bots connectiviy - https://phabricator.wikimedia.org/T277549#11499217 (10fgiunchedi) 05Open→03Declined I don't think we've experienced irc bots connectivity problems lately at least
[11:04:21] <wikibugs>	 06cloud-services-team, 10Toolforge, 07Kubernetes: Extremely high latency over NFS between kubernetes node and bastion host - https://phabricator.wikimedia.org/T256426#11499223 (10fgiunchedi) 05Open→03Declined I'm not aware of recent issues with nfs latency on toolforge, declining though feel free to...
[11:05:49] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: can the db server 'maps-test-2' be deleted? - https://phabricator.wikimedia.org/T368233#11499226 (10fgiunchedi) 05Open→03Invalid db instance is gone
[11:06:20] <wikibugs>	 06cloud-services-team, 06Infrastructure-Foundations, 06SRE: ACPI kernel failure on debian installer last step - https://phabricator.wikimedia.org/T357896#11499230 (10fgiunchedi) 05Open→03Invalid I'm not aware of recent troubles with this problem
[11:06:29] <wikibugs>	 06cloud-services-team (FY2025/2026-Q3-Q4), 10Horizon, 10Striker, 06serviceops, 06SRE: Move cloudweb to Ganeti VMs and repurpose the servers as wikikube nodes - https://phabricator.wikimedia.org/T392478#11499233 (10taavi) See also: {T411783}
[11:07:57] <wikibugs>	 06cloud-services-team (FY2025/2026-Q3-Q4), 10Horizon: CloudVPS: VMs created with non-allowed characters in the hostname fail to be autosigned by puppet - https://phabricator.wikimedia.org/T192059#11499235 (10fgiunchedi)
[12:45:14] <wikibugs>	 (03CR) 10Filippo Giunchedi: [C:03+1] toolforge: k8s: prepare_upgrade: Check that functional tests pass [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1223633 (owner: 10Majavah)
[12:46:50] <wikibugs>	 (03CR) 10Filippo Giunchedi: toolforge: k8s: prepare_upgrade: Automatically set downtime (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1223634 (owner: 10Majavah)
[12:48:30] <wikibugs>	 (03CR) 10Majavah: [C:03+2] toolforge: k8s: prepare_upgrade: Check that functional tests pass [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1223633 (owner: 10Majavah)
[12:48:33] <wikibugs>	 (03CR) 10Majavah: [C:03+2] toolforge: k8s: prepare_upgrade: Automatically set downtime [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1223634 (owner: 10Majavah)
[12:49:10] <wikibugs>	 (03CR) 10Majavah: toolforge: k8s: prepare_upgrade: Automatically set downtime [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1223634 (owner: 10Majavah)
[12:50:08] <wikibugs>	 (03CR) 10Filippo Giunchedi: [C:03+1] toolforge: k8s: Fix check for first node to be upgraded [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1223652 (owner: 10Majavah)
[12:50:19] <wikibugs>	 (03CR) 10Filippo Giunchedi: [C:03+1] toolforge: k8s: Remind user about hostname being upgraded [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1223653 (owner: 10Majavah)
[12:50:51] <wikibugs>	 (03CR) 10Majavah: toolforge: k8s: prepare_upgrade: Automatically set downtime (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1223634 (owner: 10Majavah)
[12:50:53] <wikibugs>	 (03CR) 10Filippo Giunchedi: [C:03+1] toolforge: k8s: Retry other errors when polling for version upgrade (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1223656 (owner: 10Majavah)
[12:51:09] <wikibugs>	 (03CR) 10Filippo Giunchedi: [C:03+1] toolforge: k8s: worker: Fix cookbook path comments [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1223657 (owner: 10Majavah)
[12:51:19] <wikibugs>	 (03CR) 10Filippo Giunchedi: [C:03+1] toolforge: k8s: worker: Remove special handling for SGE bastion [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1223658 (owner: 10Majavah)
[12:51:38] <wikibugs>	 (03Merged) 10jenkins-bot: toolforge: k8s: prepare_upgrade: Check that functional tests pass [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1223633 (owner: 10Majavah)
[12:51:42] <wikibugs>	 (03CR) 10Filippo Giunchedi: [C:03+1] toolforge: k8s: prepare_upgrade: Automatically set downtime (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1223634 (owner: 10Majavah)
[12:52:02] <wikibugs>	 (03CR) 10Majavah: [C:03+2] toolforge: k8s: prepare_upgrade: Automatically set downtime [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1223634 (owner: 10Majavah)
[12:52:06] <wikibugs>	 (03Merged) 10jenkins-bot: toolforge: k8s: prepare_upgrade: Automatically set downtime [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1223634 (owner: 10Majavah)
[12:52:10] <wikibugs>	 (03CR) 10Majavah: [C:03+2] toolforge: k8s: Fix check for first node to be upgraded [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1223652 (owner: 10Majavah)
[12:52:17] <wikibugs>	 (03CR) 10Majavah: [C:03+2] toolforge: k8s: Remind user about hostname being upgraded [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1223653 (owner: 10Majavah)
[12:55:48] <wikibugs>	 (03Merged) 10jenkins-bot: toolforge: k8s: Fix check for first node to be upgraded [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1223652 (owner: 10Majavah)
[12:56:35] <wikibugs>	 (03Merged) 10jenkins-bot: toolforge: k8s: Remind user about hostname being upgraded [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1223653 (owner: 10Majavah)
[12:59:54] <wikibugs>	 (03PS2) 10Majavah: toolforge: k8s: Retry other errors when polling for version upgrade [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1223656
[12:59:55] <wikibugs>	 (03PS2) 10Majavah: toolforge: k8s: worker: Fix cookbook path comments [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1223657
[12:59:55] <wikibugs>	 (03PS2) 10Majavah: toolforge: k8s: worker: Remove special handling for SGE bastion [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1223658
[13:01:26] <wikibugs>	 (03CR) 10Majavah: [C:03+2] toolforge: k8s: Retry other errors when polling for version upgrade (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1223656 (owner: 10Majavah)
[13:01:35] <wikibugs>	 (03CR) 10Majavah: [C:03+2] toolforge: k8s: worker: Fix cookbook path comments [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1223657 (owner: 10Majavah)
[13:01:43] <wikibugs>	 (03CR) 10Majavah: [C:03+2] toolforge: k8s: worker: Remove special handling for SGE bastion [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1223658 (owner: 10Majavah)
[13:05:27] <wikibugs>	 (03Merged) 10jenkins-bot: toolforge: k8s: Retry other errors when polling for version upgrade [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1223656 (owner: 10Majavah)
[13:05:47] <wikibugs>	 (03Merged) 10jenkins-bot: toolforge: k8s: worker: Fix cookbook path comments [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1223657 (owner: 10Majavah)
[13:05:47] <wikibugs>	 (03Merged) 10jenkins-bot: toolforge: k8s: worker: Remove special handling for SGE bastion [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1223658 (owner: 10Majavah)
[13:07:05] <wikibugs>	 (03PS1) 10Majavah: toolforge: Prevent duplicate SALs for generic copy_to_registry [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1224068
[13:16:27] <wikibugs>	 06cloud-services-team, 06SRE, 10Wikimedia-Mailing-lists: aborrero@wikimedia.org still subscribed to ops@lists.wikimedia.org - https://phabricator.wikimedia.org/T413883#11499539 (10Andrew) 05Open→03Invalid The alerts stopped after a few hours, so likely he was auto-unsubscribed.
[13:50:58] <wikibugs>	 (03CR) 10FNegri: [C:03+1] toolforge: Prevent duplicate SALs for generic copy_to_registry [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1224068 (owner: 10Majavah)
[13:51:15] <wikibugs>	 (03CR) 10Majavah: [C:03+2] toolforge: Prevent duplicate SALs for generic copy_to_registry [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1224068 (owner: 10Majavah)
[13:55:03] <wikibugs>	 (03Merged) 10jenkins-bot: toolforge: Prevent duplicate SALs for generic copy_to_registry [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1224068 (owner: 10Majavah)
[13:59:03] <wikibugs>	 06cloud-services-team, 10Toolforge: Upgrade tools cluster to Kubernetes 1.31 - https://phabricator.wikimedia.org/T413797#11499762 (10taavi)
[14:52:22] <wikibugs>	 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS: metricsinfra-prometheus-3 using more space than metricsinfra-prometheus-2 - https://phabricator.wikimedia.org/T412926#11499954 (10fgiunchedi) The situation has normalized around Dec 17th in the sense that -3 blocks now match -2 in size. I'm still at a los...
[14:54:14] <wikibugs>	 (03CR) 10Jforrester: "check experimental" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1223178 (owner: 10L10n-bot)
[14:55:05] <wikibugs>	 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS: metricsinfra-prometheus-3 using more space than metricsinfra-prometheus-2 - https://phabricator.wikimedia.org/T412926#11499965 (10fgiunchedi) {F71462049}
[15:00:08] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Alert on metricsinfra prometheus storing a few days of metrics - https://phabricator.wikimedia.org/T413985 (10fgiunchedi) 03NEW
[15:14:07] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: cloud: neutron l3 agent: improve failover handling - https://phabricator.wikimedia.org/T268335#11500078 (10taavi) 05Open→03Resolved Closing as there are no more remaining subtasks.
[15:19:29] <wikibugs>	 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS: metricsinfra-prometheus-3 using more space than metricsinfra-prometheus-2 - https://phabricator.wikimedia.org/T412926#11500098 (10fgiunchedi) I run a `promtool tsdb analyze` (from prometheus 3 since the command for debian prometheus doesn't report highest...
[15:27:27] <wikibugs>	 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS: metricsinfra-prometheus-3 using more space than metricsinfra-prometheus-2 - https://phabricator.wikimedia.org/T412926#11500151 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi I'm not sure how to audit this further, my current hypothesis:  Someth...
[15:28:41] <wikibugs>	 10Cloud-Services, 07Epic: Nova-network to Neutron migration - https://phabricator.wikimedia.org/T167293#11500161 (10taavi)
[15:33:22] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Alert on metricsinfra prometheus storing a few days of metrics - https://phabricator.wikimedia.org/T413985#11500196 (10taavi) SGTM, with the team label and dashboard updated
[15:48:56] <wikibugs>	 (03merge) 10taavi: Autovoice sirenbot in -operations/-sre [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/33
[16:23:01] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Alert on metricsinfra prometheus storing a few days of metrics - https://phabricator.wikimedia.org/T413985#11500381 (10Volans) p:05Triage→03Medium
[16:43:34] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10WikiCite, 10Wikidata, 10Wikidata-Query-Service: Raise quota on wikiqlever so that an instance with 256 GB RAM and 3 x 4 TB SSD can be launched - https://phabricator.wikimedia.org/T413097#11500444 (10fnegri) > Can you assign 1 TB of disk and 32 GB of...
[16:51:31] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10WikiCite, 10Wikidata, 10Wikidata-Query-Service: Raise quota on wikiqlever so that an instance with 256 GB RAM and 3 x 4 TB SSD can be launched - https://phabricator.wikimedia.org/T413097#11500474 (10Physikerwelt) >>! In T413097#11500444, @fnegri wrot...
[16:58:06] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10WikiCite, 10Wikidata, 10Wikidata-Query-Service: Raise quota on wikiqlever so that an instance with 256 GB RAM and 3 x 4 TB SSD can be launched - https://phabricator.wikimedia.org/T413097#11500503 (10Volans) @Physikerwelt Do you need also the VCPU lim...
[17:05:32] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10WikiCite, 10Wikidata, 10Wikidata-Query-Service: Raise quota on wikiqlever so that an instance with 256 GB RAM and 3 x 4 TB SSD can be launched - https://phabricator.wikimedia.org/T413097#11500555 (10Volans) >>! In T413097#11500474, @Physikerwelt wrot...
[17:07:02] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10WikiCite, 10Wikidata, 10Wikidata-Query-Service: Raise quota on wikiqlever so that an instance with 256 GB RAM and 3 x 4 TB SSD can be launched - https://phabricator.wikimedia.org/T413097#11500559 (10Physikerwelt) The flavor g4.cores8.ram32.disk20 wou...
[17:07:52] <logmsgbot_cloud>	 !log volans@cloudcumin1001 wikiqlever START - Cookbook wmcs.openstack.quota_increase by 944 gigabytes, 16384 ram (T413097)
[17:07:56] <stashbot>	 T413097: Raise quota on wikiqlever so that an instance with 256 GB RAM and 3 x 4 TB SSD can be launched - https://phabricator.wikimedia.org/T413097
[17:08:00] <logmsgbot_cloud>	 !log volans@cloudcumin1001 wikiqlever END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) by 944 gigabytes, 16384 ram (T413097)
[17:09:49] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10WikiCite, 10Wikidata, 10Wikidata-Query-Service: Raise quota on wikiqlever so that an instance with 256 GB RAM and 3 x 4 TB SSD can be launched - https://phabricator.wikimedia.org/T413097#11500566 (10Volans) Limits increased to 1TB of disk and 32G of...
[20:21:40] <wikibugs>	 10Tool-wikicontest, 06Indic-TechCom: Add Alembic for Database Migrations - https://phabricator.wikimedia.org/T413623#11501178 (10Agamyasamuel)
[20:59:09] <wikibugs>	 10Cloud-VPS (Project-requests), 10Wikidata, 10Wikidata-Query-Service: Request creation of wikiqlever VPS project - https://phabricator.wikimedia.org/T377655#11501311 (10Physikerwelt) We now have 32GB RAM and 1TB disk, which I attached to the qlever1 instance, which I resized to 32GB memory following http...
[21:00:32] <wikibugs>	 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge: Start collecting uptime metrics for toolforge - https://phabricator.wikimedia.org/T412349#11501314 (10bd808) We were tracking uptime things with watchmouse in the olden days. I vaguely remember us deciding that a replacement wasn't really valuable when wa...
[21:10:32] <wikibugs>	 10Tool-gitlab-account-approval: Consider extending trust to users with an email on the zuul trust list - https://phabricator.wikimedia.org/T401488#11501345 (10bd808) >>! In T401488#11451434, @matmarex wrote: >>>! In T401488#11071022, @bd808 wrote: >> This could be reasonably considered blocked in the near term b...
[21:30:15] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10WikiCite, 10Wikidata, 10Wikidata-Query-Service: Raise quota on wikiqlever so that an instance with 256 GB RAM and 3 x 4 TB SSD can be launched - https://phabricator.wikimedia.org/T413097#11501400 (10Physikerwelt) Thank you. That worked.  Do you happe...
[23:00:31] <wikibugs>	 10Cloud-VPS (Project-requests), 10Wikidata, 10Wikidata-Query-Service: Request creation of wikiqlever VPS project - https://phabricator.wikimedia.org/T377655#11501746 (10Physikerwelt) Preparing qlever data ` physikerwelt@qlever1:/srv$ sudo mkdir qlever physikerwelt@qlever1:/srv$ sudo chown physikerwelt ql...
[23:10:57] <wikibugs>	 10Cloud-VPS (Project-requests), 10Wikidata, 10Wikidata-Query-Service: Request creation of wikiqlever VPS project - https://phabricator.wikimedia.org/T377655#11501778 (10Physikerwelt) running the same step from another instance (with access to nfs) for comparison ` /data/scratch/qlever$ time docker run -i...