[00:01:51] 10Wikibugs: Allow more folks to submit/merge config patches - https://phabricator.wikimedia.org/T364436#9779405 (10bd808) 05Open→03In progress p:05Triage→03Medium a:03bd808 [00:10:58] 10Wikibugs: Allow more folks to submit/merge config patches - https://phabricator.wikimedia.org/T364436#9779415 (10bd808) 05In progress→03Resolved Changes made: * {nav settings > repository > protected branches > main}: allowed to merge == developers + maintainers * {nav manage > members > invite a group... [00:14:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [00:24:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [01:44:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [01:54:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:12:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:22:41] RESOLVED: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:42:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:52:41] RESOLVED: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:43:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:53:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:27:37] 10Toolforge (Toolforge iteration 09): [toolforge] Investigate authentication - https://phabricator.wikimedia.org/T363983#9780000 (10fnegri) I think using idp.w.o is my favourite solution, as it can potentially be a true "single" sign-on that all applications can rely on. Using CAS without any extra library is pr... [09:32:13] 10Toolforge: [jobs-api] Prefix all endpoints with `/tool/` - https://phabricator.wikimedia.org/T363346#9780013 (10fnegri) [09:56:08] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS: Migrate eqiad1 hypervisors to Neutron OVS agent - https://phabricator.wikimedia.org/T364457 (10taavi) 03NEW [09:56:40] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS: Create new g4 flavors to support hypervisor migration from Linuxbridge to OVS Neutron agents - https://phabricator.wikimedia.org/T364458 (10taavi) 03NEW [09:57:26] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS: Migrate eqiad1 cloudnets to Neutron OVS agent - https://phabricator.wikimedia.org/T364459 (10taavi) 03NEW [10:15:23] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS: Deploy OVS test setup in codfw1dev - https://phabricator.wikimedia.org/T358761#9780138 (10taavi) To move an instance from linuxbridge to OVS, the following UPDATE needs to be manually run on the database: ` mysql:root@localhost [neutron]> UPDATE ml2_port_... [10:25:45] 06cloud-services-team, 10Toolforge: [toolforge,storage] Provide per-tool access to cloud-vps object storage - https://phabricator.wikimedia.org/T358496#9780156 (10fnegri) >> A bit of a non-related question, should we create app credentials for every toolforge user too? > I don't think that's needed; I would en... [10:56:40] 14Grid-Engine-to-K8s-Migration, 10Tools, 06All-and-every-Wikisource, 03Wikimedia-Hackathon-2024: Migrate phetools from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319965#9780256 (10Epigeneticist) >>! In T319965#9773286, @Soda wrote: >>>! In T319965#9773263, @Epigenetic... [11:03:46] 10Cloud-VPS, 06Infrastructure-Foundations, 10Spicerack, 10SRE-tools: spicerack.puppet.PuppetHostsError: Unable to find CSR fingerprints for all hosts, detected errors are: Another puppet instance is already running and the waitforlock setting is set to 0; e... - https://phabricator.wikimedia.org/T361218#9780281 [11:42:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [11:42:48] 10Wikibugs: Allow more folks to submit/merge config patches - https://phabricator.wikimedia.org/T364436#9780428 (10Jdforrester-WMF) Thanks! [11:52:41] RESOLVED: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:23:46] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge: Decision Request - Toolforge policy agent enforcement model - https://phabricator.wikimedia.org/T362872#9780530 (10aborrero) More thoughts on validation vs mutation: 1: In the validation approach, one could consider that we are 'leaking' a Tool... [12:29:35] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge: Decision Request - Toolforge policy agent enforcement model - https://phabricator.wikimedia.org/T362872#9780536 (10aborrero) Another data point. Kyverno has native support for mutating existing resources: https://kyverno.io/docs/writing-policie... [12:31:29] 10Tool-spacemedia, 14Toolforge Build Service: Toolforge logs -f almost always ends in error - https://phabricator.wikimedia.org/T364468 (10Don-vip) 03NEW [12:31:55] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge: Decision Request - Toolforge policy agent enforcement model - https://phabricator.wikimedia.org/T362872#9780555 (10aborrero) [12:33:57] 10Tool-spacemedia, 10Toolforge: Toolforge logs -f almost always ends in error - https://phabricator.wikimedia.org/T364468#9780561 (10Don-vip) [12:35:07] 10Tool-spacemedia, 10Toolforge: Toolforge jobs logs -f almost always ends in error - https://phabricator.wikimedia.org/T364468#9780562 (10Don-vip) [12:46:00] 10Tool-spacemedia, 10Toolforge: Toolforge jobs logs -f almost always ends in error - https://phabricator.wikimedia.org/T364468#9780596 (10Don-vip) Seems to be a duplicate of T359953 [13:00:29] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge: Decision Request - Toolforge policy agent enforcement model - https://phabricator.wikimedia.org/T362872#9780678 (10aborrero) Given the new data, I think I'm now more in favor of option 2: mutation. I scheduled a decision meeting for 2024-05-14. [13:12:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:22:41] RESOLVED: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:42:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:52:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:42:17] 10Tools, 06Diffusion-Repository-Administrators, 10Projects-Cleanup: Consider archiving Gerrit repository "labs/tools/multichill" (20140806) - https://phabricator.wikimedia.org/T351599#9781222 (10Multichill) Repo is still very active, just moved to github long time ago, see https://github.com/multichill/tooll... [15:44:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:47:20] 06cloud-services-team, 10Toolforge, 10Pywikibot: Remove old symlinks to trunk/rewrite/compat/pywikipedia in /shared - https://phabricator.wikimedia.org/T192733#9781225 (10Multichill) >>! In T192733#9774988, @Xqt wrote: > Not related to Pywikibot: > https://codesearch.wmcloud.org/pywikibot/?q=trunk%7Crewrite%... [15:50:45] 06cloud-services-team, 10Toolforge, 10Pywikibot: Remove old symlinks to trunk/rewrite/compat/pywikipedia in /shared - https://phabricator.wikimedia.org/T192733#9781243 (10Multichill) 05Open→03Declined No activity for 5 years. I'm just going to go ahead and mark this as declined. If people want to use... [15:54:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:04:17] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge: Decision Request - Toolforge policy agent enforcement model - https://phabricator.wikimedia.org/T362872#9781279 (10aborrero) I just came up with what an implementation of option 3 could be: https://gitlab.wikimedia.org/repos/cloud/toolforge/tool... [16:44:49] 10Wikibugs: Wikibugs' gitlab connector stops working without a strong sign of why - https://phabricator.wikimedia.org/T364490 (10bd808) 03NEW [16:47:52] 10Wikibugs: Wikibugs' gitlab connector stops working without a strong sign of why - https://phabricator.wikimedia.org/T364490#9781475 (10Jdforrester-WMF) Possibly caused by https://sal.toolforge.org/log/Aq_tWI8BGiVuUzOd6v44 (GitLab rebuild/restart for upstream security release)? [16:48:44] 10Wikibugs: Wikibugs' gitlab connector stops working without a strong sign of why - https://phabricator.wikimedia.org/T364490#9781476 (10bd808) Figuring out how to add a reasonable [[https://wikitech.wikimedia.org/wiki/Help:Toolforge/Jobs_framework#Configuring_health-checks_for_jobs|health check]] to the continu... [16:57:27] 10Wikibugs: Wikibugs' gitlab connector stops working without a strong sign of why - https://phabricator.wikimedia.org/T364490#9781507 (10bd808) >>! In T364490#9781475, @Jdforrester-WMF wrote: > Possibly caused by https://sal.toolforge.org/log/Aq_tWI8BGiVuUzOd6v44 (GitLab rebuild/restart for upstream security rel... [16:58:51] 10Wikibugs: Wikibugs' gitlab connector stops working without a strong sign of why - https://phabricator.wikimedia.org/T364490#9781509 (10bd808) [16:58:52] 10Wikibugs, 03Wikimedia-Hackathon-2024: Report GitLab merge request events to IRC - https://phabricator.wikimedia.org/T362500#9781508 (10bd808) [17:04:54] (03reopen) 10bd808: This is a test of the gitlab irc reporter [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/33 [17:05:02] (03close) 10bd808: This is a test of the gitlab irc reporter [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/33 [17:06:49] 10Wikibugs: Wikibugs testing task - https://phabricator.wikimedia.org/T90594#9781523 (10bd808) [17:09:00] 06cloud-services-team, 06Infrastructure-Foundations, 10Puppet-Infrastructure: Ownership confusion on cloud-local puppet servers - https://phabricator.wikimedia.org/T364492 (10Andrew) 03NEW [17:12:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:22:41] RESOLVED: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:45:17] 06cloud-services-team, 06Infrastructure-Foundations, 10Puppet-Infrastructure: Ownership confusion on cloud-local puppet servers - https://phabricator.wikimedia.org/T364492#9781648 (10Andrew) a:05jhathaway→03Andrew [18:47:31] 10Tool-yearinreview, 10New-Engagement-Experiments, 03Wikimedia-Hackathon-2024: Improve the year in review tool for next year - https://phabricator.wikimedia.org/T362898#9781793 (10Reputation22) Thanks guys!! Following back on the discussion which me and @theprotonade had. We can scale up the project, and hav... [18:48:13] 10Tool-yearinreview, 10New-Engagement-Experiments, 03Wikimedia-Hackathon-2024: Improve the year in review tool for next year - https://phabricator.wikimedia.org/T362898#9781795 (10Reputation22) This would be helpful for user engagement and growth for Wikimedia [18:50:11] 10Tool-yearinreview, 10New-Engagement-Experiments, 03Wikimedia-Hackathon-2024: Improve the year in review tool for next year - https://phabricator.wikimedia.org/T362898#9781798 (10Reputation22) >>! In T362898#9767206, @Jdlrobson wrote: >> Move repository to GitLab > I'm keen to keep the canonical repo on git... [19:01:19] 10Tool-yearinreview, 10New-Engagement-Experiments, 03Wikimedia-Hackathon-2024: Improve the year in review tool for next year - https://phabricator.wikimedia.org/T362898#9781828 (10KStoller-WMF) >>! In T362898#9781795, @Reputation22 wrote: > This would be helpful for user engagement and growth for Wikimedia... [19:40:26] 10Cloud-VPS (Project-requests): Request creation of Huma VPS project - https://phabricator.wikimedia.org/T364509 (10Ladsgroup) 03NEW [19:41:50] 10Cloud-VPS (Project-requests): Request creation of Huma VPS project - https://phabricator.wikimedia.org/T364509#9781968 (10Ladsgroup) [19:42:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:47:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:52:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:57:41] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:04:51] 06cloud-services-team, 10Toolforge: [toolforge,storage] Provide per-tool access to cloud-vps object storage - https://phabricator.wikimedia.org/T358496#9782006 (10Andrew) I'm hitting a roadblock with the service user plan -- because of keystone's belt-and-suspenders approach to security, I can override the pol... [20:43:50] 06cloud-services-team, 10Toolforge: [toolforge,storage] Provide per-tool access to cloud-vps object storage - https://phabricator.wikimedia.org/T358496#9782046 (10fnegri) > The per-tool service user would not have a password, only app credentials. If I understand it correctly, this might be entirely impossibl... [21:11:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:16:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:24:58] 10Toolforge, 10Gerrit, 10GitLab, 10Phabricator, 10Continuous-Integration-Config: Look for ways to consolidate "we trust this human" access lists - https://phabricator.wikimedia.org/T364516 (10bd808) 03NEW [21:26:41] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:30:53] 10Toolforge, 10Gerrit, 10GitLab, 10Phabricator, 10Continuous-Integration-Config: Look for ways to consolidate "we trust this human" access lists - https://phabricator.wikimedia.org/T364516#9782150 (10bd808) The Developer account <-> SUL user mappings that #bitu is now maintaining in LDAP are likely to be... [21:32:09] 10Toolforge, 10Gerrit, 10GitLab, 10Phabricator, 10Continuous-Integration-Config: Look for ways to consolidate "we trust this human" access lists - https://phabricator.wikimedia.org/T364516#9782160 (10bd808) #tool-gitlab-account-approval is attempting to leverage existing allow lists from Gerrit, Phabrica... [21:51:40] 06cloud-services-team, 10Toolforge: [toolforge,storage] Provide per-tool access to cloud-vps object storage - https://phabricator.wikimedia.org/T358496#9782192 (10Andrew) >>! In T358496#9782046, @fnegri wrote: >> The per-tool service user would not have a password, only app credentials. > > If I understand it... [22:12:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:13:43] 06cloud-services-team, 10Horizon: Horizon puppet panel needs better exception handling - https://phabricator.wikimedia.org/T302537#9782221 (10Andrew) 05Open→03Resolved [22:17:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:23:49] 06cloud-services-team, 10Cloud-VPS (Project-requests): Request creation of Huma VPS project - https://phabricator.wikimedia.org/T364509#9782231 (10bd808) +1