[01:17:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:07:06] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/weapon-of-mass-description] - 10https://gerrit.wikimedia.org/r/1028463 (owner: 10L10n-bot) [05:17:56] FIRING: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:04:53] 10Tools, 06translatewiki.net, 10Language-Team (Language-2024-April-June), 03Localization Infrastructure FY2023-24, and 2 others: Make Wikidata Image Positions tool translatable on translatewiki.net - https://phabricator.wikimedia.org/T363626#9776419 (10Nikerabbit) [08:22:24] 06cloud-services-team, 10Data-Platform-SRE (2024.05.06 - 2024.05.26), 13Patch-For-Review: Streamline the creation of ceph storage cluster user accounts - https://phabricator.wikimedia.org/T364105#9776598 (10BTullis) This has ben deployed. I will monitor to make sure that it's OK and then resolve. [08:31:28] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance metricsinfra-puppetserver-1 in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [08:51:28] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance metricsinfra-puppetserver-1 in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [08:52:47] 06cloud-services-team, 06Data-Platform-SRE: ceph: introduce puppet logic to purge stale keyfiles - https://phabricator.wikimedia.org/T328010#9776659 (10Gehel) [08:54:55] 06cloud-services-team: cloudceph: reimage some servers - https://phabricator.wikimedia.org/T328011#9776678 (10Gehel) [09:17:56] FIRING: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:25:56] FIRING: [4x] SystemdUnitDown: The service unit user@0.service is in failed status on host cloudcephosd1004. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [09:30:56] FIRING: [6x] SystemdUnitDown: The service unit user@0.service is in failed status on host cloudcephosd1004. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [09:43:28] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance metricsinfra-puppetserver-1 in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [09:45:24] 10Cloud-Services: cloudcephosd: user@0.service is in failed status - https://phabricator.wikimedia.org/T364376 (10aborrero) 03NEW The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ and replace it with a more specifi... [09:45:56] FIRING: [6x] SystemdUnitDown: The service unit user@0.service is in failed status on host cloudcephosd1004. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [09:46:18] 10Cloud-Services: cloudcephosd: the service unit user@0.service is in failed status - https://phabricator.wikimedia.org/T364376#9776815 (10aborrero) [09:46:24] 06cloud-services-team, 10Cloud-VPS: cloudcephosd: the service unit user@0.service is in failed status - https://phabricator.wikimedia.org/T364376#9776816 (10taavi) [09:47:01] 10Cloud-Services, 06cloud-services-team: cloudcephosd: the service unit user@0.service is in failed status - https://phabricator.wikimedia.org/T364376#9776817 (10aborrero) [09:47:17] 10Cloud-Services, 06cloud-services-team: cloudcephosd: the service unit user@0.service is in failed status - https://phabricator.wikimedia.org/T364376#9776818 (10aborrero) p:05Triage→03Low [09:48:04] 10Cloud-Services, 06cloud-services-team: cloudcephosd: the service unit user@0.service is in failed status - https://phabricator.wikimedia.org/T364376#9776819 (10dcaro) The sessions seems to be a root login from cumin2002 (for all nodes): ` May 07 09:20:04 cloudcephosd1010 systemd-logind[2594]: New session 96... [09:53:28] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance metricsinfra-puppetserver-1 in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [10:41:21] 06cloud-services-team, 10Cloud-VPS, 10Toolforge: Learn how to do what Taavi does - https://phabricator.wikimedia.org/T362443#9776953 (10dcaro) For the "wikireplicas" meeting, the notes are: May 7, 2024 | Wiki replicas Attendees: c_7b9ad6d28760abb302f0909412d1ed85b8d1db6ade03cbf2242fededb17164f1@group.cale... [10:47:43] 10Cloud-Services, 06cloud-services-team: cloudcephosd: the service unit user@0.service is in failed status - https://phabricator.wikimedia.org/T364376#9776978 (10dcaro) It might be related to {T199911} [10:56:25] 10Tools, 03Wikimedia-Hackathon-2024: Re-implement WikiShootMe as a customisable frontend JS app - https://phabricator.wikimedia.org/T364142#9776995 (10Aklapper) a:03TuukkaH [11:07:10] 06cloud-services-team, 10Data-Platform-SRE (2024.05.06 - 2024.05.26): Streamline the creation of ceph storage cluster user accounts - https://phabricator.wikimedia.org/T364105#9777044 (10BTullis) 05Open→03Resolved [11:20:56] FIRING: [3x] SystemdUnitDown: The systemd unit user@0.service on node cloudcephosd1010 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [11:21:01] 06cloud-services-team: SystemdUnitDown - https://phabricator.wikimedia.org/T364381 (10phaultfinder) 03NEW [11:30:56] RESOLVED: [5x] SystemdUnitDown: The service unit user@0.service is in failed status on host cloudcephosd1005. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [11:42:11] 06cloud-services-team: SystemdUnitDown - https://phabricator.wikimedia.org/T364381#9777155 (10dcaro) a:03dcaro [11:42:26] RESOLVED: [5x] SystemdUnitDown: The systemd unit user@0.service on node cloudcephosd1005 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [11:42:34] 06cloud-services-team: SystemdUnitDown - https://phabricator.wikimedia.org/T364381#9777157 (10dcaro) →14Duplicate dup:03T364376 [11:43:52] 10Cloud-Services, 06cloud-services-team: cloudcephosd: the service unit user@0.service is in failed status - https://phabricator.wikimedia.org/T364376#9777159 (10dcaro) [11:47:26] 10Cloud-Services, 06cloud-services-team: cloudcephosd: the service unit user@0.service is in failed status - https://phabricator.wikimedia.org/T364376#9777161 (10dcaro) 05Open→03Resolved a:03dcaro @MoritzMuehlenhoff mentioned on irc that this is probably related to the upgrade of glibc + systemd issu... [12:02:33] 06cloud-services-team, 10Toolforge (Toolforge iteration 09), 13Patch-For-Review: toolforge: review pod templates for PSP replacement - https://phabricator.wikimedia.org/T362050#9777192 (10CodeReviewBot) aborrero merged https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/75 jobs-api:... [12:05:21] 06cloud-services-team, 10Toolforge (Toolforge iteration 09), 13Patch-For-Review: toolforge: review pod templates for PSP replacement - https://phabricator.wikimedia.org/T362050#9777195 (10CodeReviewBot) project_1317_bot_df3177307bed93c3f34e421e26c86e38 opened https://gitlab.wikimedia.org/repos/cloud/toolforg... [12:06:17] 06cloud-services-team, 10Toolforge (Toolforge iteration 09), 13Patch-For-Review: toolforge: review pod templates for PSP replacement - https://phabricator.wikimedia.org/T362050#9777196 (10aborrero) TODO: webservice [12:06:52] !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [12:07:03] !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [12:21:15] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [12:21:28] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [12:23:28] 06cloud-services-team, 10Toolforge (Toolforge iteration 09), 13Patch-For-Review: toolforge: review pod templates for PSP replacement - https://phabricator.wikimedia.org/T362050#9777203 (10CodeReviewBot) aborrero merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/278... [12:24:33] 06cloud-services-team, 10Toolforge (Toolforge iteration 09), 13Patch-For-Review: toolforge: review pod templates for PSP replacement - https://phabricator.wikimedia.org/T362050#9777204 (10aborrero) Before patch https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/278 with only... [12:29:11] 06cloud-services-team, 10Toolforge (Toolforge iteration 09), 13Patch-For-Review: toolforge: review pod templates for PSP replacement - https://phabricator.wikimedia.org/T362050#9777214 (10CodeReviewBot) aborrero opened https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/83 jobs: fix... [12:31:02] 06cloud-services-team, 10Toolforge (Toolforge iteration 09), 13Patch-For-Review: toolforge: review pod templates for PSP replacement - https://phabricator.wikimedia.org/T362050#9777224 (10CodeReviewBot) aborrero merged https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/83 jobs: fix... [12:32:26] 10ToolforgeBundle: Upgrade to Symfony 7 - https://phabricator.wikimedia.org/T361554#9777229 (10Samwilson) PR: https://github.com/wikimedia/ToolforgeBundle/pull/68 I'm working with #tool-wscontest and #ws_export as example tools for upgrading, to ensure I upgrade the bundle correctly. [12:33:13] !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [12:33:23] !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [12:33:34] 06cloud-services-team, 10Toolforge: [toolforge,storage] Provide per-tool access to cloud-vps object storage - https://phabricator.wikimedia.org/T358496#9777233 (10dcaro) I'm working on {T363983}, and I think both are very related, specially when we talk about how to authenticate to access the buckets/storage.... [12:34:40] 06cloud-services-team, 10Toolforge (Toolforge iteration 09), 13Patch-For-Review: toolforge: review pod templates for PSP replacement - https://phabricator.wikimedia.org/T362050#9777238 (10CodeReviewBot) project_1317_bot_df3177307bed93c3f34e421e26c86e38 opened https://gitlab.wikimedia.org/repos/cloud/toolforg... [12:34:42] 06cloud-services-team, 10Toolforge: [toolforge,storage] Provide per-tool access to cloud-vps object storage - https://phabricator.wikimedia.org/T358496#9777246 (10dcaro) I'm currently leaning on using idp.w.o for toolforge api auth, and leaving the openstack auth only when required for the storage or database... [12:34:51] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [12:35:03] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [12:38:04] 06cloud-services-team, 10Toolforge (Toolforge iteration 09), 13Patch-For-Review: toolforge: review pod templates for PSP replacement - https://phabricator.wikimedia.org/T362050#9777250 (10aborrero) [12:41:00] 10tool-wscontest: The score command throws deprecated warning - https://phabricator.wikimedia.org/T348270#9777255 (10Samwilson) PR for upgrading to Symfony 7: https://github.com/wikisource/wscontest/pull/69 [12:41:40] 10ToolforgeBundle: Upgrade to Symfony 7 - https://phabricator.wikimedia.org/T361554#9777257 (10Samwilson) PR for WS Contest: https://github.com/wikisource/wscontest/pull/69 (not ready for merging because it references the `symfony7` branch of the bundle). [12:47:58] 10Tools, 06translatewiki.net, 10Language-Team (Language-2024-April-June), 03Localization Infrastructure FY2023-24, and 2 others: Make Wikidata Image Positions tool translatable on translatewiki.net - https://phabricator.wikimedia.org/T363626#9777262 (10Nikerabbit) [12:49:01] 06cloud-services-team, 10Toolforge (Toolforge iteration 09), 13Patch-For-Review: toolforge: review pod templates for PSP replacement - https://phabricator.wikimedia.org/T362050#9777263 (10CodeReviewBot) aborrero merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/279... [13:17:56] FIRING: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:28:41] (03PS1) 10Ebrahim: Add Persian Wikipedia to configs [labs/tools/fileprotectionsync] - 10https://gerrit.wikimedia.org/r/1028824 [13:41:40] 10Toolforge (Toolforge iteration 09): [toolforge] Investigate authentication - https://phabricator.wikimedia.org/T363983#9777461 (10fnegri) I'm trying to understand pros & cons of the different protocols, especially OIDC (OpenID Connect) vs CAS. I played a bit with OIDC in a previous job, and I remember it as f... [14:08:11] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:39:11] 10PAWS: Might be able to remove referer rewrite - https://phabricator.wikimedia.org/T329210#9777654 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/paws/pull/281 [14:39:13] 10PAWS: Might be able to remove referer rewrite - https://phabricator.wikimedia.org/T329210#9777655 (10rook) 05Open→03Resolved [14:39:21] vivian-rook closed https://github.com/toolforge/paws/pull/281 [14:54:28] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance toolsbeta-puppetserver-1 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [15:14:58] 10Toolforge (Software install/update): Upgrade golang buildpack to 1.22 - https://phabricator.wikimedia.org/T363854#9777742 (10dcaro) As a quick workaround, have you tried specifying the newer version? https://github.com/heroku/buildpacks-go?tab=readme-ov-file#go-version [15:18:28] FIRING: [2x] InstanceDown: Project toolsbeta instance toolsbeta-test-k8s-worker-10 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [15:19:06] FIRING: ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_toolserver_org_redirects_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:19:55] FIRING: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.toolsbeta.eqiad1.wikimedia.cloud:6443 almost out of cpu #page - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [15:20:58] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance toolsbeta-puppetserver-1 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [15:22:08] 06cloud-services-team, 10Toolforge (Toolforge iteration 09), 13Patch-For-Review: toolforge: review pod templates for PSP replacement - https://phabricator.wikimedia.org/T362050#9777757 (10CodeReviewBot) aborrero opened https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/37 k... [15:23:28] RESOLVED: [2x] InstanceDown: Project toolsbeta instance toolsbeta-test-k8s-worker-10 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [15:24:06] RESOLVED: ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_toolserver_org_redirects_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:24:23] FIRING: ToolforgeKubernetesNodeNotReady: Kubernetes node toolsbeta-test-k8s-worker-10 is not ready - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady [15:29:25] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.openstack.cloudvirt.vm_console [15:29:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:29:30] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99) [15:29:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:29:55] FIRING: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.toolsbeta.eqiad1.wikimedia.cloud:6443 close to running out of cpu - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [15:44:55] RESOLVED: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.toolsbeta.eqiad1.wikimedia.cloud:6443 close to running out of cpu - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [15:57:52] 10Cloud-VPS (Debian Buster Deprecation), 06Moderator-Tools-Team, 06The-Wikipedia-Library: Replace deprecated Buster VMs in Cloud VPS - https://phabricator.wikimedia.org/T364399 (10jsn.sherman) 03NEW [15:59:14] 10Cloud-VPS (Debian Buster Deprecation), 06Moderator-Tools-Team, 06The-Wikipedia-Library: Replace deprecated Buster VMs in Cloud VPS - https://phabricator.wikimedia.org/T364399#9777990 (10jsn.sherman) [16:18:42] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [16:18:53] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [16:23:09] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [16:23:20] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [16:26:56] 06cloud-services-team, 10Toolforge: [toolforge,storage] Provide per-tool access to cloud-vps object storage - https://phabricator.wikimedia.org/T358496#9778159 (10Andrew) So many questions! >>! In T358496#9777233, @dcaro wrote: > I'm working on {T363983}, and I think both are very related, specially when we t... [20:14:10] (03CR) 10Andrea Denisse: [V:03+2 C:03+2] ssl: Remove unnecessary dummy key for the kibana hosts [labs/private] - 10https://gerrit.wikimedia.org/r/1026693 (https://phabricator.wikimedia.org/T360414) (owner: 10Andrea Denisse) [20:14:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:24:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:42:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:47:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:52:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:57:41] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:55:31] 10Wikibugs: Allow more folks to submit/merge config patches - https://phabricator.wikimedia.org/T364436 (10bd808) 03NEW