[07:36:31] I received a bunch of pages around 4:30 UTC, they all auto-resolved in a minute or so [07:36:48] morning, what were they about? [07:37:01] they were all like "cloudvirtXXXX/nova-compute proc minimum", on 8 different cloudvirts [07:37:13] maybe a network glitch? [07:37:21] I suspect they could all be in the same row [07:37:34] you can see them in #-cloud-feed [07:37:37] (or in victorops) [07:38:28] this seems correlated https://gerrit.wikimedia.org/r/c/operations/puppet/+/1161148 (similar timing) [07:38:31] no, they were in different rows [07:39:09] oh that's possible, good catch [07:44:04] I think andrewbogo.tt rebooted all the daemons for that yep https://sal.toolforge.org/admin?d=2025-06-19 [07:44:48] that adds up yes [09:01:36] quick tofu review (new project) https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/249 [09:03:28] LGTM though, so if nobody reviews in ~30min I'll go ahead with it :) [09:03:39] thanks taavi [09:03:41] yeah, I don't see much point in getting additional reviews for those, but done [09:06:51] Looking for +1s for https://phabricator.wikimedia.org/T396840 and https://phabricator.wikimedia.org/T397266 , seem ok to me [09:07:17] tofu failed :S [09:07:25] │ Error: Error creating openstack_identity_project_v3: Expected HTTP response code [201 202] when accessing [POST https://openstack.eqiad1.wikimediacloud.org:25000/v3/projects], but got 504 instead:

504 Gateway Time-out

[09:07:27] looking [09:07:33] should be safe to retry right? [09:07:49] yeah [09:08:48] the project is actually there in horizon, I guess it did not get to create the security groups and such, checking [09:09:48] hmm... retry fails with [09:09:53] │ Error: Error creating openstack_identity_project_v3: Expected HTTP response code [201 202] when accessing [POST https://openstack.eqiad1.wikimediacloud.org:25000/v3/projects], but got 409 instead: {"error":{"code":409,"message":"Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is toolsbeta-logging, project id [09:09:54] 94d453f99e23466e98f26025cf9a647e.","title":"Conflict"}} [09:10:24] Is there a way to 'refresh' the tofu state from openstack? [09:10:41] should I create one of those 'import' entries in the tofu code? [09:11:51] that, or ssh to a cloudcontrol where the repo is cloned and run `tofu state import` by hand [09:12:08] okok [09:12:11] s/`tofu state import`/`tofu import`/, https://opentofu.org/docs/cli/import/usage/ [09:13:17] that sounds nicer to me [09:17:14] hmmm.... that syntax is tricky [09:17:24] https://www.irccloud.com/pastebin/91600AG0/ [09:17:45] but there's such `resource` entry in the project module [09:18:04] https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/blob/main/modules/project/project.tf#L2 [09:19:48] you need the full opentofu resource path, which should've been logged just before the error snippet you copy-pasted here [09:21:34] `module.project["toolsbeta-logging"].openstack_identity_project_v3.project[0]` fails too [09:21:50] (that's what it showed) [09:24:53] that should work [09:24:57] mind if i try? [09:25:01] sure [09:25:12] thanks [09:27:41] this worked for me: [09:27:41] taavi@cloudcontrol1006 /srv/tofu-infra $ sudo tofu import 'module.project["toolsbeta-logging"].openstack_identity_project_v3.project[0]' f048baba9d7e46a49dcc72221af0da13 [09:27:52] the openstack project id in your command was completely wrong [09:28:03] oh, what's that id? [09:30:14] the openstack internal project id? [09:30:22] where did you get the id in your command from? [09:30:46] I got it from the error [09:30:54] │ Error: Error creating openstack_identity_project_v3: Expected HTTP response code [201 202] when accessing [POST https://openstack.eqiad1.wikimediacloud.org:25000/v3/projects], but got 409 instead: {"error":{"code":409,"message":"Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is toolsbeta-logging, project id [09:30:54] 94d453f99e23466e98f26025cf9a647e.","title":"Conflict"}} [09:31:47] that is the id of the project that tofu tried to create but failed because the name was already in use [09:32:50] anyway, you now need to run tofu again to create the rest of the resources [09:33:07] so that id is from tofu or openstack? [09:33:35] from openstack, but for a project that does not exist [09:33:47] so some kind of temporary id? [09:34:24] so basically tofu tells openstack "hey, create a project with this name 'toolsbeta-logging'" [09:34:53] openstack gets that request, and because tofu did not specify a project id, it generates one randomly [09:35:02] (that's the ID included in the error message you saw) [09:35:14] and only after that it realizes that project name is actually in use already [09:35:26] so that project ID would have been used if the project was created successfully then [09:35:38] but it wasn't, because there was an another project already using the name [09:37:18] sounds like a high chance of leaking ids xd [09:37:32] done \o/ [10:33:26] * dcaro lunch