[02:28:15] !log toolserver-legacy deleting relic-buster as per discussion on T260835 [02:28:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolserver-legacy/SAL [02:28:21] T260835: Stop using letsencrypt::cert::integrated on toolserver-legacy - https://phabricator.wikimedia.org/T260835 [03:14:38] hey gus [03:14:56] is this a free cloud pc [03:15:01] or something [03:15:15] cause i need it for blender [03:15:31] !help [03:15:31] If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-kanban [05:05:16] [telegram] TGG85: you can learn more about our project and services at https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_Introduction [05:06:46] [telegram] Our compute resources are for projects supporting the Wikimedia movement [13:32:09] Hi folks, we've noticed that outbound emails from accounts-appserver5 no longer seem to be sending, and this looks like it started at some point between 15:53:57 UTC and 17:04:32 UTC yesterday. [13:33:12] The error in /var/log/exim4/mainlog is: 2021-01-06 12:57:57 1kx8Nx-0006si-0Q == redacted@example.org R=smart_route defer (-32): lookup of host "mx-out01.wmflabs.org" failed in smart_route router [13:48:11] stwalkerster: I think arturo was doing some work on the mail hosts yesterday [13:51:03] according to https://openstack-browser.toolforge.org/project/cloudinfra mx-out01.wmflabs.o should resolve to 185.15.56.18 [13:51:14] but at least the toolforge bastions do not resolve it at all [13:52:10] *** Can't find mx-out01.wmflabs.org: No answer [13:52:22] Yeah, same on accounts-appserver5 [14:06:02] let me take a look [14:16:56] hmm... I think you can use instance-mx-out01.cloudinfra.wmflabs.org instead for now, but there's something fishy going on with the dns records (mx-out01.wmflabs.org is not publicly resolvable seems, but 185.15.56.18 points to it, and to the previous one) [14:17:51] OK, what's the best way to switch to it? disable puppet and manually edit /etc/exim4/exim4.conf ? [14:18:04] let me check [14:18:57] thanks :) [14:23:18] dcaro: wouldn [14:23:40] 't that break the ssl records? or is that an alternative name in the cert too? [14:30:56] yep, the latest changes only allow wmflabs.org wmcloud.org and wikimedia.cloud [14:31:38] mx-out01.cloudinfra.eqiad1.wikimedia.cloud seems the best alternative [14:31:49] (also using the latest domain name) [14:34:45] OK, so just to confirm, I would be changing route_list = * mx-out01.wmflabs.org:mx-out02.wmflabs.org to route_list = * mx-out01.cloudinfra.eqiad1.wikimedia.cloud:mx-out02.cloudinfra.eqiad1.wikimedia.cloud in /etc/exim4/exim4.conf? [14:35:28] yep, I'm trying to figure out if that should be done by puppet or manually [14:36:14] I guess I could do it manually to check it's going to work while we figure out if this should be overridden somewhere in puppet. Puppet should just override my change once it does a sync [14:36:41] yep, that'd be a way of checking :) [14:38:13] did that work? [14:39:13] yep, my test message arrived [14:40:28] awesome, I'm opening a task to keep track and see if what happened to the record was intended [14:41:51] I've just cleared the mail queue on that box before puppet reverts my change [14:42:14] did puppet revert your change then? [14:43:23] so far, no, but I don't think puppet has run yet [14:43:29] I'll force a run and see if it does [14:44:15] yeah, it's reverted it [14:44:33] ack, let me look into that [14:45:02] https://phabricator.wikimedia.org/P13659 [14:48:44] okok, just added to your project hiera and rerun, it's back in [14:48:55] (under puppet -> project on horizon) [14:49:16] that will also set it up on all the other hosts in the project btw. [14:50:15] oh, awesome, thanks :) [14:51:22] Is this likely to be an issue for other WMCS projects too? [14:52:29] yep, looking, thanks for pinging btw [14:53:14] !log account-creation-assistance manually configured mx servers to use wikimedia.cloud domain on project hiera (T271322) [14:53:18] dcaro: is this a cert thing or is the dns record missing? (I definitely don't remember changing dns) [14:53:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Account-creation-assistance/SAL [14:53:18] T271322: [mx] check what happened to mx-out01.wmflabs.org - https://phabricator.wikimedia.org/T271322 [14:54:12] andrewbogott: dns record missing [14:54:24] andrewbogott: good morning :) [14:54:28] good morning! [14:54:42] I guess we don't have a way of knowing if/when it last existed [14:58:55] andrewbogott: who/where are the records set? designate? [14:59:09] yeah [14:59:26] I mean, a public address under .org would be manually created [14:59:39] yep, that was wmflabs.org [14:59:40] probably in the wmflabsdotorg project [14:59:57] (maybe we should move to wikimedia.cloud instead, being the current cloud domain) [15:01:31] I don't know much about that setup at all. I assume it used a public IP because that was somehow easier to be reached cloud-wide? [15:01:37] public IP/public domain [15:01:48] so the equivalent to that would be .wmcloud.org [15:02:59] I think that keith herron built that initial setup so he would be the one to ask about reasoning [15:04:10] ack [15:14:52] hmm... I see the mx-out01.wmflabs.org entry on the openstack poroject wmflabsdotorg [15:23:28] okok, so the records are stored on mysql, there there's no entry mx-out1.wmflabs.org under the records table, there's though a 'domain' with that name under the domains table [15:54:42] oh! hm... [15:54:51] maybe when I created that domain it clobbered the existing record [15:54:57] I will look into that after the meeting [15:56:23] ack [15:56:28] thanks :)