[08:55:07] andrewbogott: no, this is maps-puppetmaster, you already added it. It was also rebooted and I have my own personal homedir just fine. It just cannot create this one there. must be acl or mount config [08:55:59] andrewbogott: i already checked and the puppetmaster manifest has that homedir hardcoded, so i can't bypass it by having it write to another location either. [10:20:42] bstorm_: arturo hey, if you have a minute can you check https://gerrit.wikimedia.org/r/c/operations/puppet/+/480590 ? [10:20:50] These two tables are being dropped [10:24:40] let me check [10:25:46] Amir1: it LGTM, but I would like brooke to check that too before merging, she may know better if we need to modify anything else [10:31:29] arturo: Thanks! [10:34:19] :-) you are welcome [14:26:57] andrewbogott: and a similar issue on /data/project [14:26:58] hartman@maps-tiles1:/data/project$ sudo mkdir tiles-new [14:26:58] mkdir: cannot create directory 'tiles-new': Permission denied [14:39:50] thedj: have you tried reboot already? It might be that things just need a remount. [14:56:28] andrewbogott: yup, tried that. just did another hard reboot just to be sure, but same problem [14:56:45] ok, I will look in a bit [14:57:54] chasemp: are 'sec-mwtest1' and 'sec-mwtest2' still useful or can they be deleted? [14:58:48] andrewbogott: still useful but I can recreate if it's a pain [14:59:03] it's fine, I'm just about to move them to eqiad1 [14:59:24] andrewbogott: kk tx [15:00:27] andrewbogott: on second thought just delete and I'll recreate, there is a debugging scenario there that's still relevant but I don't know when I'll get to it [15:00:35] ok! [15:02:28] !log orig moving instances to eqiad1-r [15:02:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Orig/SAL [15:03:24] !log security-tools moving instances to eqiad1 [15:03:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Security-tools/SAL [15:04:26] !log phragile moving instances to eqiad1-r [15:04:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Phragile/SAL [15:05:05] !log security-tools deleting sec-mwtest1 and sec-mwtest2 on chase's advice [15:05:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Security-tools/SAL [15:14:37] btw thedj, did 'maps' have a local puppetmaster before? Or is this a new thing you're adding on? [15:18:51] andrewbogott: this is new [15:19:01] why a separate puppetmaster? [15:19:30] temporary, to test out manifests without having to make 30 or so changesets to operations/puppet and bother everyone with my incompetence [15:19:40] Krenair suggested it [15:19:55] 'k [15:24:09] !log wikistream moving instances to eqiad1-r [15:24:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikistream/SAL [15:24:44] !log otrs moving instances to eqiad1-r [15:24:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Otrs/SAL [15:26:00] thedj: I'm not entirely sure what's happening but I worked around it by symlinking that homedir into /srv. Puppet seems happy now. [15:41:33] yeah I probably should have noted that a separate instance just for puppetmaster may be overkill for small projects [15:41:59] though it would keep the instances you care about clean of puppetmaster stuff [15:55:05] andrewbogott: maybe the server maps remote root users to nobody ? Apparently that's a safety feature in some nfs setups... [19:00:41] arturo: you there? got a problem with wmflabs mailer apparently [19:00:51] or bd808 - not sure who to ask [19:00:52] I'm here [19:01:09] so I'm trying to T212327 without success [19:01:09] T212327: Beta Cluster mailer not sending emails apparently - https://phabricator.wikimedia.org/T212327 [19:01:35] perhaps you know where the logs are stored and could see which problem we have here? [19:01:44] I see nothing at kibana4 [19:03:26] Hauskatze: I know very little about the Beta Cluster. Is that deployment-prep? [19:03:36] arturo: yes [19:04:21] first step would be to identify which server is trying to send the email, and using which relays, and check them as well [19:04:47] I believe Krenair is our deployment-prep expert here :-) [19:08:44] arturo: I guess that maybe deployment-mediawiki0[7-9], but not sure [19:09:14] I'll ssh to them and see what I can find in the meantime [19:10:07] deployment-mediawiki-0[7-9] [19:10:11] sorry [19:11:05] Hauskatze: have you verified that the emails are leaving the deployment-prep hosts? I think that's the part that needs to be known before arturo or someone else with access to the mail relay hosts gets involved in debugging [19:11:23] bd808: if you tell me how to I can try [19:12:24] I think deployment-prep has a centralized mediawiki logging host, but honestly I haven't been active in that project for years now [19:12:45] fluorine2 [19:12:51] I checked there but found nothing [19:13:03] also at kibana4.wmflabs (logstash-beta) [19:13:05] nothing [19:15:42] I see at https://tools.wmflabs.org/openstack-browser/project/deployment-prep that there's a mail.wmflabs.org [19:16:05] mail.beta.wmflabs.org I mean [19:16:07] Hauskatze: try "tail -f /var/log/exim4/mainlog" on the shell of the outgoing host in depl-prep.. then click the reset button while doing that [19:16:50] mutante: but still I'm not sure on which server [19:17:02] I'll wait for Krenair I guess, if he wants to [19:18:42] yea, ok. or somebody would have to check the same for incoming on mx then [19:35:19] hello, in horizon can shut off instances be restarted if need be? I think yes, but double checking before I do so [19:37:16] okay found one that was shut down yes, it has a "restart" control. thanks :-) [19:55:17] https://www.irccloud.com/pastebin/a3Eb2rLg/ [19:55:42] Hauskatze: looks like deployment-prep has its own mail relay [19:57:01] mx02 looks indeed the mailer gtirloni [19:57:08] I think Krenair is having a look at it? [19:57:25] looks like it [20:05:14] just a question out of curiosity, forgive my ignorance. I couldn't find the answer anywhere. Does neutron adds SDN functionality to the current cloud system or the cloud already had it in the old region? [20:05:49] I think we've had SDN this whole time, but an old version of it. [20:06:19] oh thanks [20:06:27] I think nova-network counts as SDN, yes [20:07:05] (nova-network is the old thing being replaced by neutron) [20:10:10] cool [20:10:41] Amir1, it's probably worth noting that control over it is pretty much reserved for cloudadmins though [20:10:55] so you shouldn't expect anything like AWS VPCs [20:11:19] Nah, I just want to understand the infra better [20:12:18] ok [20:14:03] Amir1: more here if you're interested -- https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Neutron [20:14:31] gtirloni: thanks!