[00:26:08] !log utrs moving VMs to eqiad1-r [00:26:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Utrs/SAL [13:44:38] andrewbogott: Available? [13:55:25] Hydriz: may I help you? [13:55:55] That would be great, can you check if there is anything wrong with the dumps project? [13:56:06] wrong if which sense? [13:56:09] Horizon reports an incorrect usage for eqiad1-r region [13:56:20] In terms of instances, ram and cpu [13:56:31] * arturo checking [13:56:57] Relevant Phab task: https://phabricator.wikimedia.org/T204503 [13:58:40] Hydriz: `project quotas seems to be incorrectly showing 6 instances when it should be 4` <--- this, right? [13:58:54] Yep [13:59:13] More specifically with the RAM and CPU parts [14:00:02] And not really quotas, it's the project usage, apologies for the wrong choice of words [14:06:33] arturo: quotas getting out of sync is a known issue, the (terrible) fix is to set things to -1 in the database and let it recalculate. [14:06:38] I can do it unless you're interested [14:06:54] oh... [14:06:54] btw I can ssh to dumps-stats just fine [14:07:31] andrewbogott: please do it, I'm about to look for lunch [14:07:41] ok [14:08:58] andrewbogott: Yeah dumps-stats is fine, it was dumps-0 which I had deleted already [14:18:36] Hydriz: try creating a new VM now? [14:19:36] andrewbogott: Launching dumps-0 [14:23:25] usage displays are still messed up there but at least you're unblocked [14:26:03] andrewbogott: Yep I've logged in nicely, thanks! [14:38:00] andrewbogott: I wonder if we should set it to -1 for all projects (for it to recalculate) after all projects have been migrated to eqiad1-r? [14:38:50] Well, setting it to -1 didn't work right this time. I wouldn't mind forcing recalcs if I knew how to do it reliably. [14:39:07] Looks like that cache has been (wisely) removed in future versions since it's never in sync [15:00:47] KPADSIBIJSWJREDMAQUIMNJKNJJHLCQNEEPGSNWJ Technical Advice IRC meeting starting in 60 minutes in channel #wikimedia-tech, hosts: @Thiemo_WMDE & @chiborg - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [15:18:25] !help is there some delay between adding a new SSH key in the Wikitech interface, and having it recognized on bastion.wmflabs.org? [15:18:25] Sorry, you are not authorized to perform this [15:18:51] wm-bot seems drunk. [15:19:46] urandom: I believe Puppet has to run first before the keys are replaced [15:20:33] :1 [15:20:40] Sorry +1 [15:20:44] I suspect the same. [15:21:00] So, what kind of delay are we talking about? [15:21:18] it runs at 0,30 every hour, right? depends on when it was changed [15:22:16] It was changed 10 mins ago [15:22:45] so we'll try in other 10 mins or so [15:30:10] personal keys are stored in ldap so should be available immediately. [15:37:25] andrewbogott: I believe you, since it's past :30 and it still doesn't work :) [15:38:31] andrewbogott: `/usr/sbin/ssh-key-ldap-lookup ` from a cloud VM shows the key though [15:38:45] and, ` urandom: what are you trying to connect to? [15:40:12] andrewbogott: a VM [15:40:22] that doesn't really narrow it down [15:40:22] kask.services.eqiad.wmflabs [15:41:41] using what bastion? [15:41:47] bastion.wmflabs.org [15:42:35] so, the most frequent cause of this problem is people omitting the @hostname and having a different username on their local machine [15:43:01] to narrow things down I'd also suggest ssh'ing directly to the bastion to see if there's an issue with the proxy command. [15:43:07] clarakosi: what does `whoami` return on your notebook? [15:43:19] andrewbogott: yeah, we did that and it failed [15:43:30] clarakosi: is it something other than your wikitech username? [15:43:44] (I shouldn't have assumed they'd be the same) [15:43:48] I see 'invalid user juanangel' [15:43:59] ahhh yeah its different. `candrew` [15:44:20] ok, try: `ssh clarakosi@kask.services.eqiad.wmflabs` [15:45:24] same thing. permission denied. [15:45:57] clarakosi: and `ssh clarakosi@bastion.wmflabs.org` ? [15:46:55] looks like that worked? [15:47:09] clarakosi: did you get a shell on bastion? [15:47:31] no its still denied [15:50:27] try -vvv and let's see why it thinks it's denied [15:50:37] KPADSIBIJSWJREDMAQUIMNJKNJJHLCQNEEPGSNWJ Technical Advice IRC meeting starting in 10 minutes in channel #wikimedia-tech, hosts: @Thiemo_WMDE & @chiborg - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [15:50:54] wm-bot are you ok? [15:51:09] i.e. `ssh -vvv clarakosi@bastion.wmflabs.org` [15:51:12] clarakosi: ^^^ [15:51:24] and it'll produce a lot of output [15:51:33] Try -i [15:51:44] no wait the last one worked [15:51:51] \o/ [15:51:52] sorry I put in the wrong passphrase [15:52:08] clarakosi: are you having to enter a passphrase each time? [15:52:17] yes [15:52:29] oh...you shouldn't have to do that [15:52:37] but my macos-fu is weak [15:53:05] I think that’s expected if a key has a passkey [15:53:20] clarakosi: OK, so you got a shell on bastion and the prompt changed to `eevans@bastion-01:~$` ? [15:53:29] macos doesn't have an agent? [15:53:41] yup now it says `clarakosi@bastion-01:~$` [15:53:48] ok, type `exit` [15:54:01] or `+d` [15:54:06] ok. connection closed [15:54:07] to get back to your terminal [15:54:27] and try: `ssh clarakosi@kask.services.eqiad.wmflabs` again [15:55:00] lol denied [15:55:26] any chance you mistyped the passphrase? [15:55:34] I take it you had to enter it yet again? [15:55:40] Didn't even ask for it this time [15:55:51] automatically denied it [15:55:59] ?? [15:56:11] maybe up-arrow and try that bastion login again? [15:56:32] clarakosi: try ssh -I clarakosi@kask.services.eqiad.wmflabs [15:57:55] clarakosi: `ssh clarakosi@bastion.wmflabs.org` (and I'm also curious whether you're prompted for that passphrase again) [15:57:56] Oh [15:58:03] You have to proxy through bastion [15:58:27] paladox: that should be happening [15:58:42] So unless you have something that picks up kask.services.eqiad.wmflabs it won’t work [15:59:02] paladox: still nothing [15:59:05] https://wikitech.wikimedia.org/wiki/Help:Access#Accessing_instances_with_ProxyCommand_ssh_option_(recommended) [15:59:19] paladox: we setup a `~/.ssh/config` earlier [15:59:20] urandom: it does ask for the passphrase with that one [15:59:37] Yup [15:59:46] clarakosi: OK, so it *does* prompt you for a password logging into bastion, and the login works? [15:59:55] and it does not for the VM, and it fails? [16:00:05] if so, that does sound like the proxycommand isn't working [16:00:59] Yup logging into bastion asks for passphrase and works but with the VM it automatically fails. Says permission denied. [16:01:15] clarakosi: `cat ~/.ssh/config` [16:01:38] Host *.eqiad.wmflabs [16:01:39] ProxyCommand ssh -a -W %h:%p clarakosi@primary.bastion.wmflabs.org [16:01:43] Does that work? [16:02:41] clarakosi: also: `ls -l ~/.ssh/config` [16:03:42] paladox: that didn't work [16:03:52] Oh :( [16:04:07] Can you ssh into any other instance [16:04:12] That your a member of? [16:04:31] this our first attempt at logging into any of them [16:12:13] we're in! [16:12:19] paladox: you were right!! I just had to move it up the config page [16:12:29] Thank you! [16:12:32] paladox: clarakosi added your proxycommand to the top of the file, and it worked [16:13:05] I've no idea, we matched hers to mine before this (and it works for me), but umm... it works, so... yeah. :) [17:31:09] !log contintcloud deleted project (T209644) [17:31:11] gtirloni: Unknown project "contintcloud" [17:31:11] T209644: Delete contintcloud WMCS project - https://phabricator.wikimedia.org/T209644 [17:32:36] !log project-contintcloud deleted project (T209644) [17:32:36] gtirloni: Unknown project "project-contintcloud" [17:33:00] !log admin deleted contintcloud project (T209644) [17:33:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [21:02:33] Hi, I have a new phone and I am trying to move 2fa to the new phone. I deactivated 2fa successfully but now I can not enable it. I get an exception but can not see the details. Is someone with access to the error logs around? [21:03:35] " If you report this error to the Wikimedia System Administrators, please include the details below. PHP fatal error: [21:03:36] Argument 1 passed to Monolog\Processor\WebProcessor::__invoke() must be an instance of array, null given" [21:04:35] andrewbogott ^^ [21:04:43] !log ircd delete instance udpmx-01 (not used, created by me in May 2016 [21:04:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ircd/SAL [21:07:58] physikerwelt, interesting [21:08:21] can you file a task? [21:09:31] sure will do I was hoping to make progress on [21:09:33] https://phabricator.wikimedia.org/T204509 [21:12:29] https://phabricator.wikimedia.org/T210669 [21:12:49] should I classify it as cloud or monolog task? [21:13:30] I got the impression that there are two problems... [21:18:23] physikerwelt: we had another report of that error today. I'm going to merge your report into the other one. I think the bug is in OATHAuth itself, but not 100% sure yet [21:19:33] I just tried and the same thing happens to me. It's easy to reproduce! [21:19:44] bd808: thank you. Are you aware of a workaround? [21:20:17] physikerwelt: not yet sadly. It probably bad code in OATHAuth that we need to fix or revert [21:21:32] ok sorry andrewbogott I did not think about that successfully reproducing this leads to being unable to access horizon... [21:21:47] it's ok :) It needs fixing soon regardless [21:25:07] ok have a great day. And thanks for your amazing support [21:31:25] bd808: did you merge that ticket with something? [22:46:57] How do I switch on nginx for my new stretch server? I.e. I have a new webproxy but I don't know which directory nginx is resolving it to? [23:02:19] notconfusing: I'm not sure I understand your question. Are you asking how to start nginx or something else? [23:03:14] yes. for instance on my ubuntu serevr whgi.wmflabs.org used to point to a specific static directory. i set a new proxy whgi2.wmflabs.org to the stretch machine, but how do i tell nginx where to look? bd808 [23:03:59] nginx seems to be running, but im not sure where the config file is to define where the traffic on port 80 should go [23:05:03] the config would be in /etc/nginx/... [23:06:55] bd808: i don't have such a directory on whgi.eqiad.wmflabs [23:07:19] notconfusing: I don't see nginx installed there [23:08:03] do i need to install nginx myself? because if i go to http://whgi2.wmflabs.org/ i see nginx already running [23:08:34] that's the reverse proxy server, not something on your instance [23:09:02] so should i install nginx or apache locally? [23:09:03] traffic goes internet -> proxy -> your instance [23:09:19] if you want to serve http, yes :) [23:09:29] ok, thanks for clearing that up [23:10:02] thank you bd808 i was so confused...