[00:04:16] Hello, I can't connect to Cloud VPS via ssh. I've WIndows 10 [00:04:20] !help [00:04:20] If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-kanban [00:07:42] Zoranzoki21: folks are going to need more details to help you. What errors are you seeing, and what if anything have you changed since the last time that ssh worked for you? [00:09:04] bd808: I have changed laptop and system :D.. I have Windows 10.. [00:09:08] When I type ssh srwiki-dev.srwiki-dev.eqiad1.wikimedia.cloud [00:09:12] It shows this: [00:09:23] CreateProcessW failed error:2 [00:09:23] posix_spawn: No such file or directory [00:11:20] Content of my "config" file: https://pastebin.com/arPyw9uN [00:11:38] Zoranzoki21: the first hit I got for that error message in a web search is https://serverfault.com/questions/956613/windows-10-ssh-proxycommand-posix-spawn-no-such-file-or-directory [00:14:21] Now this happens: [00:14:21] ssh: Could not resolve hostname jumphost: No such host is known. [00:14:21] ssh_exchange_identification: Connection closed by remote host [00:14:48] https://pastebin.com/963ZQmYD [00:15:01] Zoranzoki21, I think you may have missed the point of the advice there [00:15:15] `jumphost` is not an actual host, it's the part you fill in with your jump host (bastion) [00:15:32] The idea was to take your existing proxycommand config and add the full path to the ssh executable [00:16:28] Ahh my mistake [00:16:32] You're right Krenair [00:16:42] Sorry! [00:16:48] you probably want something like `C:\Windows\System32\OpenSSH\ssh.exe -W %h:%p zoranzoki21@bastion.wmcloud.org` [00:16:58] Krenair: Right, I've fixed it and works :) [00:17:10] :) [00:17:54] oh and [00:17:55] -a [00:20:54] Krenair: You mean that I should add "-a" after "-W", right? [00:21:26] er, before -W [00:25:14] Krenair: This happens now: [00:25:15] all-A-records NIY [00:25:15] ssh_exchange_identification: Connection closed by remote host [00:25:58] you mean adding -a broke it? [00:26:56] It happened, but I've copied your content now, instead of using one from answer on serverfault.com, and it works [00:27:05] and added -a before -W [00:27:16] so... everything is working? [00:27:43] Yes.. Now I have another question related to wmflabs.org -> wmcloud.org [00:28:21] When I visit https://srwiki-dev.wmcloud.org/IfTemplates.json it shows me DNS_PROBE_FINISHED_NXDOMAIN. When I chanfge wmcloud.org to wmflabs.org, everything works. I have something to change, right? [00:30:23] uh, yeah, I haven't been playing as much attention to that change as I should have [00:30:27] this is a proxy thing [00:30:43] is there an option in the proxy UI in horizon to make a proxy under wmcloud.org? [00:33:06] Krenair: Yea, I've created it now. It still shows DNS_PROBE_FINISHED_NXDOMAIN, but I think that this is expected. [00:33:27] As it needs some time to starts working, right? [00:34:04] that'll just be cached somewhere [00:34:23] alex@alex-laptop:~$ dig srwiki-dev.wmcloud.org @ns0.openstack.eqiad1.wikimediacloud.org +short [00:34:23] 185.15.56.49 [00:34:43] it should work, give it like an hour at least [00:35:33] Okay, thanks. And "ping" command shows me same IP. :) [00:38:28] bd808: I can see now and m1 and g2 flavors in Horizon. Is this expected? [00:39:46] It was on cloud-announce [00:39:52] "tl;dr #3: New ‘g2’ VM flavors will soon be available in Horizon, at which point you are discouraged from using the old ‘m1’ names." [00:40:41] "In the days preceding this move, you will see several new flavor options appear in the Horizon interface for new VMs. They will have standard stats-based names preceded by ‘g2’, for example ‘g2.cores1.ram2.disk80'. These new flavors will be bound to the Ceph backend such that any new VMs created with those flavors will be run on new hypervisors and stored on the Ceph backend. You're encouraged to start using these new flavors as [00:40:41] soon as they appear." [00:40:45] Krenair: Right, I know. Should I move my VMs on g2 or to wait for WMCS to do it for me. What is better? [00:41:06] Zoranzoki21: wait [00:41:50] bd808: Okay. :) [00:43:03] IIRC you can't change flavours of an existing VM yourself anyway so you'd be talking about replacing them with fresh VMs [00:43:14] which is needlessly painful considering "Remaining VMs (on cloudvirts1012 through 1030) will be moved to Ceph in future weeks. Keep an eye out for emails announcing such moves." [00:43:54] (and other better references to a flavour change in the email that I should've used as an example instead) [00:45:27] Krenair: I know that I can't change flavours on existing VMs. :) Okay, I'll wait on WMCS, but I wanted to help them to have less work to do. [00:45:36] *on WMCS to do it [09:43:23] !log admin [codfw1dev] installed qemu security update in codfw1dev cloudvirts (T262386) [09:43:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:47:38] !log admin installed qemu security update in eqiad1 cloudvirts (T262386) [09:47:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:41:19] !log admin [codfw1dev] trying to get the bonding working for labtestvirt2003 (T261724) [10:41:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:41:23] T261724: cloudgw: evaluate / validate setup in codfw1dev - https://phabricator.wikimedia.org/T261724 [10:59:33] !log tools.wikiloves Deploy latest from Git master: dfbe5ea (T262375), daf759f [10:59:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikiloves/SAL [13:51:34] !help Hello, I'm moving my VMs to new storage platform (ceph). I want to install phpmyadmin, so I can easily manage databases, but when I want to do it, I get "phpmyadmin : Depends: php-twig (>= 2.9) but 1.24.0-2+deb9u1 is to be installed" [13:51:34] If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-kanban [13:52:19] Zoranzoki21: phpmyadmin is not allowed on wmcs — it has many long-standing security issues [13:52:32] andrewbogott: Okay, I won't install it then [13:52:55] when you say you're moving your VMs, do you just mean you're re-creating them with new flavors? [13:54:01] andrewbogott: Yes, and moving to use wmcloud instead of wmflabs. [13:54:55] Have you considered adminer instead of phpmyadmin? [13:56:47] Zoranzoki21: if you like I can migrate the VMs to ceph directly for you and save you a rebuild. [13:57:03] In theory you can change the hostname in place as well although that might be messier [14:01:22] andrewbogott: Thanks, you can move discordwiki :). srwiki-dev I've moved already [14:01:37] ok! I'll have a look in a few minutes [14:03:57] andrewbogott: Okay, thanks! [14:06:59] Zoranzoki21: what project is that in? [14:07:15] andrewbogott: discordwiki from srwiki-dev [14:07:20] srwiki-dev in srwiki-dev I've moved already [14:08:01] ok, moving now [14:13:07] andrewbogott: tysm [14:18:33] Zoranzoki21: all don [14:18:35] *done [14:19:00] andrewbogott: works, tysm [14:21:11] !log admin draining cloudvirt1001, migrating all VMs with wmcs-ceph-migrate [14:21:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [15:21:30] !help Hello, when I want to connect to my instance, I get this: "ssh: connect to host srwiki-dev.srwiki-dev.eqiad1.wikimedia.cloud port 22: No route to host" [15:21:31] If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-kanban [15:21:32] !help [15:21:32] If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-kanban [15:21:38] Oops [15:22:15] I've tried to "ssh" to bastion.wmcloud.org and then from there to try, but no works. Connecting to bastion works, but connecting from bastion to instance, no. [15:23:00] Zoranzoki21: that's one of the ones you just built? [15:23:13] andrewbogott: Yes [15:23:35] were you able to log in originally? [15:24:21] andrewbogott: I had to rebuild it again, and originally it was possible. But currently, it isn't. [15:24:38] ok, looking... [15:24:41] I got in with my root key. Looks like puppet had just finished running there. Was it rebooted? [15:25:07] Zoranzoki21: try now? [15:25:27] no works. [15:26:56] When I connect firstly to bastion and then to instance: https://pastebin.com/8vLXjN3Q [15:27:50] When I connect to instance without manual connecting to bastion: https://pastebin.com/xGNMwJGW [15:28:19] Zoranzoki21: it works for me (with my personal key). When you try to connect I don't see any activity on srwiki-dev [15:28:36] so I think something is messed up with your proxy setup. Can you connect to other cloud-vps hosts? [15:29:14] andrewbogott: interestingly, ping srwiki-dev.srwiki-dev.eqiad1.wikimedia.cloud from bastion-eqiad1-01.bastion.eqiad.wmflabs is returning ICMP unreachable [15:29:21] andrewbogott: When I want to connect to "discordwiki" instance in same project, it works. [15:29:32] Maybe I should restart instance, um? [15:29:56] bd808: huh, so it works from -restricted but not from -01 [15:29:59] what's that about? [15:30:05] Zoranzoki21: there is something going on with routing from the bastion you are using. [15:30:33] and -01 isn't the bastion I moved [15:32:29] bastion-eqiad1-02.bastion.eqiad.wmflabs shows "permission denied" [15:32:43] When I want to connect to srwiki-dev from there [15:37:23] I'll delete and create instance again... [15:39:07] Zoranzoki21: can you avoid deleting it for a few minutes? [15:39:20] We're in a meeting but i'd like to investigate what's going wrong before we fix it :) [15:39:22] andrewbogott: Oh, I've just done it. [15:39:28] already [15:39:32] ok, no worries :) please let me know if you see the issue again [15:39:42] Okay, I'll wait few minutes before creation [15:49:47] andrewbogott, bd808: Instance works now, I don't know how, but it works. :) [15:49:56] ok! [20:41:24] andrewbogott or bd808 hi, would it be possible to boost the performance of my project? each user request takes under 1 second on my own machine, and over a minute on dibabel pod [20:41:50] (or are you the wrong people to ping about this? my apologies if so) [20:44:24] yurik: you can give your webservice more cpu and ram yourself (up to your tool's quota) with the `--mem` and `--cpu` flags to `webservice`. [20:45:00] oh, thanks, that might help. I'm guessing the HD is a spinning rust, not SSD, right? [20:45:10] its NFS [20:45:28] ok, thx. Looking up my quota... [20:45:32] (googling) [20:45:43] Toolforge is horrible for anything that does a lot of disk IO [20:46:15] bummer, i have a few hundred local sqlite queries i need to run [20:46:32] i might have to spin up a memcached :( [20:46:32] sqlite is really horrible on Toolforge [20:46:47] ToolsDB will almost always be faster [20:47:08] hmm, how can i migrate to it? [20:47:23] change your code? :) [20:47:45] wait, toolsDB is a platform or something else? [20:47:52] ToolsDB is docuementes at https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#User_databases [20:48:43] bleh, I'm using SqliteDict -- which makes sqlite work as a simple key/value store [20:48:56] with values being arbitrary python objects (pickled) [20:49:25] i would rather migrate to a memcached-based store with a sqlite backend [20:49:54] memcached has no notion of multi-tenancy so we do not offer it in Toolforge [20:50:50] We do have a shared Redis that can be used (with some caveats) -- https://wikitech.wikimedia.org/wiki/Help:Toolforge/Redis_for_Toolforge [20:54:03] ah, great, thx, that might work well [20:55:33] heh, it doesn't sound like Redis has a multi-tenancy concept either - judging by the fact that I have to prefix everything with a random key :))) [20:58:24] yurik: mostly memcached has no way to disable its "list all keys" api and redis does. But yes neither of them are actually multi-tenant [20:58:47] bd808 does it mean you disabled "list keys with a prefix" API ? [20:58:52] i do use that functionality [20:59:12] (i didn't see anything about it in the docs) [21:00:47] yurik: https://redis.io/commands/keys is disabled. We should document that on the wiki pags [21:00:50] *page [21:01:16] sigh, more hacks to implement :( [22:25:45] it is in the docs already: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Redis_for_Toolforge#Security [23:32:17] AntiComposite, yep - https://wikitech.wikimedia.org/w/index.php?title=Help%3AToolforge%2FRedis_for_Toolforge&type=revision&diff=1881339&oldid=1865672 [23:32:22] as of a few hours ago [23:32:54] "e have also disabled the Redis commands that let users 'list' keys." has been there for years [23:33:21] i must have missed it when searching, thx