[01:23:37] Zppix, yes [01:23:57] changing context from default to toolforge = you're now talking to an entirely separate k8s cluster [01:24:14] you won't be able to see both simultaneously [01:39:18] Thanks Krenair [01:41:59] I recently had a brand new cloud ops project created. However, just logging into horizon for the first time today, I noticed there are 3 instances already created? [01:42:43] Is that normal? [01:43:40] TheSandDoctor, no, what's the project name? [01:44:25] @Krenair commons-corruption-checker [01:44:30] https://phabricator.wikimedia.org/T241635 [01:44:43] It's empty TheSandDoctor: https://tools.wmflabs.org/openstack-browser/project/commons-corruption-checker [01:44:52] can you give me an example of an instance you see in horizon? [01:45:05] bastion-eqiad1-02 [01:45:26] um [01:45:30] that won't be a part of your project [01:45:36] that will be a part of the bastion project [01:45:44] sounds like you logged into horizon but never switched to your project? [01:46:28] you'd be right. I didnt realize I had to and was rather confused (never logged into horizon before) [01:46:33] thanks @Krenair [01:46:56] how else would it know which project you want to look at? [05:11:14] I was happily on my vps just now and got booted off. (Closed by remote host) [05:11:34] Now trying to reconnect gives me Permission denied (publickey) [05:11:49] Same correct password and it passes first authentication prior to jump [05:14:43] debug1: Authentication succeeded (publickey). [05:14:43] Authenticated to primary.bastion.wmflabs.org ([185.15.56.13]:22). [05:14:47] fails after that [05:15:33] cycles through my keys and then says [05:15:41] https://www.irccloud.com/pastebin/Sdfvl70q/ [05:30:02] !help ^ [05:30:02] TheSandDoctor: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [05:30:44] TheSandDoctor: I'll have a look [05:31:31] TheSandDoctor: try again now? [05:32:05] same @andrewbogott [05:32:14] yep, I'm watching the logfile [05:36:45] I think I'm able to reproduce the problem, investigating still [05:37:13] ok. thanks @andrewbogott [05:37:17] rather odd haha [05:37:22] any chance you uninstalled python? [05:37:35] I moved it and put 3.8 in [05:37:38] that break things? [05:37:40] ok, that's what broke it [05:37:44] darn [05:37:52] you can have multiple versions of python installed, no need to break the existing python interpreter [05:37:55] so instance bricked now? [05:37:56] lots of system things rely on that [05:38:07] nah, I can probably reinstall python2 if that won't ruin your day [05:38:15] no it wont [05:38:23] thanks @andrewbogott and apologies [05:38:41] https://www.irccloud.com/pastebin/rl6cOgKR/ [05:38:45] ^ definitely the problem :) [05:39:14] I see [05:39:21] lol oops [05:39:36] didnt realize it depended on python2 [05:39:43] didnt think that that would brick anything haha [05:39:44] boo [05:41:47] TheSandDoctor: try now? [05:42:12] in. [05:42:16] great [05:42:16] thanks @andrewbogott [05:42:23] didnt think that that would break it haha - sorry [05:42:30] I just did 'apt-get install —reinstall python-minimal' to get the binary put back in /usr/bin/ [05:42:34] no worries! [05:43:40] @andrewbogott How did you get in without it booting you off? [05:43:49] I have a root key :) [05:43:53] :P [05:44:02] it bypasses the ldap lookup (which is a python script) [05:44:06] handy for cases like this [05:44:11] definitely [05:44:26] Sounds like that python script should be tested with and switched to python3 [05:44:41] hopefully that is updated to python 3 at some point. 2 is being discontinued very soon I thought(?) [05:44:47] 19 days ago. [05:44:51] ^^ [05:45:10] @andrewbogott is the script public by any chance? [05:45:17] or is that a restricted/closed thing? [05:45:20] (well, kinda, there's one release left, but there's a change freeze between now and then) [05:45:29] * TheSandDoctor might try updating if it is former [05:45:39] cat /usr/sbin/ssh-key-ldap-lookup [05:45:48] but it's puppetized, will be reverted if you change it [05:45:53] :( [05:46:08] so....someone with the permissions needs to fix it for one & all :P [05:46:14] * TheSandDoctor nominates @andrewbogott [05:46:15] :P [05:46:28] patches for https://phabricator.wikimedia.org/T229920 welcome [05:46:55] subscribed and I shall take a look @andrewbogott. thanks for the link [05:47:20] pretty much all those scripts are in https://gerrit.wikimedia.org/r/#/admin/projects/operations/puppet [05:47:54] thanks! [05:47:59] * TheSandDoctor will take a look shortly [05:48:32] * andrewbogott -> lunch [06:08:21] !log codesearch temporarily taking everything down to free up disk space (T243121) [06:08:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Codesearch/SAL [06:08:24] T243121: CodeSearch shows unable to contact hound on visit - https://phabricator.wikimedia.org/T243121 [06:16:30] Took a look at the script, the only changes that need to be made are the /usr/bin/python at the top and the print statement on line 134. That and bumping pyyaml and python-ldap to a python3 version [14:47:44] https://github.com/wikimedia/operations-mediawiki-config/blob/master/wmf-config/LabsServices.php#L15 probably want to change that? [17:41:47] !log tools.wikifile-transfer Added User Preference (T241736) [17:41:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikifile-transfer/SAL [17:41:51] T241736: wikifile-transfer: Add "save preference" - https://phabricator.wikimedia.org/T241736 [18:26:56] That's strange. On windows ssh-ing into my instance and using 'sudo' asks for a password, yet on mac it doesnt [18:27:17] I dont think I have a password(?) I connect via keys [18:27:49] nvm. [18:27:51] that's on me [18:27:59] for some reason it decided not to forward me [18:31:14] == [18:31:22] https://www.irccloud.com/pastebin/LFketnkT/ [18:31:40] For some reason I get that on windows 10 when following https://wikitech.wikimedia.org/wiki/Help:Accessing_Cloud_VPS_instances#Windows_10 [18:35:17] Even when adding to my ssh config file the entries from that link it doesnt like it on windows (tbf though, it didnt on mac either - but the above command worked there) [18:35:21] https://www.irccloud.com/pastebin/CuCKkIQr/ [18:48:10] Deploying VideoCutTool back-end in toolforge at https://tools.wmflabs.org/video-cut-tool-back-end/ is failing multiple times, It's a node.js tool and node version is v12.14.1. I realised toolforge is only supporting node version <=node10 got to know from https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#Running_npm_with_webservice_shell, Requesting WM Cloud services to enable to deploy nodejs tools version [18:48:10] 12.14.1 sooner. [19:04:20] !help [19:04:20] TheSandDoctor: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [19:11:30] * Gopa created phab regarding this: https://phabricator.wikimedia.org/T243159 [19:14:11] TheSandDoctor: Still having issues with Win10, or is it something else? [19:15:19] I haven't had any problems using my config on win10, but you have to deliberately enable the openssh agent in services.msc if you are using the agent (not always entering the passphrase for your private keys) [19:16:59] Gopa: thanks for the task. We typically support what Debian has in their internal repos. For the time being that's likely to remain the same, but we'll take a look at backports and things. [19:18:18] bstorm_: Thanks and I wish to see our VideoCutTool back-end sooner in live :) [19:19:33] TheSandDoctor: if you come back, I'm interested in any info about sudo problems on this task: T205463 Just in case it has to do with more than just a random LDAP connection thing....so far, it seems to be LDAP connections and not OS/shell [19:19:34] T205463: sudo randomly prompts for passwords in cloud instances when LDAP lookups fail - https://phabricator.wikimedia.org/T205463 [19:25:08] @bstorm_: sorry, was afk for a sec. I do have ssh enabled on windows 10 and can connect to tools just fine [19:25:17] I just get those errors specifically with the jump [19:25:24] It doesn’t like the second half of the command. [19:25:58] (Also, I will be in an incident log on bastion as I thought I was on my instance and didn’t realize it didn’t connect through when I tried to sudo switch user) [19:26:04] So my apologies on that [19:26:07] @bstorm_: [19:26:35] How about using `ProxyCommand ssh -q -a -W %h:%p` instead of ProxyJump? [19:26:59] Ahh ok [19:27:03] I get what you are saying [19:27:27] I almost always use ProxyCommand...I thought the windows OpenSSH does ProxyJump well, but maybe it does something odd? [19:27:45] You have the OpenSSH Agent enabled, too, right, not just OpenSSH? [19:28:08] I realize my config using ProxyCommand. [19:28:15] *is using* [19:29:11] Looking. One sec [19:29:48] I know it's the one piece that is disabled by default on Windows10. [19:30:24] OpenSSH client is enabled [19:30:31] dont see anything about agent anywhere [19:30:36] nor in the available ones to add [19:30:46] * TheSandDoctor just sees OpenSSH server [19:30:50] Trying WindowsKey-F and search for "services.msc" [19:31:04] *try...why can't I type today? [19:31:42] OpenSSH Authentication Agent? [19:31:46] is that what you mean? [19:31:51] Yeah :) [19:31:54] That would be it [19:32:01] disabled [19:32:03] huh [19:32:06] Enable that [19:32:07] set to automatic? [19:32:11] yes [19:32:13] That's my config [19:32:33] That should make your config more like a Mac or Linux box [19:32:53] Then see if that affects proxy-jump (I don't recommend agent forwarding, but having an agent locally might affect that) [19:33:06] same error for ssh `commons-corruption-checker-main.commons-corruption-checker.eqiad.wmflabs` [19:33:10] Ok [19:33:35] same error also for `ssh -J thesanddoctor@primary.bastion.wmflabs.org thesanddoctor@commons-corruption-checker-main.commons-corruption-checker.eqiad.wmflabs` [19:33:55] w/ [19:33:59] https://www.irccloud.com/pastebin/4ZDOgE17/ [19:34:35] In that case, I'd change the .ssh/config part about that to use: `ssh -o ProxyCommand="ssh -W %h:%p thesanddoctor@primary.bastion.wmflabs.org" thesanddoctor@commons-corruption-checker-main.commons-corruption-checker.eqiad.wmflabs` [19:34:56] Sorry not the config...the command. Ugh. I'm sleepy today. [19:35:11] Try this command instead: `ssh -o ProxyCommand="ssh -W %h:%p thesanddoctor@primary.bastion.wmflabs.org" thesanddoctor@commons-corruption-checker-main.commons-corruption-checker.eqiad.wmflabs` [19:35:32] Let me open windows10 and play with it as well [19:35:36] first one worked. trying second [19:35:40] Ah good [19:35:44] They are the same command [19:35:54] kk [19:36:03] Ok, so perhaps your version of ssh doesn't like ProxyJump...just ProxyCommand [19:36:14] how would I modify the config to clean that up a bit? [19:36:17] I'm curious about my own version because I use ProxyCommand a lot [19:36:19] because that's a long one [19:36:24] Here [19:36:25] and how do I check my version? [19:36:35] ssh --version [19:36:45] nope [19:36:48] that doesn't work lol [19:37:34] you can ssh with the "-v" option, and it will tell you the local version string [19:37:51] powershell I was able to find the version [19:37:54] https://www.irccloud.com/pastebin/sXAauzhW/ [19:37:55] On my mac it's "SSH-2.0-OpenSSH_7.9" (in the massive output that comes with sshing to something with -v) [19:38:02] Wow. That's old [19:38:03] Ok [19:38:06] that's windows [19:38:10] their version [19:38:16] Yeah [19:38:22] * TheSandDoctor wishes it was newer [19:38:28] So in your config, you'd use this [19:38:34] but....that's microsoft for you :P [19:38:46] I wonder about my own version. I think it is a later one [19:38:52] But I'm using "Insiders Release" [19:39:25] `ProxyCommand ssh -q -a -W %h:%p thesanddoctor@primary.bastion.wmflabs.org` [19:39:33] Instead of the "ProxyJump" line [19:39:39] Then you can use the config [19:39:43] ah. I am on education and the semi annual stream. Microsoft tends to roll out versions that break everything from what I've read [19:39:50] * TheSandDoctor is leery of the 1909 upgrade [19:39:57] I should add something to the doc for this [19:40:00] * TheSandDoctor knows in the past they have bricked machines with their rollouts [19:40:07] please do @bstorm_ [19:40:07] Yeah, that's fair :) [19:40:27] something like how on windows they have a stupid old version that wont work with the "modern" command [19:40:29] :P [19:41:15] Yeah, I'm checking my version. I could have sworn I had a rather recent one, but I have a lot going on in my config (including WSL2 because then I have full Linux) [19:42:04] changing that in the config worked. [19:42:09] thanks @bstorm_! [19:42:09] :) [19:42:19] Awesome! [19:42:45] I'll see if I can find a version matrix so I can find out which Windows builds have which version of SSH [19:43:51] Thanks [19:43:59] I’d be interested in seeing that as well [19:44:16] Local version string for me is ` SSH-2.0-OpenSSH_for_Windows_7.7` [19:44:29] Huh [19:44:33] That’s different [19:44:38] Windows_7 [19:44:45] Haven’t seen that in a while 😂 [19:45:03] Well, this is OpenSSH 7.7, thankfully :) [19:45:23] So I guess they update it with one of the new releases? Which version are you on? [19:46:07] My build is 19041.1 [19:46:23] But that's not a stable release [19:46:26] That's insiders :) [19:47:03] That is a fair bit inside haha [19:47:11] 1909 is the latest release [19:47:12] Haha [19:47:35] 😁 [19:47:42] I’ve read they don’t even fix some of the major bugs in 1909 [19:48:09] So `ssh -V` checks version. I can add something to the doc that has you check that first and then decide on proxyjump [19:48:37] https://github.com/PowerShell/Win32-OpenSSH/issues/1172 [19:48:40] That’s a good idea [19:48:58] https://www.laptopmag.com/news/microsofts-fix-for-severe-windows-10-security-flaw-isnt-installing-for-many-users [19:49:08] They don’t roll out very well, do they? [19:49:32] What do you get with `ssh -V` ? [19:49:44] No they really do not [19:50:16] They were better at rollouts in the past. It seems like they are too busy building spyware Cortana things lately :) [19:50:25] :P [19:50:32] * TheSandDoctor basically uninstalled Curran’s [19:50:39] Cortana* [19:50:48] Yeah, I did my best. [19:50:56] Education edition does a lot for that too [19:51:10] Allows a ‘0’ setting for their spyware [19:51:16] Instead of usual ‘1’ [19:51:22] What is the full version string on yours for `ssh -V`? [19:51:23] * TheSandDoctor likes anti beacon [19:51:30] Out of curiosity. [19:51:32] I will look that up for you. One sec [19:51:42] That's cool. Makes me interested in the education edition for my kid. [19:52:00] Her "home" version is terrible [19:52:39] OpenSSH_for_Windows_7.6p1, LibreSSL 2.6.4 [19:53:15] but I would also recommend looking at AntiBeacon (made by same people who make spybot search & destroy). Costs some money but allows you to disable a lot of crap [19:53:22] and you can completely uninstall cortana btw [19:53:42] https://www.pcworld.com/article/3109900/you-can-remove-cortana-from-windows-10-but-its-tricky.html [19:53:51] Ok, so you are using exactly the version in that bug report here https://github.com/PowerShell/Win32-OpenSSH/issues/1172 [19:54:00] It's 7.6 [19:54:16] * TheSandDoctor is most concerned about how win10 key logs [19:54:21] Heh. [19:54:33] researchers have found it to send packets every keypress [19:54:44] ugh. Yeah. It's a nightmare [19:54:56] that's why I load up on the antispyware [19:54:57] :P [19:55:40] privacytools.io (can't really endorse, but they have some good info) points out a possibly ok binary program that tries to close privacy leaks in win10. I've found it missed some things, but helped a bit. I've mostly been hacking my own registry and GPOs to close stuff [19:56:19] 👍 [19:56:23] * TheSandDoctor has done same [19:56:24] Again, I haven't dug into source code on that program (don't think it is even open), but I was willing to test it on a VM lol [19:56:41] Fair :) [19:56:47] * TheSandDoctor also did some registry edits to lower the levels [19:56:52] They don’t make it easy [19:56:59] But you can get level 0 with education [19:57:08] Education is literally enterprise but rebranded [19:57:29] And best part is that my uni gives everyone with an account a free copy [19:57:35] Indefinite license [19:57:36] :) [19:58:32] @bstorm_: [19:58:40] Very nice [19:58:57] I wonder if they do that for homeschoolers? Probably not :-D [19:59:32] I don't think the Foundation wants to get us Windows licenses, either. Open source and all that. [19:59:37] bstorm_: while you're around, do you know the answer to https://phabricator.wikimedia.org/T229920#5815022 ? [20:01:00] * bstorm_ answers with an opinion [20:01:06] I wonder if all of those are still needed... [20:01:22] As versions we've upgraded from/past [20:01:32] We probably only need one version back for the emergency rollback that will hopefully not happen. [20:01:37] But we'll get to it? [20:01:54] We also might need some of that stuff for clients more than anything. [20:10:38] TheSandDoctor: I think this covers what we learned https://wikitech.wikimedia.org/w/index.php?title=Help:Accessing_Cloud_VPS_instances&diff=1851200&oldid=1850703 [20:10:40] Thanks! [20:11:25] Thanks @bstorm_ [20:11:27] :) [20:11:41] 11:59 AM I don't think the Foundation wants to get us Windows licenses, either. Open source and all that. [20:11:42] Darn [20:11:59] 😁 [20:13:31] @bstorm_: should probably be documented somewhere not to overwrite the default python install [20:14:00] * TheSandDoctor did that last night and if it weren’t for @andrewbogott he would’ve had a bricked instance [20:14:05] @bstorm_: [20:15:19] Well, that's arguably a sysadmin best practice, which we seem to mostly avoid in our docs, hoping docs elsewhere capture it, but maybe if we have a CloudVPS doc on pythonish stuff it wouldn't be a bad footnote :) [20:15:47] Too many OSs rely on python internally [20:16:05] So it's pretty dangerous on Linux or even Mac [20:16:24] There's probably a good spot to add it somewhere [20:16:43] * TheSandDoctor is just used to windows where it doesn’t matter haha [20:16:59] You can get away with https://github.com/pyenv/pyenv [20:17:25] Yeah, Windows doesn't care at all. One of the few really nice things on that end. [20:17:50] pyenv works like rubyenv...if you've done ruby or rails stuff [20:18:58] 12:17 PM Yeah, Windows doesn't care at all. One of the few really nice things on that end. [20:19:00] 100% [20:19:10] * TheSandDoctor wishes everything was just the latest python [20:19:11] Haha [20:19:26] Because python2 is discontinued as of a few days ago [20:47:59] @bstorm_: do you recommend any Debian database management tools? [20:48:41] * TheSandDoctor knows of phpmyadmin, but hasn’t used anything else and his vps doesn’t have a floating ip [20:49:00] I suspect very few of them are debian specific [20:49:14] doesn't have a floating ip? so it's static? [20:56:39] I don’t think it has a public facing ip(?) [20:56:44] * TheSandDoctor will have to look closer [20:56:52] * TheSandDoctor isn’t sure [20:56:58] * TheSandDoctor isn’t used to cloud vps’ [21:00:37] You can certainly request one (you might need to give a reason) [21:00:49] You can still do things like SSH tunneling and stuff to access it too [21:03:09] SSH tunnelling is how I’ve been getting in as of late. Easier on the Mac than win10. Latter required some troubleshooting in here earlier @Reedy [21:03:19] It's not [21:03:26] I was doing it on windows with putty.. 10+ years ago [21:03:29] You just save the config [21:42:13] !help I umm...might've forgot to keep ssh open when I launched my instance firewall. Looks like I have to nuke it? [21:42:13] TheSandDoctor: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [21:42:23] (allow ssh) [21:48:50] In theory, there should be a way to pop a console shell (at least for cloud staff) to fix it [21:56:58] @Reedy do you happen to know who here is cloud staff (other than andrew who is afk)? [21:57:17] I suspect most of them are AFK because it's a weekend [21:57:27] Brooke who was helping you earlier is too [21:57:38] I would suggest just to file a task and see if someone can help [21:57:43] kk [21:59:50] @Reedy https://phabricator.wikimedia.org/T243161 [21:59:56] thanks for your help [22:33:27] TheSandDoctor: happen to know the command to open the firewall? [22:41:40] andrewbogott: opensesame [22:42:16] * andrewbogott can never remember iptables syntax [22:42:28] * andrewbogott packing for a flight so will leave this in the hands of others [22:42:59] andrewbogott i have the syntax :) [22:43:05] @andrewbogott: sudo ufw allow ssh [22:43:06] sudo iptables -A INPUT -p tcp --dport 22 -j ACCEPT [22:43:17] @paladox: nope. Not one I’m using [22:43:21] I’m using ufw [22:43:33] yup, but that'll open it i think? At least that did with ferm. [22:43:36] you can also stop ufw [22:43:48] @andrewbogott: I’m out at the moment so can’t confirm for a bit [22:43:49] sudo service ufw stop [22:44:06] That too would work and allow me back in ^ [22:47:33] ok, done — gotta go! [22:52:56] Thanks @andrewbogott ! It said something like “rule added” in response, right? [22:53:03] One or two entries (either or) [22:53:16] * TheSandDoctor will check either way when he gets home [22:53:26] And I’ll close ticket if all’s good. Thanks again