[00:00:14] andrewbogott: It doesn't [00:00:19] Had to get it off of the web [00:00:35] TParis: it went invite only ~2 years ago :) [00:00:50] I think the redirect expired [00:01:01] It must have been 2 years since I've been on then [00:01:03] ebernhardson: ok, well, loathe as I am to suggest this… try EBernhardson in horizon? [00:01:31] * andrewbogott closed all his logging windows and now regrets it [00:01:31] TParis: what failure are you getting with ssh to login-stretch.tools.wmflabs.org? [00:01:50] note the .tools. in the middle there. that has tripped up a couple of people [00:01:55] publickey denied [00:02:18] I'm trying to connect from bastion 3 [00:02:35] TParis: you won't want to hop between bastions, just ssh directly to whatever bastion you need. [00:02:35] hmmm.. same for me :/ I wonder if our ldap is hosed there [00:02:41] That's what makes them bastions :) [00:03:07] login-stretch.tools.wmflabs.org works for me [00:03:33] andrewbogott: nope, tried all 3 variants. no failures showing up in authentication.log anymore [00:03:46] I guess I don't understand the infrastructure, then, Andrew. I'm on tools-bastion-03 at the moment and I'm trying to get my tools off trusty before March. My tool jobs are listed under crontab on bastion. I am trying to move them to stretch. [00:04:19] andrewbogott: maybe there is some soft-lockout from failing too many times and it's just not trying anymore? [00:04:33] * ebernhardson is randomly guessing now [00:05:37] TParis: the instructions on that page aren't entirely clear… they have a step where they say to 'exit' twice, which I take to mean 'log out from everything and start over on your local machine' [00:05:46] TParis: I'm watching the auth.log on login-stretch.tools.wmflabs.org, so I may be able to see the problem if it fails for you again. I don't think you can directly ssh from bastion-03 to it [00:06:34] "Failed hostbased for tools.tpbot" -- yeah, that does not work [00:07:01] ebernhardson: if you were locked out it would tell you [00:07:01] ok, looks like I'm in [00:07:14] the instructions I wrote are not clear apparently. The idea like andrewbogott said is to connect from your local computer to both [00:07:40] I assumed I had to run those commands from bastion [00:08:03] So, what is the difference between the login methods? All of my files are in both places? [00:08:34] TParis: the grid engine grid that you will see [00:08:35] hmm, so tailing all the other logs (tail -f /srv/mw-log/* | grep -i ebernh) does show requests being made... [00:08:57] the NFS mounts are the same, but the old and new bastions connect respectively to the old and new job grids [00:09:01] Okay [00:09:28] ebernhardson: ok, once more? [00:10:01] andrewbogott: done, with Ebernhardson [00:10:58] Thanks bd808 & andrewbogott for suffering my silly questions ;) [00:11:07] not silly! [00:11:09] not silly at all TParis [00:11:31] ebernhardson: just to double check, logging in from wikitech definitely works now? With 2fa and all? [00:11:48] andrewbogott: ya i logged out and logged back in after setting it, lemme do it one more time [00:12:20] andrewbogott: logged into wikitech with 2fa, works [00:12:30] ok, turning on yet more debug channels... [00:14:26] ebernhardson: let's try once more [00:14:33] thank you for your patience! [00:15:37] andrewbogott: done, failed login. I notice that again while i provide Ebernhardson to horizon (verified in network tab of browser) the mediawiki log still says: labweb1002 labswiki 1.33.0-wmf.19 api INFO: API POST StrikerBot 2620:0:861:1:208:80:154:23 T=25ms action=oathvalidate format=json user=EBernhardson totp=014569 token=[redacted] formatversion=2 [00:15:43] (with the B capitalized still) [00:16:00] is pip not available on here? [00:16:37] TParis: your tools don't actually run on the bastion, so installing packages there won't help much. If you need uninstalled packages then… I will refer you to bd808 for advice :) [00:19:30] ebernhardson: I was assuming that this was now failing before it got as far as checking the 2fa token. But it sounds like you're seeing otherwise? [00:19:43] * andrewbogott opens up that log again [00:20:41] andrewbogott: i'm seeing the api request log, but no attempt to auth the token [00:20:55] andrewbogott: suggests to me horizon is still sending EBernhardson, but now mediawiki says that user doesn't exist? [00:21:59] the log says the request failed, but not why [00:22:02] yeah, that's a reasonable possibility [00:22:17] I'll rename you and we'll see :) [00:22:21] :) [00:24:13] try now? [00:24:41] andrewbogott: magic! [00:24:48] omg what a mess [00:25:09] new rule: all names are a-z. Thats it. Too bad [00:25:11] ok, I'll put notes about what I did on your ticket, so when you start noticing other disasters that happen as a side-effect of the renaming we'll know how we got there :/ [00:25:21] :) [00:29:56] ebernhardson: can you verify that you can still log into a VPS? [00:30:37] good call, checking [00:30:50] ebernhardson@bastion.wmflabs.org: Permission denied (publickey). [00:30:54] lemme try reuploading [00:31:28] what host are you connecting to? [00:32:07] andrewbogott: deployment-deploy01.deployment-prep.eqiad.wmflabs [00:32:11] i should just login to bastion though, sec [00:32:16] yeah, that's simpler [00:32:49] andrewbogott: deleted key, re-added, can login to both now [00:32:53] great [00:32:55] thanks for checking [00:33:04] I'm going to declare victory for now :) [00:33:14] sounds good, i'll find out over the coming weeks whats broken :) [03:04:20] @seen TParis [03:04:20] bd808: Last time I saw TParis they were quitting the network with reason: no reason was given N/A at 3/1/2019 2:51:58 AM (12m21s ago) [10:39:58] !log math shut down drmft instance again (was active by mistake due to workload reallocation) Will be deleted soon anyway [10:40:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Math/SAL [10:43:22] !log design T204502 shutdown lsg-01 instance (was active by mistake due to workload reallocation) Will be deleted soon anyway [10:43:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Design/SAL [10:43:26] T204502: cloudvps: design project trusty deprecation - https://phabricator.wikimedia.org/T204502 [11:24:18] I’m trying to migrate my tomcat webservice to stretch, but it seems to die directly after start (or on the first request). Any ideas how to debug or fix this? [11:43:30] ireas: nothing on the logs? [11:43:47] I suggest to create a phab task [11:48:20] arturo: nothing in the service.log. is there somewhere else I could look? [11:56:50] HELP [11:57:08] pungppk_: ? [11:57:13] may I help you? [14:43:39] Are the issues with NFS (from 4 days ago) uninstalling programs and changing database accesses? [14:44:55] I can't even login to my production instance, it hangs [14:46:06] AmandaNP: what instance? [14:46:39] stuff screwed up on utrs-database2 and can't login to utrs-production2 [14:47:39] I'm not likely to be much help regarding the db but I'll see if I can figure out what's happening with utrs-production2 [14:48:07] did you try rebooting it already? [14:48:22] andrewbogott: like I had to resintall pythons mysqldb package on utrs-production2 cause it was missing [14:48:31] and no [14:48:39] ok, I'll try that next [14:50:24] andrewbogott: are you gonna reboot the db one? would make sense too with the crap i'm getting [14:50:41] !log utrs rebooting utrs-production2 to resolve nfs-mounting issues [14:50:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Utrs/SAL [14:50:59] AmandaNP: less likely to help there but I'll look at that next [14:51:12] kk thanks worth a shot [14:52:21] of course you can reboot it too [14:52:46] fair i could. /me goes to find login [14:55:21] !log utrs reinstalled python-mysqldb on utrs-database2 because it's coming up missing [14:55:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Utrs/SAL [14:56:41] utrs-production2 seems happier now; can you log in? [14:57:55] !log utrs rebooting utrs-database2 [14:57:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Utrs/SAL [14:57:59] sec [14:58:25] yes in now [14:59:46] great [15:21:06] AmandaNP: did the reboot help with the db issues? [15:21:26] no I have to reset my db password, can't even login from the command line [15:21:43] you probably have sudo right? [15:22:02] yes. and i'm sure the password is right i double checked a credential file [15:22:28] hm, you have the database stored on NFS? [15:22:39] That'll work but I'd expect the performance to be kind of terrible [15:23:50] * andrewbogott back in a bit [15:24:54] !log utrs reset db password for deltaquad due to inability to login with right password [15:24:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Utrs/SAL [15:30:57] that fixed it [15:33:03] andrewbogott: on production, i'm getting 400 errors trying to access enwiki's api [15:33:10] or was [15:33:17] maybe the reboot solved that? [15:37:55] I think it did from wget [15:38:50] !log utrs wget tested on utrs-production2 to verify errors in apache log are clear. Everything looks good [15:38:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Utrs/SAL [15:41:19] good god it uninstalled another python package too [15:42:47] it even took out pip ffs [15:43:49] !log installed missing "pip" [15:43:49] AmandaNP: Unknown project "installed" [15:43:57] !log utrs installed missing "pip" [15:43:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Utrs/SAL [15:44:41] !log utrs installed missing requests_oauthlib via pip [15:44:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Utrs/SAL [15:45:15] finally back to operating [15:48:37] great! I can't at all explain why you would be missing packages unless you switched from an NFS-mounted homedir to a locally-mounted one, or the reverse [15:51:29] AmandaNP: were those packages installed in $home with pip, or installed system-wide? [15:52:16] system wide [15:52:43] like even pip itself was missing [15:53:51] python-mysqldb was definitely system wide [16:08:11] that I cannot explain :( Glad it's working again though! [18:11:18] hi guys, who can I ask about a Membership requests for tools server? [18:15:05] dsaez: https://wikitech.wikimedia.org/wiki/Help:Getting_Started#Get_started_with_Toolforge describes the process, but if you have questions, feel free to ask! [18:17:47] hi valhallasw, here Diego from the Research team. I'm working with an student, and he has follow all that instructions, but for some reason his requests is still looking as "pending", I'm wondering if he is missing something or if it is juts mater of time. [18:18:52] dsaez: mostly a matter of time; it's a manual process. Let me take a look! [18:19:06] cool! thanks! [18:24:17] dsaez: done! sorry for the delay. [18:24:29] great! thank you very much! [18:55:17] The myrcx tool is down. We use it for WHOID [18:55:26] *WHOIS. What happened to it? [19:02:28] Cyberpower678: The webserver itself seems to be running, although it's not showing me any content either. https://phabricator.wikimedia.org/p/Samtar/ is the maintainer of the tool, so they would be the best to ask. [19:04:39] I just sifted through the tools public_html folder. It looks to be deleted [19:04:44] valhallasw`cloud: ^ [19:07:12] Cyberpower678: T172745 seems to indicate that Smatar shut the tool down well over a year ago? [19:07:13] T172745: Tool "myrcx" loads jquery from ajax.googleapis.com - https://phabricator.wikimedia.org/T172745 [19:07:14] Cyberpower678, the sole maintainer is Samtar [19:07:15] Cyberpower678: fwiw, the access log shows 404s on /whois in december _2017_, and no accesses that are not 404s, so it seems like that has been the case for a long time [19:37:12] Is https://tools.wmflabs.org/splinetools/whois/ip/ still actively maintained? [19:40:32] Kb03: maintainer info is here -- https://tools.wmflabs.org/admin/tool/splinetools -- looks like you need to ask https://wikitech.wikimedia.org/wiki/User_talk:Reticulated_Spline that question [19:42:47] Kb03: that being said... I do not see a lot of sign in the tool's homedir of activity after January 2015 [19:42:55] Bah, I have to create a Dev account [19:42:57] looks like the SUL user of that name became Richard0612 who became User:firefly ? [19:43:36] to do what exactly? [19:43:52] Leave a talk page message [19:44:13] on wikitech? yes you need an account to edit there [19:44:21] I'll ask on enwiki [19:44:42] I'm browsing through the sharedIP templates seeing if any of the linked tools are no longer working [20:49:02] hi, I try to login to login-stretch.tools.wmflabs.org and I get connection refused, where tools-login.wmflabs.org still works [20:49:09] different ssh config? [20:55:38] akoopal: I think you need to remove the .tools. from the hostname [20:55:55] hm, maybe not, it works for me [20:55:57] * andrewbogott tests some more [20:56:52] akoopal: ok, I've confirmed that you had it right the first time. can you try again? [20:57:14] nope [20:57:38] I have an older RSA key, maybe not supported anymore [20:57:44] yeah, that's possible [20:58:02] unless you have .ssh/config specified for one but not both hosts, I would guess your key needs updating [20:58:22] yep, "userauth_pubkey: key type ssh-dss not in PubkeyAcceptedKeyTypes [preauth]" [20:58:45] bd808: I left him a talk page message but it appears he's been inactive since July [20:59:22] andrewbogott: [20:59:25] debug1: Authentications that can continue: publickey,hostbased [20:59:25] debug1: Next authentication method: publickey [20:59:25] debug1: Offering RSA public key: /home/andre/.ssh/id_rsa [20:59:25] debug1: Authentications that can continue: publickey,hostbased [20:59:27] debug1: Offering DSA public key: /home/andre/.ssh/id_dsa [20:59:30] debug1: Authentications that can continue: publickey,hostbased [20:59:32] debug1: Trying private key: /home/andre/.ssh/id_ecdsa [20:59:35] debug1: No more authentication methods to try. [20:59:38] the rsa key is in my authorized keys [21:02:38] All the log says is [21:02:50] https://www.irccloud.com/pastebin/GMQQYJeT/ [21:03:36] I have to go in a minute, but for starters I think you should generate a new key as per https://www.mediawiki.org/wiki/Gerrit/Tutorial#Set_Up_SSH_Keys_in_Gerrit [21:03:43] akoopal: the new debian host is much stricter with SSH keys [21:03:45] and try passing it explicitly with -i [21:03:51] (and upload the new key of course) [21:03:59] valhallasw`cloud: creating a new key atm [21:04:26] andrewbogott: DSA is no longer accepted, RSA with non-standard key lengths is also rejected [21:05:33] valhallasw`cloud: ok — the one generated by the doc link I just pasted will work though won't it? [21:06:45] valhallasw`cloud: following https://wikitech.wikimedia.org/wiki/Production_shell_access#Generating_your_SSH_key [21:07:05] yes, RSA should be fine as long as you don't change or mistype the key length :-) [21:07:07] going for rsa 4096 bits [21:07:34] can I edit authorized_keys myself? [21:07:41] or do I need to upload? [21:08:01] yes, from wikitech you can change the ssh keys [21:08:21] https://wikitech.wikimedia.org/wiki/Special:Preferences#mw-prefsection-openstack [21:11:57] thanks, I am in :-) [21:15:00] valhallasw`cloud: oh man. thanks for pointing that out. I will add a migration FAQ about ssh keys and also send an email about it. [21:15:31] * bd808 knows he wrote a "deprecate DSA keys" bug 2+ years ago and then did nothing about it [21:15:34] hmm, the sshd config as dsa as a host key. [21:15:38] Should that be removed? [21:15:50] https://github.com/wikimedia/puppet/blob/production/modules/ssh/templates/sshd_config.erb#L18 [21:15:53] bd808: I'm not exactly sure about the new requirements -- these are just the two that I have seen happen [21:18:57] valhallasw`cloud: I think I can poke around in the sshd config and change logs and figure out some reasonable guidelines. I don't think we need to search for "absolute truths", jsut make it a bit less likely that people get stuck [21:19:17] *nod* [21:19:36] I should even be able to script a search for keys that are likely to have problems in the LDAP datastore [21:20:50] we should start by listing the new requirements [21:32:49] we have T168433, but if there are RSA key length restrictions that would be good to figure out as well [21:32:49] T168433: Deprecate DSA (ssh-dss) SSH keys for Cloud VPS and Toolforge users - https://phabricator.wikimedia.org/T168433 [22:18:14] bd808: took me a while to find the log file again, but this is what I got with an old 1023 bit (clearly I mistyped this when I generated the key... many years ago) RSA key: error: userauth_pubkey: could not parse key: Invalid key length [preauth] [22:18:18] bd808: but in retrospect, that might be because of the 1023 rather than the short length itself [22:18:47] https://stackoverflow.com/questions/48055006/ssh-add-invalid-key-length suggests anything < 1024 is rejected in recent sshds [22:19:17] *nod* that seems like a reasonable assumption [22:19:28] (labs might have rejected them for a longer while -- I haven't used this key for ages, and never used it for labs) [22:19:47] * bd808 looks around for this "labs" thing ;) [22:19:50] https://wikitech.wikimedia.org/wiki/News/Toolforge_Trusty_deprecation#Ssh_login-stretch.tools.wmflabs.org_fails_with_'Permission_denied_(publickey)' [22:20:30] bd808: I might start saying words like 'toolserver' :-p [22:21:29] valhallasw`cloud: heh. at least that doesn't have the 4-letter curse in it [22:21:42] labs [22:22:05] * bd808 feels the trolls getting restless [22:22:17] :) [22:22:58] I haven't renamed anything for over 6 months! I must be getting soft in my old age [22:23:28] It's because I haven't handed down any orders to rename stuff yet. [22:23:58] (Er, not that you should be expecting that, or anything.) [22:24:12] I think we are close to being ready to start rolling out new TLDs [22:24:27] :o [22:24:32] Hey, there is a proposal to rename the entire movement. Perhaps it was instigated by bd808 [22:24:33] wmcloud.org and eventually toolforge.org [22:24:43] are we ready for one-domain-per-tool? [22:24:59] zhuyifei1999_: I think we are close! [22:25:09] * zhuyifei1999_ is excited :) [22:25:25] I don't want to bring out toolforge.org until we can support per-tool sub-domains [22:28:45] https://wmcloud.org is nice [22:31:53] we will get rid of the "f" and the "labs" with that one. :) And like chicocvenancio trolled, probably just in time to work on changing to cloud-for-wikipedia.org or something ;) [22:32:44] oh that [22:33:13] Hey we can keep WMF if it becomes the Wikipedia mission foundation or something [22:33:18] bd808: remember, the m is for knowledge [22:33:33] harej: LOL [22:34:09] my sweatshirt says the the W was for knowledge. I guess I was lied to! [22:35:05] (this was a horrible short run made during the knowledge engine regime) [22:38:23] where are the labs host keys listed? [22:38:47] https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints [22:39:09] why didn't that show up on search? :/ [22:39:16] Platonides: if you find one missing there, poke me and I'll get it added. We are not always great about keeping that up to date [22:40:02] Platonides: what did you search for? It might just be a work choice problem [22:40:20] * bd808 got is quickly via Firefox awesombar magic [22:40:51] host keys, tools-login, login-stretch... [22:47:52] Platonides: I added an intro paragraph that hopefully with help searches [22:48:27] thanks