[00:05:10] Alright, I managed to get it set up with secure-auth [00:05:19] paladox: thanks for your help [00:05:28] DB is now up and running. [01:02:47] andrewbogott: I've given back the 120 GB of disk space I accidentally took on my xLarge instance. [01:10:16] (03PS1) 10Legoktm: Add integration/docroot [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/386566 [01:10:33] (03CR) 10Legoktm: [C: 032] Add integration/docroot [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/386566 (owner: 10Legoktm) [01:10:54] (03Merged) 10jenkins-bot: Add integration/docroot [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/386566 (owner: 10Legoktm) [03:35:23] hello Toolforge admins... I am trying to create a private database but evidently the credentials I was given are invalid [03:36:07] I tried `sql tools` and `mysql --defaults-file=$HOME/replica.my.cnf -h tools.db.svc.eqiad.wmflabs` [03:36:24] both give `Access denied for user 's53550'@'10.68.16.44' (using password: YES)` [03:36:49] any ideas? [04:37:07] tgr: (regarding yesterday) yeah, that one [05:54:32] (03CR) 10jenkins-bot: Add WLM favicon [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/385151 (owner: 10Lokal Profil) [10:22:43] !log tools.heritage Deploy latest from Git master: 07ea3ce (T176712), a4d73d8, d0ddf7a, 4b9991d (T176465), 83259c9, 14d6a1d [10:22:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL [10:22:49] T176465: ErfgoedBot crashes because it does not find builtins - https://phabricator.wikimedia.org/T176465 [10:22:49] T176712: [th-th] Base Class for Thailand - https://phabricator.wikimedia.org/T176712 [13:12:47] i'm using /mvs/ for the first time to try to edit wikipedia.. my login is refused [13:12:54] error is "Insecure Request Forbidden - use HTTPS " [13:13:02] mvs doesn't ask for the scheme (http:// or https://) [13:13:09] has anyone used mvs on wikipedia? [13:43:21] (03CR) 10Lokal Profil: "HUm. This worked when I tried it in docker but not when deployed on labs (I created a symlink in public_html)" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/385151 (owner: 10Lokal Profil) [14:50:47] musikanimal: That sounds like the script that provisions new users on the database servers got stuck somewhere. Is this from a tool account that you created recently? [14:51:12] bd808: yes it is recently created [14:52:09] can you file a bug with the tool name and assign it to madhuvishy? She is both on-call and skilled at fixing these db account creation bugs [14:52:55] what usually has happened is that a user with a very old LDAP account has recently joined tools. There is a weird bug in the script that can get stuck because of that. [15:03:29] ok will do [15:03:41] thank you :) [15:42:04] quiddity, zhuyifei1999_: did we ever end up making a "tools people want created" place in Phabricator? [15:56:26] bd808: I don’t think I’ve ever thought of that [15:56:55] bd808, there's https://phabricator.wikimedia.org/tag/technical-tool-request/ but it needs better advertising [15:57:03] discussed in https://phabricator.wikimedia.org/T134103 [15:57:16] quiddity is the knower of things! [15:57:22] lol [15:59:44] context was that I stumbled across T56703 and wanted to tag it somehow [15:59:45] T56703: Create web version of metainformation table on replica servers - https://phabricator.wikimedia.org/T56703 [16:11:23] I remembered it clearly, because it contains T135321 which I'm quite proud of (as a fun and well-written request). >.> [16:11:23] T135321: Vocabulary assistant for Simple Wikipedia, based on Upgoer Text Editors - https://phabricator.wikimedia.org/T135321 [17:30:48] I've using a SSH tunnel for database connection. Everytime there's some idiot start running massive loads the tunnel connection slows down. Is there a better server I can connect to? [17:33:30] Dispenser: tools-dev is generally a pretty quiet bastion. Also please try to use less abrasive phrasing. The people you are addressing are real, have feelings, and are not purposefully attacking you. [17:34:25] They're just making me look incompetent :-/ [17:36:31] relying on an ssh tunnel to access the databases is probably the root issue here. [17:36:51] Are you running things that would otherwise not be allowed inside of Toolforge or Cloud VPS? [17:37:12] (that's not a problem, just trying to understand the use-case) [17:37:26] Yes and legal approved it [17:37:57] cool [17:38:38] we have talked about trying to find some VPN solution to make getting to the Wiki Replicas easier, but that talk hasn't gone very far [17:38:46] I'm wondering why Dabfix is take so long executing simple queries [17:39:48] which database servers are you connecting to? The new *.{anaytics,web}.db.svc.eqiad.wmflabs ones or the old *.labsdb ones? [17:39:52] https://dispenser.info.tm/~dispenser/cgi-bin/dabfix.py?page=Chris_Wong " Routine template_redirect() failed" That's a basic SELECT statement [17:40:26] old .labsdb with the old /etc/hosts file for lookup and iptables configuration [17:40:58] I would recommend switching to the new cluster [17:41:09] Doesn't support user tables [17:41:53] true, but those are going away in about 7 weeks permanently [17:42:31] except for tools.db.svc.eqiad.wmflabs which will not be changing [17:43:14] labsdb1001 and labsdb1003 are ancient, dying hardware. Not a lot we can do about slowness there [17:43:18] So how many tools are we killing? [17:43:48] Numerically, not many -- https://tools.wmflabs.org/tool-db-usage/ [17:44:02] by importance to the movement, I'm honestly not sure [17:45:15] doing in-app vs in-database joins is entirely possible, but may not be trivial to implement depending on the data and how it is used [17:45:32] So that would kill my half my tools, https://en.wikipedia.org/wiki/Wikipedia:WikiProject_Red_Link_Recovery and https://en.wikipedia.org/wiki/Wikipedia:Disambiguation_pages_with_links [17:45:54] Are temporary table even allowed? [17:46:45] I don't think they are with the current permissions on the new servers, but that's something we could talk with the DBAs about [17:47:38] So a script like https://commons.wikimedia.org/wiki/User:Dispenser/Wrong_Extension would not work on the new servers? [17:48:16] I'm not sure what the performance/stability ramifications of allowing temp tables would be [17:48:17] * bd808 looks [17:48:23] I don't think it would work today, no [17:50:03] That's gonna kill half the functionality. That script ran 40 minutes on the Toolserver, Ran 5-10 minutes on Labs when new, runs 48 minutes today. Even if the new DBs are faster demand will adjust. [17:50:15] that looks like something that could be done with an in-app join or maybe a select [17:50:40] *subselect [17:51:03] Yes, it did, but iteration time between testing was to fuck long [17:53:03] I get that you are not happy about changes. I could re-list here all the reasons that this decision was made, but I have the feeling that you will still be upset and will still lash out at me. [17:54:16] I'm glad to have a reasoned discussion. [17:54:25] It seems it would be impossible to build applications which store user data in a database. [17:54:49] no, there is a dedicated database server for doing that [17:55:25] what is being removed is the ability to add arbitrary data on the wiki replica servers [17:56:07] I have article work list and I'd like some information on the articles in that list. How would I do it [17:57:40] open two cursors: one to tools.labsdb and one to $wiki.labsdb. Take a batch of articles from the tools.labsdb cursor and use them to run a query on the $wiki cursor [17:57:50] repeat until done [18:03:28] we are also looking into ways to replicate curated data sets across the new wiki replicas cluster. This would not work for workqueue types of use-cases but may work for things like getting access to wikidb and ORES data in a single query. [18:03:44] The last time I implemented a batcher the cursor died after 20 minutes, losing the results of the 1 hour SELECT. Othertimes its been laggy with the 100 ms ping. [18:05:59] http://dispenser.info.tm/~dispenser/sources/missing_entries.sql Its my newest report. It uses tables to get the runtime to an acceptable 30 minutes [18:08:26] I suppose the answer is [18:08:34] Add 100 MB of Application framework [18:33:16] Dispenser: sigh [18:46:08] Has there been any work on the availability of `datasets_p` yet? Nothing much has happened on T173511, so I suspect the answer is no? SuggestBot’s datasets of inlink counts might be good candidates, but I’m happy having them on tools.labsdb too. [18:46:08] T173511: Implement technical details and process for "datasets_p" on wikireplica hosts - https://phabricator.wikimedia.org/T173511 [18:59:21] mutante: your exec puppet role works perfectly. Got my bot exec nodes upgrades quickly. [18:59:40] mutante: All of my instances are now using Stretch with PHP 7.0 [20:09:07] Hi! I can not to conect me on toolforge.. I got access [20:09:52] I watching on help page. On end I got in putty message to I can not to connect [20:34:28] Nettrom: there really has not been much discussion yet on how we will do curated datasets properly. There is my strawdog proposal but the DBAs and I haven't really had time to work out the full details. [20:35:38] Zoranzoki21: I'm not sure I understand your question. Can you rephrase it? [20:35:53] OK.. Wait [20:35:58] putty is giving you an error message? [20:36:01] Yes [20:36:17] When I click connect [20:36:20] what is the message and what host are you trying to connect to? [20:36:57] On login.tools.wmflabs.org I want to connect [20:38:33] Zoranzoki21: ok. and the error message you are seeing? [20:39:36] bd808: Sounds like the possibility of `datasets_p` is quite far out then. I’ll happily wait in the meantime and see what happens. [20:40:34] Server unexceptedly closed network connection [20:40:39] I got it [20:41:11] Zoranzoki21: you got it meaning it works now? [20:41:18] No, no work [20:41:32] I click ok and putty is inactive [20:41:41] ok. what's your shell account name. I'll look for errors in the server logs [20:41:46] zoranzoki21 [20:42:37] Nettrom: it probably will be quite a while. The DBA team is really busy right now. Well they are always really busy, but right now with things other than Wiki Replicas. [20:44:57] Zoranzoki21: I can't find your username in auth.log. I'm watching the log right now, so can you try to connect again? [20:45:16] I will try.. Wait [20:47:20] Again I got same message [20:48:42] Zoranzoki21: I see an error message. It's related to an ip address which might be yours. [20:48:57] I need to lookup what this particular error message actually means [20:49:29] If my IP address need: 178.253.245.202 [20:50:30] ok. I wasn't going to post that here incase you felt it was private information :) [20:50:47] This is the error that is being logged -- https://access.redhat.com/solutions/83933 [20:51:43] I have windows 7. No linux [20:53:31] the error is on our side (which is Linux). This seems like an error that would happen for a lot of people unless there is something about how PuTTY is connecting that is especially weird. [20:53:39] * bd808 googles for clues [20:55:24] I will try to generate new private key [20:55:34] With puttygen [20:55:39] Zoranzoki21: it is not a problem with your key [20:56:04] i recommend that you use ssh on windows :). [20:56:14] git-for-windows includes ssh [20:56:25] I already have git-for-windows [20:56:29] For gerrit [20:56:36] the problem is that our ssh server on login.tools is comparing the reverse DNS lookup of your IP address with some hostname string that PuTTY is sending and being upset because they do not match [20:57:04] fwiw UseDNS yes [20:57:06] is set [20:57:13] this would probably be the case for 90% or more of our users (including me) [20:57:14] I don't actually know why exactly [20:58:33] that line has been there for 2 years... [20:58:39] I tryed with git-for-windows.. I got: Permission denied (publickey,hostbased). [20:58:50] this really feels like PuTTY weirdness [20:58:59] Zoran@Zoran-PC ~/Documents/GitHub [20:58:59] $ ssh login.tools.wmflabs.org [20:58:59] Warning: Permanently added 'login.tools.wmflabs.org,208.80.155.163' (ECDSA) to the list of known hosts. [20:59:00] Permission denied (publickey,hostbased). [20:59:14] I'm late to the conversation, but, tried giving putty just the public IP of the bastion rather than the dns name? [20:59:46] andrewbogott: this is a check on the server side [21:00:30] I'm curious about teh normal VPS bastions [21:00:37] Zoranzoki21 is connecting and then sshd is killing the connection because of PTR mismatch to some forward hostname provided by the client [21:00:46] ah, I see, the 'some hostname string that PuTTY is sending' refers to the hostname of the client [21:00:49] so, nevermind! [21:01:05] I swear we saw this before and it was a a red herring then but I cannot remember teh details [21:01:12] chasemp: I'm betting this is because we use host auth to hop inside the project [21:01:14] that's not helpful but I'm frustrated w/ my own brain [21:01:22] bd808: probably [21:01:29] the use of that flag [21:01:38] can you proxy off the normal vps bastion to tools-bastion-03? [21:01:44] I can't recall [21:01:48] not ideal [21:01:52] Resolved [21:01:58] yeah, you can (I do) [21:02:09] Thank you [21:02:14] I typed in git-for-windows $ ssh -l zoranzoki21 login.tools.wmflabs.org [21:02:16] Zoranzoki21: oh! What did you change? [21:02:26] I typed in git-for-windows $ ssh -l zoranzoki21 login.tools.wmflabs.org [21:02:36] ok. that reinforces that this is probably a PuTTY config issue [21:02:40] yeah I remember last time it was something similar [21:02:48] I went down this rabbit hole a few months ago [21:02:51] * bd808 should get a windows testing box [21:02:52] I...should hve documented it then [21:02:55] me too [21:03:00] putty makes no sense [21:04:48] bd808: hehe this is why i use windows as the native os [21:13:53] * paladox never used putty always found it never worked :). [21:14:17] I actually ran it for years and years when I was a windows user. [21:14:46] PuTTY, PAgent, and PLink are all pretty good programs once you figure out how to set them up. [21:15:54] I've used it too! 2 years as a Windows user Java Dev ;) [21:16:08] but can't recall a thing now [21:17:02] I used it too :) [21:17:13] every day for yearrrrrrrrrrrrrrrs [21:17:41] i have an old windows laptop, that i rarly use now. I only use ssh from git-for-windows. [21:17:42] and cygwin :) [21:18:17] windows was too slow for me. But when they introduced ubuntu onto windows, that introduced native ssh :)) [21:18:48] I have an unbuntu vm [21:20:21] windows 10, includes ubuntu nativly [21:20:26] and other linux os too [21:20:30] without a vm [21:21:05] Haha you think i would use win 10 [21:37:06] Hmmm.. I can not to connect on my private testwiki vezba.in.rs with ssh.. I got message: ssh: connect to host vezba.in.rs port 22: Connection refused [21:37:11] Where to ask about it problem? [21:37:40] Zoran@Zoran-PC ~/Documents/GitHub [21:37:40] $ ssh -i /c/Users/Zoran/Desktop/id_rsa.ppk -l zoranzoki21 vezba.in.rs -p 22 [21:37:40] ssh: connect to host vezba.in.rs port 22: Connection refused [21:45:39] Zoranzoki21: ask your hosting provider [21:45:53] ok.. I will ask.. Thank you [23:05:44] bd808: sorry, I'll try and get to your PR tonight [23:06:15] legoktm: no worries. I was trying to get the tests to pass :) [23:07:09] I think the downside of the change to the travis config I made is that flake8 is not run. [23:07:15] * bd808 looks for how to fix that [23:14:49] * bd808 fixed things so flake8 runs too