[06:31:20] !log tools.extjsonuploader Deployed new version. T250344. [06:31:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.extjsonuploader/SAL [09:31:22] !log admin removing invalid backups from cloudvirt1024 (196 in total) (T269419) [09:31:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:31:26] T269419: cloudvirt1024: /srv full 99% - https://phabricator.wikimedia.org/T269419 [15:37:46] Hi, I recently joined the abstract wikipedia project and was working with toolforge. I want to download a file from within my tool but not sure how to proceed. I found the command `scp .toolserver.org:remote-filename . ` from an older wiki. Not too familiar with ssh/scp, what is the proper command to use? [16:09:33] tanny411: if it references the Toolserver that documentation must be old indeed… [16:09:40] I’m trying to find newer documentation on wikitech [16:09:56] Yes, it refers to Toolserver [16:10:29] tanny411: https://wikitech.wikimedia.org/wiki/Help:Access_to_Toolforge_instances_with_PuTTY_and_WinSCP might help you [16:10:52] yeah, that’s the main one I found so far but it looks Windows-specific? [16:11:12] i am actually shh-ing directly with terminal. yes thats windows [16:11:21] yeah, I think the docs sort of assume that if you aren't on windows you already know about scp [16:12:07] https://wikitech.wikimedia.org/wiki/Help:Accessing_Cloud_VPS_instances#ProxyJump_(recommended) has some good tips (ProxyJump is awesome) [16:12:12] though it’s more about Cloud VPS than Toolforge [16:13:01] tanny411: what command do you use to SSH into toolforge? [16:13:03] Searching the web for something like "using scp" should give lots and lots of tutorials with different levels of detail [16:29:41] Lucas_WMDE: I use `ssh tanny411@login.toolforge.org`. [16:29:41] bd808: I did, i cant figure out what to put in place of [user@]SRC_HOST:] for toolforge. I tried `tools.my_tool@tools-sgebastion-07:` but i dont think this is it. [16:36:57] tanny411: you can only connect with ssh/scp as your user, not as a tool user. So you would use your personal username, but could get files in/out of a tool by targeting the tool's $HOME with the location (meaning something like `scp bd808@login.toolforge.org:~tools.my_tool_name/some/file/path .`) [16:41:07] Thank you so much, this is it! [16:44:15] !log tools.quickcategories deployed e236b28c74 (error handler for upcoming ToolsDB maintenance) [16:44:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.quickcategories/SAL [16:56:59] bd808: Is there an update on easy disk expansion for VMs? [16:58:03] Skynet: there is an active proof of concept in our testing cluster. It looks like there will be some user facing ability for this next quarter (January-March) [16:58:24] Sounds good. Can't wait. :-) [16:58:42] The DB is slowly hitting it's limit as it grows to new Wikipedias. [16:59:11] Skynet: don't forget to cleanup your old performance samples. :) [16:59:37] bd808: Yessir, happens regularly. [16:59:39] :-) [17:06:14] !log clouddb-services switching the secondary config back to clouddb1002 in order to minimize concerns about affecting ceph performance T266587 [17:06:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Clouddb-services/SAL [17:06:17] T266587: ToolsDB replication is broken - https://phabricator.wikimedia.org/T266587 [17:18:53] Is there any default configuration in a Cloud VPS that would limit the number of outgoing connections that can be made, specifically within PHP? [17:20:17] nothing I can think of other than the ulimit of the user running php. you can only open so many files before the kernel will stop you and each connection is functionally a file [17:20:54] We’re finding on InternetArchiveBot that it is having trouble making connections to external hosts, but things like curl still work, suggesting that something is failing in PHP before reaching the network stack [17:21:14] Is there a direct relationship between calling external URLs and file limits? [17:21:23] “and each connection is functionally a file” [17:24:28] !log clouddb-services settings toolsdb to readonly to prepare for shutdown T266587 [17:24:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Clouddb-services/SAL [17:24:31] T266587: ToolsDB replication is broken - https://phabricator.wikimedia.org/T266587 [17:24:36] hare: the linux kernel tracks open files with a limit on how many each user can open and a ceiling of how many can be open on the entire system. A socket connection is tracked in the same pool as a local system file. That limit should be something like 8K by default. [17:25:30] `ulimit -a` in a shell will tell you what that shell session's limits are [17:26:08] the limits are lower than Debian defaults on the Toolforge bastions deliberately [17:26:14] !log tools.majavah-bot disable crontabs for tasks needing ToolsDB access due to maintenance [17:26:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.majavah-bot/SAL [17:28:08] !log clouddb-services shutdown toolsdb T266587 [17:28:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Clouddb-services/SAL [17:28:48] My colleague is excited by your lead so you may be on to something [17:29:48] !log clouddb-services stopped mariadb on the replica T266587 [17:29:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Clouddb-services/SAL [17:29:51] T266587: ToolsDB replication is broken - https://phabricator.wikimedia.org/T266587 [17:29:53] What we were finding is that connections would be open and shut in as little as 23 microseconds, which sounds like not even an honest attempt is being made [17:31:49] !log clouddb-services sync started from toolsdb to its replica server T266587 [17:31:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Clouddb-services/SAL [18:21:28] !log paws move paws to sqlite while toolsdb is down. [18:21:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [18:34:40] !log clouddb-services restarted sync from toolsdb to its replica server after cleanup to prevent disk filling T266587 [18:34:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Clouddb-services/SAL [18:34:43] T266587: ToolsDB replication is broken - https://phabricator.wikimedia.org/T266587 [21:17:53] Hey, is ERROR 2003 (HY000): Can't connect to MySQL server on 'tools.db.svc.eqiad.wmflabs' (111 "Connection refused") expected? [21:20:33] yes, it is down for maintainace [21:21:02] https://lists.wikimedia.org/pipermail/cloud/2020-December/001358.html. [21:21:19] *maintenance [21:27:59] SantiComposite: originally thought since 1700 UTC to now is pretty long, but...fair enough [21:28:50] "we aren't sure exactly how long-expect multiple hours" [21:28:55] it's been 4 [21:38:04] Yes, toolsdb maintenance can be lengthy because of the amount of data. [21:39:07] There's 1.3TB of data to move [21:39:53] I don't want to mislead with an ETA, but it's more than halfway now [21:40:29] The wikitech page for deployment-prep: contact address: qa@lists.wikimedia.org Mailman: No such list "qa" [21:41:15] mutante, have an idea for a better contact address? [21:43:02] balloons: Hmm. Not really. There is this: https://lists.wikimedia.org/mailman/listinfo/betacluster-alerts [21:44:30] balloons: I think maybe just removing the email part and instead linking to https://phabricator.wikimedia.org/project/view/497/ would be best we have [21:45:34] or the members subpage of that https://phabricator.wikimedia.org/project/members/497/ [21:51:54] mutante, yes, perhaps linking the project is the best [21:54:49] balloons: ah.. almost just did it but the project pages are templated [21:56:07] it links to |s={{:Release_Engineering/SAL}} though so .. maybe that would match the team to contact [22:00:13] !log deployment-prep adjusted 'puppet prefix' deployment-jobrunner to use "role::beta::mediawiki::jobrunner" instead of "role::mediawiki::jobrunner" - goes together with gerrit:649707 - no instance currently exists called 'deployment-jobrunner' [22:00:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [22:01:48] mutante, it's set in https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/Documentation [22:02:07] (the mailman link) [22:03:48] SantiComposite: thanks! but seems there is another level :) {{{Contact Address}}} [22:04:01] that's just the parameter that gets passed in [22:05:32] https://wikitech.wikimedia.org/w/index.php?title=Nova_Resource:Deployment-prep/Documentation&diff=1891755&oldid=1867358 [22:06:40] SantiComposite: oh nice, even using {{phab} to get to a page that is a project and not a ticket! thanks a lot [22:06:50] np [22:28:40] I've sent a message to the operator of the tool, but https://checkwiki.toolforge.org/ throws "Could not connect to database: Can't connect to MySQL server on 'tools-db' (111 "Connection refused")"....is there an issue on the server/db side, or is it an issue with the tool? [22:29:01] Aah, nvm [22:29:01] https://lists.wikimedia.org/pipermail/cloud-announce/2020-December/000347.html [22:29:13] yup, maintainance still ongoing [22:30:15] it is not expected to be back up especially soon -- someone will probably let the mailing list know when it is done [22:31:29] Well, as long as there was a reason behind the issue I faced, then I'm all good and calm [22:32:31] toolsdb is apparently 1.3TB -- that takes a bit of time to move :) [22:33:25] That's about the size of my external hard drive. Would have though it actually been a bit bigger [22:34:10] but yeah, I know when I try to transfer files across to new computers from it...takes ages to copy everything over [22:37:05] Moving a database is typically more complex than moving a mere binary file of the same size [22:37:59] that is exactly what they're trying this time [22:39:26] (ref: https://phabricator.wikimedia.org/T266587#6639552 ) [22:40:28] and, if it does work this time (which it hopefully will), add another hour for additional maintainance [23:26:33] mutante, balloons: the qa@ list was closed a few years ago in favor of wikitech-l