[09:14:18] hi there! I would like to dump a fairly large SQL database from a wmflabs tool (https://tools.wmflabs.org/editgroups/) [09:14:46] because this is a long and IO-intensive operation, I thought I would do that in Kubernetes [09:15:57] but the Docker images available there do not seem to have mysqldump installed, so I am assuming Kubernetes is not meant to be used for that [09:16:42] -> where should I run mysqldump for a large tool SQL database? [09:19:46] using a docker image for it seems pointless indirection... And doesn't stop it being I/O intensive [09:19:49] How large are we talking? [09:20:48] the dump was 300 MB (gzip compressed) one year ago, I expect it should be around 1GB compressed by now [09:22:39] I think I did last year's dump on a login instance and apparently got away with it without annoying people too much, so happy to do that again if you think it is appropriate… [09:26:01] My only real advice... would probably not dump to NFS, but use /tmp or somewhere appropriate, and then copy to NFS after [09:26:47] makes sense! [09:27:51] (I might not even need to put it on NFS at all, I can just scp it to my own backups) [15:14:04] pintoch: it sounds like you need a grid job -- https://wikitech.wikimedia.org/wiki/Help:Toolforge/Grid Your instinct to run somewhere other than a bastion is correct, but as you noticed our Kubernetes infrastructure is not currently geared towards maintenance tasks like database dump generation. [15:14:32] pintoch: and dumping to /tmp and then copying the final dump to $HOME will probably give you the best performance [15:23:17] I thought grid was no longer? [15:54:51] Zppix: not at all. https://tools.wmflabs.org/sge-jobs/ -- we encourage webservices to move to Kubernetes, but we still have ~300K grid jobs running per week. [15:57:54] pintoch: if you’re ultimately going to keep the backup outside of Toolforge (which makes sense anyways), I’d bypass NFS completely and pipe the dump directly to your other system through ssh [16:05:18] depends how reliable your connection is... :D [16:05:27] this is how I do it (runs nightly through a systemd timer): https://github.com/lucaswerkmeister/home/blob/master/.local/bin/toolsdb-backup [16:46:22] what's an easy way to either disable access logs or automatically truncate them? [16:47:06] phamhi: we don't need that huge access.log file at all. I'm looking for documentation on wikitech, but haven't found anything useful so far. [17:05:44] rageoss: thanks for responding... I'll truncate that access.log file for your.. I'll also get back to you for the auto-truncating question [17:09:15] there is a ticket somewhere where valhalla worked out a logrotate command that seemed to mostly work for truncating access.log... [17:10:39] phamhi, ragesoss: https://phabricator.wikimedia.org/T68623#2012498 is the one I was thinking of [17:20:42] thanks, bd808 [17:27:30] !log tools.integraality Deploy latest from Git master: 12ebc39 [17:27:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [18:38:57] !log tools.integraality Deploy latest from Git master: d214f09, 9b8dfa5 [18:39:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [18:54:02] bd808: (just saw your reply) Oh I must of gotten something confused then. Thanks for clarifying that [18:54:58] Zppix: I can totally see how https://wikitech.wikimedia.org/wiki/News/Toolforge_Trusty_deprecation might have made some of your confusion more likely [18:55:13] "When possible, we recommend migrating web services to Kubernetes instead of the new grid" [18:55:22] Ah I missed that part [19:05:37] !log tools.zppixbot rm GeoLite* to investigate T233346 [19:05:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL [19:05:40] T233346: Investigate why IP lookup takes so long and fix. - https://phabricator.wikimedia.org/T233346 [19:59:30] trying to clean up an old gerrit patch. cherry picked it, resolved merge conflicts, pushed the change up and everything looks OK but I can't submit. "problems with integrating this change" https://gerrit.wikimedia.org/r/c/operations/puppet/+/504653 [20:06:01] jeh: try now [20:06:36] that worked, what was the fix? [20:06:45] thank you! [20:07:52] jeh i think you need to rebase if a change was merged since. [20:08:08] The repo is on fast forward only. [20:08:31] I did a successful rebase, maybe another commit snuck in there [20:08:48] thanks mutante and paladox :) [20:09:25] your welcome :) [20:09:27] jeh: i clicked the rebase button too [20:11:55] jeh did it by any chance say merge conflict? [20:12:22] it did, I fixed the merge conflict and `cherry-pick --continue` [20:12:46] ok [20:12:53] did it say it for submit too? [20:13:16] well, it said merge conflict on my local host with HEAD. I don't recall if it said that on the web page [20:13:42] the submit button was grey and the highlight said "problems with integrating this change" [20:13:48] oh [20:14:18] i've never seen "problems with integrating this change"