[01:39:20] (03CR) 10GTirloni: [C: 032] gerrit-channels: Move WMCS feeds to #wikimedia-cloud-feed [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/469950 (https://phabricator.wikimedia.org/T208009) (owner: 10GTirloni) [01:39:50] (03Merged) 10jenkins-bot: gerrit-channels: Move WMCS feeds to #wikimedia-cloud-feed [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/469950 (https://phabricator.wikimedia.org/T208009) (owner: 10GTirloni) [01:44:29] !log wikibugs moved gerrit updates to #wikimedia-cloud-feed [01:44:29] gtirloni: Unknown project "wikibugs" [01:44:29] gtirloni: Did you mean to say "tools.wikibugs" instead? [01:44:35] !log tools.wikibugs moved gerrit updates to #wikimedia-cloud-feed [01:44:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL [08:12:11] hello [08:12:20] I'm the developer of lewton, a rust vorbis decoder [08:12:24] https://github.com/RustAudio/lewton [08:13:07] I'm confident that it can deal with most data given to it but I'd love to give it a good test [08:13:35] Wikipedia has a big corpus of ogg/vorbis files [08:14:22] therefore my idea was to download all of them, pass them to lewton, and compare it to libvorbis [08:14:53] I'd like a bit help on the matter though [08:15:15] I've asked brion and he said that there is the possibility of getting a cloud server [08:15:28] then I wouldn't hit your ISP bill [08:15:52] especially, I don't have 500 gb of free space on my hdd [08:17:35] (there seem to be 500 gb of ogg/vorbis files on wikipedia) [08:17:43] https://commons.wikimedia.org/wiki/Special:MediaStatistics [11:08:37] est31: hi there! that's awesome! we offer a kind of PaaS service (Toolforge) where you don't have to deal with maintaining a server yourself (which adds more responsibilities) and a IaaS service (Cloud VPS) which is the traditional virtual machine model. You can read more about them here: https://wikitech.wikimedia.org/wiki/Main_Page [11:09:44] More here: https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_Introduction [13:15:00] !log tools removing Jonas Kress (WMDE) from tools project, no longer with wmde [13:15:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:23:27] !log deployment-prep enable statsd reporting for swift [13:23:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [13:25:47] !log wikilabels wikilabels:45a687d is going staging [13:25:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL [13:31:55] !log wikilabels wikilabels:45a687d is going prod [13:31:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL [13:46:04] gtirloni: the task is a bit cpu heavy. there are 800 thousand ogg vorbis files. If on average, comparing a file takes one second, it'd take one cpu 9 days to churn through the entire set at 100% usage [13:46:19] with multiple CPUs it's faster accordingly [13:46:38] does toolforge support such cpu heavy uses? [13:47:08] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Developing#Heavy_processing [13:47:51] got it, so I think you're best next step is to verify which service fits you better (toolforge vs. cloud vps) and open a project/tool request detailing what your project is going to do, the architecture, how much data it'll process, etc -- it'll go through an approval process and we can provide more feedback [13:48:22] hmmm there is a thing called job grid [13:48:40] est31: we've a bastion host, used to login to toolforge and run commands from there. *ideally* it should be used only to run commands that interact with the grid engine / kubernetes cluster.. but sometimes people long running scripts there and that's what we're advising against [13:48:59] I see that makes sense [13:49:01] yeah, the job grid is sun grid engine.. we prefer people use kubernetes nowadays [13:50:46] if you're anticipating a lot of cpu usage, maybe Cloud VPS is a better option. you get your own VM's with CPU/memory/disk... on ToolForge your tool will be running on a shared cluster (there are limits on resource usage but still... CPU is limited at some point and can be a bit unpredictable sometimes) [13:51:46] https://phabricator.wikimedia.org/project/view/2875/ [13:54:06] ok [13:54:18] isn't it open grid engine? [13:54:58] hm, looks like OGE was an SGE fork [13:55:58] gtirloni: I've read about those data services, they only have sql access, right? [13:56:10] I want to access the raw files like this https://upload.wikimedia.org/wikipedia/commons/2/2e/Nl-onbezorgdste.ogg [13:56:13] is this okay? [13:56:19] or should I use a different way [13:59:45] I'm not sure, tbh [14:00:07] !help can Commons file be accessed directly from Toolforge/CloudVPS [14:00:07] gtirloni: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [14:00:35] well you can pull it over the web like you could from any other system [14:00:45] I'm not sure if multimedia dumps are provided...hmm [14:00:50] Technical Advice IRC meeting starting in 60 minutes in channel #wikimedia-tech, hosts: @Thiemo_WMDE & @chiborg - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [14:02:38] https://wikitech.wikimedia.org/wiki/Dumps/media [14:03:02] Heh. Yeah, they'd be huge [14:03:35] yeah, an URL like that wil go from the VM to the upload load balancer, it should have decent throughput [14:03:59] I'd second the notion that you'd use web. [14:04:06] yeah [14:04:24] I don't know of any other way to really do this. You can get the information and data about the media via the wiki replicas [14:04:49] FWIW the cloud VPS stuff (not excluding toolforge) runs in the same DC as the public facing LBs [14:05:20] so it'll be quick [14:05:25] cool [14:05:36] thanks for the info [14:05:40] Anyone who moves lots of media through their tools is also likely to get tickets from me one day to clean things up once in a while. Media projects grow fast in disk usage lol [14:05:55] speaking of networks [14:06:00] network maintenance happening soon [14:06:07] Oh yeah. [14:06:34] So if you try something in the next what, hour or so? Expect network weirdness. est31 [14:06:50] kk [14:07:11] It should be done in a hour, most likely less. [14:07:30] right now I'm still writing the tool to crawl the urls [14:07:49] there is a dump of all commonswiki page titles, I extract all ending with .ogg [14:07:53] remove everything with a / inside [14:08:18] and then hash the name [14:23:38] \o are there docs on doing some sort of rewrite rules for tools web services somewhere? [14:23:54] ie, rewrite all paths etc to just hit index.php ? [14:38:28] addshore: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web/Lighttpd#URL_rewrite [14:38:42] or you could have it in your tool/code so it's portable in the future [14:39:07] okay, but it is only included in the Lighttpd webservice ? [14:42:45] yes, I believe so [14:43:01] okay, i should be able to deal with that, I think *checks* [14:43:20] need to see if the Lighttpd webservice can do php7 already [14:48:53] !log huggle migrating project to eqiad1 [14:48:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Huggle/SAL [14:50:42] Technical Advice IRC meeting starting in 10 minutes in channel #wikimedia-tech, hosts: @Thiemo_WMDE & @chiborg - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [14:53:30] addshore: I don't think we've explored K8s ingresses in detail at this time but that'd be where something like that would go... I think your best bet is to do the redirect in your code (going from k8s+php7 back to the grid because of redirects might not be a fun experience) [14:54:43] hmm, okay, ill think about it all, aiming to try some stuff out on friday that would need something like this (nice redirecting) but with code running php7 [14:57:18] cool, let us know if you need any help [14:57:22] good luck [14:57:25] is friday a holiday for you too? [14:59:44] nope, but we have a project at work which essentially lets us work on other stuff [15:00:24] and I wanted to create some pretty api for all of the useful things I have code doing about the place that might be useful to others etc [15:01:20] nice! [15:19:59] addshore: our php5.6 & php 7.2 kubernetes containers use lighttpd internally, so https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web/Lighttpd#URL_rewrite should be what you are looking for [15:25:31] bd808: amaazing!!! [15:25:49] bd808: thanks for the racoonmendation! https://usercontent.irccloud-cdn.com/file/RnHPVPPO/image.png [15:28:47] that's a pretty shiny graphic addshore [15:29:14] bd808: coming to a phabricator near you soon (i guess) ;) [15:29:24] and to a sticker collection too [15:40:00] !log logging migrated project to eqiad1 region [15:40:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Logging/SAL [15:43:05] !log mobile migrating project to eqiad1 region [15:43:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Mobile/SAL [15:56:26] !log mwstake migrating project to eqiad1 region [15:56:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Mwstake/SAL