[00:39:21] any db guys around? [00:41:46] jackmcbarn: hey [00:42:03] legoktm: there you are [00:42:09] did you look at the api thing yet? [00:42:18] * legoktm goes to do that [09:57:52] Hey all. [09:58:17] I sent an e-mail to translators-l@ on Saturday, and it still hasn't reached the list. [09:58:21] Any ideas what might be going on? [11:00:32] apergos: hello [11:01:35] hello svick [11:01:53] wonder if there's a parent around for our last gsoc meeting (but not our last ever meeting :-) ) [11:02:25] pinged [11:02:33] ok, i'll wait [11:03:12] he'll show up in a sec [11:05:22] parent5446: hello [11:05:42] Hey [11:06:17] so, documentation is on a good way, it will be done by the GSOC deadline (today 19 UTC) [11:07:24] yay! [11:08:18] parent5446: i belive you wanted to have a look at the code when that's done [11:08:42] Yep, and a few others as well if I remember. [11:10:07] after the official end of the program, your time is of course your own, but there will be things like code cleanup, changes from testing and features that didn't get done... how should we handle that? [11:11:11] Well GitHub conveniently has inline file comments for commits. That could come in handy. [11:11:40] parent5446: doesn't gerrit have that too? [11:12:16] Yes but Gerrit views things as patches, and if I'm correct in GitHub you can just browse through all files. [11:12:26] I don't want people to have to go to github to look at previous comments etc [11:13:04] parent5446: you can't just leave comments on files on github, but. you can only leave comments on diffs for each commit. [11:13:10] parent5446: so not *that* different from gerrit [11:13:28] Oh, well nevermind then. [11:13:59] apergos: i probably won't work much this week, but i'll try to be responsive to bugs/questions/whatever; after that, i plan to continue working, probably starting with compressing comments [11:14:29] this is very good, my evil plan is working :-) [11:15:29] some of the times you are working on code, you might hang out in here [11:15:29] yeah there's wikimedia-dev I guess but meh, this is kinda the community/code interaction space [11:16:07] ok [11:17:01] maybe other folks will have questions for you as we get closer to production use [11:17:31] also you might see questions from people in here that you can easily answer (there are a lot of beginner questions peoople ahve about mw) and when you're feeling in the mood to answer [11:17:37] Not sure if it's on-topic, an anon user (xtalmath) is seeing equation render errors on enwiki. [11:17:59] on various pages or on a specific page? [11:18:02] yeah I see just "Failed to parse (unknown error): \begin{pmatrix} c t' \\ x' \end{pmatrix} = \begin{pmatrix} \cosh \varphi & - \sinh \varphi \\ - \sinh \varphi & \cosh \varphi \end{pmatrix} \begin{pmatrix} ct \\ x \end{pmatrix} = \mathbf \Lambda (\varphi) \mathbf v" [11:18:09] https://en.wikipedia.org/wiki/Rapidity [11:18:43] actually can someone else look at that, or me later, cause meeting right now [11:18:45] I tried to purge the page and it worked for me, but the user is still affected. [11:19:18] svick: since this is the last gsoc meeting, do you have questions for me, comments, things you want to bring up? [11:19:47] For anyone interested, xtalmath in in #wikipedia [11:20:19] apergos: i probably won't be on IRC much [11:20:39] apergos: i have few tiny things [11:21:00] sure [11:21:57] i think i noticed that you use .dd extension for incremental dumps; i've been using .id for incremental dumps and .dd for diff dumps, and the magic numbers in the files are that way too (MWID and MWDD) [11:22:34] we'll probably redo the options and all that tbh [11:22:51] what do you have in mind? [11:24:13] I haven't actually run any incremental dumps yet [11:24:13] just converting (full) xml dumps to the new format [11:24:35] options usually with long or short form, with single or double hyphens, standard getopt style [11:24:49] right [11:25:42] when you talk about incremental vs diff dumps, well either they are full dumps or they are incremental (diff) [11:26:17] these are details of course, functionality is much more important [11:26:49] well, the whole project is called “incremental dumps”, so that's what i've calling the normal dumps in the new format (because they can be updated incrementally) [11:27:01] yes, I got that :-) [11:27:11] but incremental dumps really are the dumps that are applied to fulls :-D [11:27:30] just like incremental backups etc [11:28:03] 00(Sorry for the interruption, but the problem I mentioned earlier was solved) [11:28:03] and indeed that is the central point of the project is to produce those, which you do [11:28:35] then we need some new names :-) (before we go to production and people start using it, not necessarily right now) [11:28:43] “dumps in new format” is not very catchy [11:29:37] ah in case there was any doubt in your mind, which there should not be, you are of course getting 'pass' or 'success' or whatever gsoc needs in order to consier that you completed well etc [11:30:24] yep, we wil want to rename em [11:30:24] and we absolutely do *not* want to ask people on the list for names [11:30:24] because we will never get done with that discussion (*cough*bikeshedding*cough*) [11:30:35] *consider [11:31:29] I nominate dumpfile.bikeshed and dumpfile.ibikeshed [11:31:35] apergos: right, thanks, i wanted to ask that just to make sure, i know that if there was some problem, you would tell me earlier than few hours before the end :-) [11:32:24] so the old ones were "xml dumps", these could be called binary or random access or the name of your girlfriend (I mean possible, right? people do any kind of weird thing) [11:32:40] I would have told you much much earlier, so that whatever it was could be fixed [11:32:44] we like success [11:32:59] right [11:33:38] * apergos offhandedly stabs parent5446 [11:33:49] when we're doing that ranimg, we also probably want to change the name of the executable, i noticed that idumps.com is some kind of porn site [11:33:56] hahahaha [11:33:59] XD [11:34:32] I would go there but not really feeling it right now... [11:35:11] mwbindump (not too huge, prolly not porn) but there are lots of possibilities [11:36:11] that sounds reasonable [11:38:11] if you're creative you'll find some phrase that happens to have your gf initials and it will be mw[initials]dump :-P [11:38:12] anyways... [11:38:46] what else gsoc-wise? [11:39:01] you already made sure your t-shirt details are in etc? [11:40:01] if you're mostly not irc, should I be sending email, or leave questions on your mw user talk page (or one of your dump sub-pages)? [11:40:26] yeah, the details seem to okay [11:41:35] apergos: all ok with wikipedia [11:41:41] tested. [11:42:39] apergos: you have do some stress testing ? [11:42:40] apergos: whatever works for you: jabber, email or talk pages (i'll make sure i'm watching them and have email sending set up); and i can join IRC if there is some discussion, but yeah, i won't be there most of the time [11:42:51] for testing [11:43:49] cortexA9: meeting now, sorry [11:44:01] ok apergos sorry. [11:44:15] svick: I probably won't do jabber too much, I'm on there all the time but stuff like this is better done where other people can see it later [11:44:32] apergos: sure, ok [11:44:32] it's totally fine to nag me there or whatever if you need something though [11:45:04] ok [11:48:19] thanks a lot for mentoring me, it has been a fun summer (and the parts that weren't fun are the fault of C++, not yours :-)) [11:48:24] :-D [11:48:48] well I am excited that we have a new contributor (you) and shiny new dumps which are really going to change the way people use them [11:49:32] my secret goal is still to be able to run incrementals on all projects once a day [11:49:37] how nice it would be.... [11:49:58] we shall see! [11:50:16] yeah, we'll see, i think we could get close to that [11:50:29] let's do it! [11:53:52] so, if there isn't anything else, see you when there's something to talk about [11:55:00] See you! [11:55:09] can someone time https://commons.wikimedia.org/w/api.php/?action=query&list=allcampaigns for me? [11:55:09] I'm good and thanks for all the hard work! [11:55:09] and see how lng that takes? [11:55:14] not sure if it is my shitty connection or shitty code [11:55:20] see you on the lists and on the wiki! [11:56:17] 31 loosely counted seconds (not with a stop watch) [11:56:27] aaughhhh [11:56:31] * YuviPanda runs to find a profiler [11:56:37] yep [11:57:04] apergos: https://test.wikipedia.org/w/api.php/?action=query&list=allcampaigns? [11:57:30] < 1 sec [11:57:34] hmm, since that is not even remotely close to as bad, I'm going to guess it is a stupid algorithm somewhere. I don't even have any algorithms [11:57:51] :-D [11:57:57] query someplace? [11:58:04] they all are just directly indexed [11:58:12] parser calls, perhaps [11:58:15] I mean query scans all rows and test is small and commons is huge [11:58:22] something like that [11:58:33] apergos: sure, but I am pretty sure that none of the queries actually scan all tables [11:58:36] I don't know what allcampaings is anyways [11:58:44] or allcampaigns either [11:58:50] apergos: at least not according to EXPLAIN [11:59:09] apergos: it's the API for UploadCampaigns, which on desktop provides things like http://commons.wikimedia.org/wiki/Campaign:wlm-nl [12:00:03] you checked these on commoos (the explains) right? [12:00:22] (nice!) [12:00:31] apergos: i checked those on the local mw install. would that actually make a dfiferent? [12:00:34] *difference? [12:00:46] in the sense of how mysql optimizes things? [12:01:34] so I am still very much a mysql n00b but maybe it would tell you something different based on sizes of indexes and things, how it would have to handle the query ... maybe? [12:01:40] perhaps [12:01:45] i've access to the analytics slaves [12:08:36] apergos: do you have access to production dbs? [12:08:42] apergos: can you run an EXPLAIN or two for me? [12:09:19] I do and I can [12:09:25] sec [12:10:04] apergos: sweet. pastebinning, moment [12:10:23] apergos: https://dpaste.de/GkfP8/ [12:11:32] apergos: and https://dpaste.de/Z7wdd/ [12:13:54] https://dpaste.de/Qf1C9/ [12:15:08] rows 1616? that's nothing [12:15:27] also seems to be indexing fine [12:15:43] all primary keys chosen [12:15:48] yeah [12:15:52] yeah sure not anything in here [12:16:06] probably all the wikitext parsing [12:16:08] then [12:16:13] T305 BinaryPig - Scalable Malware Analytics in Hadoop [12:16:16] I... should cache those. [12:16:17] T501 Lessons from Surviving a 300Gbps Denial of Service AttackT305 BinaryPig - Scalable Malware Analytics in Hadoop [12:16:17] how much parsing are you doing? [12:16:30] what are those from parent5446 ? [12:17:00] apergos: on average, about 3-5 small parses (a couple of templates, at best) for every campaign [12:17:07] apergos: and commons has ~140 campaigns now, many with 0 parses. [12:17:20] apergos: i'm *not* caching the parses (I'm caching the sql results tho) [12:18:03] apergos: is our parser really *that* slow that sql queries over the network is faster?! :O [12:18:10] so can you ask if you less campaigns and see how long that takes? [12:18:18] s/if/for/ [12:18:30] like, try one, then ten, then 50 [12:18:31] ? [12:18:37] yeah, doing [12:19:00] https://commons.wikimedia.org/w/api.php/?action=query&list=allcampaigns&uclimit=1 is instant [12:19:09] https://commons.wikimedia.org/w/api.php/?action=query&list=allcampaigns&uclimit=10 is a couple of seconds [12:19:24] https://commons.wikimedia.org/w/api.php/?action=query&list=allcampaigns&uclimit=50 is default, ~10s [12:19:36] https://commons.wikimedia.org/w/api.php/?action=query&list=allcampaigns&uclimit=500 takes fucking forever. [12:19:48] * YuviPanda times them [12:20:54] profile on the 50 one might be interesting [12:21:22] 500 (actually only ~140 exist) returns in 17s. [12:21:26] 50 in 7s [12:21:29] 5 in 0.7s [12:21:42] and this is from inside the cluster, so network not a big factor [12:21:54] network would not be the factor here. just no way [12:21:56] apergos: hmm, so I guess I go in, add a lot more profiling calls, and then wait for the next deploy? [12:22:12] mmmmmm [12:22:16] indeed, with differing sizes. still, I have a healthy mistrust of my network :P [12:22:41] can you get the relevant page titles and imprt em to your local instanc? [12:22:50] then do profiling there? [12:22:55] apergos: probably, but i'll also need to import the associated templates, etc. [12:22:58] instead of having to sit around and wait I mean [12:23:00] which'll be a paint o hunt down [12:23:04] *pain to [12:23:09] well not if you [12:23:13] export all the templates :-P [12:23:29] but typically via special:export [12:23:35] you can just check 'templates too please' [12:23:41] and boom it gives them to you [12:24:04] apergos: won't work. Campaign pages are JSON stored under a Campaign: namespace, so I highly doubt export will pick those up automatically [12:25:12] apergos: or I could just be a good boy and cache the result of the parses, but that's not even remotely close to being 'simple', because of invalidation, etc [12:25:21] wonder how many templates they have [12:25:39] not too much, actually. Just WLM stuff right now [12:25:44] I could probably make a list in about 30 mins [12:26:11] I also have no idea how to properly cache and invalidate those, though. [12:26:17] ideally would've loved to use Redis for this... [12:28:00] and when the templates change this would need to be refreshed somehow, and I doubt that linksupdate, etc will consider these as normal pages [12:28:01] grr [12:28:07] mmm [12:28:39] 17 s [12:28:49] very slow [12:30:46] cortexA9: indeed [12:35:21] apergos: hmm, there seems to be no way for me to just export all pages from Campaign: namespace? [12:35:21] :( [12:35:23] * YuviPanda does hacks [12:36:11] from the namespace? mm [12:36:21] typically it's by category I guess [12:36:28] yeah [12:37:29] gah [12:38:15] apergos: maybe the load balancing have some problems ? [12:38:45] mm, bash oneliners! [12:39:49] so many time 17 s. [12:42:10] YuviPanda: ping [12:42:17] dMaggot: pong [12:42:24] YuviPanda: https://gerrit.wikimedia.org/r/#/c/85154/ [12:42:35] dMaggot: i saw your patch to UW. Been travelling (just landed in India a few hours ago). [12:42:53] dMaggot: will take a look at it in a bit. will make sure to merge it before the cutoff for deployment (Thursday this week) [12:43:54] YuviPanda: ok, just make sure it is deployed at the same time EventLogging's change is deployed (see the linked review in that review) [12:44:35] templates on commons: 130511 [12:44:36] dMaggot: ah, yeah. [12:44:43] apergos: not much :P [12:45:04] apergos: imported all campaigns to localhost, curl now takes 1.1s for all 140 of them [12:45:20] seriously [12:45:25] 1.1 secods? [12:45:35] yeah [12:45:37] locally [12:45:40] but if it's not expanding the tempaltes [12:45:43] yeah [12:45:44] because you don't have them [12:45:49] indeed [12:46:10] let me see if I can grep my way through one set of templates [12:48:11] * apergos is gonna get vagrant set up to be able to have a vanilla installation of X wiki of your choice with all tempaltes, project namespace and mediawiki: pages in there already [12:48:20] such an obvious thing to have [12:48:57] apergos: do you already use vagrant? [12:49:07] apergos: also, 'all templates' might be a bit much :P [12:49:08] I have used it [12:49:12] it's not that much really [12:49:19] apergos: but in general, 'sync all templates' might not be a bad iea [12:49:20] *idea [12:49:22] I've done this for en and it's fine [12:49:49] having a vanilla intance into which you can now import any other page (ok there are all the extensions that need to be set up, so that's a pain) [12:49:49] oh wah [12:49:55] oh? [12:50:08] as in, I didn't expect to be able to do this for en locally [12:50:19] ah [12:50:30] however, i might be thinking with the 'omg my laptop has only 10G of free space now!' mindset that i've had for the last 1.5 years [12:50:33] I have done fun little en experiments [12:50:55] http://meta.wikimedia.org/wiki/Data_dumps/Import_examples#Import_into_an_empty_wiki_of_a_subset_of_en_wikipedia_on_Linux_with_MySQL [12:51:04] apergos: I don't think Special:Export can handle that, tho [12:51:25] this is all via scripts + the api [12:51:33] anyways that's a bit of a sidetrack for you now [12:51:42] true [12:51:48] but vagrant role for that sounds nice :D [12:51:53] role::wikimedia::commons [12:51:57] only if they can be kept in sync [12:52:56] apergos: theoretically speaking, I can get the same effect by grabbing all the Template pages and importing them in arbitrary order... [12:54:34] yes you can [12:54:56] * YuviPanda files away idea for tool that does daily 'Template only!' dumps on toollabs [12:55:13] hey YuviPanda [12:55:23] hello sumanah! [12:55:43] YuviPanda: you saw Tim's email about RFC review Tuesday 24 September at 22:00 UTC? [12:55:50] I did! [12:55:54] ok! just wanted to check [12:56:09] sumanah: I'm not participating any RFCs *right now*, but I'll probably turn up anyway - jet lag willing... [12:56:16] right, that's what I figured [12:56:51] it's slightly at Silly O'Clock here, and as Daniel mentioned probably Europe too. [13:00:55] don't do another set of dumps, we already get adds/changes for all wikis every day [13:01:01] no deletions but who cares for this use case [13:01:31] apergos: templates only? [13:01:37] all [13:01:42] but so what, just grab the templates out [13:01:43] apergos: yeah, so that's large, no? [13:01:46] ah, hmm [13:01:49] for daily changes? no [13:01:54] hmm [13:01:55] once you have the base [13:01:58] anyways: [13:02:11] "Incremental Dumps", that's where the future is [13:02:14] :D [13:02:16] ok [13:02:24] and gsoc just concluded for that, hope to get em into production by end of the year [13:02:27] also, I'm in need of a way to mass null edit pages... [13:02:32] no you aren't [13:02:45] * apergos growls [13:02:49] apergos: :P [13:03:08] apergos: found a bug in Campaign: code where the campaign doesn't get picked up if it is only imported [13:03:20] apergos: apparently article creation hooks aren't run if it is imported via Special:Import [13:03:29] apergos: so I now need to edit them to trigger the hooks [13:03:47] how many pages [13:04:02] apergos: 140 [13:04:16] oh. when you said mass null edit I thought you really meant mass null edit [13:04:20] whew [13:04:32] apergos: hehe, it is 95% of all pages on this wiki.... [13:04:32] :P [13:06:38] when your wiki si teeny tiny :-P [13:07:27] apergos: :D [13:08:05] Krinkle|detached once you reattach yourself please take a look at https://www.mediawiki.org/wiki/Requests_for_comment/URL_shortening_system_for_Wikimedia_sites_to_support_QR_codes [13:08:16] so bet there is a maintenance script [13:08:17] checking [13:08:36] * ToAruShiroiNeko bets 10$ [13:10:04] edit.php [13:10:14] input from stdin [13:10:30] ewww [13:10:41] heh [13:10:46] whiner [13:11:35] apergos: oh, edit.php in mainteanance. I thought you meant EditPage.php [13:11:37] hence the ewww [13:13:01] ah no [13:13:24] apergos: what [13:13:41] so for each title in some list you would have to get the wikitext, shovel it into stdin of that, pass the args... [13:14:06] oh :) [13:14:29] apergos: yeah, just found that out when i xarg'd the wrong things into it :P [13:14:49] touch.py of pywikipediabot too [13:15:19] i'd have to set that thing up [13:15:52] doing [13:16:33] do you need to update the "touched" timestamp in db ? [13:16:58] page_touched in page. This timestamp is updated whenever the page changes in a way requiring it to be re-rendered, invalidating caches [13:17:12] mutante: there seems to be a bug where the article create hooks aren't run if I use Special:Import [13:17:34] mutante: i'm testing something in my dev wiki, and imported all Campaign: pages (~140) and the hook not running is problematic [13:17:35] ah, i didn't listen to all, just saw "null edit" [13:17:50] mutante: so I'll need to just edit them to trigger appropriate hook [13:18:45] mutante: heh, i'm not going to flood the job queue :) [13:19:21] ehm.. what apergos said :) [13:19:21] :-D [13:21:27] so the other thing is that then there will be a bunch of stuff in objcache table in your local install I bet, [13:21:27] which means that you might get very fast response time but [13:21:27] apergos: Empty set (0.00 sec) [13:21:27] if you clear that out maybe it will behave more like prod [13:21:27] after the null edits? [13:21:27] apergos: also, 1.1s was for only 4 pages, not 140. [13:21:27] apergos: nope, still setting up pywikipediabot :P [13:21:27] ok [13:21:27] check after [13:21:27] you wanna not have that cruft lying around [13:22:01] ok [13:26:03] if you just wanted to time 50 pages you probably would have had them edited by now by hand :-P [13:26:11] i know [13:26:11] grr [13:26:15] my network's slow [13:26:43] just saying... [13:34:16] apergos: grr, I can't seem to get this to work [13:34:48] ok, maybe you want to try pywikipedia then [13:35:05] just ser up your user config file to point to your local installation with your creds [13:35:47] yeah, doign that now [13:35:52] but honestly maybe you just want to opn 15 tabs [13:35:58] 140! [13:35:59] click edit on them all [13:36:01] save them all [13:36:04] do the next 15 [13:36:08] be done in 5 mins tops [13:36:50] fiiiineeee [13:37:48] it will just take me half an hour to write this script... (two hours later) well another 10 minutes and it will be done [13:37:58] hehe :D [13:38:13] apergos: hmm, for some reason, clicking edit takes.... so far 30s and still loading [13:38:20] I blame wikieditor + ACE [13:38:39] (three hours later) just 2 more minutes and I know it will be working... [13:40:35] ugh codeeditor [13:40:41] but ace shouldn't be that heavy [13:41:33] apergos: save also takes a good 30s [13:41:34] oh wait [13:41:37] i turned off my cache [13:41:38] grr [13:41:39] that's why [13:41:41] :-D [13:42:47] apergos: wooo, 4s even on super-small loads [13:43:11] goooood [13:46:58] apergos: 29 parse calls for 18 campaigns [13:47:53] mm hmm [14:46:09] helderwiki: I assume you read [[translatewiki:FAQ]] etc.? [14:46:31] hmm... probably a long time ago... heh [14:46:41] There you are helderwiki [14:46:56] Hi! :-) [14:47:53] So I have some ideas for displaying security bugs as such. [14:48:05] Resolution of bugs will be tougher. [14:48:31] Hm... [14:48:33] Talking about the enwp gadget btw [14:49:29] I'm particularly interested in the resolution actually (since it is a lot more common) [14:49:35] Security bugs return an error, so I should be able to do those by adding a .fail() to the current query. [14:49:59] Nemo_bis: I think this is the answer I was looking for: https://translatewiki.net/w/i.php?title=FAQ#LocalisationUpdate [14:51:40] Resolved bugs may require a page html scrape for the resolution. [14:52:01] Technical_13: ah, I was just going to ask about this (what was the problem with it) [14:52:38] Just that I'm not good at it yet. :p [14:53:53] do you have a sample URL/query which returns the resolution? [14:54:20] (I know a little of regex if that helps ;-) ) [15:07:57] helderwiki: good, can you close your own thread please? :) [15:16:27] someone knows what mark stats are? https://en.wikipedia.org/wiki/User_talk:Petrb#Tool_Labs_Homepage_.2F_Wiki [15:16:42] mark's request stats [15:16:50] they don't exist anymore [15:17:19] in that case I will remove them from main page... [17:39:43] Are virtual namespaces cheap? [17:41:25] Or are they expensive on the system? [17:41:47] PHP namespaces? MediaWiki namespaces? [17:42:04] MediaWiki [17:43:17] they're not expensive, no [17:43:18] what [eco]system? servers or projects as a whole considering users too? :) [17:43:32] we just send Nemo_bis a bill [17:43:33] Thanks ori-l [17:43:48] I'll make sure to forward mine too. [17:43:59] Technical_13: Nemo_bis thinks they add cognitive overhead to users and make interfaces cluttered [17:44:24] it's not a completely unfair criticism -- there are aspects of the interface that could be re-touched to be friendlier to namespaces [17:44:33] but you probably want to talk w/him about it [17:44:41] noo [17:44:58] namespaces are cheap; we should use more [17:45:17] there are about 64 namespaces I'd like to add to Meta ;) [17:45:30]