[17:43:53] howdy pardners [17:44:11] hey brion ! [17:44:18] hey hey [17:56:06] brion: Western accent? [17:56:13] You gonna have a code rodeo? [17:56:35] hehe [17:56:45] careful you don't get thrown by that thar branch merge [17:56:53] ain't no man can break 'er yet [17:59:18] urgh. [18:00:04] *guillom looks at the jorm and wonders: "Is it alive?!" [18:00:15] that's a very good question. [18:00:24] i had. . . much whiskey yesterday. [18:00:31] I heard so. [18:01:31] reaaalllly? [18:01:40] i wasn't the only one partaking, mind you. [18:01:50] wikiwhiskey [18:06:08] jorm: if you need some hair of the dog, pour a glass on my behalf -- i'm an uncle today :D [18:06:20] this is why i took today off. [18:06:25] :D [18:06:26] yeah? that's great! [18:06:49] what kind? [18:07:07] it's a little nephew-dude [18:07:28] i had a little nephew-dude get born on my own birthday once. [18:07:33] we should get to see him when we go down for xmas [18:07:42] we almost did -- actually the due date was dec 1, which is mine :) [18:07:51] would have made it very easy to remember [18:08:21] as it is i'm just grateful for a nice even birth-year like 2010 [18:08:23] makes the math easy [18:09:59] a large percentage of my family have birthdays between Nov. 22 and Nov. 28. [18:10:44] there must be something fun in late february :) [18:11:43] yeah. mathematically, i'm *exactly* 9 months after valentine's, if one accounts for me being two weeks late. [18:12:30] must've been a successful valentine's day [18:14:18] :D [18:27:02] hi RoanKattouw [18:27:13] Hi vvv [19:28:53] hey RoanKattouw [19:29:15] Hey flipzagging [19:29:17] So [19:29:37] I guess we need to work hard on UW the next 36 hours [19:29:42] right so I haven't committed the frontend for your backend yet [19:29:49] well, I know I do, not sure about you [19:30:14] Partly depends on your needs/wishes [19:30:31] I would hope that all you might need to do would be to approve a fix I'll commit soon, and then come back at 10am CET to deploy. [19:30:38] We can go ahead and do the scheduled deployment at 1am tonight [19:30:40] Exactly [19:30:53] there is another fix I'd like to get in that deals with warning errors too [19:30:59] Frontend for the temp/thumb hack plus UW bugfixes [19:31:02] right [19:31:25] so I'm thinking I just should just turn all thumbnailing into streaming, and when it's local just use the local thumb.php. [19:32:34] so Alolita wanted me to sync up with you now, does this seem viable to you given that nothing is yet committed or should we wait until perhaps later on 11/30 to deploy [19:33:10] I think we can do this, but I may need your guidance or approval on the thumbs thing. [19:33:18] who else can deploy, BTW? [19:33:48] also, are there changes to thumb.php that I would want to port back to trunk? [19:38:02] OK first things first [19:38:20] As for viability, the sooner the code is committed and visible to me, the better [19:38:31] If you need guidance, ask for it now [19:38:43] Approval / review, same [19:38:59] Whether the thumb.php changes should be merged back to trunk depends on the changes [19:39:20] I'm assuming you created a new mode / rewrite rule to allow temp/thumb but perhaps not [19:39:26] (Why are you touching thumb.php anyway? I thought my hacks in 1.16wmf4 would be sufficient) [19:39:30] (syncing now) ? [19:39:37] that's what I'm saying [19:39:40] Well actually [19:39:43] What happens is this [19:39:57] If *anything* on upload.wm.o 404s, the 404 handler is invoked [19:40:30] It then uses a bunch of regexes to decide if it's something it can handle, and if so, of what type (regular thumb, archived thumb, PDF page, ... ) [19:40:56] regexes in the 404 handler, in PHP you mean [19:41:03] I simply expanded on the regex for the primary (regular image thumb) case to recognize temp thumbs (not very hard, it already recognized so-called archived thumbs) [19:41:05] Yes [19:41:12] The 404 handler is a PHP script in this case [19:41:12] huh [19:41:21] so that doesn't require any new mod_rewrite [19:41:24] Nope [19:41:31] All 404s are directed to this one PHP file [19:41:48] Which runs a few regexes, decides if it can create and stream a thumb file, and otherwise just runs 404.php [19:41:55] ok, well it's not crucial to backport that 404 now, but it would be nice if we did. [19:41:57] Which is a basic 404 page [19:42:10] It's in /trunk/tools , so those changes are in SVN too [19:42:18] But that's OK, it's allowed to be WMF-specific :) [19:42:44] so you didn't touch thumb.php at all [19:43:54] I did [19:44:12] Because there's a little more indirection than I told you about [19:44:16] (checking out 1.16wmf4 as we speak :) [19:44:22] The 404 handler is lazy [19:44:41] If it recognizes something with a regex, it formats the substrings it matched into a query string for thumb.php and sends it off [19:45:02] So thumb.php gets crap like ?f=Foo.jpg&archived=1 [19:45:04] yes. [19:45:11] Or, in our case, &temp=1 [19:45:16] and archived gets a slightly different path [19:45:22] bingo [19:45:33] It's easy enough to tweak the 404 handler to recognized /temp/thumb paths and send &temp=1 for them [19:45:52] Getting thumb.php to respond to &temp=1 correctly was a bit trickier [19:46:05] ok, I see you have "dirty, horrible, evil hack" here [19:46:29] OH. [19:46:32] I see what you did :) [19:46:54] Yeah [19:46:59] I'm creating a repo with all the right paths [19:47:04] Took three tries to get right, IIRC [19:47:08] right, we probably don't want this in trunk :) [19:47:27] damn [19:48:42] Let's not worry about that [19:48:49] Full deployments, sadly, are infrequenty [19:48:52] -y [19:48:58] And thumb.php is rarely changed [19:49:09] the problem I'm trying to solve is that in order to make a streaming thumbnail work, I have to refactor the code, which created thumbnails at a different point. [19:49:26] so I thought "I know, I'll just stream them all the time in the manner of whatever Roan deployed" [19:49:37] but now we're saying I can't backport this [19:49:40] or are we? [19:49:50] backport what where? [19:50:04] the thing you just wrote to stream temp thumbnails!!!!!! [19:50:08] hah [19:50:10] OK so [19:50:21] All you need is a URL pattern where these thumbs will be available [19:50:38] And count on present infrastructure (in this case, my hack, in more saner cases, a web server exposing a directory) to do the rest [19:50:52] you seem to forget that I'm already working in trunk [19:50:57] I know that [19:51:34] But we're talking about sane-ish URLs like http://upload.wikimedia.org/wikipedia/commons/thumb/temp/e/e6/Filename.jpg/123px-Filename.jpg [19:51:34] ok, there's something that I'm missing, because you seem to think this should be easy. [19:51:50] (Where of course "Filename" is something insane generated by PHP, but the rest of the URL is sane) [19:51:51] the URL doesn't matter [19:52:06] I can craft a URL with the f= archived= temp= params if necessary. [19:52:15] the user doesn't see that URL [19:52:24] they see UploadStash urls instead [19:52:25] OK so basically what you say is "if this setting is set, I'll pull thumbnails from this URL using cURL instead of rendering them" [19:52:30] no [19:52:40] Yeah so Special:UploadStash would need to stream the upload.wm.o URL through cURL [19:52:51] well, let me rephrase: [19:52:58] yes, but it's not quite what you think [19:53:24] MediaWiki's notion of transform() gives the user a URL at which they can get a transformed thumbnail. [19:53:37] so transform is more like "create a URL for transformed thingy" [19:53:41] Yes [19:53:49] Because transforms can be deferred [19:53:50] I hacked it to ACTUALLY scale the file at that point. [19:54:02] Yes, and at least on the WMF setup that ain't gonna work [19:54:05] and then stash the file in the stashed file system. [19:54:11] Right. [19:54:15] But for normal setups it's good [19:54:17] So now we are deferring it again [19:54:36] Yes [19:54:43] but in order to accomodate both cases I need either ridiculous amounts of flaggy type code, or I remove my hack and add in another URL-based thumbnailer. [19:54:53] which streams. [19:55:00] so I was hoping to use your modification to thumb.php [19:55:04] is this making sense now? [19:55:05] Aha [19:55:09] OK [19:55:32] So we both agree on what to do in the WMF env, right? [19:55:34] I can write my own, and I sort of did (in SpecialUploadStash) but it needs a "back end" URL based thumbnailer. [19:55:38] yes. [19:55:42] It's just about an elegant solution for sane envs [19:55:46] yes [19:56:04] so the right thing is to do it with a URL-based thumbnail since that's what MW always wanted in the first place. [19:56:07] anyway [19:56:09] I can do it [19:56:15] I just have to do more work this way. [19:56:21] Or move the code I already had around a bit. [19:56:22] Would you need lots of flags? [19:56:39] well I am just going to rip out the hackery I did to transform(). [19:56:52] and move it into the URL-parsing layer in UploadStash. [19:56:57] Could do [19:56:58] SpecialUploadStash that is. [19:57:20] all right I am going to go do that. It might take less time to do than discuss :) [19:57:24] So in Special:UploadStash you either render immediately & stream out from FS or you stream out from cURL? [19:57:37] that could be the way [19:57:38] OR [19:57:50] well, no, I can't use thumb.php without your evil hack [19:57:59] so I think I'll do what you said. [19:58:26] thanks [19:58:37] UW deployment soon? [19:58:58] vvv: < 24 hours [19:59:03] Yay [19:59:05] Where? [19:59:10] RoanKattouw: away for now, will commit asap. [19:59:31] Alright [19:59:32] vvv: just commons for now, but it's not the default experience. [19:59:45] I see [21:10:31] I may very well be the only person in the world who doesn't care that Neubaten canceled their shows. [21:10:51] Who's Neubaten? [21:10:54] See, you're not [21:10:55] Or, at least, among a large portion of a couple of my social circles. [21:11:16] RoanKattouw: http://en.wikipedia.org/wiki/Einst%C3%BCrzende_Neubauten [21:12:27] my ex-wife loves them. dragged me to a show once, at the fillmore. she had a blast; i spent two hours in the club's bar drinking beer and reading a book on my palm pilot. [21:12:52] haha [21:12:59] I didn't know you used to be married [21:13:06] yeah. [21:13:32] But then Rob's also divorced and he's like ten years younger than you; ever since I heard Rob was in the middle of his marriage at my age, nothing of that sort surprises me anymore [21:13:33] a pink-haired, dreadlocked, tattooed rockstar-type. [21:13:39] who is now a mad scientist. [21:13:41] haha [21:13:48] if you can believe that. [21:13:48] I would have expected nothing less of you [21:14:28] (The latter referring to your characterization of your ex-wife, not to early marriage+divorce, BTW) [21:15:19] i figured. [21:15:28] so, i hear that you have NEVER eaten fried chicken. [21:15:35] it is now my personal mission to fix that. [21:15:41] You shall have your chance [21:15:52] i got a deep fryer for my birthday! [21:15:55] Yay! [21:15:58] it is like a tiny god. [21:16:06] ...in mid+late January [21:16:08] so making chicken is super easy and fast. [21:16:30] yeah, that's what i was thinking. i'll try to figure out the optimal date when you, sam, and andrew are all together. [21:17:41] Sam's planning to leave around the 19th AFAIK [21:17:53] That's a Wednesday [21:18:02] I get in the Sunday before that, Andrew gets in on Saturday [21:22:39] What should I do if I found a filesort query in CodeReview? [21:24:19] Find the bug that whines about inefficient queries in CR and add it there [21:29:00] vvv, https://bugzilla.wikimedia.org/show_bug.cgi?id=24479 [21:29:19] Reedy: are you on it? [21:29:22] Nope [21:29:27] That's the bug for reference :P [21:29:40] It's quite a general "CR indexes suck" [21:31:44] vvv, just lob it on there if you're not gonna get round to fixing it now :) [21:32:06] Reedy: right now I'm trying to figure out how to fix it [21:32:25] what's the query on? [21:33:05] Not that that makes much sense [21:33:37] Reedy: the one that filters revisions by paths [21:34:17] vvv, presumably, that's https://bugzilla.wikimedia.org/show_bug.cgi?id=23720 then [21:34:30] As that's got a performance limit for old revisions [21:34:51] Aha [21:35:17] :) [21:36:59] vvv, fixitfixitfixit :D [21:43:08] LOLOLOLOLOLOLOL [21:43:14] Do you know what caused filesort? [21:43:28] The performance-reasoned revision limit [21:43:37] :| [21:43:41] Awesome [21:43:53] vvv, is the query not so inefficient with it remove? [21:44:32] Getting rid of a filesort doesn't /necessarily/ make a query more efficient; there's other ways for queries to suck that aren't called "filesort" ;) [21:44:37] Although usually it's a good tihng [21:44:43] RoanKattouw: e.g.? [21:44:45] Sure [21:44:59] vvv: Full table scans, unindexed where's [21:45:05] RoanKattouw, but if the query wasn't actually so bad after removing the restriction.... :) [21:45:09] Sure [21:45:14] I'm not saying it's necessarily worse [21:45:15] RoanKattouw: doesn't the latter cause filesort? [21:45:20] Nope [21:45:29] Filesort is only for sorting AFAIK [21:45:43] Happens when the ORDER BY can't be done with the selected index [21:47:55] http://dpaste.org/Ohq5/ [21:51:42] http://dpaste.org/UeM1/ [21:52:30] Mine gets a using where for code_rev too [21:52:49] Fail [21:53:09] ? [21:53:11] PRIMARY KEY (`cp_repo_id`,`cp_rev_id`,`cp_path`) [21:53:39] And it's doing a query with WHERE cp_repo_id=const AND cp_path LIKE 'const%' ORDER BY cp_rev_id (basically) [21:53:41] http://dpaste.org/Tog3/ [21:54:12] So that table would be much happier with an index on (cp_repo_id, cp_path) [21:54:47] Are you sure? [21:54:54] Yes [21:55:29] Hah, ran EXPLAIN a second time and got a different result, this one's worse [21:55:41] temporary + filesort + full scan on code_rev [21:55:41] o_0 [21:56:01] While I'm poking at CR, maybe I should poke at indexes next [21:56:17] It'd make sense to have 1 or 2 patches of indexes [21:56:22] rather than multiple [21:56:30] Exactly [21:56:37] So I should save them up [21:56:48] Sounds like a plan [21:56:49] After doing a comprehensive analysis [21:56:57] I'll log this one on the bug [21:57:13] I suppose run query, test it, find it, log it [21:57:47] Thinking about it, just doing some "normal" operations using CR locally, logging the queries, then looking at them is probably the way to go [21:58:14] Aww crap [21:58:26] Even with good indexes, this query is never gonna work right [21:58:43] Because doing this query the "right" way would sort paths before rev IDs [21:59:20] RoanKattouw: so? [21:59:38] So that would be kinda useless [22:00:04] Ooooh wait [22:00:13] The table changed purposes at some point [22:00:17] Or... [22:00:30] Well [22:00:39] Depending on my understanding of this, we may be able to replace the LIKE with a = [22:01:20] If that's right, we should add an index on (cp_repo_id, cp_path, cp_rev_id) and it should at least be possible to make the query happy [22:02:05] RoanKattouw: changed its purpose? [22:03:44] I don't really remember [22:03:50] And I've been awake for 17 hours after sleeping like 4 [22:03:56] Bad RoanKattouw [22:04:00] RoanKattouw, OOH! [22:04:02] Idea [22:04:07] RoanKattouw: :( [22:04:08] Leave it for when you're here? [22:04:17] I'm guessing we'll have some spare time... [22:04:57] I'm leery of piling up stuff for that [22:05:02] But we could do it [22:05:05] Well [22:05:10] We've only got querypages [22:05:15] for definate [22:05:16] True [22:05:25] plus I'm sure you'll get some requests to get a few bits and pieces... [22:05:31] As always [22:05:35] Ideally, we just need to get the stuff back into core [22:05:37] OK so let's say we'll do that if there's time [22:05:47] And we'll try to round up stuff we run across in the meantime [22:05:51] Sure [22:05:56] Then we (I) can hack at the API module in the near future [22:06:06] querypages API module? [22:06:13] That's the idea of it [22:06:20] To expose the various querypages THROUGH the api [22:06:26] Yeah [22:06:28] That one should be fairly trivial [22:06:35] I could almost write that on the plane [22:06:36] indeed, if we do the rest right :) [22:06:38] heh [22:06:40] (Almost, I think it'd take about 2 hours) [22:07:07] There's not a lot to do wrong when I've already written / redone the interfaces with that in mind :) [22:07:15] Although that was two years ago [22:07:25] Sure [22:07:29] But not a huge amount has changed [22:07:39] (in the grand scheme of things) [22:07:44] Exactly [22:07:51] Disappointingly little, in fact [22:08:03] Although secretly I'm glad cause it reduces merge conflicts [23:41:08] http://www.newhighmart.com/shop/item/938/Grey-Ant-x-Teva-Stiletto/