[07:04:57] anyone awake? is there some firefighting going on? [07:05:08] I'm awake, and should there be? [07:05:29] no, just wondering where Roan is... we were supposed to do a deploy [07:06:55] yes, oh that's right we should be on for that now [07:07:07] no, as far as I know it was set for 10 am my time (ie now) [07:07:43] i will hang out a bit longer, maybe he overslept [07:07:52] we demand a lot of that guy [07:08:11] he might have. we talked about it last night [07:08:16] ah speak of the sleepyhead... [07:08:19] how's tricks? [07:08:26] hey Roan [07:08:49] Morning [07:09:02] I'm sorry for being late [07:09:06] It took me ten minutes to find the right config file to edit so my wireless would work again [07:09:08] no not at all [07:09:28] ah, I was wondering if you had human failings or something [07:09:36] I do, I do [07:09:41] I was down here at 9:05 [07:09:55] omg, inexcusable [07:10:51] anyway do what you need to do to get ready [07:11:19] Just looking over the slew of commits made while I was asleep [07:12:03] Also, I need info on those config changes [07:12:24] right [07:12:32] crap I am a moron, I didn't email you [07:12:53] i've been out, just got home 15 min ago [07:13:19] haha: [07:13:21] + // omitted navy because it is believed only MultiChil uses it heavily. Could add it back [07:13:28] :-D [07:14:01] OK looks good [07:16:31] RoanKattouw: I emailed you the config. It's just one more line. [07:17:01] Cool thanks [07:17:12] RoanKattouw: yeah there's not a lot that changed today in terms of commits although Kaldari tends to commit half-working stuff [07:17:19] let me review the last things he did [07:17:35] "beginning of fix for Bug 28316" looks suspiciously like that [07:22:08] I know [07:24:06] everything looks ok except that one [07:24:12] I'd say leave out 85606 [07:24:35] but, hm [07:25:37] let me do another quick check [07:35:24] RoanKattouw: looks ok to deploy IMO, except for r85606 [07:35:52] RoanKattouw_: did you see my last comment? [07:36:04] I said "member:RoanKattouw: looks ok to deploy IMO, except for r85606" [07:36:19] er, except the member. Colloquy just added that... [07:36:35] OK [07:36:36] I will do that then [07:50:03] neilk_: In the config, should the licenses stuff stay or be removed? [07:50:28] what's the live config look like now, in LocalSettings.php ? [07:50:34] oh I know what you mean [07:50:37] yes, that should go [07:50:47] but can you pastebin me the config now just to double check [07:52:14] http://pastebin.com/uukXrE9p [07:54:01] yup, kill the licenses [07:54:12] and remove the trailing comma from prev entry, for obvious reasons [07:54:19] Not needed in PHP :) [07:54:33] I know :) [07:55:06] But if I stop doing it in one language, I'll just start slacking in all of them... [07:55:32] OK here goes [07:58:03] *apergos crosses fingers [07:58:53] OMG I ACCIDENTALLY THE WIKIPEDIA [07:59:14] and I would have gotten away with it, if it hadn't been for you meddling kids [08:00:04] ( neilk_: what trailing comma? I don't see one in the pastebin, if you mean after the feedbackPage line) [08:00:28] apergos: ha [08:00:36] well that would have been a syntax error then [08:00:42] yes it would have. [08:00:57] does no one php -l anything any more (except me)? :-P [08:01:02] I always do it [08:01:05] sweet [08:01:13] *apergos awards neilk_ a gold star [08:01:16] supposedly there is a php -l immediately before commit, or so chad implied [08:01:20] yes there is [08:01:23] er not commit I mean deploy [08:01:36] there used to be some precommit hook but apparently no longer? [08:01:37] yup. it's in the script. [08:02:09] dunno about that [08:02:18] I also usually jslint before saving. Not that jslint catches everything. [08:02:53] this is why vim still rocks, you can just add arbitrary shell commands to almost any event [08:05:23] Alright we should be live now [08:05:52] There may be a few glitches because srv196 probably didn't get the sync and may or may not still be serving requests despite that [08:06:48] RoanKattouw: I'm not seeing changes [08:07:10] Oh [08:07:12] D'oh [08:07:14] Running svn up helps [08:07:17] :-/ [08:07:20] Good morning Roan [08:07:26] ...wondering what you were doing now [08:07:29] goooooood morning! :-D [08:08:20] *apergos offers roan some chocolate. [08:08:24] breakfast of champions. [08:09:19] (09:44:24 ????) nagios-wm: PROBLEM - RAID on srv196 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [08:09:27] this is why srv196 is unresponsive. ignore it. [08:09:52] OK [08:09:56] I commented it out in the node list [08:10:05] So I don't have to wait 5 mins for it to time out every time I sync [08:10:14] it says you commented out 193 [08:10:15] As long as srv193 isn't serving requests, it's fine [08:10:17] in the log [08:10:23] For some reason Nagios thinks it's still up [08:10:25] Yeah from the node list [08:10:28] That doesn't depool it [08:10:29] which one did you comment out? [08:10:34] 196? or 193? [08:10:37] Oh, 196 [08:10:51] I actualy commented out 193 but meant 196 [08:11:00] could you fix and relog? [08:11:10] Done [08:11:13] sweet [08:11:24] I'd like to know why Nagios displays 196 as HTTP OK and SSH OK [08:11:37] I can tell that 196 is responsive to ping, for one thing [08:11:42] no idea [08:12:50] srv196 is serving HTTP for sure [08:12:50] I would ignore it and move on however [08:13:09] I just nc-ed to it and it dumped [[Barack Obama]] in my face just fine [08:13:16] neilk_: OK deployed for real now [08:13:44] RoanKattouw: right. I also svn upped my deploy branch [08:15:32] RoanKattouw: thumbnailing is definitely not working, although I'm not sure if that's UploadWizard or Commons [08:15:54] May very well be ms*'s fault [08:16:05] oh? [08:16:09] don't be blaming ms5 [08:16:13] The things that's supposed to serve thumbnails is 500 server error-ing [08:16:19] ^ [08:16:28] 1 PHP Fatal error: Class 'ApiTitleCheck' not found in /usr/local/apache/common-local/php-1.17/includes/api/ApiMain.php on line 837 [08:16:30] WTF... [08:16:34] what. [08:16:59] that's not supposed to be committed. [08:17:01] strange [08:17:10] (I am watching load, graphs, and render log: when I start seeing timeouts I'll let you know) [08:17:17] maybe there was a revert... [08:17:20] extensions/UploadWizard/UploadWizard.php [08:17:22] 44: //, 'ApiTitleCheck' [08:17:23] 49:$wgAPIModules['titlecheck'] = 'ApiTitleCheck'; [08:17:25] 50:$wgAPIListModules['titlecheck'] = 'ApiTitleCheck'; [08:17:40] Also, it shouldn't be both an action= AND a list= module [08:17:45] yeah I know I wrote this but reverted at some point [08:17:57] it was a mistake [08:18:05] (I also have Special:NewFIles open as a doublecheck.) [08:18:19] I just commented those two lines out [08:18:33] However, if you also have JS hitting that, you should fix that [08:21:01] I tried commenting them out too [08:21:06] doesn't seem to have fixed anything [08:21:12] on my other deployment branch [08:22:13] titlecheck doesn't return any grep results in 1.17wmf1/extensions/UploadWizard, other than the php file [08:22:57] RoanKattouw: I think you merged r85606 -- that was the revision I asked to leave out [08:23:04] that shouldn't cause the problem, though [08:23:08] just saying [08:23:27] I reverted it [08:24:29] It's not in there [08:24:40] Search the diff for 'largethumbnailwidth' (case-insensitive) [08:26:14] ? [08:26:36] Well, it was in whatever I got from deploy117 branch [08:28:33] ok, this is weird -- I had a conflict in includes/specials/SpecialUploadStash.php -- it was causing similar 500 errors here, can you check there? [08:28:42] just php -l includes/specials/SpecialUploadStash.php [08:28:53] ^RoanKattouw [08:29:52] No syntax errors on my local copy [08:29:58] ok [08:30:11] and svn st is clean? [08:30:16] no C's [08:30:18] Nor on fenari [08:30:39] Yes, except for the fact that I commented out those two lines in UploadWizard.php [08:32:33] anything in the logs about errors with SpecialUploadStash.php ? It is throwing 500s still, for me [08:33:12] Logs are clean [08:33:29] Except for that one ApiTitlewhatever error [08:34:16] that can't be, I'm getting 500 errors [08:34:31] we're talking about commons.wikimedia.org right, not commonstest ? [08:35:04] Yes, the live site [08:35:06] You're getting 500s from the live site? [08:35:49] ah no, firebug was lying [08:35:53] hm [08:37:56] there's some sort of error preventing it from doing thumbnails [08:38:07] however, my error message sucks... it just says "Array" [08:38:15] :-) [08:38:23] http://commons.wikimedia.org/wiki/Special:UploadStash/thumb/ms13qieg3i7inwl4qfvfhg4e8wxvbit.png/60px-ms13qieg3i7inwl4qfvfhg4e8wxvbit.png [08:39:38] I can't do that, it tells me the key is not found in stash [08:39:49] That URL only works for Neil [08:39:56] ha [08:39:59] forgot about that :) [08:40:01] That's the purpose of the stash in fact [08:40:11] also I see nothing in the (I thought it was) thumbnail error log [08:40:16] can I get you to push a one line change to make that error more informative [08:40:18] so I bet the request never makes it to the scaler [08:40:24] neilk_: Sure [08:43:32] ...as long as you tell me what to change ) [08:45:32] oh, I just thought you could pick, randomly [08:45:38] sorry, I went away and then noticed something else [08:46:01] RoanKattouw SpecialUploadStash has been changed recently, I didn't know that. Maybe this is causing the errors [08:46:29] In trunk you mean? [08:48:03] yes [08:49:54] Last change was on Mar 25, and I think I deployed that one last week [08:50:00] Indeed I did [08:56:49] RoanKattouw: yeah but there were line differences between SpecialUploadStash.php in the deploy branch and trunk [08:56:56] and some code differences [08:57:12] Is there a way to know if r81209 was merged? [08:57:12] Yeah I just merged in your change, I didn't update it to trunk state [08:57:17] !r 81209 [08:57:17] --elephant-- http://www.mediawiki.org/wiki/Special:Code/MediaWiki/81209 [08:57:31] It claims to be [08:58:12] ok, I am confused. [08:58:29] anyway you got the last change? r85610 ? [08:59:17] !r 85610 [08:59:17] --elephant-- http://www.mediawiki.org/wiki/Special:Code/MediaWiki/85610 [08:59:33] OK picking that up now [09:03:16] neilk_: Deployed [09:03:25] thanks. checking [09:05:45] RoanKattouw: ok, now that line is giving a more informative error. It's saying http-bad-status, 404, Not Found [09:06:03] in other words, the thumbnail it expected to find isn't found [09:06:13] so [09:06:24] either the calculation of the path to the thumbnail is wrong [09:06:34] or, the thumbnail never got generated [09:06:39] any other possibilities? [09:08:03] In theory this could be a regression due to the ms5 proxying setup [09:08:15] But I'd like to know the URL that 404d [09:10:14] that seems highly unlikely to me, how could UW know [09:10:35] i am disturbed to see there is difference in how trunk and deploy branch calculate the urls for thumbs [09:10:50] Are you sure that this is the cause? [09:11:52] not at all.... it should have been broken before then [09:12:02] !r 81984 [09:12:02] --elephant-- http://www.mediawiki.org/wiki/Special:Code/MediaWiki/81984 [09:12:32] That change does not purport to be merge [09:12:33] e [09:12:37] might be it [09:12:38] d [09:12:42] where do stashed thumbs go? [09:12:56] nowhere special, just the temp area of FileRepo [09:13:50] ok we've been at this for 2 hours [09:14:29] I don't see how it used to work before deploy and now doesn't [09:14:58] committing this would be nice, but this change appears to be a refactor anyway [09:15:06] not an actual change of behavior [09:15:13] unless the underlying function changed... [09:17:16] you checked it before deployment and it was working? [09:17:19] RoanKattouw, Apergos: suggestions? [09:17:33] apergos: works on my machine, works on commons.prototype trunk. [09:17:45] and works on commons.prototype deploy branch [09:17:55] Debug dump the URL that's 404ing is my suggestion [09:18:03] It's quite possible our thumb infrastructure is at fault here [09:19:43] yes please, let's at least see what it thinks it is trying to retrieve [09:19:49] without that I can't check very much on my ind [09:19:52] *end [09:20:42] RoanKattouw: what do you mean by debug dump [09:20:56] print it out somewhere you can see it, is what I meant [09:21:51] here's an example of the output [09:21:51] http://pastebin.com/Tngg7GZc [09:21:52] notthe special:uploadstash page but the actual request [09:22:15] the URL is http://commons.wikimedia.org/wiki/Special:UploadStash/thumb/p8uiv3aan9nxpvjgc7pnvxhdzlisk0z.png/26px-p8uiv3aan9nxpvjgc7pnvxhdzlisk0z.png [09:22:37] without that stuff in it. that's not enough for me to tell what ms5 actually sees [09:22:46] I can't know exactly what file it's trying to look up because the whole point is that users like me aren't supposed to know. [09:23:00] the only way to do that would be to add debugging statements or something [09:23:05] yes. exactly. [09:23:07] ok [09:23:10] please do if you don't mind [09:23:32] I've already grepped for one of those by filename in all thelogs I have access to, and there was nothing.... so... [09:23:50] (all the *relevant* logs :-P) [09:28:16] RoanKattouw: check out this paste [09:28:17] http://pastebin.com/v98zAMHp [09:28:40] OK [09:28:51] please replace the if() { ... } around line 232 of includes/specials/SpecialUploadStash.php [09:28:56] with that [09:29:37] we'll have to remember to remove this... [09:29:56] Yes [09:30:04] Deployed [09:31:38] ok now this is interesting [09:31:43] here's the pastebin of an error [09:31:44] http://pastebin.com/cyNCEP1t [09:31:52] note the URL it is trying to fetch [09:32:05] it's this one [09:32:06] http://upload.wikimedia.org/wikipedia/commons/thumb/temp/9/9f/20110407093716!phphuNG53.jpg/32px-20110407093716!phphuNG53.jpg [09:32:11] which seems to be just fine!! [09:33:42] there is virtually nothing in that directory [09:33:47] temp/9/9f [09:34:42] would you expect there to be a lot? [09:34:50] i'm not sure I understand [09:35:04] yes; I would expect more than one item sitting in there [09:35:14] from other thumb creation attempts [09:35:19] For temp files? [09:35:25] Temp thumbs are rare at this point [09:35:29] maybe someone is cleaning them [09:35:36] Although 1 per shard is a little to few maybe [09:36:08] ok how about this: last update time is Apr 3 [09:36:13] so no, nothing's cleaning them [09:36:14] ok I don't understand why we are getting an error from PHP when this file is available, unless it wasn't available at that time. Like, it took too long. [09:36:22] Doesn't look like it [09:36:29] I requested a 103px thumb and got it immediately [09:36:29] did we change anything about delaying the creation of thumbs? [09:36:55] well, PHP is saying 404. [09:36:59] uh huh [09:37:01] PHP SAYS NO [09:37:08] I'm still looking at it. give me a few minutes please [09:37:17] the more I type in here the less I am looking at the nginx setup [09:37:30] and the thumb handler and (maybe also) the squid settings [09:37:59] lol [09:38:04] I can reproduce the 404 from inside the cluster [09:38:13] But it works from the outside [09:38:30] hahahaa [09:38:34] OH LOL [09:38:38] okay that is starting to make sense [09:38:45] It's a 404 *but it has the image in the response body* [09:39:00] No. way. [09:39:10] apergos: Whatever the web server conf on ms5 (?) is, is running the 404 handler appropriately but not changing the status code from 404 to 200 [09:39:11] because the image does not exist in the filesystem; I looked. [09:39:24] Dude, http://upload.wikimedia.org/wikipedia/commons/thumb/temp/9/9f/20110407093716!phphuNG53.jpg/103px-20110407093716!phphuNG53.jpg [09:39:32] *shrug* [09:39:54] Roan is right [09:39:59] it is 404ing and returning the content [09:40:06] WTF but the file isn't there, you'rer ight [09:40:06] at least for us [09:40:12] ls -l /export/thumbs/wikipedia/commons/thumb/temp/9/9f [09:40:13] total 0 [09:40:13] drwxrwxrwx 2 www-data www-data 87 2011-04-03 00:46 20110403004620!phpEMLSBV.jpg [09:40:18] that is all that's there. [09:40:23] apergos: Yeah so the 404 handler is broken in two different ways [09:40:36] Might well be related now I think of it [09:40:51] but wait [09:40:56] what changed with this deploy [09:41:23] Nothing, I don't think. This would have happened when apergos and his minions were messing around with the thumbs infrastructure to keep it from exploding [09:41:30] well we get new thumbs [09:41:34] every day all day [09:41:49] I didn't verify that the UploadWizard was working on Commons in the last few hrs before we deployed. [09:41:51] I just assumed. [09:42:00] ...at least I don't think I did? [09:42:49] they get generated, written, served. so this is specific either to the code or to the temp dir location. [09:43:28] Probably the temp dir [09:43:33] I mean look at this [09:43:42] I've been requesting thumbs for this image in all sorts of sizes [09:43:52] They come back fine, but with a 404 status code, and they aren't created on ms5 [09:44:24] well the scaler would write them someplace [09:44:34] after it generates them, since you get served them [09:44:51] Well wait one second [09:44:54] It's not even present on ms7 [09:45:00] Where the hell is this file even *coming* from [09:45:08] http://upload.wikimedia.org/wikipedia/commons/temp/9/9f/20110407093716!phphuNG53.jpg --> 404 [09:45:10] ls -ld /export/thumbs/wikipedia/commons/thumb/temp/9/9f [09:45:10] drwxrwxrwx 3 www-data www-data 41 2011-04-03 00:46 /export/thumbs/wikipedia/commons/thumb/temp/9/9f [09:45:22] and the directory is writeable by the scaler.... [09:45:31] What I'm saying is, I can't even find the original file [09:45:41] So how the heck are the scalers even scaling it [09:46:08] well if the y scale it why aren't they writing it? they write it via nfs, there's no web service in volved [09:46:44] That's mysterious [09:46:59] But I think that failure to locate the original is even more mysterious [09:51:25] anything I can do? [09:51:39] just a sec [09:52:31] yes, could you dump the url of the image? [09:52:42] let's see if it's really what it ought to be [09:53:01] (ie the image upladed in the stash or wherever it is) [09:53:25] OK so this works http://commons.wikimedia.org/w/thumb.php?f=20110407093716!phphuNG53.jpg&w=201&temp=1 [09:53:48] apergos: I lost that one in a million tabs I can start from scratch [09:53:48] But I cannot find the original [09:54:01] so we'll find out where it really is in a minute... neil? [09:54:08] yeah from scratch is fine [09:54:08] neilk_: Does UploadStash succeed in fetching the original? [09:54:14] it never tries [09:54:20] You can make it try [09:54:26] I guess [09:54:29] I guess in theory it should be at http://upload.wikimedia.org/wikipedia/commons/temp/9/9f/20110407093716!phphuNG53.jpg but that's 404 for me [09:54:35] let me start over with a new file [09:54:39] sure [09:54:42] And neither the file nor any of its thumbs are in the normal FS locations on ms7/5 [09:56:18] here's an original [09:56:19] http://commons.wikimedia.org/wiki/Special:UploadStash/file/qpwsk103fcpy2jpmjlw4s8e1qkcd61i.png [09:56:51] firebug says that was 200 OK [09:57:41] and do you have a pastebin of what the underlying image url is from in the mw code? [09:58:11] AHA: [09:58:14] PHP fatal error in /usr/local/apache/common-local/php-1.17/includes/specials/SpecialUploadStash.php line 120: [09:58:16] Call to a member function getFile() on a non-object [09:58:50] apergos: I don't know what the underlying object was -- we only added that to failed methods [09:59:00] but i can show you one for the failed thumbnail [09:59:15] http://upload.wikimedia.org/wikipedia/commons/thumb/temp/3/36/20110407100209!php8lAwPN.png/120px-20110407100209!php8lAwPN.png [09:59:19] I want it for the original image [09:59:22] unfortunately. [09:59:26] Which implies that .... wtf, $this->stash is null?!? [09:59:29] I don't know what that is, unless we add some other thing [09:59:33] the fatal tells us [09:59:34] thanks [09:59:56] http://upload.wikimedia.org/wikipedia/commons/temp/3/36/20110407100209!php8lAwPN.png 404s [10:00:07] And yeah the fatal seems to point us somewhere [10:00:26] Oh, how helpful [10:00:32] Fatal errors don't include the srv number [10:00:52] it's landed in [10:01:23] stash being null is supposed to be impossible [10:01:25] /export/upload/wikipedia/commons/temp/thumb/3/36/20110407100209!php8lAwPN.png [10:01:51] Oh so this one is there? [10:02:01] Ion ms7 [10:02:04] not on ms5 [10:02:18] oO [10:02:24] So the thumb is being created on ms7 instead of ms5? [10:02:29] yes. so that's mw screwin g it up. [10:02:41] Yeah [10:03:30] OK the stash is null problem seems to have been a fluke [10:03:46] well that should not even be possible [10:03:51] I don't see the code path for that [10:04:12] Me neither [10:04:17] It failed once, and worked after refresh [10:04:19] No idea why [10:04:24] So that's probably a red herring [10:04:33] It seems the fs path for temp thumbs is being built wrong [10:04:37] dunno, it is a pretty smelly red herring [10:05:06] Note that that link wouldn't have worked for me anyway [10:05:10] Cause it'd be accessing my stash, not yours [10:05:27] right [10:06:01] if either of you have Chrome or Firebug you can do this yourself, just go to UploadWizard, upload some file (NOT from Commons, get some file which has never been on Commons), examine API returns [10:06:02] well let's fix that at least [10:07:25] So, just to recap [10:07:36] we didn't merge this change to deploy branch? [10:07:39] http://www.mediawiki.org/wiki/Special:Code/MediaWiki/81984 [10:08:08] it changes things a bit, although I think it's just a refactor with no impact on functionality. [10:08:23] other than that, from the app end I will have to just sit back and wait... [10:08:55] can we (you, someone) please fix the fs path for temp thumbs issue? [10:09:05] that is a mw code issue [10:09:40] I'm looking at the path issue nwo [10:09:44]