[00:00:46] *^demon 	kicks Reedy a little harder
[00:00:59] <Reedy>	I just fixed one
[00:01:07] <Reedy>	though, diffs are skewy
[00:01:20] <^demon>	Yeah, gonna say
[00:01:49] <Reedy>	Linker thing will be new
[00:03:16] <neilk_>	Roan: yup, you were right, more or less.
[00:03:38] <Reedy>	Fixed also
[00:03:59] <^demon>	Maybe someone shouldn't have been so impatient to deploy trunk ;-)
[00:04:13] <^demon>	Diff on http://www.mediawiki.org/wiki/Special:Code/MediaWiki/105948 still broken.
[00:04:24] <^demon>	Missing line #s
[00:04:37] <Reedy>	Not as broken as it was before ;)
[00:05:54] <Reedy>	Ok, what the hell did Hashar also change
[00:06:02] *Reedy 	reverts for the time being
[06:40:48] <TimStarling>	> var_dump(unserialize(null))
[06:40:48] <TimStarling>	bool(false)
[06:40:58] <TimStarling>	this doesn't even issue a warning
[06:49:27] <Nikerabbit>	yes unserialize accepts all kinds of crap
[09:17:23] <hashar>	!branches
[09:17:23] --elephant--	I don't know anything about "branches".
[12:34:28] <petan|wk>	RoanKattouw: could you send me a list of commands you use for current irc bot?
[12:34:45] <petan|wk>	Reedy: is there any problem with this
[12:34:46] <petan|wk>	https://bugzilla.wikimedia.org/show_bug.cgi?id=32819
[12:34:58] <RoanKattouw>	I'm not sure I know the exhaustive list
[12:35:06] <RoanKattouw>	But these are the ones I know:
[12:35:07] <petan|wk>	I mean the commands like @trusted
[12:35:09] <petan|wk>	etc.
[12:35:14] <RoanKattouw>	!petantest is Test 1 2 3
[12:35:14] --elephant--	Successfully added keyword: petantest
[12:35:20] <RoanKattouw>	!petantest del
[12:35:20] --elephant--	Successfully removed keyword: petantest
[12:35:24] <RoanKattouw>	!petantest alias petantest2
[12:35:24] --elephant--	com.amidaniel.mwbot.MWBotException: Unable to resolve keyword: petantest2
[12:35:28] <RoanKattouw>	Oh, lol
[12:35:35] <petan|wk>	ok these are implemented already heh
[12:35:41] <RoanKattouw>	Even del and alias?
[12:35:45] <petan|wk>	yes
[12:35:53] <RoanKattouw>	What about piping and `e1 replacement?
[12:36:03] <petan|wk>	piping too
[12:36:08] <petan|wk>	dunno what e1 is
[12:36:12] <RoanKattouw>	!e
[12:36:12] --elephant--	http://www.mediawiki.org/wiki/Extension:`e1
[12:36:17] <RoanKattouw>	!e VisualEditor
[12:36:17] --elephant--	http://www.mediawiki.org/wiki/Extension:VisualEditor
[12:36:22] <petan|wk>	that will be implemented in 5 minute
[12:36:23] <petan|wk>	minutes
[12:36:30] <RoanKattouw>	:)
[12:36:34] <petan|wk>	anything else?
[12:37:11] <petan|wk>	I don't know what the @trust commands are
[12:37:20] <petan|wk>	I implemented @trustadd now and trustdel
[12:37:25] <RoanKattouw>	I think it's really just @trust and @untrust
[12:37:31] <RoanKattouw>	But no one cares if those aren't fully compatible
[12:37:36] <RoanKattouw>	They're used very infrequently
[12:37:40] <petan|wk>	ok
[12:37:43] <petan|wk>	no problem to change it
[12:37:54] <RoanKattouw>	There's an 80/20 rule here, as with everything
[12:38:03] <petan|wk>	heh
[12:38:05] <RoanKattouw>	There's only a few commands that people use all the time, those are the most important
[12:38:17] <petan|wk>	I will need to dump current db of keys too
[12:38:31] <petan|wk>	I don't think we want to learn it everything in channel hehe
[12:38:50] <petan|wk>	* tech it
[12:38:53] <petan|wk>	* teach
[12:39:13] <petan|wk>	I think you talked about html dumps too?
[12:39:14] <RoanKattouw>	!botbrain
[12:39:14] --elephant--	I don't know anything about "botbrain".
[12:39:16] <RoanKattouw>	meh
[12:39:22] <RoanKattouw>	http://toolserver.org/~mwbot/botbrain.html
[12:39:28] <RoanKattouw>	has the brain for the #mediawiki instance
[12:39:28] <petan|wk>	ok
[12:39:38] <RoanKattouw>	The bot brain is really just stored in a text file
[12:39:55] <petan|wk>	ok
[12:40:09] <petan|wk>	I will insert it to current db so that people could use del to overwrite it
[12:40:29] <petan|wk>	there is nothing hard coded
[12:40:37] <petan|wk>	everything is in some db
[12:41:04] <petan|wk>	I finish it today and put it to -labs for test
[12:41:24] <petan|wk>	it will be running on wmf labs I think it's better than toolserver
[12:42:09] <RoanKattouw>	yeah
[12:42:23] <petan|wk>	more space :)
[12:42:24] <petan|wk>	for logs
[12:42:35] <petan|wk>	should I log everything or just messages?
[12:42:42] <petan|wk>	what about /me
[12:42:59] <RoanKattouw>	 /me should be logged
[12:43:05] <RoanKattouw>	Joins/quits probably don't need to be
[12:43:08] <petan|wk>	I think there were some concerns about quit messages so I will leave it configurable what all would be logged
[12:43:15] <RoanKattouw>	The point is to preserve conversation
[12:43:19] <petan|wk>	I will make a config option
[12:43:24] <petan|wk>	for each channel for that
[12:43:58] <petan|wk>	since the source is in svn everyone can improve it or change :) also it's pretty easy to get access to labs so whoever else can operate it too
[12:44:06] <petan|wk>	I will create shared folder for the bot on labs
[12:44:56] <petan|wk>	btw conversation is hard to read when you don't know that person quit meanwhile etc.
[12:45:16] <petan|wk>	so that's why I think join etc should be logged too
[12:46:04] <RoanKattouw>	Joins/quits are so noisy, we tend not to log them
[12:46:09] <petan|wk>	ok
[12:46:11] <petan|wk>	no problem
[12:49:44] <Reedy>	petan, I'm not sure who exactly has the authority to sign off extensions for deploy
[12:51:22] <petan|wk>	ah, ok... maybe we should wait for Ian I don't know it too
[12:51:53] <petan|wk>	it can't be deployed even to test wiki?
[12:57:29] <hashar>	petan|wk: IIRC test wiki is on the cluster and is not really a test wiki
[12:57:35] <hashar>	more like a preproduction wiki
[12:57:40] <RoanKattouw>	That's right
[12:57:43] <RoanKattouw>	test = preproduction
[12:57:45] <RoanKattouw>	Well said
[12:57:59] <petan|wk>	ok
[12:58:06] <hashar>	long time ago we would  svn update the change we wanted from the svn repository to a file server
[12:58:12] <hashar>	tests.wikipedia.org ran from that file server
[12:58:30] <petan|wk>	ah
[12:58:31] <hashar>	once tests was fine, we could actually copy the files from that file server to the other web servers
[12:58:51] <hashar>	the process is still roughly the same :)
[12:59:18] <petan|wk>	right
[12:59:20] <hashar>	anyway, if you get your extension reviewed there is some possibility it will be deployed on the live cluster :)
[12:59:35] <petan|wk>	it was reviewed by Ian already
[12:59:41] <petan|wk>	he submitted the ticket
[12:59:45] <hashar>	might need a last pass for security review which is probably done by Roan / Tim Starling
[12:59:50] <hashar>	great :))
[13:00:00] <petan|wk>	it's like 10 lines of code
[13:00:04] <petan|wk>	really easy to review :D
[14:06:58] <petan|wk>	RoanKattouw: I will use $1 for replacement
[14:07:05] <petan|wk>	instead of `e1 ;)
[14:07:13] <RoanKattouw>	Yeah that's probably better :)
[14:09:15] <petan|wk>	!this_is_intentionally_null
[14:09:15] --elephant--	I don't know anything about "this_is_intentionally_null".
[14:09:32] <petan|wk>	!random_blah
[14:09:32] --elephant--	I don't know anything about "random_blah".
[14:09:36] <petan|wk>	hm...
[14:10:55] <petan|wk>	what html parsing you wanted it to do :O
[14:11:13] <petan|wk>	or html output or whatever you wanted
[14:11:58] <RoanKattouw>	I think the TS thing just has an HTML table
[14:12:14] <petan|wk>	that's ok to implement if I save the keys to sql I can create php script
[14:12:15] <RoanKattouw>	http://toolserver.org/~mwbot/botbrain.html
[14:12:32] <Reedy>	Why does it have to be sql?
[14:12:37] <Reedy>	php can parse a simple text file
[14:12:39] <petan|wk>	it doesn't not have to be
[14:12:47] <petan|wk>	ok, or parse text file :)
[14:12:53] <petan|wk>	php anyway heh
[14:14:12] <petan|wk>	actually I storing it to file is a lot easier
[14:14:20] <petan|wk>	s/I//
[16:14:19] <petan|wk>	hey Reedy where are you from? I am wondering how you can be so early in here, aren't you working for wmf too?
[16:14:45] <petan|wk>	I thought you are in san francisco like others...
[16:14:46] <RoanKattouw>	petan|wk: You're in UTC +1, +2 or +3 I assume?
[16:14:52] <petan|wk>	I am in gmt
[16:14:55] <RoanKattouw>	Ah
[16:14:58] <petan|wk>	europe
[16:14:59] <RoanKattouw>	Reedy is in GMT as wel
[16:15:00] <RoanKattouw>	l
[16:15:03] <petan|wk>	ah
[16:15:09] <RoanKattouw>	And I'm in +1
[16:15:12] <petan|wk>	so not in sf :)
[16:15:15] <RoanKattouw>	No
[16:15:46] <petan|wk>	hm... so many people working remotely
[16:16:09] <RoanKattouw>	Yeah, we've got a number of remote people
[16:16:42] <RoanKattouw>	We have like 8 WMF employees in Europe
[16:16:53] <Reedy>	Most of which are in NL :p
[16:16:55] <RoanKattouw>	2 people in Australia, people in India, etc
[16:17:01] <RoanKattouw>	Yeah, at least half of them :)
[16:17:08] <RoanKattouw>	But we've got 2 in France, 1 in the UK and 1 in Greece
[16:17:11] <petan|wk>	heh
[16:17:25] <petan|wk>	yes Apergos is from Greece I think
[16:17:38] <RoanKattouw>	It used to be that all employees were based in the US, AU or NL
[16:17:44] <apergos>	no, I just live here
[16:17:56] <petan|wk>	right, that's what I meant :)
[16:18:01] <apergos>	k :-)
[16:18:34] <apergos>	and my back is finally starting to unfreeze... *whew*
[16:19:15] *RoanKattouw 	googles "weather in athens" and snorts
[16:19:31] <RoanKattouw>	22 C today? That's not winter :P
[16:19:42] <petan|wk>	heh, speak for yourself ;P
[16:19:54] <petan|wk>	you are from Netherlands right?
[16:19:57] <RoanKattouw>	Yeah
[16:20:07] <petan|wk>	hm...
[16:20:14] <RoanKattouw>	Our winter isn't looking too good just yet either
[16:20:15] <apergos>	it's evening
[16:20:22] <petan|wk>	I think one of my bots could predict weather
[16:20:23] <apergos>	and without heat and without moving around a lot...
[16:20:26] <RoanKattouw>	Lots of rain mostly
[16:20:32] <RoanKattouw>	Oh, right, no heat
[16:20:32] <petan|wk>	but then site was broken and I was lazy to fix that
[16:20:45] <RoanKattouw>	15 C is cold if you don't have a working heater
[16:21:07] <apergos>	when it gets cold enough some muscles in my back and shoulders ahve a tendency to sieze up... so today was the worst of it, I could barely move at some points
[16:21:33] <RoanKattouw>	lol, if you look up "weather in tromso" it says it's "clear", then shows an icon of a brightly shining sun
[16:21:44] <RoanKattouw>	Except that the sun doesn't actually *rise* in Tromso this part of year xD
[16:21:53] <petan|wk>	:)
[16:23:35] <robla>	Reedy: you around?  I'd like to talk about the exif stuff
[16:23:57] <Reedy>	i'm currently seeing what my pulse is according to an android app
[16:24:07] *apergos 	lurks
[16:24:31] <Reedy>	But yeah, I'm here
[16:24:35] <Reedy>	going to go refill my drink though
[16:25:20] <robla>	Reedy: did Tim's list building suggestion make sense?
[16:26:59] <robla>	apergos: I know you've been following this saga (thank you!)....Tim suggested that we should be able to build a list of all images with EXIF rotation.  I'm assuming that hasn't already been done by anyone, correct?
[16:27:07] <apergos>	I have a list
[16:27:15] <apergos>	it's all images with rotation 3, 6 and 8
[16:27:28] <robla>	oh really?  where does that live?
[16:27:35] <apergos>	which supposedly (I did not verify it) are the values mw looks at
[16:27:41] <apergos>	on my laptop :-P
[16:27:50] <apergos>	hold on a sec
[16:29:16] <apergos>	300k file gzipped, where can I put it?
[16:29:28] <Reedy>	~/public_html ?
[16:30:13] <apergos>	on...?
[16:30:59] <petan|wk>	your laptop :P
[16:31:07] <Reedy>	fenari works for me :p
[16:31:19] <Reedy>	just a web dir seemed easier for people to grab it from :)
[16:33:35] <apergos>	/home/ariel/incoming/imagetitlestofilteruniq.txt.gz
[16:34:00] <apergos>	hmm just a sec
[16:34:07] <apergos>	that might be only against the images I was gonna delete
[16:35:00] <robla>	delete thumbs for, I'm assuming
[16:35:49] <apergos>	no that's the right list
[16:35:57] <apergos>	yes, thumbs :-D
[16:36:08] <apergos>	actually I'm in a bad mood so I thought I'd remove the images too....
[16:36:24] <robla>	that'll show 'em for getting the metadata wrong!
[16:36:27] <apergos>	this is only commons
[16:36:36] <apergos>	I did not check any of the images local to the projects
[16:36:44] <robla>	urgh....there is that
[16:36:54] <apergos>	(I am also not engaged in doing cleanup on those thumbs)
[16:36:58] <robla>	ok...so, 27835 images
[16:37:05] <apergos>	if/when you want that list I can do it with a minimum of effort
[16:37:24] <apergos>	uh huh
[16:37:48] <apergos>	I have a tool that lets me run queries n all the wikis, collect up the gzs and then process them
[16:37:51] <robla>	apergos: roughly what percentage of images live on commons versus out in the other wikis
[16:37:56] <robla>	?
[16:38:30] <apergos>	it's the vast majority but I don't have numbers ight now
[16:38:35] <apergos>	I have some old numbers around
[16:38:52] <robla>	nothing too precise....over 2/3?
[16:38:58] <RoanKattouw>	<10% I think
[16:39:31] <robla>	<10% out on other wikis, I'm assuming
[16:39:43] <RoanKattouw>	Yeah, that's my guess
[16:39:50] <RoanKattouw>	We have like 10 or 11 million files on Commons now?
[16:39:58] <RoanKattouw>	I would be surprised if we had more than a million on all other wikis combined
[16:40:08] <apergos>	surely over 2/3
[16:40:29] <RoanKattouw>	Oh, enwiki has 822k files
[16:40:31] <RoanKattouw>	More than I thought
[16:40:46] <apergos>	yes, they have a lot relatively speaking
[16:40:56] <RoanKattouw>	Yeah, de and fr have very few
[16:40:59] <RoanKattouw>	en+de+fr =~ 1M
[16:41:08] <robla>	so...I don't imagine we'll need the other wiki list right away, but we'll need it eventually
[16:42:51] <apergos>	that's fine
[16:43:03] <robla>	apergos: if you create /home/ariel/public_html, copy the file there, we should be able to access it at http://noc.wikimedia.org/
[16:43:04] <apergos>	give me a couple days of lead time
[16:43:17] <robla>	http://noc.wikimedia.org/~ariel that is
[16:43:18] <apergos>	I had no idea we had user html set up for noc
[16:43:24] <apergos>	sec
[16:44:08] <apergos>	done
[16:44:31] <apergos>	I see, no index page needed
[16:44:46] <apergos>	please let me know when you've grabbed it, I guess I shouldn't just leave crap in there all the time
[16:45:05] <robla>	well....thing is....
[16:45:11] <apergos>	yees?
[16:45:32] <robla>	I'd like to make that list available in the commons talk discussion
[16:45:36] <apergos>	um
[16:45:54] <apergos>	do we really want a pile of people downloading from fenari? especially from nfs
[16:46:08] <apergos>	?
[16:46:34] <robla>	it's not going to be like putting it on the enwiki homepage or anything.  I imagine a couple dozen people will be interested in it
[16:46:36] <apergos>	btw that list is from dec 8, I assume it's "good enough" though
[16:46:44] <robla>	yeah, it's certainly good enough
[16:46:55] <apergos>	ok.  please put a timeout on it, like "please get it in a week" or something
[16:48:45] <robla>	here's the discussion thread
[16:48:46] <robla>	http://commons.wikimedia.org/wiki/Commons_talk:Rotation
[16:49:34] <apergos>	right
[16:49:50] <apergos>	there's a few other discussions floating around
[16:55:17] <robla>	posted comment here: http://commons.wikimedia.org/wiki/Commons_talk:Rotation#List_of_images
[16:55:36] <apergos>	great
[16:56:03] <robla>	Reedy: could you coordinate with the existing fixup efforts and (if possible) accelerate the fixup?
[16:56:03] <apergos>	yeah the issue is that some of the rotations are "correct" and some are not, depending on whether the initial image was rotated with nice software or with broken software >_<
[16:56:14] <apergos>	which means a human eye has to look at them
[16:56:45] *robla 	goes afk for a bit
[17:05:47] <Reedy>	apergos, I'm somewhat out of the loop with all this
[17:06:03] <Reedy>	What needs doing now? Manual checking of that list of pages, optionally marking for rotation?
[17:06:07] <apergos>	ah
[17:06:27] <apergos>	well if they have a bot backlog and problems clearing it (because it runs too slowly)
[17:06:39] <apergos>	which is something I have heard folks say a few days ago, that's one thing that could be done
[17:07:11] <apergos>	the other is unfortunately screening all remaining images... some of which have old thumbs and some of which have new ones.  so it's worse than anyone would like.
[17:07:38] <apergos>	I would pop into wikimedia-commons and ask how the rotate bot is coming along though (and if there are similar efforts)
[18:03:40] <Inez>	Hello
[18:04:45] <RoanKattouw>	neilk_: Could you respond to https://www.mediawiki.org/wiki/Special:Code/MediaWiki/105974#c27521 ?
[18:05:44] <hashar>	neilk_: also added you to https://www.mediawiki.org/wiki/Special:Code/MediaWiki/104495  :)
[18:16:44] <Krinkle>	Did everybody forget the meeting was moved or was it not really moved ?
[18:17:00] <Krinkle>	afaik it's 10.15am PDT, and meeting was moved from 11-11.30 to 10-10.30 ?
[18:18:01] <Nikerabbit>	what meeting?
[18:18:06] <Krinkle>	Features
[18:18:16] <Nikerabbit>	wasn't it on wednesday?
[18:18:35] <Krinkle>	uhuh, I see that now.
[18:20:09] <alolita>	krinkle: hi
[18:20:21] <alolita>	confusion about the features meeting?
[18:20:30] <Krinkle>	yeah, so there's one today and one tomorrow
[18:20:33] <Krinkle>	correct ?
[18:20:51] <alolita>	krinkle: due to the remote folks having such a tough time dialing in - i've split the sessions
[18:21:10] <Krinkle>	the etherpad is still named after today's date. But content adapted to tomorrow's date. So.. SF-members are not going to etherpad/IRC anymore ?
[18:21:22] <Krinkle>	alolita: wasn't the meeting changed to an IRC meeting ?
[18:21:26] <alolita>	krinkle: so we do tuesday 30 mins for sf only; wed for remote
[18:21:44] <Krinkle>	ok. so sf only is going to be in conference room again.
[18:22:03] <alolita>	krinkle: sf folks can participate in the irc meeting as needed
[18:22:10] <Krinkle>	ok
[18:22:37] <alolita>	krinkle: tuesdays 11am - i would still like the sf team to sync up since they are here face-to-face
[18:23:13] <Inez>	I will put my update on etherpad around 11:30.
[18:23:22] <alolita>	inez: thanks!
[18:48:42] <guillom>	So, starting this week, the features team meeting will happen on Wednesdays? That means no meeting today? Just checking.
[18:48:53] <RoanKattouw>	I hear the SF and non-SF meetings are split now
[18:50:11] <guillom>	Wait, what?
[18:50:23] <guillom>	That makes absolutely no sense.
[18:50:43] <RoanKattouw>	I know
[18:50:54] <RoanKattouw>	But I'm too busy with OMG-there's-a-deadline-TODAY to care
[18:52:12] <guillom>	Yes, sorry, my WTFing wasn't directed towards you.
[18:52:26] <RoanKattouw>	That's alright
[18:52:38] <RoanKattouw>	I guess I was explaining to myself why I hadn't balked at it
[18:53:59] <Krinkle>	I've updated the etherpad header
[18:54:16] <Krinkle>	http://etherpad.wikimedia.org/FeaturesTeam20111213
[18:59:28] <guillom>	Krinkle, thanks. Where did you get the information from?
[19:00:09] <Reedy>	guillom, he'd have tok kill you
[19:00:19] <Krinkle>	guillom: Alolita. Part of it in the mail from earlier today (cc-ed to a bunch of us), and from a few lines up in this room.
[19:00:23] <Krinkle>	irc room*
[19:00:28] <guillom>	ok, thanks
[19:00:34] <Krinkle>	you're welcome
[19:00:49] <Krinkle>	alolita: I've updated the etherpad header, can you verify it's correct ?
[19:00:59] <Krinkle>	on the pad for today
[19:02:06] <guillom>	Well, I still think this is a truly terrible idea, but it's Not My Problem, after all, as long as the notes in etherpad are filled in.
[19:31:48] <aharoni>	OK, nobody's speaking, so i'll just say something. Notice the comment about test.wikipedia.org in the Internationalization team report.
[19:31:54] <aharoni>	it may be useful for future deployments.
[19:38:32] <Krinkle>	Citing aharoni's subject from pad:
[19:38:33] <Krinkle>	Firefox doesn't load fonts from a different domain. [http://www.w3.org/TR/css3-fonts/#same-origin-restriction Blocking cross-site font loading is actually the standard behaviour] and Firefox is the only browser that implements it. So or.wikipedia.org cannot load fonts from bits.wikimedia.org. We couldn't notice it in translatewiki.net and '''not even in test.wikipedia.org''', which loads everything from the same domain.
[19:38:39] <Krinkle>	</quote>
[19:40:04] <Krinkle>	aharoni: From what I remember Google's font api works fine in Firefox, so there's a way around it.
[19:40:29] <Krinkle>	The way around it (or rather, the "standard" way) is to make sure that the stylesheet referencing the font file and the stylesheet itself are on the same domain.
[19:40:41] <aharoni>	yes, the server may allow it.
[19:40:44] <Krinkle>	Which is the case if ResourceLoader is properly used
[19:40:55] <Krinkle>	as then both the css and the font will come from bits.wikimedia.org
[19:41:17] <RoanKattouw>	Hah, yeah
[19:41:19] <Krinkle>	from what I can tell, it only fails if the font from bits.wm.o is referenced from a user script or site script that isn't loaded with ResourceLoader.
[19:41:26] <brion>	^demon, git repo looks nice so far :D
[19:41:46] <brion>	i'm just giddy that the actual checkout dir size is only moderately bigger than a svn checkout and it's got FULL HISTORY
[19:41:51] <brion>	\o/
[19:41:55] <brion>	\o\
[19:41:56] <brion>	 /o/
[19:42:00] <brion>	daaaaaancin'
[19:42:13] <^demon>	:)
[19:42:37] <brion>	extensions will be fun to convert though... mwhahaha
[19:42:39] <Krinkle>	aharoni: RoanKattouw So this shouldn't be an issue in production, not even on or.wikipedia.org
[19:42:59] <RoanKattouw>	I guess
[19:43:04] <RoanKattouw>	Hmm
[19:43:09] <RoanKattouw>	That means we wouldn't need CORS
[19:43:10] <Krinkle>	but.. doesn't' debug mode load css from local load.pph ?
[19:43:11] <RoanKattouw>	Nikerabbit: Are you seeing this?
[19:43:20] <RoanKattouw>	I don't think so
[19:43:29] <RoanKattouw>	I can't imagine it would
[19:43:33] <Krinkle>	right
[19:43:36] <aharoni>	just try loading http://or.wikipedia.org and see the error log.
[19:43:44] <Krinkle>	oh, right, it only does that on test.wikipedia.org (obviously)
[19:43:46] <Krinkle>	nevermind
[19:43:59] <Krinkle>	aharoni: checking now :)
[19:44:26] <RoanKattouw>	[20:44:18.530] downloadable font: download failed (font-family: "Utkal" style:normal weight:normal stretch:normal src index:2): bad URI or cross-site access not allowed
[19:44:28] <RoanKattouw>	source: https://bits.wikimedia.org/w/extensions-1.18/WebFonts/fonts/Orya/utkal.ttf?version=0.13
[19:45:22] <RoanKattouw>	Ooooooooohh
[19:45:26] <RoanKattouw>	It's embedded CSS
[19:45:29] <RoanKattouw>	The CSS's origin is bits
[19:45:37] <RoanKattouw>	But we embed it in a <style> tag in the DOM
[19:45:41] <Krinkle>	doh
[19:45:50] <Krinkle>	Hm..
[19:45:55] *RoanKattouw 	glares at whichever browser it was that didn't support dynamically inserted <link> tags
[19:46:17] <Krinkle>	RoanKattouw: I don't think that was the issue
[19:46:17] <RoanKattouw>	Although, well, I guess the <style> thing is desirable for other reasons (fewer requests) anyway. But I digress
[19:46:20] <Krinkle>	the issue was http reuqests
[19:46:28] <Krinkle>	well, 'issue'. w/e
[19:46:40] <RoanKattouw>	Anyway
[19:46:42] <Nikerabbit>	here
[19:46:50] <RoanKattouw>	Nikerabbit: Never mind, we were idiots
[19:47:04] <RoanKattouw>	We were thinking this should work because the CSS comes from bits and the font is on bits
[19:47:21] <RoanKattouw>	But because of some RL magic, the conceptual origin of the CSS != the browser-observed origin
[19:47:45] <Krinkle>	( Google Font API works in Firefox, they load css + font from external domain)
[19:48:05] <Nikerabbit>	yeah and the css is dynamically generated
[19:48:09] <Krinkle>	so cross-domain font isn't the problem, as long as css is cross domain as well. Which we do, both are from bits.wikimedia.org, but the problem is ??? ^ Roan says it above.
[19:48:09] <RoanKattouw>	Aye
[19:48:22] <aharoni>	Hi Alolita. You're just in time. We're discussing a major issue with WebFonts not loading in Firefox.
[19:48:27] <RoanKattouw>	Hmm
[19:48:31] <Nikerabbit>	Krinkle: google turns up same errors with google font api, but maybe they have fixed that
[19:48:37] <alolita>	amir: hi
[19:48:39] <RoanKattouw>	I wonder whether @font-face works in IE8?
[19:48:50] <Nikerabbit>	@font-face is what webfont uses
[19:48:53] <RoanKattouw>	Yeah
[19:48:54] <Krinkle>	^
[19:48:57] <RoanKattouw>	I'm asking, does it work in IE8
[19:49:09] <Krinkle>	Yes
[19:49:17] <Nikerabbit>	I would imagine that it works all way back to IE6
[19:49:29] <brion>	should work back to ie6 yeah
[19:49:31] <brion>	using the .eot fonts
[19:49:42] <Krinkle>	Indeed, in a way it works in IE6. It needs a custom version of the font, but it works.
[19:49:52] <Krinkle>	(which WebFonts provides, it provides 3 versions of each font)
[19:50:10] <RoanKattouw>	Oh, hmph
[19:50:16] <RoanKattouw>	I was thinking about data URI embedding for the fonts
[19:50:23] <RoanKattouw>	But that's a no-go for fonts >24KB in IE8
[19:50:36] <RoanKattouw>	Anyway
[19:50:37] <Nikerabbit>	hah, fonts are way bigger than that
[19:50:39] <RoanKattouw>	I guess we just need CORS
[19:50:39] <Krinkle>	yeah
[19:50:59] <RoanKattouw>	Access-Control-Allow-Origin: *   on bits *should* be fine, right?
[19:51:06] <brion>	is anybody seeing the error listed in https://www.mediawiki.org/wiki/Special:Code/MediaWiki/105925#c27530 ?
[19:51:12] <RoanKattouw>	Or at least for the font files
[19:51:30] <Krinkle>	RoanKattouw: Hm.. doesn't it need permission both ways ?
[19:51:56] <RoanKattouw>	No
[19:52:01] <RoanKattouw>	Permission in the other direction is implied
[19:52:12] <RoanKattouw>	The target domain needs to give permission in a header
[19:52:20] <Krinkle>	imho we need proper CORS for all wmf domains at some point regardless. Inevitable for future front-end projects.
[19:52:21] <RoanKattouw>	The originating domain gives implicit permission by serving JS that makes the request
[19:52:24] <RoanKattouw>	Yes
[19:52:38] <Krinkle>	But for now this might work.
[19:54:13] <brion>	allow-origin: * on bits and upload would simplify my life yes :D
[19:54:46] <Krinkle>	brion: Not sure, but the good doesn't look right. It is pre-defined in the class, but calling setCode( .. ) then getHtmlCode( .. ) would trigger it.
[19:55:02] <Krinkle>	or not
[19:55:08] <Krinkle>	yeah, it would
[19:55:13] <Krinkle>	is_null on undefined propertu
[19:56:17] <brion>	hmmm
[19:56:28] <brion>	ah unset!
[19:56:32] <brion>	suspicious indeed :D
[19:56:39] <brion>	it should just set to null instead
[19:56:45] <RoanKattouw>	gerrit-wm	New patchset: Catrope; "Set a CORS header for static files (needed for WebFonts)" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/1501
[19:56:46] <Krinkle>	perhaps change to !isset( .. ) || is_null() ? (or don't unset() )
[19:56:47] <RoanKattouw>	There
[19:56:54] <RoanKattouw>	Get ops to review and deploy that
[19:59:24] <brion>	Warning: post-commit hook failed (exit code 8) with no output.
[19:59:26] <brion>	^demon,  ^
[19:59:39] <^demon>	Mother f'ing....
[19:59:40] <^demon>	Again?
[20:00:01] <brion>	yyyyup
[20:00:59] <aharoni>	RoanKattouw, i'm not much of an Apache guru, but is it ok to allow "*"? maybe just wikimedia domains?
[20:01:38] <RoanKattouw>	aharoni: It's just static image/font files
[20:01:43] <RoanKattouw>	On a cookieless domain
[20:02:34] <brion>	any chance of cors "*" on upload.wikimedia.org as well? i need that for gadgets and other JS tools to edit uploaded svg and raster images etc
[20:02:48] <brion>	i've had to make do with proxies :P
[20:04:32] <RoanKattouw>	upload.wm.o is an evil Sun server, no idea where its config lives
[20:04:36] <RoanKattouw>	Pretty sure it's not Apache
[20:05:13] <RoanKattouw>	brion: We've wanted CORS on upload for a while. Problems are I don't know where the config is, and then there's Squid cache
[20:05:22] <RoanKattouw>	So we'd need to configure Squid to add the header to cached responses too
[20:06:16] <RoanKattouw>	For bits, in theory that's needed too but in practice we can make do
[20:07:12] <brion>	RoanKattouw, well if we put them up months ago and just left them, our caching problem would be gone ;)
[20:07:34] <brion>	by waiting, we ensure that it never gets done
[20:07:36] <RoanKattouw>	True
[20:07:39] <RoanKattouw>	Well, bug Ariel
[20:07:41] <brion>	heh
[20:07:43] <RoanKattouw>	There is a BZ ticket, too
[20:07:43] <Nikerabbit>	okay
[20:07:44] <brion>	apergos, poke poke
[20:07:51] <Nikerabbit>	what's the current state?
[20:07:54] <RoanKattouw>	https://bugzilla.wikimedia.org/show_bug.cgi?id=28700
[20:08:03] <RoanKattouw>	Nikerabbit: Current state is there is a CORS patch in gerrit
[20:08:10] <RoanKattouw>	https://gerrit.wikimedia.org/r/1501
[20:08:17] <RoanKattouw>	An ops person needs to review and deploy that
[20:08:21] <brion>	\o/ an rt ticket is now open
[20:08:29] <Nikerabbit>	RoanKattouw: I'll let our team and erik know this
[20:08:38] <RoanKattouw>	I did
[20:08:43] <Nikerabbit>	you did? even better
[20:08:57] <brion>	heh i already mentioned in june that if it'd been done the squid change would be unnecessary by then
[20:08:58] <RoanKattouw>	Erik and Siebrand, at least
[20:09:54] <Nikerabbit>	okay, I'll update the team also
[20:14:34] <Nikerabbit>	thanks all
[20:18:46] <brion>	mmmm bits
[20:25:09] <aharoni>	RoanKattouw, is it supposed to be deployed?
[20:25:25] <RoanKattouw>	Ryan_Lane: ---^^ ?
[20:25:32] <RoanKattouw>	It might need a cache purge
[20:25:36] <Ryan_Lane>	whenever puppet runs
[20:25:39] <RoanKattouw>	Oh, right
[20:25:41] <RoanKattouw>	That could take hours
[20:25:45] <Ryan_Lane>	yep
[20:25:47] <RoanKattouw>	There aren't very many bits Apaches, are there?
[20:25:53] <Ryan_Lane>	nope
[20:25:55] <RoanKattouw>	Any chance we can run puppet there now-ish?
[20:26:00] <Ryan_Lane>	go for it
[20:26:10] *Ryan_Lane 	implies not it
[20:26:15] <RoanKattouw>	By "we" I kind of meant you
[20:26:16] <RoanKattouw>	Meh
[20:26:27] <Ryan_Lane>	you have root :)
[20:26:28] <RoanKattouw>	Excuse me while I submit a grad school application first :D
[20:26:36] <Ryan_Lane>	I have other projects too
[20:26:53] <Ryan_Lane>	and every single day something comes up that totally screws my schedule
[20:26:56] <RoanKattouw>	Right
[20:26:59] <RoanKattouw>	puppet -tv right ?
[20:27:02] <Ryan_Lane>	yep
[20:27:21] <RoanKattouw>	Dear Ganglia, go to hell
[20:27:24] <RoanKattouw>	There was an error collecting ganglia data (127.0.0.1:8654): fsockopen error: Connection refused
[20:27:59] <RoanKattouw>	                $bits_appservers = [ "srv191.pmtpa.wmnet", "srv192.pmtpa.wmnet", "srv248.pmtpa.wmnet", "srv249.pmtpa.wmnet", "mw60.pmtpa.wmnet", "mw61.pmtpa.wmnet" ]
[20:28:01] <RoanKattouw>	According to puppet
[20:28:18] <Ryan_Lane>	heh
[20:30:15] <RoanKattouw>	Dear mw60, screw you too
[20:31:55] <RoanKattouw>	Ryan_Lane: mw61: Could not parse options: ambiguous option: -tv
[20:35:36] <Krinkle-away>	wtf, Special:Search bugzilla
[20:35:52] <Krinkle>	https://bugzilla.wikimedia.org/buglist.cgi?title=Special%3ASearch&quicksearch=CORS&list_id=64771
[20:38:11] <RoanKattouw>	Ryan_Lane: Nm, figured it out
[20:39:03] <Krinkle>	trunk/tools/bugzilla doesn't contain the skin anymore.. Hm.. checked out from github on the server ?
[20:40:17] <RoanKattouw>	OK
[20:40:23] <RoanKattouw>	Nikerabbit: Please update /all/ font versions
[20:41:35] <Nikerabbit>	that is needed?
[20:42:41] <RoanKattouw>	Hmph
[20:42:44] <RoanKattouw>	And touch the font files too
[20:43:35] <Krinkle>	ah found it. It is in trunk/tools/bugzilla indeed (trunk/tools/bugzilla/bugzilla/bugzilla-4.0/template/en/custom)
[20:43:49] <Krinkle>	"ack" doesn't' search .tmpl by default
[20:44:00] <^demon>	That's why you have -a :)
[20:44:20] <Krinkle>	athx
[20:44:39] <RoanKattouw>	I think it's needed
[20:44:43] <RoanKattouw>	Caching is quite aggressive
[20:44:49] <^demon>	I have -a in my .ackrc
[20:45:21] <aharoni>	^demon, it kinda beats the point of ack :)
[20:45:49] <^demon>	aharoni: No, it still skips .svn and such.
[20:46:13] <Krinkle>	^demon: Thanks,
[20:46:15] <Krinkle>	!r 106088
[20:46:15] --elephant--	http://www.mediawiki.org/wiki/Special:Code/MediaWiki/106088
[20:46:19] <RoanKattouw>	Nikerabbit: Why don't we try it with one font file
[20:46:25] <RoanKattouw>	Touch it, sync it, increment its version
[20:46:53] <Nikerabbit>	RoanKattouw: the version number is supposed to sync with upstream... but I can just update the JS to append &1 to every font
[20:47:05] <Krinkle>	I have no idea what deployment method is for bugzilla, is it just svn up ?
[20:47:35] <^demon>	No, a root has to go and copy out files manually to kaulen.
[20:47:50] <Krinkle>	^_^
[20:48:02] <RoanKattouw>	Nikerabbit: Whatever, as long as you touch the font file and change its URL, in that order
[20:48:07] <RoanKattouw>	Hmm
[20:48:13] <RoanKattouw>	Actually, let me just touch the font ifle first
[20:48:17] <RoanKattouw>	And try some stuff from the cmd line
[20:48:24] <Nikerabbit>	okay
[20:50:07] <RoanKattouw>	Ok where the hell is my header
[20:50:37] <RoanKattouw>	Oooh wait
[20:51:11] <RoanKattouw>	Needed to do a graceful restart of Apache, of course
[20:52:43] <RoanKattouw>	YAY
[20:53:02] <RoanKattouw>	Nikerabbit: OK I have *confirmed* that if you touch a font file, and request it with a different URL, it works
[20:53:16] <RoanKattouw>	http://bits.wikimedia.org/w/extensions-1.18/WebFonts/fonts/Orya/utkal.ttf?version=0.14&b
[20:53:19] <RoanKattouw>	Serves a CORS header
[20:54:32] <Nikerabbit>	then lets hope that is good enough for Firefox
[20:54:50] <RoanKattouw>	Aye
[20:55:22] <Nikerabbit>	RoanKattouw: just to recap 1) touch all font-files & sync 2) change the url and sync?
[20:56:02] <RoanKattouw>	Yes
[20:56:19] <Nikerabbit>	doing that now then
[21:05:33] <Nikerabbit>	RoanKattouw: *hug*
[21:05:52] <RoanKattouw>	Is it working?
[21:05:54] <aharoni>	Nikerabbit, RoanKattouw, Krinkle, Ryan_Lane, brion - thanks a lot for the help.
[21:05:59] <aharoni>	it works.
[21:06:03] <Krinkle>	cool!
[21:06:05] <RoanKattouw>	Yay
[21:06:41] <Ryan_Lane>	RoanKattouw: yw
[21:11:15] <brion>	\o/
[21:30:50] <Krinkle>	robla: Can you invite my wmf address instead of my gmail (re: google calendar invite update)
[21:31:40] <robla>	Krinkle: sure...done
[21:32:53] <Krinkle>	thx
[22:16:35] <Reedy>	Bah, Dario has gone
[22:26:25] <RoanKattouw>	TimStarling: Could you take a look at https://gerrit.wikimedia.org/r/#change,1501 ? It's already been deployed but I'd like you to look at it anyway jus to be sure
[22:32:14] <TimStarling>	it's for bits?
[22:33:25] <RoanKattouw>	Yes, but it applies to all Apaches
[22:33:34] <RoanKattouw>	So it applies to foo.wikipedia.org but not upload.wm.o (cause that's nginx)
[22:33:50] <RoanKattouw>	It serves CORS headers for the file extensions listed a few lines up
[22:34:46] <TimStarling>	if it's deployed then why don't I see the header in responses? was apache not reloaded?
[22:34:50] <RoanKattouw>	It should probably be put in a bits-specific config at some point
[22:35:01] <RoanKattouw>	Not across all servers, no
[22:35:05] <RoanKattouw>	Also, responses are cached
[22:35:09] <RoanKattouw>	Varnish caches very aggressively
[22:35:19] <RoanKattouw>	I manually ran puppet and apache graceful on the bits app servers
[22:35:25] <TimStarling>	yeah I tried getting it straight from apaches.pmtpa.wmnet
[22:35:32] <RoanKattouw>	Oh, right
[22:35:34] <TimStarling>	is there some server where it's deployed?
[22:35:45] <RoanKattouw>	srv191, but that's a bits Apache
[22:35:48] <RoanKattouw>	Also, bits in general
[22:36:20] <RoanKattouw>	It's not deployed on any non-bits Apaches that I know of, but you can make that happen by running puppetd --test && apache2ctl graceful
[22:36:35] <RoanKattouw>	(I guess puppet should really graceful Apache itself if it changes the config file)
[22:38:37] <TimStarling>	looks fine to me
[22:39:06] <RoanKattouw>	Alright, great
[22:39:18] <RoanKattouw>	It's not about whether it's functional, really, we got it to work for WebFonts, but more like "is this a good idead"
[22:39:22] <RoanKattouw>	*idea
[22:39:50] <RoanKattouw>	Brion and I also want CORS headers for all files served from upload.wm.o but that's more of a yak shave
[22:40:36] <brion>	it's not much a yak shave, it's a matter of whether you can do certain things or not at all from JavaScript gadgets
[22:40:47] <brion>	without maintaining an external proxy just to get at files
[22:41:16] <RoanKattouw>	No I mean actually deploying CORS to upload is a yak shave
[22:41:28] <RoanKattouw>	Because you need to configure Squid to add the header to cached responses
[22:41:38] <RoanKattouw>	Otherwise it's gonna take months before CORS wprks reliably
[22:41:42] <brion>	or... just add it on the web server and wait
[22:41:48] <RoanKattouw>	Yeah could od
[22:41:49] <brion>	cause it's been many months already ;)
[22:41:56] <RoanKattouw>	What's three months :D
[22:41:58] <TimStarling>	it's probably OK as long as it's just static files
[22:42:21] <brion>	if we're gonna end up waiting another 6-10 months for the squid tweak, i'd say just do the apache tweak and be done with it. at least then it'll start working at some point :)
[22:42:37] <TimStarling>	you just have to be careful that you don't accidentally send the header on script output that happens to have .png in the path info
[22:42:56] <RoanKattouw>	TimStarling: On a wiki domain, that is a concern, yes
[22:43:01] <brion>	shouldn't be much script output on upload should there?
[22:43:15] <brion>	other than the redirects through thumb.php which return actual images or error pages
[22:43:17] <RoanKattouw>	I think upload should be safe from script output yeah
[22:43:26] <RoanKattouw>	Even then, it's a cookieless domain
[22:43:44] <brion>	no coooookies
[22:43:45] <TimStarling>	yes, it's ok, I mean for deploying CORS in general
[22:43:50] <RoanKattouw>	Right
[22:44:03] <RoanKattouw>	Well we are talking about deploying CORS to semi-trusted content on upload
[22:44:09] <RoanKattouw>	(user-uploaded, sanitized by MW)
[22:44:51] <RoanKattouw>	But that doesn't create an attack vector AFAICT unless the script doing the cross-domain request is really naively written
[22:44:57] <brion>	an example user is my svgedit extension which embeds an svg editor. to work as a gadget now it needs to use a proxy to load the SVG images from upload.wikimedia.org
[22:45:20] <brion>	with the CORS in, it can use an XMLHTTPRequest to load them
[22:45:25] <brion>	then they get saved back via API to the local domain
[22:49:32] <brion>	that reminds me i gotta whip together my SafeGadgets proposal sometime, for a limited-access iframe embedding interface for gadgets hosted on standalone domains >:D
[22:49:47] <brion>	let them run arbitrary javascript, but only have restricted access to the hosting wiki
[22:49:56] <TimStarling>	would there be a problem if it was untrusted content, rather than semi-trusted?
[22:50:12] <brion>	for the upload CORS case, untrusted shouldn't make a difference
[22:50:35] <brion>	basically it's a way for upload.wikimedia.org to say "i allow other sites to load my files, and do what they wish with them"
[22:50:41] <TimStarling>	as long as there's no authentication it should be fine, right?
[22:50:52] <brion>	you can already <img> or <script> or <link rel="stylesheet"> without the CORS, but you can't XHR them
[22:50:54] <brion>	yeah
[22:51:05] <brion>	and as a cookieless domain, that wouldn't be a worry in theory either
[22:51:10] <TimStarling>	it's basically equivalent to using a proxy
[22:51:15] <brion>	but by leaving auth off, it's a guarantee
[22:51:18] <brion>	yep
[22:51:55] <RoanKattouw>	Yes, auth would be off
[22:52:03] <RoanKattouw>	You need a separate header to allow auth
[22:52:33] <RoanKattouw>	Right, and you can already execute JS from another domain, so it's fnie
[22:52:41] <RoanKattouw>	Even with auth on, it's upload.wm.o, which is cookieless
[22:57:19] <TimStarling>	have you seen the updates on bug 31576?
[22:57:47] <RoanKattouw>	I haven't no, looking
[22:57:49] <TimStarling>	I'm haven't logged the magic word missing exception at all since I deployed r105964, I don't understand why not
[22:57:56] *RoanKattouw 	is 1k e-mails behind on bug mial
[22:58:28] <RoanKattouw>	When did you deploy it?
[22:58:38] <RoanKattouw>	There are some problems with the job system I've been meaning to talk about
[22:58:50] <RoanKattouw>	You know how, in theory, each job runner thread runs for 5 minutes, then terminates?
[22:58:53] <TimStarling>	around the time I made the bug comment, UTC 4:00
[22:59:06] <RoanKattouw>	In practice most of the 5 threads do that, and then 1 or 2 run for 60-100 minutes
[22:59:32] <TimStarling>	yeah, that sucks
[22:59:33] <RoanKattouw>	Because a single RefreshLinksJob2 job can require parsing up to 500 pages, which can easily take 4-5 hours if they're all Barack Obama-like pages
[23:00:31] <TimStarling>	we should go back to plain refreshlinks but use a job queue system that doesn't fall over like MySQL does
[23:00:33] <RoanKattouw>	This also means that deployed code updates can be delayed by up to 2h
[23:00:40] <RoanKattouw>	Right, you mean a job for every page?
[23:00:45] <RoanKattouw>	(Effectively set the batch size to 1)
[23:00:46] <TimStarling>	yeah
[23:00:53] <TimStarling>	refreshlinks2 is refreshlinks in batches
[23:01:05] <RoanKattouw>	Yeah, I figured the reason it wasn't done before is because of the write volume on the DB
[23:01:05] <TimStarling>	it was invented to reduce the MySQL write load
[23:01:31] <RoanKattouw>	So I've found out the hard way that this causes the job runners to take up to two hours to pick up your debugging hacks
[23:02:04] <TimStarling>	try /etc/init.d/mw-job-runner restart
[23:02:08] <RoanKattouw>	I know
[23:02:19] <RoanKattouw>	But that kills the existing job and loses its remainder
[23:03:19] <AaronSchulz>	yep
[23:03:45] <brion>	for statusnet i did a background job runner than pulled jobs from an ActiveMQ queue server. we didn't use the capability much but i threw in a feature where you could tell it to exit by sending a special queue type; giving you a chance to restart it after each individual job piece is done
[23:03:58] <RoanKattouw>	Anyway -- you're saying that exception isn't thrown at all
[23:04:00] <brion>	it was on a separate control channel so wouldn't get delayed by other jobs
[23:04:17] <RoanKattouw>	So either we're magically no longer producing bad links (this has happened before, no bad links for hours or even a day)
[23:04:24] <RoanKattouw>	or your exception is in the wrong place
[23:04:36] <RoanKattouw>	You can test this by reenabling the logging I've hacked in
[23:05:01] <RoanKattouw>	# Debugging hack for bug 31576. Enabling this triggers logging to /home/wikipedia/logs/{bug31576,bug31576bt,bug31576jq}.log
[23:05:02] <RoanKattouw>	$wgEnableBug31576Debugging = false;
[23:05:07] <AaronSchulz>	I'm going to change RefreshLinksJob2 to always convert to RefreshLinksJob (as already in non-cli mode)
[23:05:13] <RoanKattouw>	CommonSettings.php:2406
[23:05:23] <RoanKattouw>	AaronSchulz: That will probably lead to DB overload per Tim
[23:06:36] <RoanKattouw>	TimStarling: Given that you're throwing the exception in load(), I can see how the MagicWord cache might be corrupted somewhere along the way and bypass the exception
[23:07:14] <TimStarling>	yes, but you see the wfDebugLog() that I removed that's in the same place?
[23:07:25] <TimStarling>	we were getting loads of log lines from that for weeks
[23:07:29] <AaronSchulz>	RoanKattouw: really?
[23:07:47] <TimStarling>	maybe it stopped before I deployed my change
[23:07:53] <RoanKattouw>	Oh, hah
[23:08:09] <RoanKattouw>	Well, are we sure we're logging exceptions from maintenance scripts correctly?
[23:08:42] <TimStarling>	I checked it on fenari with eval.php
[23:08:51] <RoanKattouw>	ewww
[23:08:53] <RoanKattouw>	2011-12-13 23:08:19 mw26 enwiki: /wiki/Special:WhatLinksHere/Media:Blason_ville_fr_Burlats_(Tarn).svg   Exception from line 40 of /usr/local/apache/common-local/php-1.18/includes/Namespace.php: MWNamespace::getTalk does not make any sense for given namespace -2
[23:08:53] <TimStarling>	I requested an invalid magic word and it logged that exception
[23:08:58] <RoanKattouw>	2011-12-13 22:21:30 fenari enwiki: [no req]   Exception from line 281 of /home/wikipedia/common/php-1.18/includes/MagicWord.php: Error: invalid magic word 'asdkfasd'
[23:09:00] <RoanKattouw>	Ah yes
[23:09:06] <TimStarling>	yeah that second one was my test
[23:09:38] *RoanKattouw 	files a bug about the first one
[23:09:40] <TimStarling>	my deployment was at 01:12
[23:10:18] <TimStarling>	the last exception was 00:13, but there are plenty of hour-long gaps in the logs
[23:10:42] <RoanKattouw>	Yeah
[23:10:50] <RoanKattouw>	Usually it's one or a few job runners that go on a spree
[23:10:57] <RoanKattouw>	Then it's quiet for a while, then the next one starts
[23:12:28] <RoanKattouw>	Well thanks for looking into this anyway
[23:12:32] <TimStarling>	have you seen bug 32170? it really points to a LocalisationCache issue
[23:12:36] <TimStarling>	not MagicWord::load()
[23:12:42] <RoanKattouw>	I had this idea you might Just Know where to look
[23:12:56] <TimStarling>	I stared at all the relevant code for a while, I still don't have a good theory on the root cause
[23:13:07] <RoanKattouw>	Yeah, I know about that one too
[23:13:19] <RoanKattouw>	It's strange that it only affects job runners
[23:13:42] <TimStarling>	the first thing I checked was the username, but they're all running as apache
[23:13:54] <RoanKattouw>	I hadn't investigated that one (or the category bug, bug 31577) because I assumed they'd all be related
[23:13:56] <RoanKattouw>	Oh, hey
[23:13:58] <RoanKattouw>	apache
[23:14:04] <RoanKattouw>	Wait a damn second
[23:14:13] *RoanKattouw 	ssh'es into a job runner to try something
[23:14:59] <RoanKattouw>	Hmm
[23:15:21] <RoanKattouw>	The LC files are world-readable, and MW shouldn't ever need to write to them itself
[23:15:38] <RoanKattouw>	Ah, LU files
[23:15:42] <RoanKattouw>	Not even on recace
[23:15:47] <TimStarling>	I checked LU too
[23:15:58] <TimStarling>	LU isn't even working at the moment, it's just logging failures
[23:16:05] <RoanKattouw>	I know
[23:16:09] <RoanKattouw>	It's my project for tomorrow
[23:16:11] <TimStarling>	and the LU hook doesn't touch the magicWords key
[23:16:17] <RoanKattouw>	Right
[23:16:41] <RoanKattouw>	I just figured there might be some weird failure if the LU files (which are CacheDependency-s for LC) are unreadable
[23:17:24] <TimStarling>	the LC CDB files can't be wrong, if they were, every run of runJobs.php would fail, not just one per hour
[23:18:36] <TimStarling>	I stared at ForkController for a while, it has the potential to screw things up in interesting ways
[23:18:44] <RoanKattouw>	Oh, hmm
[23:18:51] <TimStarling>	and it's mostly only used for runJobs
[23:18:52] <RoanKattouw>	Yeah I just tried eval.php as Apache, no obvious failuers
[23:18:59] <TimStarling>	but I couldn't see any way it would cause what we're seeing
[23:21:16] <TimStarling>	it's all very mysterious
[23:21:49] <TimStarling>	I also looked at the CDB code inside PHP
[23:21:54] <AaronSchulz>	hehe
[23:22:03] <RoanKattouw>	lol
[23:22:12] <TimStarling>	if the CDB read fails then this will happen
[23:22:19] <TimStarling>	but there's no obvious reason why it would randomly fail
[23:24:58] <TimStarling>	I tried searching the syslogs, I couldn't find any relevant PHP errors from around the same time as the magic word missing errors
[23:27:07] <TimStarling>	there's no proper error checking in the CDB handler in PHP, so there's a few kinds of errors that could lead to this without being logged
[23:27:51] <AaronSchulz>	RoanKattouw: remember the empty value in wikiversions? ;)
[23:28:07] <RoanKattouw>	Oh yeah!
[23:28:12] <RoanKattouw>	That was a freak occurrence too
[23:29:40] <TimStarling>	in a CDB file?
[23:31:11] <TimStarling>	was the blank entry actually in the file, or did it just appear from the file on read?
[23:31:40] <RoanKattouw>	No
[23:31:42] <RoanKattouw>	It came out of nowhere
[23:31:45] <RoanKattouw>	And went away
[23:32:09] <AaronSchulz>	RoanKattouw: you didn't resync it?
[23:32:12] <RoanKattouw>	I saw some errors on fatalmonitor, about require_once(/usr/local/apache//index.php)
[23:32:19] <RoanKattouw>	Or something like that
[23:32:28] <RoanKattouw>	I did absolutely nothing and the error went away
[23:32:35] <RoanKattouw>	I only saw it after the fact, anyway
[23:32:42] <RoanKattouw>	By the time I saw it it'd already magically fixed itself
[23:32:49] <RoanKattouw>	There were ~4 instances, all on the same server
[23:32:59] <RoanKattouw>	I did sync the server for good measure, but the issue had subsided before that
[23:34:01] <TimStarling>	we could switch to the PHP version of CDB
[23:34:08] <TimStarling>	it has better error checking
[23:34:18] <TimStarling>			if ( $buf === false || strlen( $buf ) !== $length ) {
[23:34:19] <TimStarling>				throw new MWException( __METHOD__.': read from CDB file failed, file may be corrupted' );
[23:34:46] <RoanKattouw>	Is there a performance hit involved?
[23:34:54] <TimStarling>	yes
[23:35:00] <RoanKattouw>	I would expect the PHP version to be slower, but maybe not insanely so
[23:35:04] <TimStarling>	but I think we've got some apache CPU headroom
[23:35:09] <RoanKattouw>	Like, not as bad as the diff stuff
[23:35:12] <RoanKattouw>	Oh, heh, yeah :)
[23:37:20] <RoanKattouw>	Well we could try that