[00:00:31] <Krinkle>	Ryan_Lane: Just in case there's another Ryan (I didn't get your irc nickname during the conversation) - it was you I spoke with, right ?
[00:00:48] <Ryan_Lane>	I dunno, when? :)
[00:00:56] <Krinkle>	1 hour ago
[00:01:00] <Ryan_Lane>	where?
[00:01:04] <Krinkle>	Skype ?
[00:01:07] <Ryan_Lane>	nope :)
[00:01:10] <Krinkle>	OK. Thanks
[00:01:15] <Ryan_Lane>	there's also kaldari
[00:01:21] <Ryan_Lane>	and ryan faulkner
[00:02:07] <Krinkle>	neilk_: Which Ryan sat next to you after we spoke ?
[00:02:19] <neilk_>	that was Kaldari
[00:02:32] <neilk_>	he prefers to just be called Kaldari
[00:02:32] <Krinkle>	Oh, my bad.
[00:05:31] <Krinkle>	Alright, I'm gonna close up for today
[00:05:31] <Krinkle>	nn
[01:29:07] <robla>	hey folks, demon had something come up, and won't have time for a bit to look at this: https://bugzilla.wikimedia.org/show_bug.cgi?id=27287
[01:29:46] <robla>	it's a test2 problem with CentralAuth that might actually be a real 1.17 problem, but it could just be a problem with the test infrastructure
[01:30:09] <robla>	is there anyone with the access, time, and inclination to look at it?
[01:35:09] <TimStarling>	I will do it if I have time
[01:35:35] <TimStarling>	but first of all I want to make sure we have profiling set up in a way that will let us detect what the main problem is
[01:38:51] <robla>	TimStarling: yeah, keep going on that front.  Chad may get back before you're done
[01:40:40] <robla>	btw, just refreshed the review list: http://www.mediawiki.org/wiki/MediaWiki_roadmap/1.17/Revision_report
[01:40:58] <robla>	7 revs left:  3 unreviewed, and 4 fixmes
[02:33:31] <apergos>	wgLogAutocreatedAccounts should probably be set to true (default is false) in the 1.17 config files if we want autocreated accounts logged on test2
[02:34:39] <apergos>	I'm happy to add it to um I guess InitialiseSettings in a test2 stanza if we want it
[03:13:49] *^demon 	is back now :)
[03:14:34] <apergos>	see the backread (I didn't add it as I didn't want to muck with test2 when Tim was poking at it)
[03:22:18] <^demon>	We should set it for test2wiki, confirm that's the fix, and then set the default to true cluster-wide.
[03:22:44] <^demon>	Victor changed behavior in r64853.
[03:41:07] <robla>	excellent, thanks apergos!
[03:43:29] <apergos>	if that default makes sense
[03:43:36] <apergos>	also that is only for um the autocreated ones
[03:44:06] <apergos>	the other issue (accounts created from special:login) I don't know about
[03:44:34] <apergos>	(sorry I was away looking at a media store demo, or rather a demo of a testing infrastructure, it is coming along!)
[03:52:17] <apergos>	I added it in InitaliseSettings enabled for test2 only and will sync it
[03:52:46] <^demon>	Alrighty
[03:54:17] <apergos>	let's see how that does
[03:56:01] <apergos>	lookie there, two accounts created automatically
[03:56:01] <^demon>	Weird, I didn't login via cookie...
[03:56:14] <apergos>	(one is mine)
[03:56:34] <^demon>	Might be on my end.
[03:56:46] <^demon>	I've been having intermittent login issues for the last 2 weeks or so
[03:56:51] <apergos>	guess I'l try creating an account the old way now and see how that is
[03:57:50] <^demon>	Hmmm.
[03:57:56] <^demon>	I wonder if my bot account isn't SUL'd
[03:58:06] <apergos>	new user account showed up
[03:58:12] <apergos>	created the old way
[03:58:17] <^demon>	Ha, I never bothered merging into SUL.
[03:58:31] <apergos>	where's that bug report
[03:58:33] <Brian_S>	Yay, I saw -tech.
[03:58:40] <Brian_S>	It wasn't enabled?
[03:58:42] <Brian_S>	oO
[03:58:44] <apergos>	nope
[03:58:49] <Brian_S>	Weird.
[04:00:07] <apergos>	nice job. go merge yerself ^demon
[04:00:16] <apergos>	I mean that in the nicest way of course :-D
[04:00:30] <^demon>	haha, I have to unblock it first.
[04:00:34] <apergos>	woops
[04:00:38] <^demon>	I blocked my bot account on enwiki ages ago
[04:01:15] <apergos>	I had looked earlier and I remembered that on test when we enabled 1.17 I had created an account the manual awya and it worked (so it seemed unlikely to be the code causing the issue)
[04:01:30] <apergos>	what is left in the queue?
[04:01:51] <robla>	apergos: which queue?
[04:02:01] <apergos>	fixes. reviews. whatever
[04:02:12] <apergos>	stuff to do before 10 pm PST
[04:02:33] <robla>	well....I'm looking at this problem that Ryan_Lane|away reported: https://bugzilla.wikimedia.org/show_bug.cgi?id=27310
[04:02:50] <^demon>	Hmm, it fixes the logging-when-auto-created.
[04:02:52] <apergos>	hmph
[04:02:58] <^demon>	Still isn't logging on login.
[04:03:04] <robla>	it's on trunk and may not be on 1.17, but was just about to test it
[04:03:19] <apergos>	^demon: I created an account via special login and it showed up
[04:03:35] <apergos>	see recent changes, look at
[04:03:38] <^demon>	Created, or logged in to a SUL'd account?
[04:03:41] <apergos>	 TestAcctCreation1.17
[04:03:45] <apergos>	I did both:
[04:03:49] <^demon>	I tried logging in as ^demonBot2, but I didn't get a creation.
[04:03:53] <apergos>	first I logged in via cookie (that's ??????????????)
[04:03:56] <apergos>	then I logged out.
[04:04:09] <apergos>	next I created an account (that's TestAcctBlah), that worked
[04:04:14] <apergos>	what I did not try is
[04:04:24] <apergos>	create an account while already logged in
[04:05:11] <apergos>	in fact I'm still logged in as testacctblah
[04:05:18] *apergos 	logs out 
[04:06:05] <^demon>	What about logging in normally, not via cookie.
[04:06:10] <^demon>	That should also create a local log entry
[04:06:13] <^demon>	But doesn't seem to be
[04:06:30] <apergos>	hmm, let's see if that's how it works elsewhere
[04:07:11] <robla>	password reset seems fine at least
[04:07:31] <^demon>	Hmm, actually it doesn't seem to do that in 1.16wmf4 either.
[04:07:43] <^demon>	Tried logging in to frwiki, didn't log as an auto-creation
[04:07:49] <apergos>	I don't see it in production so *gong*
[04:07:56] <apergos>	yeah I just did it wikisource
[04:08:16] <^demon>	Hmm, wonder how long that's been a regression.
[04:08:24] <apergos>	heheheh
[04:08:37] <apergos>	long enough to ignore it for now
[04:08:49] <^demon>	It's certainly not a regression in 1.17
[04:08:50] <apergos>	you can modify the one bug (I did not close it)
[04:09:01] <apergos>	or add a new one but let's save til later
[04:10:14] <robla>	I think I encountered a great bug in my script to update the rev list....looks like we're getting few enough revisions to review that I've got an uninitialized variable  :)
[04:10:24] <apergos>	nice :-D
[04:10:27] <apergos>	what a good bug to find!
[04:10:38] <robla>	I thought so
[04:10:46] <robla>	I don't mind finding that one about now
[04:10:48] <^demon>	apergos: I commented.
[04:11:01] <apergos>	who knew two years ago when I was watching brion sweat through deployment that someday...
[04:11:01] <^demon>	So, we should go ahead and set that to true for all wikis.
[04:11:17] <^demon>	Or change it in DefaultSettings to true. He changed behavior in that commit, and it's unintuitive.
[04:11:20] <apergos>	I'd be suffering through the same crap :-D
[04:11:33] <apergos>	yer call ^demon
[04:11:58] <apergos>	I kinda wonder if we didn't have people complaining about the autocreate stuff
[04:12:24] <^demon>	I can't imagine it's expected behavior. But obviously no one was missing it...
[04:12:35] <apergos>	(too many entries as people were bouncing around all over the place)
[04:12:55] <apergos>	hmm that's true
[04:15:22] <TimStarling>	I think he changed it in response to some sort of steward request
[04:15:40] <apergos>	ok
[04:15:46] <TimStarling>	you should ask him why, send him an email
[04:15:52] <robla>	bummer, not quite as rosy as I'd thought:  http://www.mediawiki.org/wiki/MediaWiki_roadmap/1.17/Revision_report  6 revs left
[04:16:37] <robla>	we need an "okfornow" tag  ;-)
[04:16:44] <apergos>	heh
[04:18:43] <^demon>	r81896 is only fixme because of a db2 complaint
[04:18:47] <^demon>	db2 doesn't matter :)
[04:20:26] <robla>	^demon: is that the only problem with that checkin?
[04:21:04] <^demon>	Yeah the rest of it was wfProfile fixes that Roan already merged.
[04:21:05] <robla>	I guess everything else is a profile in or out call
[04:22:09] <^demon>	Ok, e-mailed Victor.
[04:22:12] <robla>	ok...we won't worry about that.  the obsessive compulsive side of me wants to create a "1.17wmf1ok" tag, but I'll resist the urge
[04:22:12] <apergos>	cool
[04:23:02] <^demon>	robla: It doesn't really matter for release either.
[04:23:07] <robla>	this one looks like the only one not ok for deploy: http://www.mediawiki.org/wiki/Special:Code/MediaWiki/70498
[04:23:18] <robla>	or at least, no one has declared it safe yet
[04:23:57] <^demon>	r70095 *should* be ok. We're not using upload-by-url (yet). The generally applicable bits look fine.
[04:24:52] <mdale>	is there a SF location for deployment ?
[04:25:04] <apergos>	location?
[04:25:06] <robla>	mdale: not really
[04:25:10] <apergos>	I'm on my bed in the wikisuites :-P
[04:25:16] <mdale>	( Danese ) is asking
[04:25:29] <apergos>	paaarty in thw wikisuites!
[04:25:36] <apergos>	(russ and steven would kill me)
[04:26:16] <robla>	This IRC channel was kinda it for the last deploy
[04:27:04] <robla>	I totally forgot to dial into the conf line during the deploy
[04:27:09] <robla>	(last time)
[04:27:20] <^demon>	Did anyone?
[04:27:29] <robla>	I'm not sure...no one complained
[04:27:43] <apergos>	I think we were all typing
[04:27:47] <apergos>	worked well enough
[04:28:24] <robla>	yeah.  there are times when having a headset and being able to talk and type is helpful, but IRC worked well enough
[04:29:40] <robla>	btw, what about r70498?
[04:29:48] <robla>	!r 70498
[04:29:48] --elephant--	http://www.mediawiki.org/wiki/Special:Code/MediaWiki/70498
[04:30:12] <^demon>	I hadn't even looked at it yet.
[04:31:12] <robla>	that's the only one that no one seems to have mentally attached the "goodenoughfornow" tag to
[04:31:25] <apergos>	need a special page with wikitext headings
[04:31:33] <apergos>	to look at on test2
[04:32:26] <^demon>	Use Special:Version
[04:32:42] <apergos>	I did but I don't see edit tags there
[04:33:01] <apergos>	 edit section links are added to wikitext headings on special pages   it says but I don't see that
[04:33:12] <^demon>	Oh hrm.
[04:33:40] <apergos>	of course if it's actually fixed
[04:33:43] <robla>	ah, I see: http://www.mediawiki.org/wiki/Special:Code/MediaWiki/79185
[04:33:53] <robla>	it was fixed there, and merged into 1.17
[04:33:57] <apergos>	in that rev like in a prev comment...
[04:34:07] <apergos>	well there we go then
[04:34:09] <robla>	so this one looks like it's resolved, actually
[04:34:12] <^demon>	I think version is the only one.
[04:36:15] <apergos>	so someone gonna close that? :-P
[04:37:56] <robla>	well, I actually misfollowed the version history.  the comment ialex made may still not be fixed, but I guess we're in a "works for me" situation, right?
[04:38:16] <apergos>	http://test2.wikipedia.org/wiki/Special:Version
[04:38:27] <apergos>	yup
[04:39:35] <robla>	ok...I'll put it back, then
[04:43:03] <robla>	alright...going afk for a little bit
[04:43:13] <apergos>	okey dokey
[04:49:14] <^demon>	This time I'm gonna watch James Bond movies during deployment :)
[04:50:58] <apergos>	ok, that's just plain weird :-D
[04:51:06] <apergos>	oohhh... time to put on the news
[04:51:30] <^demon>	I started with The Spy Who Loved Me
[04:51:46] <apergos>	it's been a long time since I've seen them
[04:51:53] <apergos>	I used to watch those bond marathons on tnt
[04:52:02] <apergos>	waaaay back in the day
[04:52:23] <^demon>	I think Spike has the rights to Bond marathons these days
[04:52:37] <^demon>	I've got them all
[04:52:51] <apergos>	dang
[04:52:53] <apergos>	that's hardcore
[04:53:49] <apergos>	sports news, bah
[04:54:01] <apergos>	but it will be the top of the hours in a couple minutes and then we'll get more ...
[04:54:01] <^demon>	Oh, it's been my career goal since I was like 5 to be a secret agent
[04:54:11] <^demon>	Travel around the world drinking martinis and blowing stuff up.
[04:54:11] <apergos>	rumors about egypt!
[04:54:19] <apergos>	hahaha
[04:54:45] <apergos>	I think you have to be involved in organized crime for that
[04:56:44] <apergos>	woah really, in an hour? bah, there goes my afternoon nap...
[04:57:09] <^demon>	An hr?
[04:58:01] <^demon>	I think I'm going to run to the store real quick and grab some snacky sustenance.
[04:59:26] <TimStarling>	http://noc.wikimedia.org/cgi-bin/pcache-hit-rate.py
[04:59:32] <TimStarling>	finished in plenty of time
[05:00:36] <^demon>	I'll be back in like 20 or so.
[05:07:10] <apergos>	(making food)
[05:07:43] <apergos>	ok!
[05:20:11] <^demon>	(back)
[05:20:36] <robla>	ditto
[05:27:13] <apergos>	food in about 10 minutes I think...
[05:45:14] <TimStarling>	anyone mind if I take a more active role this time around?
[05:45:47] <waihorace>	Is the first window of 1.17 upgrade start?
[05:46:39] <^demon>	TimStarling: I take the silence to mean no objections.
[05:47:03] <robla>	TimStarling: what would the difference be?
[05:47:32] <TimStarling>	I'll be the one moving wikis and syncing things
[05:47:53] <TimStarling>	Roan is not here anyway
[05:48:11] <robla>	ah, I see.  sounds reasonable
[05:49:09] <TimStarling>	I'll move usability.wikimedia.org when the appointed time comes up in 7 minutes
[05:49:59] <TimStarling>	that's basically an alpha test since nobody uses it
[05:50:11] <TimStarling>	hi roan, just saying usability first in 5 minutes
[05:51:03] <TimStarling>	then after that, the next batch can be strategy, simple x2 and meta
[05:51:15] <RoanKattouw>	OK
[05:51:18] <TimStarling>	that's hopefully a beta test since people do use meta
[05:51:18] <^demon>	Do any of the wikis in round 1 use FlaggedRevs?
[05:51:23] <RoanKattouw>	Let me just make some breakfast real quick
[05:51:55] <RoanKattouw>	Yes, we should be a little more comfortable with ourselves by the time we do meta
[05:51:57] <TimStarling>	then the third set will be the rest of them, that give us our load test, of sorts
[05:52:00] <RoanKattouw>	Because central stuff lives there
[05:53:39] <^demon>	Ah, enwikinews and enwikibooks use it. My followup question is moot then.
[05:53:40] <waihorace>	Is it only the 11 selected wikis will upgrade today? Countdown, it just have 2 minutes left...
[05:54:02] <^demon>	Time is an illusion.
[05:54:09] <apergos>	I see stats/1.17-phase1 and 1.17-phase1
[05:54:13] <apergos>	is there a difference?
[05:54:29] <apergos>	or to be more precise
[05:54:41] <apergos>	what is the difference, since I see ther eis one?
[05:54:58] <apergos>	waihorace: yes, it's only those ones, and only for a while.
[05:55:30] <waihorace>	It is 6 UTC now...
[05:55:51] <TimStarling>	can we maybe have a little bit less waihorace this time around?
[05:56:02] <RoanKattouw>	And a little more breakfast for Roan :)
[05:56:10] <TimStarling>	anyway usability is on 1.17 now
[05:56:14] <RoanKattouw>	Oh?
[05:56:25] <RoanKattouw>	Oh you logged it in -ops
[05:57:54] <TimStarling>	still the PHP diff engine?
[05:58:17] <RoanKattouw>	?
[05:58:54] <RoanKattouw>	Hmm, abg of >500 calls to wfMsgReal per req, is that normal?
[05:59:34] <TimStarling>	in which db?
[05:59:35] <RoanKattouw>	It's elevated
[05:59:38] <RoanKattouw>	1.17-phase1
[05:59:53] <RoanKattouw>	Wait, five HUNDRED
[05:59:54] <RoanKattouw>	On enwiki the average is 40
[06:00:01] <mark>	heh
[06:00:03] <TimStarling>	those are the pre-1.17 wikis
[06:00:12] <TimStarling>	they are running on 1.16
[06:00:14] <RoanKattouw>	Oh
[06:00:29] <TimStarling>	let's hope 1.17 has an efficient wfMsgReal()
[06:01:05] <RoanKattouw>	Ah yes
[06:01:24] <RoanKattouw>	OK so what we're seeing in 1.17-phase1 is a mix of 1.17 (usabilitywiki) and 1.16 (others) currently
[06:01:36] <TimStarling>	yes, but I haven't cleared the profiling data yet
[06:01:51] <TimStarling>	so it's mostly from all those wikis for the last 6 hours or so
[06:02:02] <TimStarling>	I took a snapshot in /ng, I'll clear it now
[06:02:16] <RoanKattouw>	OK
[06:02:35] <TimStarling>	also you may like http://noc.wikimedia.org/cgi-bin/pcache-hit-rate.py
[06:02:58] <TimStarling>	that's the same set of wikis, whether or not they are running 1.17
[06:03:24] <RoanKattouw>	hit-rate.py is nice
[06:03:29] <RoanKattouw>	Which wikis does it refer to?
[06:03:44] <TimStarling>	1.17-phase1.dblist
[06:03:47] <RoanKattouw>	The hit rate is shows seems atrociously low
[06:04:03] <TimStarling>	it agrees with the data in stats/1.17-phase1 though
[06:04:04] <RoanKattouw>	enwiki had a 78% hit rate under normal conditions earlier this week, right?
[06:04:58] <TimStarling>	74%
[06:05:27] <RoanKattouw>	How are you counting pcache_not_possible? Not at all?
[06:05:33] <TimStarling>	smaller wikis get a smaller portion of the cache space, because they are further down the frequency/rank graph
[06:05:39] <waihorace>	http://usability.wikimedia.org/wiki/Special:CommunityVoice and I got HTTP 500 using IE8
[06:05:39] <TimStarling>	not at all
[06:05:44] *RoanKattouw 	shuts up and reads the source instead
[06:05:45] <RoanKattouw>	Right
[06:05:51] <RoanKattouw>	     12 Fatal error:  Cannot use object of type stdClass as array in /usr/local/apache/common-local/php-1.17/extensions/CommunityVoice/Modules/Ratings.php on line 28
[06:06:05] <RoanKattouw>	I know exactly what happened there
[06:07:14] <^demon>	I'm already on it
[06:07:25] <^demon>	Should be $row->vot_category;
[06:07:27] <RoanKattouw>	I am too
[06:07:33] <RoanKattouw>	Reverting r75677 on CommunityVoice
[06:07:48] <RoanKattouw>	But ^demon  will be faster, he wins
[06:08:33] <RoanKattouw>	So, there was a bug reported against 1.17wmf1 about logging account creations
[06:08:55] <^demon>	Part of it was seemingly on purpose because of a change Victor made.
[06:09:07] <robla>	!r 75677
[06:09:07] --elephant--	http://www.mediawiki.org/wiki/Special:Code/MediaWiki/75677
[06:09:07] <apergos>	already dealt with but only enabled for test2, we'll see about the rest later
[06:09:13] <RoanKattouw>	apergos: ?
[06:09:18] <apergos>	^^
[06:10:09] <apergos>	https://bugzilla.wikimedia.org/show_bug.cgi?id=27287  for the comments if you want them
[06:10:21] <RoanKattouw>	Right
[06:10:51] <apergos>	doublewiki is quite the memory hog I see (that's irrelevant, just watching the messages scroll by)
[06:11:04] <RoanKattouw>	It is, yes
[06:11:11] <RoanKattouw>	apergos: Where did you set that var exactly?
[06:11:32] <apergos>	initalisesettings in a stanza at the end. default left as false
[06:12:09] <^demon>	RoanKattouw: Fix is in trunk. Review + merge plz.
[06:12:28] <^demon>	apergos: It's a memory hog in 1.16 as well. I'm not sweating it right now.
[06:12:55] <apergos>	oh me neither, just being amused
[06:13:14] <TimStarling>	I pushed out a fix as a live patch
[06:13:51] <apergos>	PHP Fatal error:  Allowed memory size of 83886080 bytes exhausted (tried to allocate 51927 bytes) in /usr/local/apache/common-local/wmf-deployment/languages/LanguageConverter.php on line 390
[06:13:52] <apergos>	hmm
[06:14:14] <TimStarling>	we're mostly interested in errors in 1.17 at this stage
[06:14:39] <apergos>	couldn't tell if that was 1.16 or 1.17
[06:14:46] <RoanKattouw>	apergos: wmf-deployment in the path
[06:14:53] <apergos>	eh duh
[06:14:54] <apergos>	:-D
[06:15:06] <RoanKattouw>	Oh right, I forgot
[06:15:12] <RoanKattouw>	Running svn up I get lots of changes pulled in
[06:15:20] <RoanKattouw>	Mostly the profiling stuff from yesterday
[06:15:23] <TimStarling>	I made a script called sync-file-17, it's like sync-file but for 1.17
[06:16:33] <RoanKattouw>	OK
[06:16:48] <TimStarling>	PHP Warning:  filemtime() [<a href='function.filemtime'>function.filemtime</a>]: stat failed for /usr/local/apache/common-local/wmf-deployment/extensions/Vector/modules/./images/open.png in /usr/local/apache/common-local/php-1.17/includes/resourceloader/ResourceLoaderFileModule.php on line 361
[06:16:54] <TimStarling>	note mixed paths
[06:17:17] <RoanKattouw>	^
[06:18:24] <RoanKattouw>	<whym>	I got "<vector-simplesearch-search" in the search box when I use localization.
[06:18:39] <RoanKattouw>	TimStarling: Can I run s-c-a ?
[06:18:48] <TimStarling>	I'll run it as root
[06:19:09] <RoanKattouw>	OK
[06:20:36] <robla>	https://bugzilla.wikimedia.org/show_bug.cgi?id=27318
[06:21:35] <robla>	getting reports of that bug ^ on #wikimedia-tech as well.  I haven't seen it myself, though
[06:21:40] <RoanKattouw>	I have
[06:21:47] <RoanKattouw>	Use ?uselang=fr on usabilitywiki
[06:22:14] <robla>	is this possibly a localization cache issue?
[06:22:25] <RoanKattouw>	Blegh
[06:22:31] <RoanKattouw>	 /* exception 'DBQueryError' with message 'A database error has occurred.  Did you forget to run maintenance/update.php after upgrading?  See: http://www.mediawiki.org/wiki/Manual:Upgrading#Run_the_update_script Query: DELETE FROM `msg_resource` Function: MessageBlobStore::clear Error: 1213 Deadlock found when trying to get lock; try restarting transaction (10.0.6.44)
[06:22:57] <RoanKattouw>	And of course RL happily sets a Cache-Control header on that
[06:23:09] <RoanKattouw>	I wanted to fix cache settings for errors at some point but forgot
[06:23:54] <RoanKattouw>	It's in MessageBlobStore::clear() which is called from LocalisationCache::recache()
[06:24:15] <RoanKattouw>	I'm guessing DELETE FROM msg_resource; is generally a deadlock-prone query though
[06:26:18] <Shirley>	Roan's online right now?
[06:26:35] <RoanKattouw>	Well yeah
[06:26:47] <Shirley>	I don't usually expect you for a few more hours.
[06:26:59] <robla>	Shirley: we've started a deploy
[06:27:01] <apergos>	I don't see that with el btw
[06:28:02] <RoanKattouw>	I've written a patch for the caching issue, now testing it
[06:28:15] <TimStarling>	the module_deps table seems ok now, maybe it was just a transition thing?
[06:29:00] <RoanKattouw>	Could've been
[06:29:10] <RoanKattouw>	My local patch for not caching RL errors works, committing
[06:29:52] <RoanKattouw>	Hmm, I'm thinking
[06:30:21] <TimStarling>			// TODO: Give this some more thought
[06:30:22] <RoanKattouw>	This basically does if ($warnings !== '' || $exceptions !== '' ) { don't cache } which means any file-doesn't-exist warning has a quite large impact
[06:30:30] <RoanKattouw>	TimStarling: Yeah, I meant to ask you about it
[06:31:18] <TimStarling>	recache() is called for each language separately
[06:31:32] <TimStarling>	was there some reason for clearing all languages?
[06:31:58] <RoanKattouw>	Just me not realizing
[06:32:10] <robla>	TimStarling: nadeesha_calcey reported the bug on test2 a bit before we started here, so it may not be a transition thing
[06:32:17] <RoanKattouw>	However, the deadlock prone-ness remains
[06:32:36] <TimStarling>	put it in a transaction
[06:32:42] <Shirley>	robla: BTW, I ended up filing a bug for rewriting Wikimedia's sync scripts.
[06:32:44] <apergos>	woops
[06:32:47] <Shirley>	'cause I couldn't find one.
[06:32:56] <RoanKattouw>	TimStarling: OK, I'll wrap it in begin(), end() and add a clause for the language
[06:33:03] <TimStarling>	begin() and commit()
[06:33:10] <RoanKattouw>	TimStarling: What are your thoughts on whether nocache any single PHP warning is a good idea
[06:33:16] <RoanKattouw>	Yeah, commit(), of course. Wake up, Roan
[06:33:46] <TimStarling>	PHP warnings shouldn't appear in the output
[06:34:10] <TimStarling>	so they shouldn't need any special cache headers
[06:34:49] <RoanKattouw>	They currently appear in /* comments */
[06:34:49] <apergos>	I wonder what that pile of segfaults was from
[06:34:52] <apergos>	gone now anyways
[06:34:53] <RoanKattouw>	But I can remove them I guess
[06:34:59] <RoanKattouw>	Exceptions I would like to stay
[06:35:13] <RoanKattouw>	Oh wait
[06:35:17] <RoanKattouw>	Warnings appear in debug mode only anyway
[06:35:50] <TimStarling>	maybe don't show warnings if display_errors is off
[06:35:51] <RoanKattouw>	TimStarling: Also, I'll need an index on mr_lang. Can you create one across all DBs?
[06:36:05] <apergos>	and now a batch of these:
[06:36:08] <apergos>	Feb 11 06:40:38 10.0.2.202 apache2[13387]: [error] [client 208.80.152.85] request failed: error reading the headers
[06:36:19] <TimStarling>	that means RL will be in keeping with site security policy
[06:36:32] <RoanKattouw>	Right
[06:36:38] <TimStarling>	do you have an ALTER TABLE command?
[06:38:20] <RoanKattouw>	CREATE INDEX msg_resource_lang ON msg_resource (mr_lang);
[06:38:28] <TimStarling>	it's a bit more complicated than you think to properly clear msg_resource on recache
[06:38:35] <Ryan_Lane>	in case rob la is still around, my bug report pretty much only affects the LDAP extension, though it's a general bug in mediawiki
[06:38:44] <TimStarling>	because of fallbacks
[06:38:44] <RoanKattouw>	TimStarling: How so?
[06:38:49] <RoanKattouw>	Ugh
[06:38:56] <robla>	Ryan_Lane: yup...got it, thanks!
[06:38:56] <RoanKattouw>	Well, actually, no
[06:39:12] <RoanKattouw>	Or yes
[06:39:22] <RoanKattouw>	Drat
[06:39:46] <TimStarling>	also, recache() is called when there is no local cache, and you are clearing a shared cache
[06:40:05] <TimStarling>	so every time a message file changes, you'll be clearing it 200 times
[06:40:10] <TimStarling>	once for each server
[06:40:14] <RoanKattouw>	Right
[06:40:17] <RoanKattouw>	Hence the deadlocks
[06:40:44] <TimStarling>	could that have caused the performance problems we saw last time?
[06:40:53] *RoanKattouw 	is wondering whether we should've gone with CDB for msg_resource after all
[06:41:54] <RoanKattouw>	I'll at least wrap this stuff in a transaction
[06:42:21] <TimStarling>	I think we will need more than just a transaction before we continue deployment
[06:42:24] <apergos>	how often are message files changing?
[06:42:44] <RoanKattouw>	apergos: Every time we deploy code xD
[06:42:48] <RoanKattouw>	As in every time we upgrade the site
[06:43:03] <apergos>	mmm hmmm
[06:43:05] <RoanKattouw>	The cache invalidations should stop at some point though
[06:43:35] <TimStarling>	there are 355 messages files, and 146 apaches
[06:43:46] <TimStarling>	give or take
[06:43:51] <TimStarling>	so 51830 cache invalidations
[06:44:03] <RoanKattouw>	Right
[06:44:10] <RoanKattouw>	Yeah the transaction is a very temporary patch
[06:45:34] <RoanKattouw>	TimStarling: Alright, shall I write a quick CDB replacement for msg_resource?
[06:46:06] <TimStarling>	there's still the fallback issue
[06:46:45] <RoanKattouw>	Yes, I'll have to do something similar to LCStore
[06:46:50] <RoanKattouw>	CDB for WMF, DB for mortals
[06:46:58] <TimStarling>	if RL did its own cache invalidations, and it did them when a message was requested instead of hooking into LocalisationCache, it could deal with fallbacks elegantly
[06:47:22] <RoanKattouw>	That'd require RL knowing when to invalidate
[06:47:49] <RoanKattouw>	And knowing that it has to beforehand so the module's mtime can be updated to reflect the message chance
[06:47:52] <RoanKattouw>	*changes
[06:48:42] <RoanKattouw>	But it seems that code is kind of broken in trunk anyway
[06:48:52] <TimStarling>	LocalisationCache doesn't know when to invalidate, it just does it once per request at the first wfMsg() call
[06:49:20] <TimStarling>	once per language, I should say
[06:50:08] <RoanKattouw>	Yes, because it has CacheDependency and all that jazz
[06:50:09] <TimStarling>	RL could do its checks in the startup module, that's cached pretty aggressively isn't it?
[06:50:16] <TimStarling>	hmm
[06:50:22] <RoanKattouw>	The startup module is cached for 300 seconds
[06:50:33] <TimStarling>	but it's not enough, right?
[06:50:42] <TimStarling>	you need the mtimes of the modules that are included in the HTML
[06:50:52] <TimStarling>	and for that you would need to check on every request
[06:51:05] <RoanKattouw>	We accept that invalidations will take 5 minutes to propagate
[06:51:30] <RoanKattouw>	So if a module's mtime changes, the client might not find out about that for 5 mins. Once it's got the new mtime, it'll be set
[06:51:51] <RoanKattouw>	What I would like to have here, though, is for l10ncache to know when a certain language was last cached
[06:52:24] <RoanKattouw>	Or, better, the invalidation timestamp it comes up with using CacheDependency
[06:53:27] <TimStarling>	the text of the messages is included with the JS source of the module, right?
[06:53:37] <TimStarling>	so each module request has a language parameter?
[06:54:02] <RoanKattouw>	Yes
[06:54:23] <TimStarling>	can the mtime be different for different languages?
[06:54:41] <RoanKattouw>	Yes
[06:54:52] <RoanKattouw>	See ResourceLoaderModule::getMsgBlobMtime( $lang )
[06:54:59] <RoanKattouw>	(Its logic is subtly broken, I just noticed)
[06:55:16] <TimStarling>	so when you are writing a <script> tag to the HTML, you can check the timestamp of the Messages*.php file for the language in question
[06:55:19] <TimStarling>	and any fallbacks
[06:55:36] <TimStarling>	so maybe 3 stats at the most
[06:55:38] <RoanKattouw>	We don't write <script> tags with timestamps to the HTML, ever
[06:55:52] <RoanKattouw>	All timestamps are done through the startup module
[06:56:05] <RoanKattouw>	(which of course is also passed a lang code)
[06:56:28] <TimStarling>	right, I was right 10 minutes ago, so you can just check in the startup module
[06:56:55] <RoanKattouw>	Yes, I'd have to update the timestamps for all modules AND clear msg_resource somehow, right?
[06:57:12] <TimStarling>	yes
[06:57:35] <TimStarling>	I don't see the need to integrate with LocalisationCache
[06:57:59] <RoanKattouw>	Hmm I guess calling MessageBlobStore::clear() and fixing getMsgBlobMtime so it returns NOW when a module has messages but no msg_resource row should do it
[06:58:16] <RoanKattouw>	And parameterize clear() by language too
[06:58:23] <TimStarling>	hmm
[06:58:46] <TimStarling>	I suppose if LocalisationCache gave you a timestamp for a given language, that would speed things up slightly, and make the code a bit simpler
[06:58:51] <RoanKattouw>	Yes
[06:59:07] <RoanKattouw>	That would require modifying CacheDependency a little bit
[06:59:17] <RoanKattouw>	Because, don't forget, there are extensions msg files too
[06:59:30] <TimStarling>	add CacheDependency::getLastModified()
[06:59:58] <RoanKattouw>	How will that work for GlobalDependency and ConstantDependency?
[07:00:06] <RoanKattouw>	I guess they should just return NOW or the epoch
[07:00:23] <TimStarling>	or 01-01-1970
[07:00:32] <TimStarling>	if they returned NOW they would break everything
[07:00:37] <TimStarling>	it has to be a long time ago
[07:00:49] <RoanKattouw>	I meant, return NOW or the epoch depending on whether their global/constant has changed or not
[07:00:57] <RoanKattouw>	Where the epoch is 1-1-70
[07:00:59] <TimStarling>	right
[07:01:12] <TimStarling>	yes that would be fine
[07:01:16] <RoanKattouw>	Hmm
[07:01:26] <RoanKattouw>	I guess the reason this fixes it is because the startup module is cached
[07:01:33] <Platonides>	will having been serving 2038 dates hide those fixes?
[07:02:03] <RoanKattouw>	Yeah if something is returning dates in the future, that'll break many things
[07:02:21] <RoanKattouw>	There's the applied assumption that wfTimestampNow() is the highest possible timestamp in some places, I guess
[07:02:34] <RoanKattouw>	However it will only 'break' as in 'not invalidate cache'
[07:02:46] <RoanKattouw>	Not as in 'melt the servers with 5,000 recaches'
[07:02:57] <TimStarling>	so are you going to do the CacheDependency change and also use CDB?
[07:03:13] <RoanKattouw>	The CacheDependency change is easiest
[07:03:26] <RoanKattouw>	Using CDB obviates it, doesn't it?
[07:03:58] <RoanKattouw>	Cause if MessageBlobStore uses CDB, it'll be a local cache just like l10ncache and integrating them won't be an issue
[07:05:34] <TimStarling>	I'm not sure how CDB caching would work exactly
[07:06:22] <TimStarling>	what happens if every server has a different mtime on its messages files?
[07:06:22] <RoanKattouw>	The same way as l10ncache, right?
[07:06:31] <RoanKattouw>	Hmm right
[07:06:38] <RoanKattouw>	That'd screw things up
[07:06:41] <TimStarling>	which is not unrealistic
[07:06:52] <TimStarling>	if you run scap as non-root there will be some skew
[07:07:10] <RoanKattouw>	Well, we still have a centralized cache for each language's startup module in Varnish
[07:07:29] <RoanKattouw>	However, on each recache stuff will drift, causing unnecessary Varnish cache fragmentation and revalidation
[07:07:30] <TimStarling>	so it'll just pick a random server and use that
[07:07:54] <RoanKattouw>	Because it's very likely the next recache will use a different server
[07:08:10] <TimStarling>	this seems complicated for a fix midway through a deployment
[07:08:41] <TimStarling>	here's an idea: remove MBS::clear() from 1.17wmf1, and run "TRUNCATE TABLE msg_resource" from scap
[07:09:20] <^demon>	I got a reply from Victor w.r.t. logging of locally created CA accounts.
[07:09:26] <RoanKattouw>	That's a much better idea, Tim
[07:09:37] <^demon>	(we can discuss that after the ResourceLoader fun)
[07:09:48] <RoanKattouw>	That means I don't have to mess around with loadDependencyValues() and __sleep() black magic at 8am
[07:10:44] <mark>	we certainly don't get to deal with __sleep() this 8 am ;-(
[07:11:10] <RoanKattouw>	TimStarling: What about Message*.php syncs, though? I guess this TRUNCATE should have its own shell command
[07:12:34] <TimStarling>	I'll write a script that checks the modification times and does the TRUNCATE
[07:13:15] <RoanKattouw>	Checks the mtimes of what against what?
[07:13:25] <RoanKattouw>	http://www.mediawiki.org/wiki/Special:Code/MediaWiki/81933
[07:13:56] <TimStarling>	it would check to see if any files have changed since last time it was run
[07:14:24] <RoanKattouw>	Right
[07:14:40] <RoanKattouw>	So use a special file in /home whose timestamp you set to max( all timestamps ) ?
[07:14:46] <mark>	when 1.17 was deployed last week, Wikimedia used 4 kW more power
[07:15:09] <RoanKattouw>	1.17 is killing the environment
[07:15:12] <mark>	I say, mediawiki 1.17 is...
[07:15:14] <mark>	yeah that.
[07:15:18] <apergos>	heh
[07:15:36] *^demon 	files a bug to make MediaWiki more Green(tm)
[07:16:27] <TimStarling>	RoanKattouw: yes
[07:16:43] <RoanKattouw>	lol, communication fail re syncs
[07:18:09] <TimStarling>	I gave up waiting for you to do it
[07:18:26] <RoanKattouw>	Same
[07:18:33] <RoanKattouw>	:)
[07:19:28] <robla>	so what's next?
[07:19:41] <RoanKattouw>	Alright, visiting usabilitywiki in weird languages works for me now
[07:19:48] <RoanKattouw>	Waiting for Tim's TRUNCATE script I guess
[07:19:52] <TimStarling>	I write script, other people fix other things
[07:20:04] <TimStarling>	is there anything else that needs fixing?
[07:20:19] <^demon>	Well we can discuss Victor's reply to my question about removing logging of locally created accounts.
[07:20:26] <robla>	ah, right
[07:20:27] <apergos>	what'd he say?
[07:20:50] <^demon>	""It was a response to https://bugzilla.wikimedia.org/show_bug.cgi?id=23126 . Stewards complained that locked accounts can continue to autocreate on other wikis. I decided to turn off autocreation logging because it:
[07:20:58] <^demon>	1) allows vandals to flood local logs with "X plays with blocks" nonsense,
[07:20:59] <^demon>	2) is useless (I did not find the use case).
[07:20:59] <apergos>	urgh
[07:20:59] <^demon>	If we find reason why autocreation logging may be useful, I do not really mind it being turned on again.
[07:21:37] <RoanKattouw>	Is it configurable?
[07:21:39] <^demon>	Yes
[07:22:00] <^demon>	So it's really a matter of deciding how we want to configure it.
[07:23:06] <^demon>	!r 64853
[07:23:06] --elephant--	http://www.mediawiki.org/wiki/Special:Code/MediaWiki/64853
[07:23:11] <^demon>	^ The change was done here
[07:23:33] <apergos>	right
[07:25:26] <apergos>	do we have to solve this right now?
[07:25:39] <apergos>	that is, it's a political issue basically. we know how to set a variable
[07:25:47] <RoanKattouw>	Yeah
[07:26:10] <RoanKattouw>	I guess we can just set it to keep the current behavior to prevent drama
[07:26:17] <guillom>	Hello
[07:26:23] <guillom>	What's the status of the upgrade?
[07:26:24] <Platonides>	hi guillom
[07:26:27] <RoanKattouw>	Then switch it later, or offer communities to opt out, or something
[07:26:43] <^demon>	guillom: Running on usabilitywiki right now :)
[07:26:45] <apergos>	the current config file does that; it's only true on test2
[07:26:46] <robla>	guillom: we're fixing some problems we discovered in the process of deploying to usability
[07:27:03] <RoanKattouw>	Hmmm
[07:27:10] <RoanKattouw>	mark: Bits doesn't support HTTPS, does it?
[07:27:14] <mark>	no
[07:27:18] <RoanKattouw>	Right
[07:27:24] <guillom>	robla, ok; I can go around the 1st-wave wikis & add a local notice, if we're sure we're going to deploy there.
[07:27:30] <RoanKattouw>	So I currently run load.php through secure.wm.o/wikipedia/foo/w/load.php on secure
[07:27:32] <guillom>	(good thing usabilitywiki is closed, too ;)
[07:27:42] <RoanKattouw>	But I don't think there's any server-side caching for such URLs, is there?
[07:27:49] <mark>	no
[07:28:05] <mark>	given that secure is only one server, I wouldn't worry about it
[07:28:19] <RoanKattouw>	So everything that calls secure runs on that one server?
[07:28:22] <robla>	guillom: sure, I can help out with that.
[07:28:29] <mark>	no, but is proxied through that server
[07:28:29] <RoanKattouw>	I thought it was a gateway/proxy
[07:28:37] <mark>	and it's https
[07:28:48] <mark>	it'll be difficult to overload the bits app servers backend through that
[07:28:56] <guillom>	robla, so, what's the current plan? Deploy to all the 1st wave wikis in the coming hours?
[07:28:59] <mark>	or the app servers in this case I guess
[07:29:11] <RoanKattouw>	Oh, right
[07:29:29] <RoanKattouw>	You're saying the secure server will die before the app servers take a noticeable hit from the extra load
[07:29:35] <robla>	guillom: yup.  let's pick a different IRC channel to coordinate the messaging in
[07:29:37] <mark>	yeah
[07:29:40] <guillom>	sure
[07:29:40] <RoanKattouw>	Because it's all alone and outnumbered
[07:29:58] <RoanKattouw>	(Man we really need to get our shit together with HTTPS some time)
[07:30:28] <robla>	guillom and I will be in #wikimedia coordinating on messaging
[07:31:15] <apergos>	it and everything else on there
[07:34:22] <TimStarling>	script committed
[07:34:34] <TimStarling>	have you solved every other problem yet?
[07:34:39] <TimStarling>	;)
[07:35:13] <apergos>	yes; we're working now on mubarak out of egypt, world peace will be next
[07:35:47] <RoanKattouw>	What does +			$lb->reuseConnection( $db ); do?
[07:35:52] <TimStarling>	silly egyptians
[07:35:58] <RoanKattouw>	It sounds like it might help me fix another problem
[07:36:12] <TimStarling>	the man's been there for 30 years, he was hardly going to leave after 2 days of peaceful protest
[07:36:29] <RoanKattouw>	The script is OK if reuseConnection() does what I expect
[07:36:51] <apergos>	which means they will only step things p now
[07:36:52] <RoanKattouw>	Aha
[07:36:53] <apergos>	*up
[07:36:55] *RoanKattouw 	reads documentation
[07:37:05] <RoanKattouw>	It does exactly what I thought it did, and what I needed it to do
[07:37:15] <TimStarling>	LoadBalancer caches connections to every DB server
[07:37:23] <RoanKattouw>	Yeah I saw
[07:37:26] <RoanKattouw>	I could use this in PrefSwitch
[07:37:28] <TimStarling>	alright
[07:37:53] <Platonides>	looks ok
[07:39:03] <^demon>	Hmm. Actually, I kind of agree with Victor. I don't really see a use-case for spamming the local logs with autocreations. And it does create problems for stewards & co. when dealing with problematic usernames.
[07:39:46] <RoanKattouw>	Shall we proceed to the next wiki?
[07:39:58] <robla>	sounds like it's time
[07:40:07] <apergos>	are the truncations complete by now?
[07:40:20] <TimStarling>	I'm running the script
[07:40:30] <TimStarling>	it's done
[07:40:36] <apergos>	ok
[07:40:37] <RoanKattouw>	Alright
[07:40:40] <RoanKattouw>	Next up is ...
[07:40:50] <apergos>	simple?
[07:41:06] <TimStarling>	strategy simple simple meta
[07:41:25] <^demon>	Have we agreed on the setting of logging of autocreations?
[07:41:25] <RoanKattouw>	I say we do strategy and 2x simple
[07:41:26] <TimStarling>	the beta set
[07:41:34] <RoanKattouw>	Hmm yeah
[07:41:38] <RoanKattouw>	meta might as well go in too I guess
[07:42:01] <Bryan>	morning
[07:42:02] <apergos>	I;d rather hold on meta
[07:42:05] <RoanKattouw>	^demon: Fine by me, I don't really care. If people complain we can tell them it's a feature and that they can change it back with consensus
[07:42:09] <apergos>	do it after this round
[07:42:22] <RoanKattouw>	Yeah let's put meta in its own round by itself
[07:42:29] <RoanKattouw>	To see if central things are affected
[07:42:48] <TimStarling>	should we do a small RTL wiki instead?
[07:43:04] <RoanKattouw>	We didn't pre-notify any RTL wikis
[07:43:14] <RoanKattouw>	Something that Amir made a point of on wikitech-l
[07:43:18] <robla>	lemme check if he.wikisource is possibly ready
[07:43:26] <apergos>	I'd rather do a simple and then an rtl
[07:43:37] <RoanKattouw>	If they're not ready, we could just set $wgLanguageCode to he or something on test2wiki
[07:44:29] <robla>	yay for google translate: http://he.wikisource.org/wiki/%D7%95%D7%99%D7%A7%D7%99%D7%98%D7%A7%D7%A1%D7%98:%D7%9E%D7%96%D7%A0%D7%95%D7%9F
[07:44:32] <TimStarling>	ok, I'm switching strategy, simple.wp, simple.wiktionary now
[07:44:50] <RoanKattouw>	OK
[07:45:19] <apergos>	there go the failed stats
[07:45:49] <apergos>	ergh, with the mixed paths
[07:46:01] <^demon>	Yep.
[07:47:17] <TimStarling>	maybe there is a cache from the failed deployment earlier in the week?
[07:48:09] <RoanKattouw>	Ah yes, memcached probably
[07:48:13] <^demon>	I think they're mostly (all?) coming from srv191.
[07:48:45] <TimStarling>	true
[07:48:49] <mark>	what is coming from srv191?
[07:48:51] <apergos>	248
[07:48:55] <^demon>	No, I lied. Also seeing 249 and 248.
[07:49:01] <mark>	bits app servers
[07:49:07] <TimStarling>	no, it's just that they come in big batches
[07:49:08] <mark>	so resource loader
[07:49:09] <RoanKattouw>	Wait no, file dependencies are supposed to come from module_deps
[07:49:13] <TimStarling>	all in the same second
[07:49:23] <apergos>	yes, a pile at once
[07:49:27] <TimStarling>	so if you look at the last 20 lines, it looks like they are all from the same server
[07:49:38] <mark>	what are you talking about?
[07:49:47] <TimStarling>	Feb 11 07:51:50 10.0.2.191 apache2[16335]: PHP Warning:  filemtime() [<a href='function.filemtime'>function.filemtime</a>]: stat failed for /usr/local/apache/common/wmf-deployment/skins/common/images/tipsy-arrow.gif in /usr/local/apache/common-local/php-1.17/includes/resourceloader/ResourceLoaderFileModule.php on line 361
[07:49:49] <^demon>	Yeah I noticed that as soon as I scrolled back up a bit ;-)
[07:49:56] <RoanKattouw>	Aha, found it
[07:49:58] <RoanKattouw>	mysql> select count(*) from module_deps where md_deps like '%wmf-deployment%';
[07:49:59] <RoanKattouw>	7 on strategywiki
[07:50:03] <mark>	aha
[07:50:15] <apergos>	ahh
[07:50:18] <RoanKattouw>	Deleted those rows, it should stop now
[07:50:32] <TimStarling>	can that table be safely truncated?
[07:50:40] <RoanKattouw>	Yes
[07:50:41] <mark>	tipsy-arrow.gif does not exist there
[07:50:43] <RoanKattouw>	It's a caching table
[07:50:54] <RoanKattouw>	mark: Well yeah, cause it's mixed paths
[07:50:59] <mark>	ok
[07:51:03] <TimStarling>	it doesn't exist in 1.16, only in 1.17
[07:51:23] <RoanKattouw>	This means that if and when we move php-1.17 to wmf-deployment, we'll have to keep a symlink in place
[07:51:53] <RoanKattouw>	TimStarling: Are you gonna truncate that table on all wikis or shall I?
[07:53:07] <TimStarling>	I'm rewriting foreachwikidb
[07:53:14] <RoanKattouw>	No need
[07:53:17] <TimStarling>	there was a time when that file made sense
[07:53:28] <RoanKattouw>	Put the query in an SQL file, then run it with sql.php and foreachwiki
[07:53:29] <TimStarling>	truncate away then, I'm going to rewrite this while I'm thinking of it
[07:53:41] <TimStarling>	once upon a time, there was no all.dblist, that's when it made sense
[07:54:24] <TimStarling>	my dinner is ready
[07:54:48] <RoanKattouw>	Running
[07:54:50] <TimStarling>	nothing exciting is happening, should I just go and eat it?
[07:55:01] <RoanKattouw>	Go ahead
[07:55:03] <RoanKattouw>	I have another RL bug to fix anyway
[07:55:24] <TimStarling>	ok, back in 15 minutes
[07:57:02] <Bryan>	yay, CSS is back
[08:03:29] <RoanKattouw>	My RL fix is in http://www.mediawiki.org/wiki/Special:Code/MediaWiki/81936
[08:03:38] <RoanKattouw>	TimStarling: Once you've finished dinner, could you review it? ----^^
[08:12:57] <RoanKattouw>	Ugh, JS error in lqt.js due to missing wgWikiEditorPreferences
[08:15:03] <TimStarling>	back
[08:21:29] *RoanKattouw 	stabs Firefox
[08:21:33] <RoanKattouw>	It's a great browser, I just wish it didn't hog memory like crazy and crashed less often
[08:21:46] <mark>	after a day or two, mine gets sluggish
[08:21:55] <RoanKattouw>	Yeah
[08:21:56] <mark>	like, every time you want to scroll a page or type text, it delays for one second
[08:22:02] <mark>	lags
[08:22:08] <RoanKattouw>	Probably something to do with it using 48% of my memory
[08:22:19] <TimStarling>	you probably want $msgBlobMtime = wfTimestamp( TS_MW )
[08:22:35] <TimStarling>	since it converts to unix on the next line, it seems a bit contradictory
[08:22:46] <RoanKattouw>	Right
[08:22:54] <RoanKattouw>	Can just use wfTimestampNow() then, for clarity
[08:22:57] <TimStarling>	and you probably want $msgBlobMtime === false instead of !
[08:23:01] <RoanKattouw>	The format doesn't matter since we convert anyway
[08:23:15] <RoanKattouw>	Yeah I wasn't sure what selectField() returned when there is no row found
[08:23:41] <RoanKattouw>	And I figured values like '' or 0 are invalid and extremely unrealistic, respectively
[08:23:45] <TimStarling>	false
[08:23:59] <RoanKattouw>	Since the timestamp field is supposed to be filled with a real timestamp, and this software didn't exist yet in 1969
[08:24:00] <TimStarling>	well, it's DBMS-specific
[08:24:08] <RoanKattouw>	Hmm that's right
[08:24:35] <TimStarling>	I checked whether mr_timestamp could be set to 01-01-1970, it seemed not
[08:24:44] <TimStarling>	but it's probably better to use ===false for clarity
[08:25:20] <TimStarling>	other than that, it looks ok
[08:25:23] <RoanKattouw>	Addressed both comments in a follow-up commit
[08:25:29] <TimStarling>	thanks
[08:25:38] <RoanKattouw>	I'll merge it to 1.17wmf1 now and push it
[08:26:11] <TimStarling>	I set it to resolved
[08:26:27] <RoanKattouw>	Thanks
[08:27:58] <TimStarling>	still no action on #wikimedia-tech
[08:28:09] <TimStarling>	are there any serious outstanding issues?
[08:28:48] <Bryan>	well, nobody is editting simple
[08:29:03] <Bryan>	perhaps take a more active wiki for the next batch?
[08:29:10] <TimStarling>	I think we can just do the rest of them
[08:29:27] <^demon>	Let's still do meta by itself.
[08:29:29] <^demon>	Maybe next?
[08:29:41] <RoanKattouw>	Hold on
[08:29:53] <RoanKattouw>	There is an issue with the skin/extension paths on secure
[08:30:49] <RoanKattouw>	Hmm, I don't understand, $wgExtensionAssetsPath is set correctly but LQT is loading from the wrong place on strategywiki
[08:31:23] <RoanKattouw>	$wgLiquidThreadsExtensionPath = "{$wgScriptPath}/extensions/LiquidThreads";
[08:31:27] <RoanKattouw>	Thanks, Andrew *grumble*
[08:31:28] <TimStarling>	where are you seeing this?
[08:31:43] <TimStarling>	ah right
[08:32:36] <RoanKattouw>	Fixing it already
[08:33:08] <apergos>	I'll ask it since I don't have the answer: where were you seeing it?
[08:33:54] <RoanKattouw>	What the ...
[08:33:57] <RoanKattouw>	It's still not working
[08:34:11] <RoanKattouw>	strategy wiki, user talk page
[08:34:18] <RoanKattouw>	on secure
[08:34:25] <RoanKattouw>	It's loading lqt.js from /w/extensions
[08:34:42] <RoanKattouw>	On the normal site ... *turns off HTTPSEverywhere* ...
[08:34:45] <apergos>	secure... ok
[08:35:31] <RoanKattouw>	Also broken
[08:37:04] <RoanKattouw>	Aha
[08:37:06] <RoanKattouw>	wmf-config/liquidthreads.php
[08:39:54] <RoanKattouw>	Hmm
[08:40:13] <RoanKattouw>	So I'm still getting JS errors in LQT but I guess you guys can proceed to roll out to other non-LQT wikis
[08:40:27] <TimStarling>	ok
[08:41:19] <TimStarling>	switching to 1.17 on metawiki
[08:44:57] <Bryan>	looks good
[08:45:06] <TimStarling>	CentralNotice seems to still work
[08:45:17] <Waihorace>	Suggest to re-enable the CentralNotice for upgrade
[08:46:26] <RoanKattouw>	Whee, LQT works on strategy now
[08:46:38] <RoanKattouw>	Waihorace: I think someone enabled CentralNotice on the wikis that were being switched over
[08:46:39] <guillom>	Waihorace, no, we're using local sitenotices today
[08:46:42] <apergos>	what did you change in the meantime?
[08:46:48] *RoanKattouw 	looks over at guillom and robla
[08:46:50] <RoanKattouw>	Ah, right
[08:47:24] <guillom>	It doesn't really make sense to spam 800+ wikis since we're upgrading only a dozen
[08:48:06] <Waihorace>	Maybe it should be turn on again before the second window?
[08:48:34] <guillom>	yes
[08:48:39] <guillom>	but that's next week :)
[08:49:41] <Jamesofur>	interesting. if you click the link to see the list of people in a global group you're getting the whole global user list
[08:49:44] <Jamesofur>	http://meta.wikimedia.org/wiki/Special:GlobalGroupPermissions/global_rollback
[08:50:17] <guillom>	Jamesofur, you mean http://meta.wikimedia.org/wiki/Special:GlobalUsers/global_rollback
[08:50:28] <Waihorace>	linking error, http://meta.wikimedia.org/w/index.php?title=Special%3AGlobalUsers&username=&group=Global_rollback&limit=50 works
[08:50:34] <Jamesofur>	yeah
[08:50:38] <Waihorace>	It should be fix
[08:50:58] <Jamesofur>	I was just linking to the place where I was gettin gthe error (clicking that link)
[08:51:33] <Waihorace>	But it works well on usability
[08:52:37] <RoanKattouw>	Hmm, wait
[08:52:56] <RoanKattouw>	I think we need to do something about the LocalisationUpdate cronjob
[08:53:04] <RoanKattouw>	Remember how I said 'be careful with maintenance scripts'?
[08:53:14] <Waihorace>	OK, I know the problem, it should not be lowercase.
[08:53:28] <Waihorace>	http://meta.wikimedia.org/wiki/Special:GlobalGroupPermissions/Global_rollback can work well but not http://meta.wikimedia.org/wiki/Special:GlobalGroupPermissions/global_rollback
[09:01:00] <Brian_S>	Are there any changes to how RevisionDeletions of revisions are visible to admins?
[09:01:29] <RoanKattouw>	I think certain things changed in revdel, but I have no idea what
[09:01:36] <RoanKattouw>	Try searching the RELEASE-NOTES file for it I guess
[09:02:35] <TimStarling>	I'll fix Special:GlobalUsers
[09:03:02] <Brian_S>	I can see one difference, it's not greyed out and stuff
[09:03:18] <Brian_S>	like you can now click the revision
[09:03:44] <RoanKattouw>	Ah, never mind, LU will be fine
[09:03:49] <RoanKattouw>	Its cache directory is relative to $IP
[09:04:17] <RoanKattouw>	Which means the next LU runs will only update the cache for the 1.16 wikis because they write to wmf-deployment not php-1.17 but I guess that's fine
[09:04:44] <apergos>	it's fine for now
[09:07:10] <Waihorace>	<dungodung> why does Special:GlobalUserRights redirect to Special:UserRights on meta? @ #wikimedia-tech
[09:07:56] <RoanKattouw>	That's strange
[09:09:17] <RoanKattouw>	Hmm:
[09:09:20] <RoanKattouw>	8:class SpecialGlobalGroupMembership extends UserrightsPage {
[09:09:21] <RoanKattouw>	12:             parent::__construct( 'GlobalGroupMembership' );
[09:09:32] <RoanKattouw>	That's probably part of the reason why it goes wrong, GGM extends Userrights
[09:10:55] <RoanKattouw>	Aah, it calls parent::__construct()
[09:11:19] <^demon>	Parent constructor doesn't allow for a param.
[09:11:42] <Bryan>	12 	                 parent::SpecialPage( 'GlobalGroupMembership' ); 	                 parent::__construct( 'GlobalGroupMembership' );
[09:11:53] <Bryan>	r71963
[09:12:05] <^demon>	!r 71963
[09:12:05] --elephant--	http://www.mediawiki.org/wiki/Special:Code/MediaWiki/71963
[09:12:33] <^demon>	My fault, fixing
[09:13:35] <Brian_S>	Any new logs in 1.17?
[09:13:41] <Brian_S>	:o
[09:13:41] <RoanKattouw>	^demon: Already fixed in the live code
[09:14:06] <^demon>	It should call the SpecialPage constructor, not the parent.
[09:14:17] <RoanKattouw>	Already doing that
[09:16:51] <^demon>	Was globaluserrights merged somewhere else? I can't find it in 1.17wmf1
[09:17:29] <RoanKattouw>	It's not in SVN
[09:17:41] <RoanKattouw>	I haven't bothered committing my last 10 or so live hacks to SVN yet
[09:17:49] <^demon>	Ah, moved into CentralAuth
[09:17:59] <RoanKattouw>	I'll be a good boy and commit stuff, but after this deployment is over
[09:18:32] <^demon>	...so should I not bother?
[09:19:52] <Waihorace>	Is the next wiki can go?
[09:20:43] <TimStarling>	Special:GlobalGroupPermissions lets you type anything as a subpage
[09:20:46] <TimStarling>	http://meta.wikimedia.org/wiki/Special:GlobalGroupPermissions/38u9fr3m
[09:21:03] <TimStarling>	including global_rollback, which is a non-existent group
[09:21:10] <TimStarling>	Global_rollback does exist
[09:21:44] <TimStarling>	Jamesofur: was there an incorrect link to http://meta.wikimedia.org/wiki/Special:GlobalGroupPermissions/global_rollback somewhere?
[09:23:23] <Jamesofur>	no it was the link on that page, to see the users in the group
[09:23:38] <TimStarling>	yes, but how did you get to that page?
[09:25:14] <Jamesofur>	http://meta.wikimedia.org/wiki/Special:GlobalGroupPermissions
[09:25:30] <Jamesofur>	(from special:specialpages
[09:25:57] <TimStarling>	it links to http://meta.wikimedia.org/wiki/Special:GlobalGroupPermissions/Global_rollback
[09:26:02] <TimStarling>	which is correct
[09:26:12] <TimStarling>	unless someone just changed it 5 minutes ago
[09:26:59] <Jamesofur>	huh... yeah not sure it does
[09:27:09] <Jamesofur>	but that was how I got there so if it's working now
[09:27:38] <Jamesofur>	*yeah not sure, it does
[09:28:17] <TimStarling>	PHP Fatal error:  Allowed memory size of 83886080 bytes exhausted (tried to allocate 71 bytes) in /usr/local/apache/common-local/php-1.17/includes/diff/WikiDiff.php on line 1094
[09:28:34] <RoanKattouw>	Oh, how nice, the old toolbar is broken
[09:28:36] <TimStarling>	which is not surprising, it's one of the reasons I wrote wikidiff2 in the first place
[09:28:51] <RoanKattouw>	Yeah I was wondering why we would use the PHP diff engine, like, ever
[09:29:19] <TimStarling>	I checked before we started today, there was still no module
[09:29:26] <TimStarling>	I'll see if the file is there
[09:31:44] <TimStarling>	they all have the .so file, I'll just create the .ini file manually
[09:38:45] <RoanKattouw>	Meh, debug mode is broken because wikibits doesn't load early enough
[09:38:54] <RoanKattouw>	I'll just have to bite the bullet and debug in production mode then
[09:41:06] <RoanKattouw>	What the hell, jQuery.client seems to be disappearing somehow
[09:41:51] <RoanKattouw>	Oh wait, that's probably what jQuery.client gets for trying to call itself
[09:47:02] <TimStarling>	Error: that is undefined
[09:47:02] <TimStarling>	Source File: http://bits.wikimedia.org/meta.wikimedia.org/load.php?debug=false&lang=en&modules=jquery.async%7Cjquery.checkboxShiftClick%7Cjquery.client%7Cjquery.cookie%7Cjquery.delayedBind%7Cjquery.placeholder%7Cjquery.textSelection%7Cmediawiki.action.edit%7Cmediawiki.language%7Cmediawiki.legacy.ajax%7Cmediawiki.legacy.ajaxwatch%7Cmediawiki.legacy.edit%7Cmediawiki.legacy.wikibits%7Cmediawiki.util&skin=monobook&version=20110211T084634Z
[09:47:02] <TimStarling>	Line: 198
[09:47:20] <RoanKattouw>	TimStarling: Already fied
[09:48:51] <TimStarling>	half the time when I click a tab in preferences, the tab changes and the rest of the preferences pane doesn't
[09:49:12] <RoanKattouw>	That's weird
[09:49:58] <RoanKattouw>	WFM
[09:50:05] <TimStarling>	ah, worked it out
[09:50:17] <TimStarling>	it happens when mousedown is over the tab button, but mouseup isn't
[09:50:30] <RoanKattouw>	Aha
[09:50:39] <RoanKattouw>	So we have mousedown for the visual thing and click for the tab change
[09:50:42] <RoanKattouw>	That sounds like a bad idea
[09:51:36] <TimStarling>	you might expect the two things to be in the same region of the code
[09:52:09] <RoanKattouw>	I'll have a look in a second
[09:52:16] <RoanKattouw>	I'm now filing a bug about debug mode being borked
[09:52:51] <Waihorace>	OK, may anyone take any action on my suggestion about varient changes? [ crossposing #wikimedia-tech ]
[09:53:44] <RoanKattouw>	Yeah so Waihorace was suggesting we also test deploy to a wiki that uses variants (e.g. sr, zh)
[09:53:49] <RoanKattouw>	Which sounds like a good idea to me
[09:53:59] <robla>	I'm not so sure just yet
[09:54:05] <RoanKattouw>	Variants are the next hairy language-based thing after RTL
[09:54:13] <RoanKattouw>	I realize there are communications issues here
[09:54:30] <RoanKattouw>	I'm just saying I agree that we'll need to test variants functionality somehow
[09:54:42] <robla>	sure, we should definitely do that
[09:54:44] <RoanKattouw>	Which can probably also be done by switching test2wiki to Serbian and making some adjustments
[09:54:53] <robla>	that'd be perfect
[09:54:56] <Waihorace>	The URL have http://zh.wikipedia.org/zh/Article http://zh.wikipedia.org/zh-hans/Article on zhwiki and more...
[09:55:26] <RoanKattouw>	Yes
[09:55:43] <RoanKattouw>	For us English-speaking devs, sr-el vs. sr-ec is probably more convenient
[09:56:00] <RoanKattouw>	Because one is Latin and one is Cyrillic, easier to tell the difference for us than between two different Chinese alphabets
[09:56:25] <Waihorace>	And I don't know will the URL broken after upgrade and it is an experience that the URL have broken and the upgrade before (such as 1.15 => 1.16 and zh link broken that time...)
[09:56:42] <RoanKattouw>	Yeah, variant stuff has broken before, which is why we should test it
[09:56:54] <Waihorace>	But, we should test gan or zh? Seems zh to be more people is using it...
[09:57:05] <RoanKattouw>	Like I said, I prefer Serbian
[09:57:11] <apergos>	the same issues are liable to surface on sr
[09:57:33] <Waihorace>	http://gan.wikipedia.org/gan-hans is 404
[09:57:50] <RoanKattouw>	Yes, but for English-speaking devs it's probably easier to work with Latin vs. Cyrillic then Han Chinese vs. Traditional Chinese
[09:58:13] <Waihorace>	But http://zh.wikipedia.org/zh-hans/Wikipedia:%E9%A6%96%E9%A1%B5 works
[09:59:04] <Waihorace>	My purpose is to test the URL works or not. This cannot be test on gan because http://gan.wikipedia.org/gan-hans is not using but http://zh.wikipedia.org/zh-hans is using...
[09:59:10] <TimStarling>	RoanKattouw: looks like the preferences thing is fixed in trunk already
[09:59:53] <RoanKattouw>	Yeah
[09:59:58] <RoanKattouw>	Prefs was rewritten in trunk I think
[10:00:18] <RoanKattouw>	... or not
[10:00:27] <robla>	we've only got a couple hours left in our window, and we haven't deployed to nlwiki.  I think we should probably get that done before trying anything with variants
[10:00:28] <RoanKattouw>	Well it was, but the rewrite made it into 1.17wmf1
[10:00:35] <RoanKattouw>	Yes, I guess we shoult
[10:00:35] <TimStarling>	r81585
[10:00:36] <RoanKattouw>	*d
[10:00:50] <RoanKattouw>	!r 81585
[10:00:50] --elephant--	http://www.mediawiki.org/wiki/Special:Code/MediaWiki/81585
[10:01:44] <RoanKattouw>	Looks good, I'll pull it in
[10:01:55] <^demon>	TimStarling: It's not a 1.17-specific issue, but if you could look at r81950 for OggHandler. It throws an error when $metadata['streams'] is empty.
[10:02:01] <Waihorace>	Maybe we can create a new window for testing the variants... If we have time, just done it this window...
[10:03:44] <robla>	Waihorace: what I think we can do is do the variants at the very beginning of our second window.  I imagine we can knock out whatever problems we hit fairly quickly if we're focused on those problems
[10:04:31] <Waihorace>	OK, then go ahead on http://techblog.wikimedia.org/2011/02/1-17deployment-attempt2/ this wikis. We only have 2 hours left...
[10:04:32] <Bryan>	it would be good to have Philip online
[10:04:54] <TimStarling>	^demon: done
[10:05:02] <^demon>	Ah, dupes
[10:05:09] <^demon>	See, array_unique() would've made it clear :)
[10:05:14] <^demon>	I'll do that
[10:05:23] <Waihorace>	Bryan, http://zh.wikipedia.org/wiki/Special:%E7%94%A8%E6%88%B7%E8%B4%A1%E7%8C%AE/PhiLiP PhiLiP is online on
[10:05:31] <Waihorace>	Wikipedia half hours ago
[10:06:20] <TimStarling>	are we running out of things to fix yet?
[10:06:36] <TimStarling>	can I add the rest of the wikis in this phase?
[10:07:04] <RoanKattouw>	Probably
[10:07:06] <apergos>	what do we have besines nl?
[10:07:10] <apergos>	*besides
[10:07:10] <Brian_S>	http://simple.wikipedia.org/wiki/Special:NewFiles
[10:07:12] <RoanKattouw>	I'm gonna push that preferences JS fix now
[10:07:21] <robla>	we've got wikinews/wikiquote
[10:07:25] <Brian_S>	The text is funky with the sound preview
[10:07:45] <robla>	eowiki
[10:07:56] <apergos>	eo is flagged revs?
[10:07:57] *RoanKattouw 	<3 not having to update the style version appendix any more
[10:07:57] <thedj>	Brian_S: interesting.
[10:08:02] <Waihorace>	When will we continue to roll out the six wiki on the list?
[10:08:08] <RoanKattouw>	I say we do it now
[10:08:14] <robla>	I'm fine with that
[10:08:19] <thedj>	Brian_S: that's because the gallery tile height is fixed now, but the adding of the button is dynamic.
[10:08:30] <Brian_S>	Ah.
[10:08:40] <robla>	TimStarling: fire when ready
[10:08:48] <thedj>	Brian_S: we need a ticket for that, so it can be fixed in ogghandler i think...
[10:09:52] <Waihorace>	RoanKattouw, go ahead then
[10:09:54] <RoanKattouw>	TimStarling: Be sure to clear profiling data after you push
[10:10:17] <thedj>	Brian_S: Krinkle wrote the gallery changes i believe, might wanna ping him on it as well
[10:11:50] <^demon>	Who the hell is using /e in a preg_replace?
[10:11:57] <RoanKattouw>	Investigating reported pfuncs breakage on nlwiki
[10:11:59] <^demon>	Roll back eowiki, it's exploding.
[10:12:15] <^demon>	Maybe not roll back, lemme see...
[10:12:34] <apergos>	warning anyways
[10:12:42] <Bryan>	doesn't eo use some magical characters?
[10:13:00] <robla>	looks pretty splodey to me
[10:13:14] <apergos>	PHP Fatal error:  Maximum execution time of 180 seconds exceeded in /usr/local/apache/common-local/php-1.17/includes/Title.php on line 2584
[10:13:18] <apergos>	ok that's less good
[10:13:31] <TimStarling>	ok, reverting nl and eo
[10:14:31] <RoanKattouw>	     30 Warning:  preg_replace_callback() [<a href='function.preg-replace-callback'>function.preg-replace-callback</a>]: Modifier /e cannot be used with replacement callback in /us
[10:14:33] <RoanKattouw>	r/local/apache/common-local/php-1.17/languages/classes/LanguageEo.php on line 49
[10:15:04] <apergos>	yep those are the eo ones
[10:15:07] <RoanKattouw>	      1 Warning:  preg_match() [<a href='function.preg-match'>function.preg-match</a>]: Unknown modifier '/' in /usr/local/apache/common-local/wmf-deployment/includes/search/SearchEngine.php on line 1212
[10:15:09] <RoanKattouw>	      1 Warning:  preg_match() [<a href='function.preg-match'>function.preg-match</a>]: Compilation failed: unmatched parentheses at offset 8 in /usr/local/apache/common-local/wmf-deployment/includes/search/SearchEngine.php on line 1212
[10:15:10] <RoanKattouw>	      1 Warning:  preg_match() [<a href='function.preg-match'>function.preg-match</a>]: Compilation failed: unmatched parentheses at offset 21 in /usr/local/apache/common-local/wmf-deployment/includes/search/SearchEngine.php on line 1212
[10:15:14] <RoanKattouw>	Oh wait that's 1.16
[10:15:15] <RoanKattouw>	Ignore me
[10:15:27] <RoanKattouw>	Hmm this is strange
[10:15:34] <RoanKattouw>	ParserFunctions works on test2wiki: https://secure.wikimedia.org/wikipedia/test2/wiki/Main_Page
[10:16:00] <^demon>	RoanKattouw: The preg issue in search engine I saw the other day. Don't beat yourself up over it :)
[10:17:09] <TimStarling>	maybe it's that stupid preprocessor XML cache
[10:17:24] <TimStarling>	apologies to whoever wrote it
[10:17:59] <RoanKattouw>	Oooh
[10:18:18] <TimStarling>	unless it was me in which case I meant every word
[10:18:19] <RoanKattouw>	I guess purging wouldn't help against that because it caches by the hash of the wikitext or something?
[10:19:50] <apergos>	mm I think so
[10:20:07] <TimStarling>	let's confirm first
[10:20:12] <^demon>	svn up LanguageEo and we can try it again
[10:20:15] <TimStarling>	are there any bugs on the other wikis I enabled just now?
[10:20:20] <^demon>	(put the fix in r81954)
[10:20:56] <RoanKattouw>	!r 81954
[10:20:56] --elephant--	http://www.mediawiki.org/wiki/Special:Code/MediaWiki/81954
[10:20:56] <apergos>	don;t see anything in the logs
[10:21:17] <RoanKattouw>	TimStarling: Maybe we should version that XML cache then?
[10:21:56] <TimStarling>	it uses Parser:CACHE_VERSION
[10:22:05] <TimStarling>	if you change that, all parser cache objects immediately expire
[10:22:15] <TimStarling>	we can split it, if it was the problem
[10:22:35] <RoanKattouw>	Ouch
[10:22:48] <RoanKattouw>	Yeah let's not touch Parser::CACHE_VERSION :)
[10:22:52] <apergos>	heh
[10:23:15] <apergos>	we;ve had enough cache fun and games recently
[10:23:23] <RoanKattouw>	Exactly
[10:23:23] <Brian_S>	srv278 is having issues?
[10:23:41] <RoanKattouw>	I think that box is notorious
[10:23:44] <TimStarling>	there were a lot of timeouts
[10:23:51] <RoanKattouw>	srv278 has always been trouble AFAIK
[10:23:58] <Brian_S>	heh
[10:24:00] <TimStarling>	all different servers
[10:24:10] <apergos>	yes
[10:24:38] <TimStarling>	we even made an impression on overall CPU: http://ganglia.wikimedia.org/graph.php?g=cpu_report&z=medium&c=Apaches%208%20CPU&m=cpu_report&r=hour&s=descending&hc=3&mc=3
[10:24:39] <RoanKattouw>	^demon: I synced the LanguageEo.php change
[10:24:45] <TimStarling>	so there you go, nlwiki was big enough
[10:24:53] <apergos>	I've been watching them
[10:24:55] <RoanKattouw>	Yay
[10:25:00] <RoanKattouw>	Well, "yay"
[10:25:22] <apergos>	heheheh
[10:25:25] <RoanKattouw>	We succeeded in finding a set of wikis large enough to show on Ganglia but not melt the cluster
[10:25:41] <apergos>	well because we reverted pretty quickly
[10:25:46] <RoanKattouw>	However, the fact that we're seeing a rise in CPU with these wikis is a bad sign
[10:26:07] <TimStarling>	no, it's good
[10:26:15] <mark>	and almost nothing for bits/RL
[10:26:22] <RoanKattouw>	How so?
[10:26:24] <apergos>	ah that's useful
[10:26:30] <TimStarling>	better to see it here than to have an overload and have all the apaches go into swap and the site go down for 3 hours
[10:26:36] <RoanKattouw>	Well yeah
[10:26:42] <RoanKattouw>	But it means we're not out of the woods
[10:26:48] <TimStarling>	this is why we're testing, to see things like this happen
[10:26:50] <mark>	it's almost invisible on the 4 bits app servers
[10:27:04] <apergos>	excellent
[10:27:41] <RoanKattouw>	mark: You can see what's probably the initial cache population around 10:14
[10:27:54] <TimStarling>	it wasn't me, btw
[10:28:00] <^demon>	Do we want to turn eowiki back on now?
[10:28:06] <TimStarling>	when I wrote Preprocessor_DOM, I left a cache-shaped hole in the code
[10:28:10] <RoanKattouw>	haha
[10:28:23] <TimStarling>	I did a lot of profiling and decided that it wasn't worth the trouble
[10:28:27] <apergos>	I think we should (switch eo again)
[10:28:31] <TimStarling>	but someone eventually filled it in
[10:29:01] <RoanKattouw>	It has its own CACHE_VERSION constant
[10:29:21] <TimStarling>	ah right, it's Preprocessor_DOM::CACHE_VERSION, not Parser::CACHE_VERSION
[10:29:55] <RoanKattouw>	In a dirty way too
[10:30:04] <RoanKattouw>	sprintf()ing the version in front of the XML?
[10:30:16] <RoanKattouw>	Just store array( $version, $xml ) or something
[10:32:53] <RoanKattouw>	Oooh, look and there's $wgPreprocessorCacheThreshold
[10:33:09] *RoanKattouw 	creates a small user page on eowiki with a pfunc
[10:34:59] <apergos>	what's the verdict?
[10:35:32] <RoanKattouw>	I'm in favor of switching over eowiki
[10:35:48] <TimStarling>	ok
[10:35:50] <robla>	yup
[10:35:58] <RoanKattouw>	If pfuncs break there again on large pages but continue to work on small pages such as my user page, it's the XML cache thing
[10:36:06] <RoanKattouw>	And increasing the XML cache version is trivial
[10:36:17] <RoanKattouw>	eowiki pfunc test page: https://secure.wikimedia.org/wikipedia/eo/wiki/Uzanto:Catrope
[10:36:28] <apergos>	ok
[10:36:36] <TimStarling>	so it was breaking on eowiki, like nlwiki?
[10:36:42] <Bryan>	yes
[10:36:51] <RoanKattouw>	I don't know, eowiki was probably borked due to the /e regex as well
[10:36:59] *RoanKattouw 	is sad brion isn't around
[10:37:01] <TimStarling>	I'm not switching back to 1.17 on eowiki then
[10:37:11] <RoanKattouw>	(I don't blame him, it's almost 3am, but still)
[10:37:48] <TimStarling>	the XML cache thing was just an idea, it's not anything like confirmed
[10:38:05] <RoanKattouw>	Hmm yeah
[10:38:09] <TimStarling>	it could just as easily be this hack I see in Preprocessor_DOM.php which disables expansion limits under certain circumstances
[10:38:22] <Platonides>	why would you change the preprocessor CACHE_VERSION?
[10:38:33] <TimStarling>	did you find out why those fatals were happening btw?
[10:38:33] <RoanKattouw>	If we had evidence that pages < 1kb are unaffected, that'd be more solid
[10:38:38] <RoanKattouw>	No
[10:38:44] <RoanKattouw>	I guess I should get a backtrace on them
[10:38:51] <Platonides>	there are some pages where libxml break btw
[10:38:57] *RoanKattouw 	goes to get his backtrace
[10:39:01] <Platonides>	for being a too large xml
[10:39:09] <Platonides>	there's a parameter for that
[10:39:15] <apergos>	and another giant pile of seg faults, wtf
[10:39:16] <Platonides>	shouldn't be a change from 1.16 though
[10:44:03] <Platonides>	hmm https://secure.wikimedia.org/wikipedia/test2/wiki/Template:Template_doc didn't break
[10:44:40] <Platonides>	so it's probably using an earlier libxml and is completely unrelated
[10:44:57] <RoanKattouw>	I have put in logging for those preprocessor fatals
[10:45:11] <RoanKattouw>	Hasn't turned up anything yet, probably due to low traffic on 1.17 wikis
[10:45:16] <RoanKattouw>	In the meantime, Cite is broken
[10:45:27] <RoanKattouw>	I'm gonna try to reproduce it in a test case on test2
[10:45:35] <Platonides>	what was the fatal text?
[10:46:06] <RoanKattouw>	Ewww
[10:46:08] <RoanKattouw>	<a href="http://en.wikipedia.org/wiki/Wikipedia:Edit_summary" class='internal' title="Briefly describe the changes you have made here">Edit summary</a>
[10:46:31] <RoanKattouw>	Platonides: Another $this->parser->mOptions is not an object
[10:46:45] <RoanKattouw>	So I now have code that logs a backtrace if ( !is_object (... ) )
[10:47:13] <RoanKattouw>	Log file is empty so far (hasn't even been created by the logger yet in fact)
[10:47:14] <Platonides>	I thought all of them were fixed
[10:47:20] <apergos>	meh
[10:47:21] <RoanKattouw>	For some value of fixed
[10:47:33] <RoanKattouw>	If you mean prefixed by is_object( .. ) && then yes
[10:48:05] <Platonides>	I mean the extensions not calling clearState without mOptions
[10:48:27] <Platonides>	wasn't that merged?
[10:48:33] <vvv>	How would I import a page to test2?
[10:48:57] <RoanKattouw>	Special:Import
[10:48:58] <vvv>	I want to see how does 1.17 work with some enwiki FAs
[10:49:05] <RoanKattouw>	If you need +sysop someone can give it to you
[10:49:16] <RoanKattouw>	Cite works fine on test2
[10:49:17] <RoanKattouw>	https://secure.wikimedia.org/wikipedia/test2/wiki/Cite
[10:49:33] <vvv>	RoanKattouw: someone = ?
[10:49:41] <robla>	actually, I don't think I'm sysop on test2 either, otherwise I would do it
[10:49:50] <apergos>	what's yer username over there?
[10:49:55] <vvv>	VasilievVV
[10:50:04] <Platonides>	yes, r81742 was merged
[10:50:10] <robla>	apergos: RobLa-WMF while you're at it :)
[10:50:35] <mark>	we have one hour left
[10:51:01] <vvv>	apergos: thanks
[10:51:09] <apergos>	you can prolly give yourselves privs, anyways done and done
[10:51:13] <RoanKattouw>	Hmm, sup_1 and chap_2? Is that Cite or something else?
[10:51:28] <robla>	thanks apergos
[10:51:57] <RoanKattouw>	Red herring: Cite isn't broken, it's not even *used* on the broken page on hewikisource
[10:52:09] <apergos>	good to know
[10:52:42] <RoanKattouw>	Ooh look, they're using <a name="foo"> in wikitext
[10:52:51] <RoanKattouw>	Was that allowed in 1.16 and disallowed in 1.17 or something?
[10:54:43] <TimStarling>	the nlwiki problem is reproducible from eval.php, there's no need to screw up the whole wiki to see it
[10:54:56] <apergos>	example?
[10:55:40] <TimStarling>	$wgTitle = Title::newFromText('Dieupentale');;$rev = Revision::newFromTitle($wgTitle);$text = $rev->getText();$opt = new ParserOptions; $wgDebugLogFile = '/dev/stdout';
[10:55:50] <TimStarling>	$out = $wgParser->parse($text, $wgTitle, $opt )
[10:55:57] <TimStarling>	print $out->getText()
[10:56:48] <RoanKattouw>	Nice
[10:57:19] <vvv>	http://test2.wikipedia.org/w/index.php?title=Main_Page&action=edit
[10:57:25] <vvv>	Mind the edit summary field
[10:57:34] <vvv>	Is that a bug of test2 or of 1.17?
[10:58:48] <Bryan>	vvv: on simple.wp it doesn't appear
[10:59:05] <RoanKattouw>	It's probably the message having changed from being an HTML message to a non-HTML message
[11:00:46] <GerardM>	RoanKattouw can the messages available from LocalisationUpdate make things complicated ??
[11:01:06] <RoanKattouw>	GerardM: No.
[11:01:12] <RoanKattouw>	I was afraid they would, so I checked
[11:01:30] <GerardM>	Cool
[11:01:37] <RoanKattouw>	And the LU cache directories were already segregated (by accident, kinda) so they won't mess with each other
[11:01:48] <GerardM>	it may make sense to clean the files anyway
[11:01:51] <RoanKattouw>	However the nightly cronjob will only update the LU cache for 1.16
[11:02:09] <RoanKattouw>	1.17 will start out with a clean set
[11:02:17] <GerardM>	cook
[11:02:18] <GerardM>	l
[11:03:15] <^demon>	apergos: Ok, the wikiset one is fixed.
[11:03:35] <apergos>	??????????????????
[11:03:36] <apergos>	er
[11:03:38] <apergos>	excellent
[11:03:50] <apergos>	I should move that off of right shift someday I guess
[11:03:53] <RoanKattouw>	Speak of the devil:
[11:03:55] <RoanKattouw>	      1 Warning:  Illegal offset type in /usr/local/apache/common-local/wmf-deployment/includes/LocalisationCache.php on line 309
[11:03:57] <RoanKattouw>	      1 Warning:  Illegal offset type in /usr/local/apache/common-local/wmf-deployment/includes/LocalisationCache.php on line 308
[11:03:58] <RoanKattouw>	      1 Warning:  Illegal offset type in isset or empty in /usr/local/apache/common-local/wmf-deployment/includes/LocalisationCache.php on line 299
[11:04:00] <RoanKattouw>	      1 Warning:  Illegal offset type in isset or empty in /usr/local/apache/common-local/wmf-deployment/includes/LocalisationCache.php on line 228
[11:04:03] <RoanKattouw>	      1 Warning:  Illegal offset type in isset or empty in /usr/local/apache/common-local/wmf-deployment/includes/LocalisationCache.php on line 221
[11:04:16] <RoanKattouw>	Oh wait that's LC not LU
[11:04:18] <apergos>	not 1.17
[11:04:57] <RoanKattouw>	That too
[11:05:03] <RoanKattouw>	But I thought it was LU
[11:05:08] <RoanKattouw>	Never mind
[11:05:38] <RoanKattouw>	TimStarling: PHP:  Error parsing /etc/php5/cli/conf.d/wikidiff2.ini on line 2
[11:05:52] <RoanKattouw>	(Upon running php php-1.17/maintenance/eval.php )
[11:06:03] <TimStarling>	on fenari?
[11:06:33] <TimStarling>	on what server?
[11:07:12] <RoanKattouw>	Yes, fenaru
[11:07:14] <RoanKattouw>	*i
[11:07:23] <mark>	fenari doesn't even have wikidiff2.ini now
[11:07:27] <mark>	did someone delete it?
[11:07:44] <RoanKattouw>	Yes it does
[11:07:47] <RoanKattouw>	Look at the path
[11:08:12] <RoanKattouw>	It's looking in /cli not /apache2
[11:08:26] <RoanKattouw>	Ooh wait
[11:08:28] <mark>	mark@fenari:/etc/php5/cli/conf.d$ ls -l /etc/php5/cli/conf.d/wikidiff2.ini
[11:08:28] <RoanKattouw>	My apologies
[11:08:28] <mark>	ls: cannot access /etc/php5/cli/conf.d/wikidiff2.ini: No such file or directory
[11:08:32] <RoanKattouw>	I was on srv193 all this time
[11:08:34] <mark>	ok
[11:08:35] <mark>	checking
[11:08:43] <RoanKattouw>	And I didn't notice because it's /home -mounted
[11:09:06] <mark>	it doesn't like the comment
[11:09:50] <robla>	comments are for sissies....ptui!
[11:10:13] <mark>	odd
[11:10:18] <mark>	it wants ;, not #, but all others have #
[11:13:36] <RoanKattouw>	TimStarling: Meh, I tried raising the Preprocessor DOM CACHE_VERSION locally on srv193 but it didn't fix anything
[11:13:50] <TimStarling>	don't do that
[11:13:54] <TimStarling>	you'll destroy the test case
[11:14:09] <RoanKattouw>	How so?
[11:14:14] <RoanKattouw>	(I already did it)
[11:14:22] <TimStarling>	it'll regenerate the object and save it back to memcached
[11:14:32] <RoanKattouw>	With a different CACHE_VERSION, right?
[11:14:45] <TimStarling>	yes
[11:14:48] <RoanKattouw>	Oh crap, the version isn't in the mc key
[11:14:52] <TimStarling>	no
[11:14:55] <RoanKattouw>	Grrr
[11:15:09] <TimStarling>	I confirmed a while ago that the XML cache was not the problem
[11:15:18] <RoanKattouw>	OK
[11:15:26] <TimStarling>	actually the problem is in the nl localisation, which makes a lot more sense
[11:15:39] <RoanKattouw>	hah really?
[11:15:39] <TimStarling>	it explains why none of the other wikis are affected
[11:15:56] <TimStarling>	anyway, just let me finish my isolating
[11:16:20] <RoanKattouw>	Sure
[11:18:33] <Platonides>	the increased cpu happening in localization reparsing
[11:18:36] <Platonides>	that makes sense
[11:18:40] <TimStarling>	this is what you get when you put your localisations on a wiki
[11:18:52] <TimStarling>	$magicWords['eo'] = array(
[11:18:53] <TimStarling>		'expr' => array( 0, 'espr', 'esprimo' ),
[11:18:53] <TimStarling>		'if' => array( 0, 'se' ),
[11:19:12] <TimStarling>	$magicWords['nl'] = array(
[11:19:13] <TimStarling>		'expr' => array( 0, 'expressie' ),
[11:19:13] <TimStarling>		'if' => array( 0, 'als' ),
[11:19:31] <TimStarling>	it's clearly documented in MessagesEn.php that you need to include the English form as a synonym
[11:20:05] <Platonides>	it was probably localiated from translatewiki
[11:20:11] <Platonides>	tw should enforce that :P
[11:21:26] <RoanKattouw>	Aaah
[11:22:06] <RoanKattouw>	Romaine: There's the source of the {{#if: problems on nlwiki. TranslateWiki had translated #if to #als and removed the original #if as an alias
[11:22:29] <RoanKattouw>	TimStarling: Magic word translations are exported by hand, though. The TWN people check them and add aliases as appropriate
[11:22:38] <RoanKattouw>	They have that routine down for namespace names, maybe not so much for magic words
[11:22:58] <Reedy>	lol
[11:23:10] <TimStarling>	it's not just ParserFunctions, I checked another random extension with magic words and it has the same problem
[11:23:12] <Romaine>	we almost use everywhere the English parserfunctions
[11:23:17] <Romaine>	English-named
[11:23:39] <apergos>	probably true of most wikis
[11:23:47] <Romaine>	also for magic words
[11:24:21] <Platonides>	I suspect trasnlators translate magic words although it isn't then used
[11:24:46] <RoanKattouw>	Alright, so we need to audit all magic word translations for presence of the English alias
[11:25:05] *RoanKattouw 	goes ahead and automates this
[11:26:02] *robla 	smells a commit hook  ;-)
[11:26:36] <^demon>	Commit hooks are slow. Better to get translatewiki to just do it properly :)
[11:27:09] <Bryan>	foreach( $magicWords as $key => $words ) { if ( !in_array( $key, $words ) ) { wfRaiseHell(); } }
[11:27:25] <TimStarling>	how about I put a special case in LocalisationCache::mergeItem()?
[11:28:33] <Bryan>	perhaps even better, translators should not need to bother about english fallbacks, the software should
[11:29:40] <^demon>	Time for some breakfast. Back in a little while.
[11:30:15] <RoanKattouw>	 /usr/local/apache/common/php-1.17/extensions/LiquidThreads/i18n/LiquidThreads.magic.php ar useliquidthreads
[11:30:23] <RoanKattouw>	Plus a bunch of other languages in the same file
[11:30:26] <RoanKattouw>	That's all it finds
[11:30:42] <vvv>	AbuseFilter tests scoes 28/29 for 1.17
[11:31:08] <vvv>	And so they do for 1.16...
[11:31:13] <Reedy>	lol
[11:31:39] <robla>	at any rate, we've got 25 min or so in this window.  are we going to try to take another run at eowiki and/or nlwiki before the window closes?  seems like we're probably going to be pretty much out of time by the time everything is ready for that
[11:37:21] <TimStarling>	there's still another bug, the one that was causing the timeouts
[11:37:28] <RoanKattouw>	Alright, I have a list of bad magic words now
[11:37:39] <TimStarling>	we've got no test cases, we just saw the CPU spike and the error logs
[11:38:01] <TimStarling>	RoanKattouw: I've almost finished writing a merge function for it
[11:38:27] <RoanKattouw>	One that fixes them automatically?
[11:38:46] <RoanKattouw>	http://mediawiki.pastebin.com/QFQ9M5uk
[11:38:56] *mark 	branches MW 1.16 and labels it 1.18
[11:39:14] <Reedy>	mark, i wonder how many complaints that would bring "Where is X!?"
[11:39:22] <mark>	it's the latest version
[11:39:25] <mark>	it must be the best :)
[11:39:29] <Reedy>	haha
[11:39:37] <Reedy>	mark, just call it 2.0
[11:39:41] <Reedy>	Problem solved.
[11:40:31] <apergos>	:-D
[11:40:49] <Platonides>	phase3 > 2.0 ;)
[11:42:03] <vvv>	That would be phase3.14159265
[11:42:32] <apergos>	eeehhhh
[11:43:01] <Reedy>	vvv, does that have enough points of precision?
[11:43:18] <vvv>	Not sure
[11:43:38] <Reedy>	vvv, we should put it to a community vote
[11:44:00] <Reedy>	So we can waste a few weeks
[11:44:03] <apergos>	265358979626..  iirc (which I prolly don't' at this hour of the morning)
[11:44:12] <Reedy>	And get the obvious "No consensus"
[11:44:15] <RoanKattouw>	Beats me
[11:44:17] <apergos>	maybe we can propose "3"
[11:44:25] <apergos>	simpler.
[11:44:29] <RoanKattouw>	I get up to 3.141592653589
[11:44:34] <vvv>	Reedy: are you sure community has enough technical competence to determine the required precision?
[11:44:35] <apergos>	people like round numbers
[11:44:39] <mark>	I get up to 3.14
[11:44:59] <Reedy>	vvv, well, if we don't involve them, even if that is the case, they won't be happy
[11:45:38] <RoanKattouw>	OK, so now what
[11:45:52] <RoanKattouw>	TimStarling: How close are we to getting those magic words fixed?
[11:46:01] <RoanKattouw>	robla: How do you feel we should proceed?
[11:46:46] <robla>	well, I think if we've got a fix that we can deploy to eowiki, we should give it a shot
[11:47:04] <RoanKattouw>	eo is one of the languages with borked PF magic words
[11:47:15] <RoanKattouw>	Which is why I was wondering how close Tim was to fixing those
[11:47:16] <apergos>	8 minutes?
[11:47:17] <apergos>	eh
[11:48:02] <apergos>	irritating.  I woould like anotehr go at watching what happens with nl after the localization issue is fixed
[11:48:07] <apergos>	but we'd be over time
[11:48:19] <TimStarling>	I don't think the magic word issue was causing the CPU spike
[11:48:26] <RoanKattouw>	Me neither
[11:48:27] <apergos>	no. I don't either
[11:48:30] <RoanKattouw>	But it unacceptably broke nlwiki
[11:48:42] <TimStarling>	r81960
[11:48:46] <RoanKattouw>	If we fix the magic word issue, we can deploy to nlwiki and observe the CPU spike as it happens
[11:48:58] <TimStarling>	for 7 minutes?
[11:49:19] <apergos>	exactly. and that's why I'm glum
[11:49:35] <TimStarling>	we know the three lines of code are most responsible for the CPU spike
[11:49:40] <RoanKattouw>	Looks good
[11:49:45] <RoanKattouw>	Which are those?
[11:50:09] <TimStarling>	Title.php:2584, Preprocessor_DOM.php:936 and Preprocessor_DOM.php:937
[11:50:15] <TimStarling>	that is where the timeout errors came from
[11:50:22] <RoanKattouw>	Ah
[11:50:30] <RoanKattouw>		<TimStarling>	for 7 minutes?
[11:50:48] <RoanKattouw>	I know we never clearly discussed this, but I wasn't under the impression that we'd downgrade all switched wikis at 1200 UTC
[11:50:58] <apergos>	I thought we were
[11:50:58] <robla>	oh, no....
[11:51:05] <apergos>	it was a temporary switch, no?
[11:51:18] <TimStarling>	I thought it was permanent
[11:51:26] <robla>	no, I'd assumed it was to leave them running
[11:51:30] <apergos>	huh
[11:51:32] <^demon>	So did I
[11:51:36] <TimStarling>	it's a staged deployment, not a test
[11:51:42] <robla>	exactly
[11:51:45] *apergos 	goes to reread the techblog entry
[11:51:46] <RoanKattouw>	Yeah why not leave them running if they're trucking along just fine
[11:52:02] <RoanKattouw>	The techblog doesn't say anything either way, and neither did we in our meetings
[11:52:31] <apergos>	if seems to read "deployed" so ok
[11:52:39] <apergos>	let's leave em there then
[11:52:44] <RoanKattouw>	TimStarling: LocalisationCache.php look OK ; I assume you've tested them?
[11:53:12] *RoanKattouw 	is merging them already
[11:53:25] <TimStarling>	I checked the nlwiki localisation of ParserFunctions
[11:53:37] <TimStarling>	and I tested the fact that the case flag is not overridden
[11:53:41] <RoanKattouw>	OK
[11:53:57] <RoanKattouw>	Merging and pushing it then
[11:54:24] <RoanKattouw>	I guess I'll also need to touch and sync ParserFunctions.i18n.magic.php to cause a l10ncache recache
[11:54:44] <TimStarling>	yes
[11:54:52] <RoanKattouw>	I'll test it after using eval.php
[11:55:06] <RoanKattouw>	I love that we can now run eval.php in 1.17 mode on a 1.16 wiki *evil laugh*
[11:55:44] *mark 	thinks of Austin Powers
[11:56:58] <robla>	it's not like anyone is holding a stopwatch, so we can go a little bit over here, but I just want to make sure I understand where we're at:
[11:57:55] <robla>	so....the magic words problem is one that we think Tim just fixed, and that made it so that we couldn't leave it enabled on eowiki, and was one reason not to leave it on nlwiki
[11:58:05] <RoanKattouw>	Yup, seems to work for #if on nlwiki
[11:58:27] <RoanKattouw>	You're implying there are other reasons to not deploy to nlwiki
[11:58:34] <robla>	we also still have an unknown performance problem that will register on ganglia if we deploy to nlwiki
[11:58:40] <RoanKattouw>	We did see a CPU spike but I don't know that it was very dramatic
[11:58:42] <RoanKattouw>	Yes
[11:58:45] <robla>	...but will it cause any user problems?
[11:58:52] <mark>	as long as it's nlwiki only, that's hopefully fine
[11:58:54] <RoanKattouw>	It'll register, but not necessarily cause any harm AFAIK
[11:59:07] <Bryan>	was the CPU spike transient or continuous?
[11:59:11] <RoanKattouw>	If it's noticeable for just nlwiki it's definitely a blocker for wholesale deployment
[11:59:15] <RoanKattouw>	Bryan: We don't know, too little data
[11:59:27] <RoanKattouw>	We switched off nlwiki after like a few minutes due to the #if issue
[11:59:29] <Bryan>	guess we should try it then
[11:59:33] <robla>	yup....ok, I think we should give it a shot
[11:59:41] <mark>	we will get much better data with nlwiki on 1.17
[11:59:49] <TimStarling>	it wsa massive
[11:59:54] <TimStarling>	the CPU spike was huge
[12:00:00] <mark>	yes
[12:00:04] <TimStarling>	and the syslog was flooded with timeout errors
[12:00:10] <mark>	that, is a problem
[12:00:11] <TimStarling>	it's probably an infinite loop
[12:00:13] <apergos>	note that our overall traffic is starting to pick up now
[12:00:26] <apergos>	we are well out of the quiet zone
[12:00:43] <RoanKattouw>	Yeah, you can see traffic getting higher even on the Apache CPU graph
[12:00:47] <robla>	so TimStarling: you're skeptical that we should push it then?
[12:00:55] <RoanKattouw>	Thinking about it, repeating that CPU spike now might not be a good idea after all
[12:01:27] <Reedy>	Bloody ubuntu.
[12:01:54] <mark>	ok
[12:01:56] <mark>	time for breakfast then
[12:02:05] <TimStarling>	I don't think we've exhausted the options for testing without deployment
[12:02:10] <RoanKattouw>	We can still do eowiki I guess
[12:02:14] <^demon>	You're a little late for breakfast mark :)
[12:02:19] <TimStarling>	we've got a few leads, like I said, we have the line numbers
[12:02:20] <RoanKattouw>	It's much smaller than nlwiki
[12:02:22] <RoanKattouw>	Yeah
[12:02:42] <TimStarling>	why do we need to deploy either of them?
[12:02:58] <RoanKattouw>	I guess we don't /need/ to
[12:03:01] <TimStarling>	if we deploy again, we should be ready to convert timeout errors to backtraces
[12:03:07] <RoanKattouw>	Yes
[12:03:22] <RoanKattouw>	But if we leave it like this, we should remove nlwiki and eowiki from the 1.17 profiling list
[12:03:24] <robla>	the only reason to do eowiki was if we wanted to see the magic words fixes in action, but that's not critical to test right now
[12:04:12] <robla>	let's leave things like they are....fix up the profiling per Roan's comment, and call it good
[12:04:16] <apergos>	it might be nice, so that gets wrapped up
[12:04:24] *^demon 	raises hand
[12:04:26] <RoanKattouw>	8 preprocessorfatal.log
[12:04:28] <RoanKattouw>	Aha
[12:04:30] <apergos>	(that was to pushing out eo)
[12:04:33] <RoanKattouw>	I have leads on the preprocessor thing now
[12:04:36] <^demon>	Why didn't hewikisource blow up with the same i18n bug?
[12:04:37] <robla>	I call on ^demon
[12:04:50] <RoanKattouw>	Maybe its magic words weren't borked?
[12:04:52] <^demon>	Or do they use localized pfuncs all over the place so we didn't notice?
[12:04:54] <^demon>	Or that.
[12:05:21] <RoanKattouw>	$magicWords['he'] = array(
[12:05:22] <RoanKattouw>	        'expr' => array( 0, '??????', 'expr' ),
[12:05:29] <RoanKattouw>	Their magic word settings were fine
[12:05:37] <robla>	yay Hebrew!
[12:06:01] <^demon>	:)
[12:06:18] <apergos>	whew
[12:06:28] <robla>	alright....I'll update the blog one last time
[12:06:45] <RoanKattouw>	Title:2854 is 		$retVal = array(); in Title::getBrokenLinksFrom()
[12:06:53] <RoanKattouw>	Timing out there makes no sense
[12:07:15] <RoanKattouw>	Argh
[12:07:26] <RoanKattouw>	I borked the backtrace thing for the preprocessor and it's just outputting '1' to the log
[12:08:04] <robla>	great job everyone!  this went really well
[12:08:31] <TimStarling>	Roan: no it isn't
[12:08:36] <TimStarling>	                $dbkey = preg_replace( '/[ _\xA0\x{1680}\x{180E}\x{2000}-\x{200A}\x{2028}\x{2029}\x{202F}\x{205F}\x{3000}]+/u', '_', $dbkey );
[12:08:46] <RoanKattouw>	Oh I was in 1.16wmf4
[12:09:16] <RoanKattouw>	Aha that makes more sense
[12:09:25] <RoanKattouw>	Have we already trimmed $dbkey to 255 chars by that point?
[12:09:29] <apergos>	we are going to have a ton of that aren't we?  I mean the "I'm in 1.1X instead of 1.1X+/-1"
[12:09:49] <RoanKattouw>	Yeah unfortunately
[12:10:00] <RoanKattouw>	1.16wmf4, 1.17wmf1, REL1_17 and trunk
[12:10:06] <apergos>	bleah
[12:11:04] <RoanKattouw>	The Preprocessor_DOM ones are adjacent lines inside a while loop in expand()
[12:12:03] <TimStarling>	expand() is complex code, it's the most complex thing I did in the preprocessor
[12:13:04] <TimStarling>	it started off being recursive, and was converted to an iterative algorithm with a virtual stack
[12:13:15] <TimStarling>	to limit the stack depth and allow more stuff to be parsed when xdebug is enabled
[12:15:30] <nadeesha_calcey>	robla: hi Rob
[12:15:41] <robla>	hi nadeesha_calcey
[12:16:21] <nadeesha_calcey>	robla: we are getting java script errors on http://beta.wikiversity.org/wiki/Main_Page in IE7 and IE8
[12:17:11] <robla>	nadeesha_calcey: what are the errors?
[12:17:29] <Reedy>	Uncaught ReferenceError: addEvent is not defined
[12:17:33] <Reedy>	That appears in Chrome
[12:17:47] <RoanKattouw>	Bug in site JS I think
[12:18:23] <RoanKattouw>	addEvent is not in wikibits.js
[12:18:43] <robla>	oh, I guess I do see it if I open a console
[12:18:54] <nadeesha_calcey>	robla: 'errors on page'   in stutus bar
[12:18:55] <Reedy>	if (texts[i].type == 'text') addEvent (texts[i], 'focus', EditTools.registerTextField);
[12:19:08] <Reedy>	nadeesha_calcey, is the error the same as the one i posted a few lines up?
[12:21:13] <RoanKattouw>	TimStarling: I will make some lunch, then investigate the Title.php timeout
[12:21:59] <TimStarling>	ok
[12:22:04] <apergos>	I am going to sleep and wishing you fruitful and low stress debugging
[12:22:08] <RoanKattouw>	I guess you're gonna sleep soon
[12:22:11] <RoanKattouw>	(Tim)
[12:22:30] <RoanKattouw>	And the West Coast folks too, since the Sun will rise there in a few hours
[12:22:37] <TimStarling>	I'll keep an eye on IRC while you make your lunch, but otherwise I'm going to start winding down I think
[12:22:45] <RoanKattouw>	OK
[12:22:46] <TimStarling>	it's been a long day
[12:22:59] <RoanKattouw>	I have to catch a 15:06 train
[12:23:04] <RoanKattouw>	So I'll be leaving the house around 14:50 my time
[12:23:33] <^demon>	Just had a crapload of segfaults again
[12:24:11] <Reedy>	Funtimes
[12:24:14] <nadeesha_calcey>	Reedy: no
[12:24:19] <^demon>	Still going, actually.
[12:24:23] <TimStarling>	from one server? or a lot?
[12:25:46] <^demon>	Lots of them
[12:26:04] <TimStarling>	maybe we're out of APC space
[12:26:28] <^demon>	First one was 12:28:04, last was at 12:29:21
[12:27:20] <apergos>	when did I see the last huge pile?
[12:27:22] <apergos>	hmm
[12:27:53] <apergos>	12:44
[12:28:02] <apergos>	doesn't correlate with anything I can see
[12:28:25] <^demon>	I copied those timestamps from the apache log.
[12:29:04] <apergos>	yeah sorry, mine must have actually been at some other hour + 44
[12:29:15] <TimStarling>	we're not out of APC space, but I'll do a restart anyway
[12:29:17] <TimStarling>	it can't hurt
[12:31:25] <robla>	alright, blog updated, and now I'm off to bed
[12:31:27] <^demon>	I'd really love to find out why wfMkdirParents() keeps trying to make these directories when they clearly already exist.
[12:31:35] <^demon>	(general issue, not 1.17)
[12:31:42] <robla>	'night everyone!
[12:31:46] <everton137>	Hi, just got thie error when saving an wikipedia page: Request: POST http://pt.wikipedia.org/w/index.php?title=Pyramid_(framework_web)&action=submit, from 208.80.152.49 via sq72.wikimedia.org (squid/2.7.STABLE7) to ()
[12:31:46] <everton137>	Error: ERR_CANNOT_FORWARD, errno [No Error] at Fri, 11 Feb 2011 12:34:27 GMT
[12:32:14] <everton137>	just reporting, as said in the page. As I undertood, here is a good place
[12:32:36] <Reedy>	everton137, did it work when you tried again?
[12:32:43] <everton137>	yes
[12:32:47] <everton137>	Reedy: yes
[12:32:55] <^demon>	I can't seem to replicate locally at all or using eval.php on fenari, hm.
[12:32:56] <TimStarling>	^demon: on NFS?
[12:33:08] <^demon>	Oh, it's probably nfs.
[12:33:12] <^demon>	Segfaults again
[12:33:17] <^demon>	Ok, so that's definitely not APC
[12:33:34] <TimStarling>	it's probably a race condition
[12:33:46] <TimStarling>	NFS would make it worse by caching stat calls in the client
[12:34:15] <^demon>	Ah, I did see a php bug or two relating to it when I was trying to find a cause (I had suspected NFS, you confirmed it)
[12:34:46] <^demon>	At least with 1.17 it only throws one warning and not two.
[12:38:40] <RoanKattouw>	Alright, I'm back
[12:38:56] <RoanKattouw>	I have to leave in about an hour, though, who's gonna be around after that?
[12:39:04] <RoanKattouw>	I guess Reedy and mark ?
[12:39:13] <Reedy>	I'll be about most of the day
[12:40:38] <RoanKattouw>	Grr
[12:40:42] <^demon>	I'll be around for another 2 hours, then class. After that I'll probably sleep for awhile :p
[12:40:49] *RoanKattouw 	wants the preprocessor to fail on the cluster again so he gets his backtrace
[12:43:09] <^demon>	Reedy and I might have tracked down the constant OOM bug in DoubleWiki
[12:43:13] <^demon>	Or at least made it better :p
[12:43:14] <Reedy>	lolol
[12:43:20] <Reedy>	And RoanKattouw untagged it 1.17 :P
[12:43:26] <^demon>	Yeah, that should go 1.17
[12:43:31] <Reedy>	Bad RoanKattouw :P
[12:43:36] <RoanKattouw>	Tag it 1.17, 1.17wmf1 then
[12:43:43] <RoanKattouw>	And paste the rev link here :)
[12:43:51] <^demon>	ThomasV was doing a preg_split() inside of a for() and doing nothing with the results.
[12:43:53] <RoanKattouw>	I think I've found the Title.php timeout
[12:43:55] <^demon>	preg. split.
[12:43:57] <RoanKattouw>	Ouch
[12:44:02] <Reedy>	http://www.mediawiki.org/wiki/Special:Code/MediaWiki/81672
[12:44:04] <RoanKattouw>	ouch. ouch.
[12:44:14] <Bryan>	DoubleWiki, ouch ;)
[12:44:26] <RoanKattouw>	I've just noticed that Title.php will run Unicode cleanup regexes before enforcing the 255/512 length limit
[12:44:26] <^demon>	DoubleOuch
[12:44:35] <RoanKattouw>	Nice one, ^demon
[12:44:39] <Bryan>	the same applies to proofreadpage, btw
[12:44:41] <RoanKattouw>	Are you sure you need sleep? ;)
[12:44:51] <^demon>	Yes, O do
[12:44:52] <^demon>	*I
[12:45:09] <Reedy>	Bryan, the ouchyness?
[12:45:15] <Bryan>	yeah
[12:45:35] <^demon>	Well at least proofread page isn't spamming apache.log every few minutes :p
[12:45:36] <Reedy>	Indeed
[12:46:50] <TimStarling>	the length limit only applies to the dbkey part
[12:47:05] <TimStarling>	so you've got to split the namespace and interwiki parts off before you can apply it
[12:47:39] <RoanKattouw>	Ugh
[12:47:49] <RoanKattouw>	You're right, I can't just limit the whole thing to 512 chars
[12:48:24] <RoanKattouw>	At least not without causing subtle regressions
[12:49:06] <TimStarling>	I doubt it would be the problem anyway
[12:49:25] <RoanKattouw>	Hmm
[12:49:37] <RoanKattouw>	Well the Title.php line number you gave me corresponds to a preg call
[12:49:47] <RoanKattouw>	I haven't actually looked at the regex yet to confirm that it's scary
[12:49:49] <RoanKattouw>	I should, I guess
[12:49:52] <TimStarling>	yeah, it's probably the slowest thing in a tight loop of title construction
[12:50:07] <TimStarling>	it doesn't mean a single call to preg_replace takes 180s
[12:50:19] <RoanKattouw>	Right, yeah
[12:50:21] <RoanKattouw>	That regex is very sane
[12:53:37] <RoanKattouw>	So the callers for secureAndSplit() are Title::newFrom* and Title::makeTitleSafe
[12:54:05] <RoanKattouw>	I'm afraid grepping for those is gonna get me a lot of hits
[12:55:35] <RoanKattouw>	It does turn up fun things, though:
[12:55:40] <RoanKattouw>	517:                    $t = Title::newFromText( 'NO TITLE' );
[12:55:48] <RoanKattouw>	4242:                   $title = Title::newFromText( 'Dwimmerlaik' );
[12:56:29] <TimStarling>	tstarling@shimmer:~/src/mediawiki/branches/wmf/1.17wmf1$ grep -r --include=*.php Title:: . | wc -l
[12:56:29] <TimStarling>	998
[12:56:45] <TimStarling>	you have successfully narrowed down the problem to about a thousand callers
[12:56:56] <RoanKattouw>	hah
[12:56:58] <RoanKattouw>	Les
[12:57:02] <RoanKattouw>	s
[12:57:04] <RoanKattouw>	Hopefully
[12:57:12] <RoanKattouw>	My regex is Title::(makeTitleSafe|newFrom)
[12:57:18] <RoanKattouw>	649
[12:57:23] <RoanKattouw>	Well yay me ^^
[12:58:33] <RoanKattouw>	In 273 distinct files, that hurts
[12:59:09] <Reedy>	RoanKattouw, What time did you say you were leaving? :D
[12:59:20] <RoanKattouw>	14:50 CET
[12:59:24] <RoanKattouw>	i.e. in about 45 mins
[12:59:29] <RoanKattouw>	And I have to pack before that
[13:00:02] <TimStarling>	that means you have 4.2 seconds per caller
[13:00:07] <Reedy>	:D
[13:00:10] <RoanKattouw>	haha
[13:00:25] <RoanKattouw>	Well LanguageConverter doesn't seem to be guilty
[13:00:48] <RoanKattouw>	Unless someone's calling findVariantLink() in a tight loop, but I doubt that
[13:01:30] <TimStarling>	if something is going to take you a very long time to do, then it makes sense to first make tools
[13:02:01] <TimStarling>	the investment will pay for itself quickly enough
[13:02:02] <RoanKattouw>	I know
[13:02:32] <RoanKattouw>	At least I already have a list with matching files, and I've removed things like maintenance/* and trackback.php
[13:02:42] <RoanKattouw>	Right now I'm just flicking through it looking for outright suspicious things
[13:02:50] <RoanKattouw>	But LangConv and OAIHarvester came up blank
[13:04:06] <^demon>	Lang Converter did change in 1.17
[13:04:19] <RoanKattouw>	Yes
[13:05:02] <TimStarling>	everything makes titles
[13:05:09] <RoanKattouw>	TimStarling: How hard is it to get OOMs and max exec timeouts to report line numbers anyway?
[13:05:15] <RoanKattouw>	Yeah, it's all over the place
[13:05:29] <TimStarling>	you mean backtraces?
[13:05:35] <RoanKattouw>	Or that, yeah
[13:05:38] <RoanKattouw>	Something useful
[13:05:38] <TimStarling>	xdebug does it
[13:05:47] <RoanKattouw>	But only in the web response, right?
[13:06:00] <^demon>	No, you can make xdebug log to places.
[13:06:03] <RoanKattouw>	I'm looking for something that we can put on the cluster to diagnose these things there
[13:06:05] <RoanKattouw>	Oh cool
[13:06:10] <TimStarling>	I wrote my own extension called wmerrors, similar to how xdebug works
[13:06:11] <RoanKattouw>	And I'm not sure we want xdebug on the cluster
[13:06:19] <TimStarling>	except it just shows a pretty error page
[13:06:25] <TimStarling>	it has a comment "todo logging"
[13:06:44] <TimStarling>	if you get that extension, put logging in it where it says "todo logging"
[13:06:52] <TimStarling>	maybe a backtrace or URL or something
[13:06:55] <TimStarling>	then you'd be set
[13:06:59] <RoanKattouw>	OK
[13:07:10] <RoanKattouw>	Give me the code and I'll get cracking
[13:07:41] <TimStarling>	trunk/php/wmerrors
[13:08:02] <RoanKattouw>	Cool
[13:08:15] <TimStarling>	we can't put xdebug on the cluster, because it's too invasive
[13:08:18] <TimStarling>	it breaks things
[13:08:32] <RoanKattouw>	Gah and no one's hitting the preprocessor fatals any more :(
[13:08:41] <TimStarling>	but PHP provides an error hook which you can do what you want with, not invasive at all
[13:10:08] <RoanKattouw>	I don't need to compile my own PHP for this do I? I can just compile that .c file to a .so file somehow and add an .ini file for it, right?
[13:10:36] <TimStarling>	are you on ubuntu?
[13:10:47] <RoanKattouw>	Yes
[13:11:00] <TimStarling>	install php5-dev if you don't have it already
[13:11:06] <^demon>	phpize!
[13:11:07] <TimStarling>	that gives you a command called phpize
[13:11:27] <TimStarling>	run phpize in the source directory, that gives you a ./configure
[13:11:37] <TimStarling>	run ./configure && make && make install
[13:11:41] <TimStarling>	then you have your .so
[13:11:44] <RoanKattouw>	I like this
[13:11:52] <RoanKattouw>	That's way easier than I expected
[13:12:11] <^demon>	wikidiff2 is just as easy
[13:16:47] <TimStarling>	you're not going to like it for long
[13:17:24] <RoanKattouw>	 /usr/lib/php5/20090626+lfs/wmerrors.so doesn't appear to be a valid Zend extension
[13:17:34] <RoanKattouw>	Fun
[13:17:38] <TimStarling>	extension= not zend_extension=
[13:17:44] <RoanKattouw>	Oh
[13:17:55] <RoanKattouw>	I just copied xdebug.ini to wmerrors.ini
[13:18:41] <TimStarling>	you're used to hacking MediaWiki, we're kind to our developers
[13:18:48] <TimStarling>	we write comments and documentation
[13:18:55] <TimStarling>	PHP is not so nice
[13:19:38] <TimStarling>	still, easier and more productive than reviewing 650 Title::newFromText() callers
[13:19:43] <RoanKattouw>	Yeah
[13:20:10] <TimStarling>	have fun, good night
[13:20:12] <RoanKattouw>	Alright, I guess I'll start hacking this on the train
[13:20:13] <RoanKattouw>	Good night
[13:20:22] <RoanKattouw>	I'm gonna pack now, I'll be sort-of-around for another 20 minutes or so
[13:20:45] <Tim-away>	get the PHP source on your laptop if you don't have it
[13:21:14] <RoanKattouw_away>	Good idea
[13:21:19] <RoanKattouw_away>	Is there an Ubuntu package for it?
[13:21:26] <Tim-away>	php5?
[13:21:34] <RoanKattouw_away>	Right
[13:21:45] <Tim-away>	they also have subversion
[13:21:53] <RoanKattouw_away>	I'll use that then
[13:22:09] <Tim-away>	http://svn.php.net/repository/php/php-src/branches/PHP_5_3
[13:22:51] <RoanKattouw_away>	Thanks
[14:43:45] <Bryan>	http://www.mediawiki.org/wiki/Special:Code/MediaWiki/81966 would fix NewUserMessage
[14:44:02] <Bryan>	if somebody could merge&deploy
[17:34:47] <guillom>	TrevorParscal, about that IRC office hour on ResourceLoader: Roan suggested Monday evening for us (Monday morning for you); does 10:00am your time work for you?
[17:35:09] <TrevorParscal>	yes
[17:35:13] <TrevorParscal>	sounds great
[17:35:29] <guillom>	ok, I'll send the announcements then.
[17:36:36] <TrevorParscal>	sweetness
[17:39:02] <TrevorParscal>	hey brion
[17:39:50] <TrevorParscal>	i'm on "deployment watch" - but I will be taking a unavailable for the next 30 mins while I transition from being in bed to being at work - could you hold down the fort?
[17:40:15] <TrevorParscal>	s/taking a//
[17:41:05] <TrevorParscal>	maybe just 10 mins
[17:41:07] <TrevorParscal>	...
[17:46:37] <brion>	hey dudes
[17:47:33] <guillom>	I'm not sure there's much to watch, is there?
[17:48:21] <TrevorParscal>	ok, back
[17:48:25] <TrevorParscal>	um - to watch?
[17:48:40] <TrevorParscal>	don't know, robla asked me to be around in case something comes up
[17:49:52] <guillom>	It's been pretty calm since the deployment ended.
[17:50:14] <guillom>	There was an issue with the NewUserWelcome extension on meta, but d.omas disabled it.
[17:51:45] <TrevorParscal>	fun
[17:51:48] <TrevorParscal>	well, that's good
[17:52:14] <Reedy>	<aharoni> Hi. A few hours ago 1.17 was deployed on he.wikisource. It works mostly fine now, but there were a few RTL problems in the beginning, for example wrong placement of the "edit" link. How were they corrected - in the core code or locally in he.wikisource?
[17:52:19] <Reedy>	from #wikimedia-tech
[18:19:44] <thedj>	hmm, seems he.wikisource.o is one of the sites with broken script to change the main page tab name.
[19:06:33] <Nikerabbit>	who has seen profiling data on wmf cluster? how many queries are the per request approximately?
[19:07:20] <apergos>	me and I didn't quite get the question
[19:07:43] <apergos>	it's stil running
[19:07:47] <apergos>	http://noc.wikimedia.org/cgi-bin/report.py
[19:08:35] <Nikerabbit>	oh they are not there
[19:09:55] <Nikerabbit>	queries 8694349, total 915928
[19:10:19] <Nikerabbit>	makes about 10 q/rec
[19:10:28] <Nikerabbit>	twn has 26 q/rec
[19:10:34] <apergos>	ah I see what you were wanting
[19:10:34] <apergos>	ok
[20:21:23] <Platonids>	I'm not able to get wmerrors_cb() run :(
[20:21:31] <Platonids>	hi Roan
[20:22:21] <RoanKattouw>	How are you testing it?
[20:22:52] <Platonids>	php -r 'asdjas();'
[20:22:54] <Platonids>	I found it
[20:23:02] <Platonids>	xdebug is not as nice as us
[20:23:06] <Platonids>	it is removing wmerrors
[20:23:28] <Platonids>	now, why is it going through the retun branch?
[20:24:10] <Platonids>	why does it skip if the sapi is not apache?
[20:24:29] <Platonids>	well, it doesn't have much sense dumping all that html
[20:25:24] <Platonids>	still, I get a blank page in apache
[20:27:49] <Platonids>	well, I'll look it later
[20:28:03] <Platonids>	probably not until tomorrow
[20:29:38] <^demon>	I'm  hitting that getFullText() on non-object in Article in 1.17
[20:30:11] <Bryan>	hey Roan & demon
[20:31:02] <Bryan>	we had to disable NewUserMessage earlier today
[20:31:17] <Bryan>	I committed a fix, but there wasn't anybody to sync it: http://www.mediawiki.org/wiki/Special:Code/MediaWiki/81966
[20:32:26] <^demon>	I'm still rather sleep deprived, but I'll look
[20:34:57] <Krinkle>	newusermessage-template should be a message for content language, right ?
[20:35:57] <thedj>	Krinkle: you wrote the new gallery right ?
[20:36:06] <Krinkle>	I did ?
[20:36:20] <thedj>	or was that diebuche ?
[20:36:36] <^demon>	Yeah, should definitely be wfMsgForContent()
[20:36:38] <Krinkle>	maybe, but not me.
[20:37:35] <Krinkle>	newusermessage-template may be translated (which is fine, since a non-WMF German wiki wouldn't have to change the default if the /de version defaults to Template:Wilkommen.
[20:38:32] <guillom>	RoanKattouw, around?
[20:38:43] <RoanKattouw>	Yeah
[20:39:10] <RoanKattouw>	Platonids: You need to configure the error file too
[20:39:19] <guillom>	RoanKattouw, do you have any idea why I would see a version of the wikieditor with fancy fonts & NTOC? http://img151.imageshack.us/img151/686/editingtalkmainpagewiki.png
[20:39:26] <guillom>	(this is a 1.17 wiki FWIW)
[20:39:29] <RoanKattouw>	Which wiki?
[20:39:34] <guillom>	en.wikinews
[20:39:40] <RoanKattouw>	That's bad
[20:39:42] <guillom>	Other people I asked don't see this
[20:39:55] <guillom>	I don't have any gadget enabled, and I've shift-refreshed multiple times.
[20:40:07] <RoanKattouw>	Preferences sticking around from the time NTOC was live?
[20:40:19] <RoanKattouw>	When you go to your enws preferences, do you see NTOC?
[20:40:39] <guillom>	 	 Enable enhanced editing toolbar
[20:40:40] <guillom>	 	 Enable dialogs for inserting links, tables and more
[20:40:47] <guillom>	is all I see in "beta" features
[20:40:52] <RoanKattouw>	So no
[20:41:08] <RoanKattouw>	Try saving your preferences (maybe change something too) and going back to the edit page
[20:41:40] <guillom>	RoanKattouw, right, that solved it.
[20:41:56] <guillom>	Thanks.
[20:50:24] <thedj>	hmm, was tiff handler updated on the 1.17 wikis ?
[20:51:18] <Bryan>	thedj: it should be
[20:51:26] <Bryan>	but note that commons is still 1.16
[20:51:52] <thedj>	shouldn't matter for foreignapi
[20:51:53] <thedj>	http://simple.wikipedia.org/wiki/File:The_Earth_seen_from_Apollo_17.tiff
[20:52:11] <thedj>	still uses the old style image descriptor below the iimage...
[20:54:04] <thedj>	it seems that none of the tiffs from commons work...
[20:54:08] *thedj 	ponders...
[20:56:35] <thedj>	are you admin on test2 bryan ?
[20:56:42] <Bryan>	yes
[20:56:52] <thedj>	can you try uploading a tiff locally, see if that works.
[20:57:04] <Krinkle>	that tiff doesn't work on Commons either though
[20:57:26] <Shirley>	"log me in globally" checkbox is nice.
[20:57:50] <Shirley>	guillom: Should Meta still have a sitenotice?
[20:58:03] <guillom>	Shirley, no, I removed it, didn't I?
[20:58:17] <Bryan>	thedj: http://test2.wikipedia.org/wiki/File:030a_Shrady_portrait.tiff
[20:58:50] <Bryan>	thedj: but all foreign stuff is broken
[20:58:56] <thedj>	yeah seems so.
[20:59:04] <Bryan>	RoanKattouw: which version do the image scalers run?
[20:59:16] <Shirley>	I saw it earlier. Probably cache, then.
[20:59:30] <guillom>	ok
[20:59:34] *guillom 	--> bed.
[20:59:38] <thedj>	i think the metadata version changed for tiff.
[21:01:00] <RoanKattouw>	Bryan: Of what, ImageMagick?
[21:01:04] <Krinkle>	http://commons.wikimedia.org/wiki/File:030a_Shrady_portrait.tiff
[21:01:05] <Shirley>	RoanKattouw: Regarding that prefs bug above, does that mean that user_properties needs to be cleared out on wikis where users might have NTOC stored?
[21:01:14] <Krinkle>	http://de.wiktionary.org/wiki/Datei:030a_Shrady_portrait.tiff
[21:01:20] <Bryan>	RoanKattouw: MW
[21:01:21] <Krinkle>	works again now on Commons and foreign
[21:01:31] <RoanKattouw>	I should look through the u_p tables for legacy prefs I guess
[21:05:51] <jorm>	herp derp derp.
[21:08:29] *Shirley 	spots a jorm.
[21:08:36] <^demon>	Nobody's called in yet?
[21:08:55] *jorm 	calls it in.
[21:10:32] <Reedy>	What're we calling in for?
[21:10:54] <jorm>	i dunno.
[21:11:00] <jorm>	pack it up, pack it in, let me begin
[21:12:14] <^demon>	priyanka, mark, rob are on.
[21:12:27] <^demon>	Reedy: scrum for launch :p
[21:12:41] <Reedy>	2003?
[21:12:56] *^demon 	nods
[21:13:21] <robla>	https://bugzilla.wikimedia.org/showdependencytree.cgi?id=26611
[21:13:29] <robla>	^ list of 1.17 blockers
[21:13:59] <Shirley>	https://bugzilla.wikimedia.org/showdependencytree.cgi?id=26611&hide_resolved=1
[21:14:12] <Reedy>	lol
[21:15:47] <RoanKattouw>	I'm not calling in if you guys don't mind; it's Friday night and I'm not 'really' here
[21:16:29] <RoanKattouw>	I also didn't bring my mic
[21:16:46] <Shirley>	jorm: Has LQT development restarted yet?
[21:17:06] <jorm>	i'm not sure what you mean by "development".
[21:17:30] <Shirley>	Sorry, I meant that I thought Andrew was going to start writing new code.
[21:17:45] <jorm>	he has been.  he's traveling now, however.
[21:17:52] <Shirley>	He was in London for Wikipedia Day and then was going out to SF to start working on code, or something like that.
[21:17:57] <Shirley>	Ah, I haven't seen any commits.
[21:18:24] <jorm>	yes.  he came out and we worked mostly on writing up an object model and a series of architecture diagrams.
[21:18:36] <jorm>	he wrote some code but i don't believe he has committed it.
[21:18:39] <Shirley>	Ah, right. I saw he was working with Neil on some of that.
[21:18:42] <Shirley>	Long road ahead, I guess.
[21:19:09] <Shirley>	I never finished getting through your design doc.
[21:19:12] <neilk_>	Shirley: I gave jorm & andrew some ideas about how to refactor their design
[21:19:20] <Shirley>	It was mostly nitpicky stuff that I can file bugs against later, though.
[21:19:26] <neilk_>	Shirley: not much participation on my end
[21:19:32] <Shirley>	neilk_: Yeah, I somewhat skim your log occasionally.
[21:19:36] <jorm>	design review stuff.
[21:19:39] <Shirley>	It puts up in my feed.
[21:19:51] <Shirley>	s/puts/pops/
[21:20:32] <neilk_>	I haven't been good about updating that lately
[21:20:34] <jorm>	i've been time splitting with a couple other things the past week or so so i've not gotten around to working on the design stuff again until today, actually.
[21:20:52] <jorm>	and i'm going to have to backburner it during the first part of next week again as well.
[21:20:59] <Shirley>	You told me which four or five projects you were working on at some point, but I've forgotten.
[21:21:06] <^demon>	hexmode: List of things locally modified in 1.17wmf1: http://dpaste.org/cCfu/
[21:21:07] <Shirley>	TradeTrack is mostly tabled, right? LQT. What else?
[21:21:44] <jorm>	AFT and controversial content right now.  also a lot of gender gap stuff lately; mostly off-list emails.
[21:21:56] <hexmode>	^demon: tyvm
[21:22:18] <Shirley>	jorm: I didn't realize anyone was looking at controversial content (from the tech side).
[21:22:24] <jorm>	yes.
[21:22:27] <Shirley>	Image blurring?
[21:22:49] <jorm>	i am preparing a series of behavior and design mockups for the board to examine.
[21:22:51] <jorm>	no.
[21:23:03] <jorm>	blurring isn't practical.
[21:23:11] <Shirley>	Aryeh wrote up some docs on that (blurring based on categories) at some point, though I disagreed with his implementation.
[21:23:23] <RoanKattouw>	Not blurring but replacing with a static image, probably
[21:23:34] <Shirley>	I was more asking: are you focused on controversial content such as images or text?
[21:23:48] <jorm>	images and media.
[21:23:55] <Shirley>	Or those dirty .oggs we have. ;-)
[21:24:27] <Shirley>	jorm: Well, if you haven't read it, this might give you some ideas: http://www.mediawiki.org/wiki/User:Simetrical/Censorship
[21:24:34] <jorm>	i've read it.
[21:24:38] <Shirley>	Ah, cool.
[21:24:45] <Shirley>	I think I had Aryeh convinced that blurring wasn't going to work.
[21:25:13] <RoanKattouw>	Erik pulled me into a meeting about the censorship thing, and I linked him to Aryeh's write-up
[21:25:59] <robla>	code review queue: http://www.mediawiki.org/wiki/MediaWiki_roadmap/1.17/Revision_report
[21:26:36] <Shirley>	But his whole implementation seemed rather prone to breakage.
[21:26:39] <Shirley>	I think that's what I wrote on the talk page.
[21:26:58] <Shirley>	I thought WMF was still waiting on Robert whathisname to write a report about controversial content.
[21:27:01] <Shirley>	Did that finally finish?
[21:27:12] <RoanKattouw>	Yes, I think so
[21:27:15] <RoanKattouw>	Harris
[21:28:00] <jorm>	yes.
[21:28:19] <jorm>	i have a meeting next week about all this where i show what i've got.
[21:28:28] <Shirley>	Hmm, I think I missed the final report.
[21:28:35] <Shirley>	Whole thing seemed rather silly.
[21:29:27] <Shirley>	The software/social side definitely needs improvement, though.
[21:29:49] <Shirley>	Cary used to deal with the complaints of "I was looking in the category for images of red dresses and oh my god."
[21:30:36] <hexmode>	^demon: out of mem errors reported or discussed anywhere?
[21:31:03] <^demon>	Just on IRC at like 5am.
[21:31:38] <^demon>	It's an issue with DoubleWiki (used on wikisources).
[21:31:41] <hexmode>	hrm... and my proxy/logger has been wonky.  but lemme check
[21:31:45] <pdhanda>	TrevorParscal: https://bugzilla.wikimedia.org/showdependencytree.cgi?id=26611
[21:31:52] <RoanKattouw>	Cary used to deal with a lot of things
[21:32:12] <^demon>	hexmode: The whole extension is a mess so don't dig too deeply into it (and it's an issue on 1.16 too, so it's not deployment-specific)
[21:32:27] <^demon>	We're pretty sure removing an unused preg_split() in a for will help a lot ;-)
[21:32:47] <hexmode>	if it were RTL issue and it is on fawiki, then it isn't 1.17, right?
[21:33:14] <Shirley>	jorm: If you have notes at some point about controversial content, I'd be interested.
[21:33:17] <robla>	hi TrevorParscal and RoanKattouw: we briefly touched on the RL issues in this list: https://bugzilla.wikimedia.org/showdependencytree.cgi?id=26611&hide_resolved=1
[21:33:37] <TrevorParscal>	cool
[21:33:38] <RoanKattouw>	No
[21:33:41] <RoanKattouw>	27302 is not a blocker
[21:33:45] <^demon>	robla: I'm also going on ~4 hours of sleep right now, so I doubt I'll get much else done today. I'll be around for the weekend though.
[21:33:46] <robla>	...but really, we're relying on you guys to make the call
[21:33:51] <hexmode>	^demon: excellent.... now, to figure out what DoubleWiki is ;)
[21:33:54] <RoanKattouw>	27323 is not a blocker and not even new in 1.17
[21:34:05] <TrevorParscal>	RoanKattouw: 27328 i can solve pretty easily
[21:34:07] <RoanKattouw>	27328 sounds nasty
[21:34:07] <robla>	^demon: thanks
[21:34:09] <RoanKattouw>	OK
[21:34:35] <robla>	I told hexmode to err on the side of making things blockers so that we could make the triage decision together
[21:34:39] <RoanKattouw>	27330 may be a RL or a CSS issue, either way I think Trevor can look at it
[21:35:09] <robla>	hexmode: can unblock stuff based on Roan's comments ^
[21:35:26] <Reedy>	I've started moving some to
[21:35:31] <Reedy>	!b 27339
[21:35:31] --elephant--	https://bugzilla.wikimedia.org/show_bug.cgi?id=27339
[21:36:11] <hexmode>	robla: right
[21:36:41] <^demon>	I got through 2 1/2 Bond movies during deployment last night :)
[21:37:08] <Shirley>	hexmode: Hampton e-mailed me about the status of the mobile rewrite. Gotta ask his permission before I can share the e-mail with you, but it'll probably help you with a good number of bugs.
[21:37:26] <hexmode>	^demon: how to make deployment worth it.
[21:37:26] <Shirley>	Or at least give you a better bigger picture about those.
[21:37:39] <hexmode>	Shirley: it would help since I need to talk to him
[21:37:47] <hexmode>	was thinking of emailing him
[21:38:03] <hexmode>	I should just do it
[21:38:06] <Shirley>	I e-mailed wikitech-l and CC'd him.
[21:38:13] <Shirley>	Yeah, I'd just go ahead and e-mail him.
[21:38:24] <Shirley>	Read what I wrote to wikitech-l first, though.
[21:38:31] <^demon>	The Spy Who Loved Me, Thunderball and about halfway through The Man With the Golden Gun before I got tired of it :
[21:39:42] <jorm>	wow.  all over the place with that, then.
[21:40:08] <^demon>	Well I picked 3 of my more favorite ones :)
[21:40:12] <jorm>	I sometimes play games as to which bond movies are the most racist.
[21:40:23] <jorm>	thunderball is very good.
[21:40:34] <^demon>	You Only Live Twice is pretty bad :p
[21:40:38] <jorm>	yes, but.
[21:40:40] <^demon>	(in terms of racism)
[21:40:56] <jorm>	"Diamonds are Forever" has a scene where a black woman is in a cage and then is transformed into a gorilla.
[21:41:22] <Shirley>	[[List of racist scenes in James Bond movies]]
[21:41:50] <^demon>	[[Diamonds Are Forever (film)]] omits the gorilla bit :(
[21:42:21] <jorm>	i have edits on that article.
[21:42:51] <^demon>	You should edit it
[21:42:54] <^demon>	To mention the gorilla.
[21:43:35] <jorm>	now i have the theme song stuck in my head.
[21:44:00] <^demon>	I had The Man With the Golden Gun in my head for a few hours after watching it.
[21:44:21] <jorm>	james bond vs. saruman.
[21:44:56] <Bryan>	saruman
[21:45:01] <^demon>	In a fight or as characters?
[21:47:25] <^demon>	I just realized I haven't eaten since breakfast ~10 hours ago. I'm going to go fix that.
[21:47:49] <RoanKattouw>	No food between 7am and 5pm, that can't be good
[21:48:22] <^demon|away>	Well I slept from like 11am-3:30 or so.
[21:48:36] <RoanKattouw>	^
[21:48:40] <RoanKattouw>	Ah
[21:49:33] <jorm>	demon: both scaramanga and saruman were played by christopher lee.
[21:56:25] <Krinkle>	http://aboutthailandliving.com/files/2009/09/scaramanga-bond.jpg :)
[21:56:38] <Krinkle>	saruman vs. bond would be funny though
[21:56:54] <Krinkle>	or saruman vs. scaramanga for that matter.
[21:59:53] <hexmode>	oldapps.com -- essential Windows testing site
[22:00:07] <^demon|away>	jorm: You should really get in to video game development again.
[22:00:18] <jorm>	why do you say that?
[22:00:23] <^demon|away>	You should make a fighting game where you pitch actors in their various roles against each other
[22:00:29] <^demon|away>	Bond v. Scaramanga.
[22:00:33] <jorm>	oh.  hah.
[22:00:41] <^demon|away>	Elrond v. Agent Smith
[22:01:06] <jorm>	gandalf v. magneto
[22:01:09] <^demon|away>	s/Bond/Saruman/
[22:01:12] <RoanKattouw>	hahah
[22:01:48] <brion_>	FINISH HIM!
[22:02:31] <^demon|away>	Tyler Durden v. Sinbad?
[22:02:43] <brion_>	wikipedia says sinbad is dead
[22:03:15] <^demon|away>	I meant Sinbad as in Brad Pitt in the '03 movie.
[22:03:57] <jorm>	no, you want him vs. his character in "Snatch"
[22:04:06] <jorm>	*that* would be a fight.
[22:04:08] <brion_>	hmm, do cartons really count?
[22:04:14] <brion_>	*cartoons
[22:04:16] <brion_>	i can't type toda
[22:04:17] <brion_>	y
[22:04:33] <^demon|away>	Obviousl
[22:04:34] <^demon|away>	y
[22:04:34] <^demon|away>	not
[22:04:54] <^demon|away>	Ok, food for real this time. I got distracted.
[22:31:49] <neilk_>	what's a good VNC client for modern macs?
[22:33:00] <neilk_>	it seems that Remote Desktop kind of ate the VNC client market on mac?
[22:34:03] <brion>	i mostly just use the built-in screen sharing viewer yeah
[22:34:14] <brion>	chicken of the vnc is the classic one for osx but it's not maintained any more
[22:34:25] <brion>	there's also one by the author of screenrecycler which is more up to date
[22:34:29] <neilk_>	yes, I'm seeing lots of stuff that ceases dev in 2006 or so
[22:34:29] <brion>	but i haven't used it much
[22:34:38] <jorm>	hrm. lemme see what i'm using.
[22:35:10] <jorm>	i'm not using anything!
[22:35:11] <brion>	http://www.jinx.de/JollysFastVNC.html <- not open source tho
[22:37:10] <neilk_>	hm, it seems that they used to include a free VNC client in 10.5, and now it's payware
[22:37:18] <neilk_>	way to kill the free software AND make a profit
[22:40:23] <brion>	neilk_, the one that ships with osx lives in /System/Library/CoreServices/Screen Sharing.app
[22:40:42] <brion>	it just doesn't have a lot of options on it :D
[22:41:09] <brion>	[apparently you used to be able to actually hack it to enable extra features, it's a pared-down version of Apple Remote Desktop sharing client which is apparently fancier]
[22:41:21] <brion>	[but now those old tricks don't work...]
[22:42:28] <neilk_>	brion: how odd, I used Spotlight to look for 'screen sharing' and came up empty, but there it is
[22:42:43] <neilk_>	brion: I figured they discontinued it in 10.6
[22:43:26] <brion>	yeah i suspect stuff in System is excluded from search results by default, as it's mostly not stuff you tend to launch yourself
[22:43:28] <neilk_>	I guess Spotlight is trying to save me from myself, no system files.
[22:43:49] <brion>	but it's still there -- the bonjour service discovery will let you launch it from Finder's networks view
[22:43:57] <brion>	or you can launch it directly if you know where it is ;)
[22:44:52] <Krinkle>	 /System, /Libary and /Users/../Library is excluded from a default search on a Mac
[22:45:12] <Krinkle>	if you navigate to /Library and choose to "search in this folder only" it works though (ie. it is indexed)
[22:46:07] <neilk_>	sweet. Well, I can theoretically connect to the local mac mini. If I knew the pw :)
[22:46:10] <neilk_>	thanks
[22:46:22] <brion>	hehe
[22:47:25] <Krinkle>	If you connect from mac to mac there's no need to install a VNC server though or a client for that matter.
[22:48:19] <Krinkle>	System Preferneces > Sharing > Screen (tick), then in Finder > Sidebar: Network > Click the Mac Mini and click something like "view screen" it then asks " Do you want to ask the local user with a dialog or log in manually"
[22:49:02] <Krinkle>	Macs talk to eachother over Bonjour within a LAN.