[00:44:03] <sumanah>	AaronSchulz: ^demon Tim-away do you have a few min for commit access review?
[00:44:18] *AaronSchulz 	hopes ^demon has time
[00:45:19] <sumanah>	there's basically 3 things to look at
[00:48:00] <sumanah>	AaronSchulz: Tim-away ^demon I'm on x2003
[00:50:16] <sumanah>	AaronSchulz: so you're saying you don't have time?
[00:50:28] <AaronSchulz>	not right atm
[00:50:52] <sumanah>	Got it.
[00:52:20] <sumanah>	brion: could I borrow you for 10 minutes, or are you on something urgent?
[00:52:30] <brion>	sumanah: what you need?
[00:52:52] <sumanah>	brion: seems Tim, Chad, & Aaron are all busy, and I could use your opinion on some commit access requests.
[00:53:36] <brion>	sure, i'll see what i can do
[00:53:40] <sumanah>	brion: ok, will pm
[00:53:42] <brion>	may not have seen recent patches tho
[00:53:50] <sumanah>	brion: that's ok.
[01:12:14] <werdna>	aaaaargh
[01:12:15] <werdna>	I hate IE
[01:12:15] <werdna>	http://werdn.us/~andrew/ie7_layout.png
[01:12:42] <werdna>	kaldari: that's just two divs within an a tag turned into a button with jQuery UI. Any idea why it's doing that?
[01:13:06] <sumanah>	kaldari: https://www.mediawiki.org/wiki/How_to_become_a_MediaWiki_hacker and https://www.mediawiki.org/wiki/How_to_become_a_MediaWiki_hacker -- more later
[01:13:08] <kaldari>	looking...
[01:13:16] <sumanah>	kaldari: I mean https://www.mediawiki.org/wiki/Category:Tutorials for the 2nd link
[01:13:26] <werdna>	to save you typing out the URL, it's http://prototype.wikimedia.org/wikis/acw/index.php/Special:ArticleCreationLanding/fadasfd
[01:14:03] <kaldari>	werdna: looks like it's not clearing a float
[01:14:47] <kaldari>	werdna: you could try putting both in the same div and using <br/> and span tags for the formatting
[01:15:17] <werdna>	lemme see
[01:15:26] <kaldari>	if you can't get the 2nd div to clear the float
[01:16:23] <kaldari>	sumanah: thanks!
[01:17:47] <werdna>	Nikerabbit: around?
[01:17:58] <werdna>	pgehres has a question about inventing fake language variants
[01:19:08] <pgehres>	werdna: theoretically they are not "fake"
[01:19:21] <pgehres>	;-)
[01:19:23] <werdna>	kaldari: hmm, same result. What's weirder: The <br/> works fine until .button() is called
[01:19:29] <werdna>	:)
[01:48:29] <kaldari>	Did werdna leave?
[01:49:32] <kaldari>	the office I meran
[01:49:34] <kaldari>	mean
[01:51:43] <Eloquence>	kaldari, yep
[14:05:10] <petan|wk>	can someone give my sysop on test2?
[14:05:25] <petan|wk>	I would like to fix the layout a bit so that it's more friendly to people who want to test
[14:06:19] <Reedy>	ya
[14:06:24] <Reedy>	username?
[14:07:32] <Reedy>	done
[14:09:19] <petan|wk>	thanks
[14:10:06] <petan|wk>	eh I can't accept revisions as sysop?
[14:10:19] <petan|wk>	btw is it using simple wiki config or custom?
[14:10:42] <petan|wk>	I guess it's custom
[14:11:38] <petan|wk>	done
[15:10:41] <Reedy>	hexmode: gadgets fixed
[15:12:21] <chrismcmahon>	Reedy: I just joined, can you say the particular issue with gadgets?
[15:12:51] <Reedy>	it's from last night ;)
[15:13:11] <Reedy>	updating the gadgets definition page didn't cause the entries in memcached to be updated
[15:13:33] <chrismcmahon>	ah, I remember that about the definitions
[15:14:11] <Reedy>	replace false with a true, in another extension, fixed
[15:19:23] <hexmode>	w00!
[15:19:45] <sumanah>	hi hexmode! how's it going?
[15:19:46] <hexmode>	Reedy: admin for https://test2.wikipedia.org/wiki/Special:UserRights/Quedel
[15:20:24] <Reedy>	done
[15:21:10] <hexmode>	sumanah: Just had my morning constitutional so I feel good :)
[15:21:53] <sumanah>	hexmode: what are you working on?  I'm doing some event followups, event planning, and git documentation/publicity
[15:22:54] <hexmode>	sumanah: I'm gonna take Tim's suggestion and try to find out if Ops can give me a proxy for that gadget
[15:23:10] <sumanah>	what/
[15:23:15] <hexmode>	and, from there, continue getting some people to test gadgets
[15:23:15] <sumanah>	I don't know what you're talking about, sorry
[15:23:40] <hexmode>	the bugzilla gadget that brought bugzilla to its knees :)
[15:23:47] <sumanah>	Aha
[15:23:58] <hexmode>	I think we have a good fix already
[15:23:59] <hexmode>	but
[15:24:12] <hexmode>	The proxy is an added protection
[15:24:18] <sumanah>	hexmode: well, you know what is happening today, right?
[15:24:29] <hexmode>	the deployments?
[15:25:01] *hexmode 	goes to check email in case he missed something
[15:26:53] <sumanah>	yes.
[15:36:16] <sumanah>	hexmode: yes, in 7 and a half hours, the next deployment
[16:05:26] <sumanah>	Thanks for the updates to https://www.mediawiki.org/wiki/Git/Workflow ^demon .
[16:05:46] <^demon>	You're welcome.
[16:06:39] <RoanKattouw_away>	^demon: git review -a ??!?
[16:06:49] <RoanKattouw_away>	Did you mean -s ?
[16:06:51] <^demon>	Whoops.
[16:06:52] <^demon>	Yeah
[16:07:28] <RoanKattouw_away>	Hmm, also the workflow page doesn't mention the working branches paradigm
[16:07:56] <RoanKattouw_away>	i.e. it should explicitly discourage people from basing unrelated changes off each other
[16:08:04] <RoanKattouw_away>	(if you do this, gerrit will also always warn you)
[16:08:26] <^demon>	Please feel free to add :p
[16:08:34] <^demon>	That was a very rough start.
[16:08:47] <RoanKattouw_away>	I'd love to expand on that when I have time
[16:08:57] <RoanKattouw_away>	And hopefully eventually replace [[labsconsole:Git]] with that too
[16:09:17] <^demon>	It's a wiki! Anyone can edit!
[16:09:22] <^demon>	Ok, time to find something to eat.
[16:09:24] <RoanKattouw_away>	Yup
[16:12:15] <sumanah>	RoanKattouw_away: I'm trying to turn a bunch of the info from https://www.mediawiki.org/wiki/Talk:Git/Conversion/issues into useful sections on the various Git conversion & workflow pages.  But that will not help with the stuff you are fixing.
[16:12:39] <RoanKattouw_away>	True
[16:12:59] <RoanKattouw_away>	My stuff is partially on [[labsconsole:Git-review]] and partially on [[labsconsole:User:Catrope/Branching guide]]
[16:13:05] <RoanKattouw_away>	But I'll integrate it myself some time soon
[16:13:06] <sumanah>	:/ scattering
[16:13:07] <sumanah>	ok
[16:13:43] <RoanKattouw_away>	I probably won't have all that much time for work this week though, I don't think I need to tell you why
[16:15:13] <sumanah>	RoanKattouw_away: :-) I presume that you don't have that much packing to do, and have done most of what's necessary already.
[16:15:29] <RoanKattouw_away>	One would think so
[16:16:03] <RoanKattouw_away>	I'm almost done with the bureaucracy, and I'm budgeting a day for packing (Friday) and a day for hanging out with old high school friends that live on the other side of the country (Saturday)
[16:16:11] <sumanah>	At this particular moment I wish I were in your shoes, about to embark on this adventure
[16:16:40] <RoanKattouw_away>	Well you've done long-distance moves before, just not internationally
[16:16:46] <sumanah>	Not the same.
[16:17:12] <sumanah>	You get to do it alone.
[16:17:20] <RoanKattouw_away>	Oh, that's right
[16:17:22] <RoanKattouw_away>	I forgot
[16:17:43] <sumanah>	I've never enjoyed that particular freedom. It makes me wistful.
[16:19:24] <RoanKattouw_away>	You must have been alone when you moved to go to college, right? (Although that was like <100 mi probably)
[16:19:51] <sumanah>	Yeah.
[16:23:35] <hashar>	RoanKattouw_away: you are doing the right choice anyway
[16:23:59] <hashar>	it is better to do those crazy things now while you are young than regrets later on when you are old :-D
[16:25:24] <sumanah>	RoanKattouw_away: ^demon|away: what is the short and not-unwieldy name for "the group of people who have +2 power in gerrit and thus can merge code into a particular branch"?
[16:26:22] <hashar>	ops
[16:26:42] <sumanah>	No, because for an extension, that might be the extension author.
[16:27:11] <^demon|away>	"project owner" is what gerrit calls it.
[16:27:57] <^demon|away>	Well, that's the auto-group defined based on the owner. It's kinda weird.
[16:28:18] <sumanah>	So, ^demon|away, each of the 20ish people that we're giving pull power for the WMF master branch will be "project owners"?
[16:28:26] <sumanah>	I'm looking for terminology to use in the workflow doc.
[16:28:46] <sumanah>	And is this defined per branch?
[16:28:55] <^demon|away>	Well, that's the "mediawiki" group. They're the owners for the mediawiki/core.git project.
[16:29:12] <^demon|away>	It can be per-branch.
[16:29:32] <sumanah>	ok, so the term in gerrit (or git?) is "project"?
[16:29:49] <^demon|away>	A repository is a "project" in gerrit, yeah
[16:59:52] <hashar>	I am really a messy guy
[17:00:09] <hashar>	I have git repo everywhere on gallium, my laptop and my desktop computer :/
[18:15:39] <robla>	yo yo yo
[18:16:33] <Nikerabbit>	yo yo yo
[18:16:37] <robla>	raindrift just walked in the door
[18:16:39] <Nikerabbit>	sorry I'm late
[18:16:47] <robla>	so he's going to be later  :)
[18:17:27] <robla>	Nikerabbit: I think all of your fixmes are done, right
[18:17:28] <robla>	?
[18:17:45] <robla>	(at least the 1.19 ones)
[18:18:03] <Nikerabbit>	hmm
[18:18:07] <Nikerabbit>	I hope so
[18:18:07] <robla>	http://www.mediawiki.org/wiki/MediaWiki_1.19/Revision_report <- from yesterday
[18:18:12] <robla>	hexmode: could you rerun?
[18:18:20] <Nikerabbit>	the logging stuff needs work but that shouldn't block 1.19
[18:18:34] <hexmode>	robla: here
[18:19:59] <robla>	Nikerabbit: could you take a look at http://www.mediawiki.org/wiki/Special:Code/MediaWiki/110909
[18:20:38] <Nikerabbit>	robla: yup that's what I was talking about
[18:20:49] <Nikerabbit>	that needs fixing, but can be done after 1.19
[18:21:12] <Nikerabbit>	just don't publicize genderized logs in the release notes
[18:21:14] <robla>	ah, ok.....do you want to mark "ok", and tag "todo"?
[18:21:45] <hexmode>	robla: should I check my logs from yesterday to see if I need to tag some revs w/ people's names? I didn't tag then b/c of lag.
[18:21:48] <Nemo_bis>	Nikerabbit, the notes still mention them
[18:21:58] <robla>	hexmode: coudl you rerun the rev report?
[18:22:18] <Nikerabbit>	Nemo_bis: where?
[18:22:23] <hexmode>	I just did, so not much is gonna change
[18:22:36] <robla>	I saw at least a coupel that will
[18:22:42] <Nemo_bis>	Nikerabbit, https://www.mediawiki.org/wiki/MediaWiki_1.19 perhaps
[18:22:53] <Nemo_bis>	I don't really know if that's up-to-date or not
[18:22:59] <Nikerabbit>	can't you fi xthat?
[18:23:00] <robla>	hexmode: did you save?
[18:23:17] <Nemo_bis>	Nikerabbit, no because as I said I don't know what is there new and what not
[18:23:27] <hexmode>	robla: that is all automated but I'll do a manual one here
[18:23:28] <sumanah>	(sorry I am late)
[18:24:27] <sumanah>	0 FIXMEs in the rev report, that's great
[18:25:01] <Nemo_bis>	Nikerabbit, except that you once told me "it will be in 1.19" about some features
[18:26:28] <sumanah>	robla: is it worth some time today making https://www.mediawiki.org/wiki/MediaWiki_1.19 more readable and clarifying which of those items is actually important?
[18:26:30] <robla>	hexmode: I see what's going on.  replag on toolserver s3 is 19hrs
[18:26:45] <hexmode>	:(
[18:26:58] <robla>	sumanah: yeah, that would be helpful
[18:27:23] <Nemo_bis>	or true even
[18:27:23] <sumanah>	robla: Chris McMahon? Or is he 100% on other release issues?
[18:27:32] <sumanah>	Nemo_bis: take a whack at it?
[18:27:49] <robla>	chrismcmahon|brb: is this something you could help out on?
[18:27:52] <Nemo_bis>	sumanah, just before you entered I said that I'm not sure what's true and what not
[18:28:15] <sumanah>	Nemo_bis: oh, sorry, missed that.
[18:28:23] <robla>	I've got a couple minutes before I disappear
[18:28:40] <sumanah>	ok, robla, anything in that list seem obviously suspect?
[18:28:46] <robla>	raindrift: I think there's an IE8 blocker or two that would be helpful to get some help on
[18:29:10] <raindrift>	sure.  got bug numbera?
[18:29:16] <raindrift>	err, numbers
[18:29:24] <robla>	https://bugzilla.wikimedia.org/show_bug.cgi?id=34409
[18:29:51] <robla>	I guess that's the only one
[18:30:07] <robla>	I'm worried about this one: https://bugzilla.wikimedia.org/show_bug.cgi?id=34147
[18:30:23] <robla>	RoanKattouw_away: are you truly _away?
[18:30:49] <robla>	oh, that's right...we need to test that one on test2.  that's also a chrismcmahon|brb thing
[18:31:01] <Nikerabbit>	murr
[18:31:41] <robla>	sumanah: come to think of it, chris is trying to get through his test plan at least once before we release
[18:31:48] <robla>	s/release/deploy/
[18:32:22] <Nikerabbit>	anything else for me?
[18:32:46] <sumanah>	Nikerabbit: anything false or misleading on https://www.mediawiki.org/wiki/MediaWiki_1.19 ?
[18:32:58] <sumanah>	Nikerabbit: or big things that are missing
[18:33:33] <Nikerabbit>	sumanah: I already removed the genderized logs which did not make into 1.19
[18:34:34] <robla>	Nikerabbit: we might use some help on something we're discussing on #wikimedia-tech
[18:34:37] <Nikerabbit>	don't really remember what else has happened, should go trough the release notes
[18:34:43] <robla>	corrupt localization files
[18:35:04] <robla>	see my comments there from around 20 min ago
[18:35:45] <Nikerabbit>	um
[18:35:48] <robla>	(10:14:27 AM) robla: AaronSchulz: here's the problem I'm talking about: (09:20:01 AM) Reedy: robla: https://upload.wikimedia.org/wikisource/mr/thumb/e/ef/%E0%A4%AB%E0%A5%81%E0%A4%B2%E0%A4%BE%E0%A4%9A%E0%A4%BE_%E0%A4%AA%E0%A5%8D%E0%A4%B0%E0%A4%AF%E0%A5%8B%E0%A4%97.djvu/page3-1200px-%E0%A4%AB%E0%A5%81%E0%A4%B2%E0%A4%BE%E0%A4%9A%E0%A4%BE_%E0%A4%AA%E0%A5%8D%E0%A4%B0%E0%A4%AF%E0%A5%8B%E0%A4%97.djvu.jpg
[18:35:49] <robla>	(10:14:31 AM) nagios-wm: RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 7.701 seconds
[18:35:49] <robla>	(10:15:09 AM) robla: (09:22:02 AM) Reedy: robla: tim logged a bug for these errors somewhere, but can't find it
[18:35:49] <robla>	(10:15:09 AM) robla: (09:22:12 AM) Reedy: robla: LU isn't actually running properly either (permission errors)
[18:36:10] <robla>	AaronSchulz: is this something that Nikerabbit might be able to help with?
[18:36:20] <robla>	or do we just need someone from ops?
[18:36:21] <AaronSchulz>	not likely
[18:36:51] <AaronSchulz>	For now, brute force rebuilding is the best option
[18:36:54] <robla>	Nikerabbit: I guess the main thing is finishing up on code review
[18:36:56] *AaronSchulz 	looks around fenari
[18:37:23] <RoanKattouw>	robla: Back now
[18:37:25] <hexmode>	robla: #34147 I *think* is fixed... Saibo has found some other problems, but nothing with that one
[18:37:34] <hexmode>	and I haven't seen it
[18:37:48] *robla 	is supposed to be in another meeting now
[18:38:00] <robla>	RoanKattouw: I was going to ask you about 34147
[18:38:06] <Nikerabbit>	mmm
[18:38:16] <robla>	although the cdb corruption problem above is also kinda up your alley
[18:38:34] <RoanKattouw>	Oh, LU has CDB problems?
[18:38:37] <RoanKattouw>	Yeah I should look at that
[18:38:49] <RoanKattouw>	I have little time for work this week but I can spare a few hours to debug that
[18:39:02] <RoanKattouw>	34147 should be considered fixed until proven otherwise, IMO
[18:39:35] <hexmode>	RoanKattouw: at this point, I'm with you :)
[18:49:02] <AaronSchulz>	RoanKattouw: just tail exception.log ;)
[18:49:08] <RoanKattouw>	heh
[18:49:18] <AaronSchulz>	eval.php doesn't give me trouble using the /home files
[18:49:29] *RoanKattouw 	takes out ye olde hour log
[18:49:35] <werdna>	:)
[18:49:39] <werdna>	ye olde...
[18:50:11] <Reedy>	Tim logged https://bugzilla.wikimedia.org/show_bug.cgi?id=33409
[18:50:18] <RoanKattouw>	by which I mean I haven't touched it in like a week
[18:50:19] <Reedy>	"Transient CDB read/write failures"
[18:50:22] <RoanKattouw>	Oh that
[18:51:13] <RoanKattouw>	2012-02-15 18:51:03 srv224 trwiki: /w/thumb.php?f=Ata_raki.jpg&width=100   Exception from line 169 of /usr/local/apache/common-local/php-1.18/includes/Cdb_PHP.php: CdbReader_PHP::read: read from CDB file failed, file may be corrupted
[18:51:15] <RoanKattouw>	Ouch
[18:51:20] <chrismcmahon>	sumanah: robla I've been assuming that https://www.mediawiki.org/wiki/MediaWiki_1.19  is true (for some value of "true").  At least I haven't found anything yet that makes me think otherwise.
[18:51:26] <Reedy>	Indeed
[18:51:35] <Reedy>	RoanKattouw: but it might all be somewhat bogus
[18:51:41] <RoanKattouw>	I'm gonna read the LU log first
[18:51:46] <Reedy>	With various servers seemingly having wrong permissions on l10n files etc
[18:51:49] <RoanKattouw>	And unless I find something there I'm just gonna rerun LU
[18:51:52] <AaronSchulz>	RoanKattouw: all just scalars
[18:51:57] <Reedy>	Rerun lu manually
[18:51:59] <Reedy>	loads of spammage
[18:52:13] *AaronSchulz 	looks at the node groups
[18:52:24] <RoanKattouw>	srv190: rsync: failed to set times on "/usr/local/apache/common-local/php-1.18/cache/l10n/.": Operation not permitted (1)
[18:52:26] <RoanKattouw>	wtf
[18:52:29] <RoanKattouw>	I fixed those perms dammit
[18:52:36] <Reedy>	indeed
[18:52:45] <Reedy>	Tim made some changes earlier this week too
[18:52:52] <RoanKattouw>	Failed to write to file '/home/wikipedia/common/php-1.19/cache/l10n/l10nupdate-ab.cache'
[18:52:56] <RoanKattouw>	That's why 1.19 broke
[18:53:13] <RoanKattouw>	but .. .the files are there? wtf?
[18:53:27] <sumanah>	chrismcmahon: ok, then, let's go forward on the assumption that it's all true, and ask, which of them are important? which will have largeish effects on end users, or, secondarily, on administrators of other MediaWiki installations?
[18:54:00] <sumanah>	chrismcmahon: no need to address this *today* but this week it would be good to make it look a little more like https://www.mediawiki.org/wiki/MediaWiki_1.18, more explanatory prose if that's called for
[18:54:24] <chrismcmahon>	sumanah: thanks for the example, got it.
[18:54:41] <AaronSchulz>	RoanKattouw: I think the l10 dir has to be made manually
[18:54:50] <AaronSchulz>	and the perms suggest it was
[18:54:53] <RoanKattouw>	Right
[18:55:02] *AaronSchulz 	vaguely remembers this with 1.18
[18:55:19] <AaronSchulz>	interesting that the one for 1.18 is owned by LU now
[18:55:20] <RoanKattouw>	It's correctly chowned and chmodded on fenari in /usr/local/apache
[18:55:30] <AaronSchulz>	but meh
[18:55:39] <RoanKattouw>	Or, well, wait, it's owned by mwdeploy
[18:56:11] <RoanKattouw>	hmm
[18:57:47] *AaronSchulz 	wonders why just scalars give the error
[18:59:49] <RoanKattouw>	Yeah it's mostly scalers
[19:00:08] <AaronSchulz>	drwxrwxr-x 2 l10nupdate l10nupdate   20480 2012-02-09 02:05 l10n
[19:00:12] <RoanKattouw>	For non-scalers only the dir itself is bad
[19:00:18] <RoanKattouw>	On scalers all the individual files are bad as well
[19:00:26] <RoanKattouw>	I think it has to be mwdeploy:mwdeploy these days
[19:00:38] <RoanKattouw>	But I changed this perms setup so many times I'm not even sure anymore
[19:01:12] <Reedy>	Yeah
[19:01:16] <werdna>	I love the unix permissions model
[19:01:20] <werdna>	it makes so much sense
[19:01:37] *AaronSchulz 	looks for the Werdna permissions model RfC
[19:03:35] <Reedy>	Computers suck
[19:04:48] <RoanKattouw>	OK, I fixed some perms and sync-l10nupdate now works for both 1.18 and 1.19
[19:04:50] <RoanKattouw>	Now gonna rerun LU
[19:08:10] <Reedy>	Still 4 hours till deployment
[19:09:16] <Reedy>	Part of me wants to just transfer mw.o
[19:10:08] *robla 	looks at deployment calendar to see if we can get away with that
[19:10:33] <RoanKattouw>	You may not want to move stuff while I'm actively running LU
[19:10:35] <RoanKattouw>	It takes a while
[19:10:51] <RoanKattouw>	I may want to reengineer things to use a buffer in /tmp
[19:10:55] <AaronSchulz>	RoanKattouw: did you skip the svn part?
[19:11:05] <AaronSchulz>	we need an option for that if one isn't there ;)
[19:11:05] <RoanKattouw>	No, SVN is already on /var/lib
[19:11:08] <RoanKattouw>	That's not the problem
[19:11:39] <RoanKattouw>	The problem is that it rewrites the file after each extension, so it's writing ~ 1/2 n^2 bytes where n = the final file size
[19:11:57] <RoanKattouw>	That in itself would be fine if we did it on /tmp , but since it does it in-place it happens on /home
[19:13:04] <robla>	Reedy: I'd want to check in with binasher before we do anything, and I don't see him in the usual places (including his chair)
[19:14:37] <Reedy>	Sure
[19:58:32] <pgehres>	Nikerabbit: are you working on stuff for the 1.19 deploy today? if not, do you have a few minutes to chat?
[19:59:12] <RoanKattouw>	raindrift: Trevor tells me you have RL problems?
[20:01:16] <Nikerabbit>	pgehres: wassup?
[20:02:06] <pgehres>	I'm working on doing country-based localizations for fundraising and trying to determine the best way to vary a message based on the country
[20:03:01] <pgehres>	my current best idea is to use language variants like pt-br and en-gb, but don't know the consequences and hassle(s) of using those
[20:04:23] <Nikerabbit>	it's not supported in translatewiki.net par few exceptions
[20:04:35] <pgehres>	yeah, we would not need or want these to go through TWN
[20:05:04] <pgehres>	we have all of the translations, I just want to make them work using wfMsg and similar functions
[20:05:35] <Nikerabbit>	they don't have automatic fallbacks
[20:05:39] <pgehres>	this would not be on the main cluster (just the payments cluster)
[20:06:22] <pgehres>	If I was reading correctly, I would need to define the languages and I could set fallbacks there, yes?
[20:08:01] <Nikerabbit>	as far as I know l10n cache is shared between all wikis
[20:08:11] <RoanKattouw>	Yes
[20:08:18] <pgehres>	we have a physically separate cluster
[20:08:32] <pgehres>	for payments
[20:09:57] <raindrift>	RoanKattouw:  i thought so, but now i can't reproduce the bug at all.  see: https://bugzilla.wikimedia.org/show_bug.cgi?id=34409
[20:10:18] <raindrift>	RoanKattouw:  so, probably, don't worry about it until we can make it break again.  :)
[20:10:37] <RoanKattouw>	Oh
[20:10:49] <RoanKattouw>	Yeah see the thing is, that can happen due to cache-induced Frankensteinism
[20:10:55] <RoanKattouw>	I've seen that on test2 before
[20:11:00] <RoanKattouw>	Or, on test, don't remember
[20:11:11] <RoanKattouw>	It fixes itself once your client-side cache expires (5 mins)
[20:12:31] <RoanKattouw>	I'll comment on the bug
[20:12:51] <Nikerabbit>	hmhmhm
[20:14:11] <Nikerabbit>	another option would be to chain messages, use foo-countrycode if it exists, otherwise just foo
[20:14:52] <pgehres>	yeah, i was trying to avoid making a ton of new messages in that i18n file that have to be ignored by TWN
[20:17:58] <pgehres>	it looks like wfMessageFallback() could be useful for this, but we are still running 1.17 on payments
[20:19:01] <Nikerabbit>	uh
[20:34:06] <Reedy>	AaronSchulz: unlink fails with a warning on non existants
[20:34:45] <AaronSchulz>	yeah
[20:34:49] <AaronSchulz>	what of it?
[20:35:18] <Reedy>	so you'd display the warning
[20:35:32] <Reedy>	before an exception though
[20:35:37] <Reedy>	probably don't overly care
[20:35:49] <AaronSchulz>	your saying if the handle exists but the file doesn't?
[20:36:04] <AaronSchulz>	how would that happen? Unless some code unlinked before but kept the handle open
[20:36:44] <Reedy>	strange things happen
[20:36:46] <Reedy>	;)
[20:37:15] <AaronSchulz>	you mean the cosmic rays?
[20:37:22] <Reedy>	fricken laser beams
[20:37:46] <RoanKattouw>	The unlink is for the tmp file
[20:37:57] <RoanKattouw>	But it can be reached before a tmp file is even created, can't it?
[20:38:30] <AaronSchulz>	$this->handle = fopen( $this->tmpFileName, 'wb' );
[20:38:38] <RoanKattouw>	Oh
[20:38:40] <RoanKattouw>	hah
[20:38:43] <RoanKattouw>	OK, then it's fine
[20:38:52] <AaronSchulz>	'w' 	Open for writing only; place the file pointer at the beginning of the file and truncate the file to zero length. If the file does not exist, attempt to create it.
[20:41:22] <RoanKattouw>	Yeah, destructive write mode
[20:41:36] <RoanKattouw>	I hadn't realized this was only for the writer class, and that the writer class uses a tmp file for everything
[20:42:44] <AaronSchulz>	all good cdb wrappers use temp files and then rename
[20:43:05] <AaronSchulz>	otherwise if it fails in the middle...boom!
[20:43:20] <RoanKattouw>	Sure
[20:43:30] <AaronSchulz>	plus having half-written stuff in any case for half a sec
[21:03:56] <Reedy>	Has the way the maintenance classes set themselves up changed?
[21:04:14] <Reedy>	It seems like they aren't loading DefaultSettings...
[21:05:21] <Reedy>	Based on some of the undefined variable noise when running mergeMessageFileList.php
[22:11:50] <raindrift>	RoanKattouw:  I have more information on bug 34409.  I think it might be a timing issue where mw.user.options.set is used before mw.user is necessarily loaded.
[22:11:57] <RoanKattouw>	Yeah I just read that
[22:12:02] <RoanKattouw>	I can't reproduce on my IE8 install
[22:12:12] <RoanKattouw>	And I'm pretty sure that race condition is supposed to be impossible
[22:12:17] <raindrift>	yeah.  me neither, but aaron can reproduce it reliably.
[22:12:35] <RoanKattouw>	Is this specific to IE8? Specific to [[Wine]]?
[22:13:11] <raindrift>	it seems, at least for Aaron, to be specific to [[Wine]] and ie8, yeah.
[22:13:21] <RoanKattouw>	It's definitely the race condition you describe, I knew that. My explanation on the bug explains why RL even allows that race condition to occur
[22:13:31] <RoanKattouw>	(missing dependency info)
[22:13:45] <raindrift>	right.  as in, with proper dependency info in place, this should be prevented.  makes sense.
[22:14:21] <raindrift>	AaronSchulz was doing a hard refresh, but he doesn't seem to be around at the moment so I can't ask him for more info.
[22:14:37] <raindrift>	(i think he's in a meeting or something)
[22:15:25] <RoanKattouw>	So yeah in my startup module response, I get this ["user.options","1293840000",["mediawiki.user"],"private"]
[22:15:38] <RoanKattouw>	Also ask him if it goes away in debug mode
[22:15:42] <raindrift>	totally.  seems right.
[22:15:49] <raindrift>	mmm.  good point.  i'll ask him that.
[22:16:13] <RoanKattouw>	And what he gets for user.options at http://bits.wikimedia.org/test2.wikipedia.org/load.php?debug=false&lang=en&modules=startup&only=scripts&skin=vector&* . I pasted what I get above
[22:16:28] <raindrift>	fwiw, the page is really different in ie8 vs chrome, say.  the page i get on chrome is using mw.loader.implement(), whereas the ie8 page calls mw.user.config.set() directly.
[22:16:30] <RoanKattouw>	(In IE you need to download the file, open it in e.g. Wordpad, and Ctrl+F for 'user.options')
[22:16:35] <RoanKattouw>	WTF!
[22:16:38] <RoanKattouw>	That's wrong
[22:16:46] <raindrift>	aha.
[22:17:00] <RoanKattouw>	Then it's not a dependency problem at all, disregard everything I said
[22:17:06] <raindrift>	okay.  :)
[22:17:07] <RoanKattouw>	It's the *page* that's bad
[22:17:20] <raindrift>	so, yeah.  line 340 of the [[wine]] article is the bit in question.
[22:18:05] <raindrift>	there's a <script> inserted there that isn't present in the corresponding line when loaded in Chrome (which is line 385 in that case)
[22:18:16] *RoanKattouw 	wonders how the hell this even worked in 1.18
[22:18:32] <RoanKattouw>	I looked at the source of a 1.18 page
[22:18:51] <RoanKattouw>	Sort of curious about how that would not have the same race condition
[22:19:01] <raindrift>	it may.  it's clearly hard to reproduce.
[22:19:04] <RoanKattouw>	We're probably with a 1.18 page calling into 1.19's loader library
[22:19:08] <RoanKattouw>	*probably dealing with
[22:19:30] <raindrift>	oh, interesting???  you mean we're getting the page from cache on test2?
[22:19:42] <RoanKattouw>	Somehow that page is coming from a cache somewhere
[22:19:55] <RoanKattouw>	Maybe that's why I can't reproduce, I'm in Europe
[22:20:19] <RoanKattouw>	But wait no you said you saw a difference between Chrome and IE too
[22:20:28] <RoanKattouw>	Maybe IE is caching the page HTML more aggressively than you think
[22:20:30] <raindrift>	when you load http://test2.wikipedia.org/wiki/Wine, what does line 340 look like?
[22:20:34] <RoanKattouw>	Or... crap this is probably IMS stuff
[22:20:41] <raindrift>	IMS?
[22:21:01] <raindrift>	fwiw, i tried spoofing an ie8 useragent from Chrome, and it didn't change anything on the Chrome side.
[22:21:08] <RoanKattouw>	Line 340:                 </div>
[22:21:33] <raindrift>	heh.  i don't have that on either of mine.
[22:21:36] <RoanKattouw>	That was as an anon, though, with wget
[22:21:36] <raindrift>	are you logged in?
[22:21:47] <raindrift>	okay.  i'm doing all of this anon, so that's right.
[22:21:49] <RoanKattouw>	I can look as a logged-in user but getting line numbers is a pain
[22:22:07] <raindrift>	do you not have the ie8 debugger?
[22:22:22] <RoanKattouw>	Oh, I was looking in Firefox
[22:23:08] <RoanKattouw>	What the f...
[22:23:13] <RoanKattouw>	I get the 1.18 version too
[22:23:21] <RoanKattouw>	But in Firefox I do not
[22:23:30] <RoanKattouw>	Oh, wait, maybe it's http vs https
[22:23:36] <raindrift>	how can you tell it's 1.18?
[22:23:56] <RoanKattouw>	Nope same thing
[22:24:13] <RoanKattouw>	Line 340 is an mw.user.options.set call that's not wrapped in an implement() call
[22:24:16] <RoanKattouw>	That's what 1.18 produces
[22:24:26] <RoanKattouw>	1.19 wraps it and puts it nearer the top of the document
[22:25:04] <raindrift>	aha.  i see.
[22:25:26] <RoanKattouw>	The IE response was served by m52
[22:25:27] <RoanKattouw>	*mw52
[22:25:33] <raindrift>	so, yeah, that's what I get in ie8.  I just loaded it, and here is the HTML source I get: http://pastebin.com/Eu9TvT3m
[22:25:34] <RoanKattouw>	My Firefox response was served by srv247
[22:25:47] <RoanKattouw>	Yeah
[22:26:17] <RoanKattouw>	That's what my IE8 sees too
[22:26:34] <raindrift>	in this case it was srv197 for me.
[22:26:37] <RoanKattouw>	huh, except that yours came from srv197
[22:27:01] <RoanKattouw>	If I Ctrl+F5 in Firefox, the server number changes
[22:27:10] <raindrift>	the chrome response i'm looking at was from srv282
[22:27:13] <RoanKattouw>	But I get the new version
[22:28:13] <RoanKattouw>	However in IE I can't get it to change no matter what
[22:28:18] <RoanKattouw>	It keeps showing me the cached version
[22:28:24] <RoanKattouw>	This makes me think IE's cache purging is broken
[22:28:37] <RoanKattouw>	It probably sends an If-Modified-Since request and gets a 304 response
[22:29:09] <raindrift>	yeah.  same here.  can't get it to change from srv197.
[22:29:50] <RoanKattouw>	I have a feeling this will go away if we edit the page, which is exactly why we shouldn't do that, to preserve our test case
[22:30:32] <raindrift>	indeed
[22:33:02] <raindrift>	it seems like ie8's sending of cache-control is broken, maybe.  it does seem to be reloading the page, and i'm quite sure i've deleted the disk cache, but perhaps it's still coming from the same squid?
[22:36:17] <RoanKattouw>	OK I had to tcpdump this
[22:36:21] <RoanKattouw>	Let's see
[22:36:32] <RoanKattouw>	IE is sending Cache-Control: no-cache
[22:37:22] <RoanKattouw>	I can't read the responses cause they're gzipped
[22:37:32] <RoanKattouw>	But it's definitely grabbing everything with Cache-Control: no-cache
[22:37:45] <raindrift>	that is also what Chrome sends, so that's right.
[22:37:55] <Reedy>	RoanKattouw: for hewikisource, didn't we want Amir to be around?
[22:38:01] <Reedy>	aggghhhh
[22:38:03] <Reedy>	robla even
[22:38:07] <raindrift>	what about Pragma?  does ie send that?
[22:38:21] <RoanKattouw>	No
[22:38:41] <robla>	Reedy: oh, hmm...there is that
[22:38:57] <Reedy>	i just noticed he doesn't seem to be around on irc
[22:39:07] <raindrift>	Chrome does - Pragma: no-cache
[22:39:14] <raindrift>	not sure if it matters here, though.  it shouldn't.
[22:39:32] <raindrift>	Is there an X-Cache header in the response?
[22:40:04] <RoanKattouw>	Yes
[22:40:08] <robla>	is AryehGregor actually about?
[22:40:12] <RoanKattouw>	All HITs
[22:40:20] <raindrift>	aha.  i get all MISSes.
[22:40:33] <RoanKattouw>	Wait a damn minute
[22:40:35] <RoanKattouw>	HIT?!?
[22:40:41] <RoanKattouw>	But, but, no-cache!
[22:40:44] <raindrift>	yeah.  that's wrong.
[22:41:14] <robla>	you will have the cached version, and you will like it
[22:41:34] <RoanKattouw>	To be fair when I use no-cache locally I get two MISSes and one HIT
[22:44:16] <RoanKattouw>	Huh no wait my IE response has one MISS
[22:44:20] <RoanKattouw>	But it's also a HIT, ... ?
[22:44:24] <RoanKattouw>	X-Cache: MISS from knsq27.knams.wikimedia.org
[22:44:26] <RoanKattouw>	X-Cache-Lookup: HIT from knsq27.knams.wikimedia.org:80
[22:44:33] <RoanKattouw>	Dear Squid, what does that even mean?
[22:46:17] <AaronSchulz>	I've had that thought myself
[22:48:59] <RoanKattouw>	Looks like Pragma: no-cache might make the difference yeah
[22:49:18] <raindrift>	that's weird.  it should only mater for http/1.0
[22:49:25] <RoanKattouw>	Anyway, it's not entirely clear to me why a 1.18 response /would/ break
[22:49:37] <RoanKattouw>	It doesn't in my IE
[22:49:54] <raindrift>	yeah.  mine neither.  Aaron's back.  let me see what he's getting.
[22:49:57] <RoanKattouw>	AaronSchulz: Does it break with ?debug=true too?
[22:50:50] <RoanKattouw>	Does http vs https make a difference?
[22:52:05] <AaronSchulz>	no; no
[22:52:33] <raindrift>	so, no bug with debug=true, and http/https makes no difference (that is, bug still exists with https, debug=false)
[22:52:54] <raindrift>	also, he's getting the same cached page i am.  loaded from srv197 in .163 seconds
[22:52:59] <RoanKattouw>	debug=true fixing it makes sense
[22:53:16] <raindrift>	the fact that that makes sense is terrible.
[22:53:33] <raindrift>	(but yeah, makes sense)
[22:55:00] <raindrift>	and yeah, if i load it with some made up query string (?foo=foo), i get the 1.19 page from srv280
[22:55:57] <raindrift>	ooh!  i got it to make the error for me again.  nifty!
[22:56:34] <RoanKattouw>	Wait, from a 1.19 response?
[22:56:47] <raindrift>	no, when i reloaded the cached response
[22:56:51] <RoanKattouw>	OK
[22:57:19] <RoanKattouw>	Hrmph, I hate that I can't debug this in debug mode
[22:58:19] <RoanKattouw>	Oh, wait a damn minute!
[22:58:42] <RoanKattouw>	Oh my God I think I just realized what caused this
[22:59:02] <RoanKattouw>	I *did* make a stupidly backwards-incompatible change in 1.19
[22:59:07] <RoanKattouw>	in mw.loader.load()
[23:00:11] <RoanKattouw>	OK I know how to fix this now
[23:01:55] <RoanKattouw>	This is going to suck slightly because I'd probably make the current 1.19 responses incompatible with my fix, hmm
[23:03:42] <raindrift>	cool!  i'm curious what the eventual cause is going to be.
[23:04:08] <raindrift>	oh, and fwiw, this is the error in question.  grabbed it while it happened to be occuring for me: https://skitch.com/raindrift/8b9ia/sfo-virtual1-vista-ie8
[23:04:51] <RoanKattouw>	Yeah
[23:04:56] <RoanKattouw>	I think I've cracked this case already
[23:05:13] <RoanKattouw>	I just need to figure out how I'm gonna fix b/c with 1.18 while not breaking b/c with the previously existing 1.19
[23:05:14] <RoanKattouw>	otoh
[23:05:21] <RoanKattouw>	We didn't actually deploy that to wikis that matter
[23:06:00] <raindrift>	yeah.  i guess the other option is global cache invalidation.  ;)
[23:06:19] <RoanKattouw>	Well
[23:06:28] <RoanKattouw>	We can afford to have minor breakage on test and test2
[23:06:34] <RoanKattouw>	Those are the only wikis with a 1.19 past
[23:06:49] <raindrift>	totally.
[23:06:49] <RoanKattouw>	So if robla is willing to delay the deployment until I'm done with my fix ..
[23:08:37] <raindrift>	he seems to be in the middle of something, but i'll check with him when he's done.
[23:09:07] <raindrift>	but, afaik, this bug is a 1.19 blocker, so deployment is delayed until it's fixed already.
[23:09:07] <AaronSchulz>	nothing is going anywhere atm
[23:09:49] <RoanKattouw>	OK
[23:09:51] <RoanKattouw>	Thanks
[23:12:30] <raindrift>	just spoke with robla.  he says that's fine, let's delay until it's done???  we're delayed for some other reasons now anyway.
[23:16:20] <Reedy>	Someone ping me when we can deploy? Ta
[23:18:19] <RoanKattouw>	5 mins now
[23:18:38] <RoanKattouw>	robla, AaronSchulz: Any objections to deploying with $wgResourceLoaderExperimentalAsyncLoading = true; ?
[23:19:51] <RoanKattouw>	It's kind of misnamed, because it's 1) not quite as experimental as it sounds and 2) actually nontrivial to reverse because it changes Squid-cached HTML
[23:19:54] <robla>	I have no idea what that does
[23:20:13] <RoanKattouw>	heh
[23:20:26] <RoanKattouw>	It loads certain scripts slightly earlier
[23:20:27] <robla>	only if we can deploy with $wgRmMinusRf
[23:20:34] <RoanKattouw>	I put it behind a feature flag for paranoia at the time
[23:20:57] <RoanKattouw>	But that was probably overkill
[23:21:19] <robla>	can we deploy that to test2 first?
[23:22:04] <RoanKattouw>	Sure
[23:22:18] <RoanKattouw>	That's probably best
[23:22:23] <RoanKattouw>	It takes a while to really deploy for anons
[23:22:27] <RoanKattouw>	same for un-deploy, unfortunately :(
[23:22:51] <AaronSchulz>	like 5min?
[23:23:00] <RoanKattouw>	No
[23:23:07] <RoanKattouw>	It changes the HTML as output by the server
[23:23:12] <RoanKattouw>	Which for anons is cached in Squid
[23:23:16] <AaronSchulz>	yeah, heh
[23:23:22] <RoanKattouw>	I really wish I'd considered this at the time
[23:23:31] <RoanKattouw>	But what I'm fixing now was even more moronic
[23:23:52] <RoanKattouw>	I added a parameter that you had to set to true to keep the old behavior. Of course the default value of this parameter is false
[23:26:27] <TrevorParscal>	RoanKattouw: i'm in the opinion that flipping that switch should be tested a bit more than a typical config change because we don't know all the ways it might interact with all extensions yet
[23:26:39] <TrevorParscal>	i expect it will be fine, but suggest using extra cuation
[23:26:47] <RoanKattouw>	True
[23:26:53] <RoanKattouw>	probably worth being cautions
[23:26:56] <RoanKattouw>	*cautious
[23:26:59] <robla>	well....
[23:27:31] <robla>	is this what we've been using on test2 already?
[23:27:44] <robla>	by virtue of this being default on?
[23:27:44] <RoanKattouw>	No
[23:28:25] <TrevorParscal>	so test2 set that param to true then?
[23:28:27] <robla>	I want to deploy in whatever state we've been running on test2, I think
[23:28:47] <raindrift>	so, it sounds like we've been seeing this new behavior on pages for which the cache is stale (and the parameter is therefore false), but not on pages that are newly rendered (where it's set to true, for "old behavior")
[23:29:58] <RoanKattouw>	robla: It's off by default
[23:30:02] <RoanKattouw>	No, you haven't
[23:30:05] <RoanKattouw>	That's different new behavior
[23:30:22] <raindrift>	oh, okay.
[23:30:43] <robla>	so RoanKattouw, do you have the fix for the JS bug you were working on?
[23:30:53] <robla>	(or is this it or something)
[23:31:45] <RoanKattouw>	Yes
[23:31:48] <RoanKattouw>	I'm about to commit it
[23:32:30] <RoanKattouw>	Hmph
[23:32:31] <AaronSchulz>	RoanKattouw: so things that did stuff like <input onChange=XObj.y()'>... and assumed that XObj was loaded already (which sucks), would that still work?
[23:32:34] <RoanKattouw>	What's the 1.19 branch point rev?
[23:33:03] <RoanKattouw>	That would probably have broken with the introduction of RL. onChange attributes are very 2005
[23:34:03] <AaronSchulz>	it would have race problems, but might work now wouldn't it?
[23:34:23] <RoanKattouw>	It would be unaffected by this
[23:34:40] <RoanKattouw>	If you're using onFoo attributes in your HTML, you're Doing It Wrong
[23:35:10] <AaronSchulz>	hmm, I don't think I quite understood trevor's explanation of the var then
[23:37:08] <robla>	!1.19
[23:37:08] --elephant--	I don't know anything about "1.19".
[23:37:45] <robla>	http://www.mediawiki.org/wiki/Special:Code/MediaWiki/110996 <- RoanKattouw
[23:38:26] <RoanKattouw>	Thanks
[23:38:35] <RoanKattouw>	Fortunately that's before the rev I just fixme'd
[23:40:11] <RoanKattouw>	robla: https://www.mediawiki.org/wiki/Special:Code/MediaWiki/111597 , will merge to 1.19wmf1
[23:41:24] <robla>	xlnt!
[23:42:28] <RoanKattouw>	hmph, I have to wait for the checkout to finish
[23:42:38] <RoanKattouw>	Reedy: Maybe you could merge r111597 into 1.19wmf1?
[23:43:21] <Reedy>	mmmm
[23:43:34] <RoanKattouw>	I was gonna do it but svn checkout is slow
[23:44:54] <Reedy>	comitting
[23:45:28] <RoanKattouw>	Thanks
[23:45:30] <Reedy>	111599
[23:47:19] <robla>	!r111599
[23:47:19] --elephant--	I don't know anything about "r111599".
[23:47:28] <robla>	!r 111599
[23:47:28] --elephant--	http://www.mediawiki.org/wiki/Special:Code/MediaWiki/111599
[23:47:57] <RoanKattouw>	Damn I'm glad I found this one
[23:48:02] <RoanKattouw>	This could've been nasty
[23:48:38] <RoanKattouw>	We're lucky someone managed to observe it
[23:48:58] <robla>	Reedy: deploying it to test2 now?
[23:49:52] <robla>	there was a pretty fragile chain that chrismcmahon will be making less fragile  :)
[23:50:32] <RoanKattouw>	What chain is this?
[23:51:03] <robla>	well, the cool part was Guillaume getting the notices in place pointing people to a central page
[23:51:09] <chrismcmahon>	robla: not sure I'm that much of  a superhero :)
[23:51:21] <RoanKattouw>	Oh, you mean the bug reporting chain?
[23:51:40] <robla>	yeah
[23:51:48] <robla>	chrismcmahon: you will be for 1.20 though
[23:51:58] <robla>	https://www.mediawiki.org/wiki/Special:Code/MediaWiki/111597
[23:52:08] <robla>	TrevorParscal: review? ^
[23:53:22] <Reedy>	done
[23:54:26] <TrevorParscal>	looks good to me