[00:01:31] TimStarling: Per robla's request, here's instructions on how to lower the RL startup module cache timeout before deployment. You want to set $wgResourceLoaderMaxage['unversioned'] = array( 'server' => N, 'client' => N ); for the wiki(s) you're deploying 1.19 to (suggested values for N are 30 or 60, default is 300). You'll need to make the config change at least 15 minutes prior to the... [00:01:33] ...switchover to be sure it's been picked up everywhere, and of course you're encouraged to revert it once 1.19 is running fine [00:02:04] ok [00:02:11] when are we switching? [00:03:18] I don't know [00:03:22] That's up to RobLa [00:03:39] uh? what just happened [00:03:40] I'm in and out of the office (I ran an errand just now and I'm leaving again in 45) so he said I should hand this off to you [00:03:50] |<-- Krinkle has left freenode (Read error: Connection reset by peer) [00:04:04] well, I hear you again so it's fine now :) [00:05:13] so....here's where we're at [00:05:29] *very* cursory testing shows that our IRC output is not perfect, but better than before [00:06:03] there are some cases where the switch statement is incomplete, and we need to make sure that we're at least returning the old text [00:07:14] robla: the same text is not required per se. as long as the used text is located under the same message-key and uses the same values for $1 and $2, etc. still. actual phrasing may be different or even localized (which is the case in 1.18 as well). [00:07:48] sorry: s/old text/what was previously in 1.19/ [00:08:07] actually, I took a closer look at it, and it looks pretty logical now [00:08:46] I hadn't noticed the bit where we set $text=null, then do a null check at the end, and set default text. that seems pretty good [00:09:28] might be a few more parameters that need switching around [00:11:02] I wonder if we could've just copied the messages across, and fixed the parameters to match.. [00:11:53] Reedy: possibly, but this means we don't have a big Message*.php automation project on our hands [00:12:08] true [00:14:03] TimStarling: could you do a code review of what we've got so far? [00:14:29] easiest thing to do would be svn diff -r112043 in the logging directory and RecentChange.php [00:14:44] I think that picks up everything without picking up the noise [00:15:07] sure [00:15:16] are you guys in the middle of a deploy? [00:15:23] Ryan_Lane: will be very soon [00:15:27] ok... [00:15:29] (hopefully) [00:15:38] fixing one last blocker [00:15:42] there's a git change I just pushed that could possible cause issues with svn [00:15:55] want me to revert for now? [00:16:24] or test really quick, and revert if necessary? [00:16:25] Ryan_Lane: you can check if you're right about it screwing up svn [00:16:31] ok [00:16:40] it's been tested in labs... [00:16:51] but, I don't want to screw up your deploy, just in case [00:17:31] testing it. if you guys run into any issues, let me know and I'll roll back the change [00:17:46] Ryan_Lane: k...thanks for the heads up! [00:17:54] bah. seems I'm missing a cherry-pick necessary for this change to go through. [00:17:56] *Ryan_Lane fixes [00:21:12] it's funny to see $actionComment left in there, what's it for? [00:21:21] not problematic, just funny [00:21:36] *robla looks [00:23:06] oh, right...that's the one you were saying was only used for IRC [00:23:13] heh...oh well [00:23:23] yeah [00:23:35] I think it's passed to a hook, but presumably any hook that uses it will be broken also [00:23:45] for the same reasons that IRC was broken [00:24:24] anyway I assume this will all be ripped out before 1.20 and replaced with something nicer and less backwards-compatible, so it's no big deal [00:28:13] *ImSexyAndIKnowIt sodomizes Reedy [00:28:18] anyway, it looks fine in overview [00:28:37] TIMStarling prepare your anus [00:28:59] TimStarling: dare we deploy and iterate on it? [00:29:49] ok [00:30:06] TrevorParscal: ^ [00:30:30] I'll reduce the startup module expiry [00:30:30] Reedy: wtf? [00:30:38] Apparently you has ops [00:30:40] -ChanServ- You are not authorized to perform this operation. [00:30:51] ah [00:30:57] I may need to revert this change. [00:31:08] Roan, Casey and Guillom aren't about atm [00:31:15] TrevorParscal I will rip your balls off if you kick me [00:31:16] of course, puppet hasn't finished running???. [00:31:37] ImSexyAndIKnowIt identify yourself, or be kicked [00:31:58] I am yo mama that's who my identity is [00:32:21] no one in here has ops in the channel? [00:32:36] Hahaha [00:32:46] Trevor does also [00:32:49] chanserv does, let's just get them to do it [00:32:54] Hence pinging him [00:32:59] TrevorParscal: http://wikitech.wikimedia.org/view/Freenode#Becoming_ops_in_a_channel [00:33:06] trevor doesn't know shit about ops [00:33:07] http://wikitech.wikimedia.org/view/Freenode#Ban_someone_from_a_channel [00:33:20] /ban *!*@mobile-166-147-*.http://mycingular.net/?? [00:33:24] err [00:33:33] /mode +b *!*@mobile-166-147-*.http://mycingular.net/?? [00:33:35] can I have +voitA please? [00:33:49] TimStarling denied [00:33:50] damn it, why does puppet take so long to run? [00:34:01] Ask the puppetmaster [00:34:05] /chanserv flags #wikimedia-dev TimStarling +voitA [00:34:13] none of us has ops [00:34:28] I thought Trevor did [00:34:54] Oh shit [00:35:01] Here it comes [00:35:09] TrevorParscal is listed as a founder in /chanserv info [00:35:10] *Ryan_Lane rolls his eyes [00:35:23] GIVE IT TO ME [00:35:28] HARD AND RAW [00:35:41] fail [00:35:52] You failed [00:35:55] You're failing more [00:35:58] Kill yourself [00:36:05] /mode +b *!*@mobile-166-147-*.http://mycingular.net/?? [00:36:15] I think he's kind of funny [00:36:23] Reedy come at me bro [00:36:24] He really is a crappy troll [00:36:26] which is more amusing [00:36:35] in a 10-year-old troll kinda way [00:36:41] Tim would you like to go out for a beer sometime [00:36:57] of course if Trevor would just paste in that /chanserv flags command he would be kickbanned in a few seconds [00:37:38] He can't even set a ban correctly how can he deputize you [00:37:43] heh. my stupid client pastes this stuff incorrectly [00:38:07] boring trolls make me sad [00:38:23] It's kinda funny [00:38:44] Last week I had robh convinced I was on the ops team [00:39:01] well, I'm glad my change didn't break anything. it's finished. let me know if you guys have any issues with svn [00:39:14] I changed out all the LDAP libraries... [00:39:35] Fancy dishing out some more rights while you're at it? [00:40:03] oh, ^demon's gitweb change went in with this too [00:40:08] Reedy: which rights? [00:40:17] Roan giving more people op in here [00:40:28] oh [00:40:31] that's a good call [00:40:50] Heh. [00:41:14] Reedy: You're an op now [00:41:23] *maplebed created RT-2494. [00:41:44] Hmmm. [00:41:53] I know I've only been around for 9.5 years but really I think I can be trusted with ops access [00:42:00] Are RT ticket titles private too? [00:42:01] pretty please? [00:42:03] TimStarling: you seem to already be on the list [00:42:05] Or just the ticket contents? [00:42:11] I think that's too many privileges for you TimStarling [00:42:16] :D [00:42:26] [modified 5 minutes, 11 seconds ago] [00:42:41] Would a RT feed of ticket resolutions be possible/useful? [00:42:43] *AaronSchulz is not an op here, but is on #mediawiki [00:42:45] I may as well preemptively ban him from labs channel [00:43:17] Joan: summary: ops need more ops [00:43:42] TimStarling: Trevor just gave you op rights like 4 minutes before you asked :) [00:43:44] Reedy: ty [00:44:52] } elseif ( $entry->getSubtype() === 'patrol-auto' ) { [00:44:58] why is that still there? ;) [00:45:30] so he did [00:45:33] thank you TrevorParscal [00:46:02] sorry for my lack of IRC skills [00:47:14] xchat gives you a kickban command in user context menus, so not much skill is required [00:48:11] TimStarling: AaronSchulz: are you absorbed by deployment stuff or do you have time for the commit access review meeting right now? [00:48:30] RoanKattouw_away: added another gerrit change we want: http://code.google.com/p/gerrit/issues/detail?id=118 [00:48:32] /kick username [00:48:36] is usually enough [00:48:39] to the gerrit changes that matter page [00:48:53] Reedy: this one was coming back after kicks [00:49:06] suew [00:49:08] *sure [00:49:16] but it gets him out for a bit ;) [00:49:21] *Ryan_Lane nods [00:50:13] sumanah: I think we're planning on deploying to commons in 5 minutes or so [00:50:18] RL is broken on commons [00:50:22] OK. nm then [00:50:55] oops [00:51:05] that's what you get for trying to do three things at once [00:54:48] Merged the bug 34508 revisions into 1.19wmf1 [00:57:08] you want to deploy them? [00:57:11] brion: if you have spare time to help a bit with commit access review, I could use it [00:57:24] Yup will do [00:58:25] wassap [00:58:31] we now have a 30s expiry time for the startup module on commons, it just found its way through varnish [00:59:07] let's do some quick smoke testing of Reedy's changes on the exisitng wikis before going out to commons [01:00:16] I'm particularly interested to see if the right stuff is making it from irc://irc.wikimedia.org/#meta.wikimedia to irc://irc.freenode.net/#cvn-meta [01:00:36] Live [01:01:56] meta is almost dead [01:04:00]

[[Special:Log/move]] move * RobLa-WMF * moved [[RobLa-WMF/Test123]] to [[User:RobLa-WMF/Test345]] [01:04:15] Seriously, who came up with the idea that it needed colour? [01:04:42] purty [01:05:00] all of our APIs need color [01:05:19] https://en.wikipedia.org/w/api.php [01:05:24] Blue and black, what more do you want? [01:06:30] alright, I'm happy that we haven't at least broken logging horribly, at least not at first investigation [01:06:32] it was a user request, I think [01:06:39] test [01:07:01] Seriously? [01:07:07] *Reedy wonders if we can kill it in 1.19 [01:07:12] anyway....should we finally deploy to commons? [01:07:27] yes, is Reedy appointed button presser today? [01:07:53] can be... Reedy, want to do the honors? [01:08:01] sure [01:08:32] chrismcmahon: you still around? [01:08:42] hi robla, been watching [01:08:54] what is commons? [01:09:12] https://commons.wikimedia.org [01:09:18] pokeswap: I think you want #mediawiki, this is mostly for wikimedia-related stuff [01:09:25] Done [01:09:27] pokeswap: Wikimedia Commons, https://commons.wikimedia.org/wiki/Main_Page but right now most of the people talking in this channel are busy with something urgent. [01:09:44] just wanted to know what it is [01:10:05] Commons mainpage is taking a while [01:10:26] can't connect to db just now on commons [01:10:28] (Can't contact the database server: Unknown error (10.0.6.32)) [01:10:40] hey everyone [01:11:18] TimStarling: 10.0.6.32 is db40? [01:11:27] nope [01:11:37] commons master has issues [01:11:38] db22 [01:11:42] commons main page is back [01:12:02] mostly MessageBlobStore::clear() I think [01:12:24] brion: I'm seeing that opensearchxmp api array_map error again. Stupid thing [01:12:32] no where conditions on that? [01:12:39] hmm [01:12:40] I'll just kill them [01:12:49] Reedy, got an err line or bt to show me? [01:13:15] brion: 57 Warning: array_map() [function.array-map]: An error occurred while invoking the map callback in /usr/local/apache/common-local/php-1.18/extensions/OpenSearchXml/ApiO [01:13:16] penSearchXml.php on line 0 [01:13:23] not very useful [01:14:24] ok this is kind of uncool [01:14:55] now we've got 100 threads trying to insert into msg_resource [01:15:22] let's assume that one of them finished already and that I can kill the rest, ok? [01:16:10] no luck, still loads of threads deleting [01:16:53] I'm going to revert to 1.18 [01:17:13] not sure if it will help [01:17:24] *AaronSchulz waits to chuckle at Special:Log [01:18:07] apparently we broke a disk on it [01:18:17] who knew that was possible? [01:19:21] ok, situation resolved, db22 is OK now [01:19:22] Reedy, any sign of related error msgs above it? exception for instance [01:19:29] nope [01:19:36] bah [01:20:10] I just ran into a merge account error on meta. File under MediaWiki:SpecialPages, right? [01:20:25]

PROBLEM - RAID on db22 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [01:22:33] sumanah: which two accounts? [01:22:40] robla: https://bugzilla.wikimedia.org/show_bug.cgi?id=34582 [01:23:05] robla: I tried to merge my meta login with my unified global login, got this error [01:23:12] "Call to undefined method SpecialMergeAccount::doUnattach" [01:23:17] hexmode: ^^ [01:23:29] this was about 5 min ago [01:24:32] // TODO: Give this some more thought [01:24:32] // TODO: Is TRUNCATE better? [01:24:42] so is there anyone here who knows this code? [01:25:03] TimStarling: are you looking at SUL or elsewhere? [01:25:20] I'm kind of focused on the thing that made the site go down at the moment [01:25:52] just looking for context for your comment... I guess I'll grep [01:26:01] MessageBlobStore::clear() [01:26:22] we can just switch to manualRecache [01:27:12] it may take an hour or so to write and test the scripts [01:28:27] also we should probably do a master switch [01:29:12] Roans comment (somewhat unsuprisingly) [01:29:44] TimStarling: might as well. you want to do that now, or do you want me to see if someone in ops can [01:30:04] Ashers out isn't he? [01:30:11] yeah [01:30:23] probably me then [01:30:42] domas is about [01:30:42] apparently asher did some work on switch.php, I don't know if he told anyone else how to use it while he was at it [01:31:06] Asher did leave instructions which someone else in ops might be able to use [01:31:07] He sent a mail around about it IIRC? [01:31:16] yeah, it's in the bug [01:31:16] the master switch is not urgent, it just has to be done at some point so that that server can be shut down and repaired [01:31:18] *robla looks up [01:31:28] It's on wikitech [01:31:38] http://wikitech.wikimedia.org/view/Switch_master [01:31:51] manualRecache is the top priority, after that's done, we can do the deployment [01:38:47] it's late everywhere except here [01:38:52] should we reschedule for tomorrow? [01:40:24] we've technically got until 7pm here in SF. we should probably not count on Reedy for much longer though [01:40:53] I'd say we keep going, and if Reedy is too exhausted to help at this point, he should drop off [01:41:07] TimStarling: ^ [01:41:48] ok [01:43:21] I should probably call it a night at 2am or so [01:43:26] AaronSchulz: can you take a look at https://bugzilla.wikimedia.org/34582 [01:43:29] Otherwise I'll be still up and it'll be 4am [01:44:46] robla: AaronSchulz, it's quite obvious why that one broke, ther isn't a doUnattch() method [01:46:16] that' [01:46:21] that is a factor, yes [01:47:29] Why would MergeAccount try to unattach something? Isn't it supposed to do the opposite? [01:47:29] TimStarling: https://svn.wikimedia.org/viewvc/mediawiki/trunk/extensions/CentralAuth/SpecialMergeAccount.php?view=markup&pathrev=24382 [01:47:34] I find that amusing too [01:47:36] 24382 [01:48:09] So apparently it never existed [01:48:58] Also, the invalidAction() goes nowhere [01:50:49] Reedy: read #wikimedia-tech [01:52:57] we can probably do this in the same loop as ExtensionMessages-$mwVerNum.php, right? [01:57:07] TimStarling: do what? [01:57:16] manualRecache [01:57:38] basically we need to change the sync scripts to run rebuildLocalisationCache.php [01:58:01] and set $wgLocalisationCacheConf['storeDirectory'] = "$IP/cache/l10n" [02:06:24] TimStarling: seems sane [02:07:59] how did you create ExtensionMessages-1.19.php before there were any 1.19 wikis? [02:09:35] did you just run mergeMessageFileList.php directly? or did you let it break on test2 and then run scap? [02:09:58] ( RoanKattouw_away is distracting him ) [02:10:27] ( now he's reading ... hi Aaron!) :) [02:10:57] TimStarling: it was Reedy that created it right? [02:11:09] *AaronSchulz didn't do that stuff this time around [02:11:12] I think so... [02:11:26] either way I think the script I'm writing will work [02:11:37] though breaking test2wiki sounds reasonable [02:12:40] I might have created it against 1.18 [02:12:57] as it's essentially the same [02:13:10] not really [02:13:25] the extension-list is the same [02:13:56] You're the only one that makes major changes to where message files are ;) [02:18:41] Updating LocalisationCache... [02:18:42] PHP Fatal error: Allowed memory size of 209715200 bytes exhausted (tried to allocate 108 bytes) in /home/wikipedia/common/php-1.18/extensions/LocalisationUpdate/LocalisationUpdate.class.php on line 37 [02:19:14] :O [02:19:26] easily fixed [02:19:45] well, if you can call any fix in three branches easy [02:37:25] fenari may be a bit toast [02:37:39] I am 70% certain it is not my fault [02:39:13] I like its memory graph: http://ganglia.wikimedia.org/latest/graph.php?h=fenari.wikimedia.org&m=load_one&r=month&s=by%20name&hc=4&mc=2&st=1329878298&g=mem_report&z=medium&c=Miscellaneous%20pmtpa [02:40:03] I saw "Passenger spawn server" in top [02:40:26] also LU just finished [02:40:49] as in the PHP part, it might be syncing [02:41:09] hmm [02:41:26] why does the memory graph say it's using so much swap [02:41:32] Swap: 10485752k total, 0k used, 10485752k free, 1414764k cached [02:42:21] it's using a negative amount according to ganglia [02:42:58] anyway it will need rebooting at this rate, I can't wait all day to get a root shell [02:43:07] It's an NFS problem looks like [02:43:16] I guess you managed to melt NFS [02:43:28] I badly need to tweak LU to not write incremental files to /home [02:43:36] but build them on /tmp and copy to /home after [02:43:46] Actually let me do that right now [02:44:21] maybe we should use a fileserver that doesn't melt just by being used a lot [02:44:21] Supposedly we're moving to a netapp [02:44:21] That's what Mark tells me [02:44:36] I've told him I will send him wine once that's done (provided it's not just as slow) [02:45:51] did you see how MessageBlobStore physically broke a disk earlier? [02:46:02] the timing was too close for it to be coincidence [02:46:25] I heard about it causing problems [02:46:31] I had no idea it physically broke a disk [02:46:37] That's disturbing [02:46:43] well, it overloaded the master, db22 [02:46:47] I'll comment it out in 1.19wmf1, we did the same in 1.17wmf1 and 1.18wmf1 [02:46:53] Ouch [02:46:54] and then nagios showed a broken disk alert on that server [02:48:28] *AaronSchulz struggles to get on hume or fenari [02:52:07] OK I just got kicked off of fenari, is it rebooting now? [04:38:09] hi Zaran, how are things going? [04:38:16] hey sumanah [04:38:20] I'm still at work :) [04:38:45] can we set up a meeting to talk about the things we discussed by email ? [04:38:49] I seem to be as well! [04:38:57] Zaran: sure, but this week is bad for me. Next week ok? [04:39:01] unless anything is urgent [04:39:09] sumanah : no next week is perfect [04:39:40] Zaran: if you prefer setting an appointment, then I'll need to look at my calendar [04:39:55] no no, don't worry [04:40:01] we'll probably find some time next week [04:40:19] ok! [04:40:47] so you're working late as well ? [04:41:20] I seem to be..... [04:41:33] Zaran: I had Monday off. So of course stuff piled up [04:41:42] I did take some breaks for dinner etc [04:41:49] sumanah : I see... [04:42:02] right now, I'm fueled by beer only [04:42:06] which is not very good [04:42:08] I just did something very exciting [04:42:13] I reorganized my TODO list [04:42:22] Zaran: is it at least good beer? [04:42:42] mirror pnd [04:43:00] http://www.mediawiki.org/wiki/User:Sumanah/TODO [04:43:07] is this the result of our work ? [04:44:02] oh, I'm on it [04:44:04] Zaran: my TODO list? I've been keeping it for months and months [04:44:05] woohoo ! [04:44:29] :-) [04:44:30] oh no it's not me, the spelling is different ;) [04:44:34] oh no! [04:44:45] I keep putting an L in [04:44:54] *sumanah fixes [04:45:06] hehe, thanks [04:51:37] Zaran: you might be interested in https://codechix-wikipedia.eventbrite.com/ or perhaps a few of your friends would be interested [04:54:57] sumanah : looks interesting, I'll send out the word [04:55:04] cool [04:55:52] I'm not sure I will be able to attend myself though [04:56:18] Understandable [06:58:10] hi ramkrsna [06:58:57] sumanah, hey [13:50:15] http://hi.wikipedia.beta.wmflabs.org no longer has gadgets. No gadgets tab in the prefs anymore (it was there yesterday) [13:50:21] Sid-G noticed this [13:50:46] :) [13:51:06] hexmode: petan: Reedy: ^^ I'm going to check some other beta sites to see if it's just Hindi [13:51:17] And the script file which used to contain twinkle does load, but without twinkle [13:52:05] http://de.wikipedia.beta.wmflabs.org/wiki/Spezial:Helferlein and http://en.wikipedia.beta.wmflabs.org/wiki/Special:Gadgets -- they're gone! why is this? [13:53:09] http://hi.wikipedia.beta.wmflabs.org/w/api.php [13:53:15] The API is also broken [13:53:21] yeah, that's weird [13:53:32] wow [13:53:45] *Reedy blames MaxSem [13:53:47] well, that means these sites are not useful at the moment for testing [13:54:14] *Sid-G sighs [13:54:39] Sid-G: thank you for reporting this [13:54:53] no problem :) [13:56:31] I've got a q: When 1.19 is deployed to wikipedia, will it look exactly like it does on the test sites? (Meaning if I'm seeing any untranslated interface messages on test, they need to be translated before deployment)? [13:57:13] live sites will have customised messages and stuff [13:57:25] if you're looking for untranslated messages, use translatewiki [13:57:47] by the way, Sid-G: you know about #mediawiki-i18n ? [13:58:06] API fixed, needs a svn up [13:58:10] Reedy: So 1.19 has some new messages over 1.18? [13:58:19] Most certainly [13:58:21] sumanah:yeah, but its mostly empty i think [13:58:44] *MaxSem blames Reedy for blaming too much [13:58:50] Reedy:k, thnx [13:58:51] Sid-G: I've been able to successfully talk with people in the i18n room [13:59:00] Sid-G: so, it may depend on the time of day [13:59:15] Sid-G: are you on the Hindi language support team? [13:59:16] hi looks mostly translated [13:59:17] sumanah: yeah, it does depend on that a lot [13:59:17] https://translatewiki.net/w/i.php?title=Special%3ATranslate&taction=translate&group=core&language=hi&limit=100&task=untranslated [13:59:30] sumanah: not officially [13:59:34] 62 messages that need updating [13:59:35] Sid-G: :-) [13:59:46] /are out of date/fuzzy [13:59:47] but, yeah I've done some translations [13:59:47] hmm [13:59:48] will check it out in a day or two [13:59:55] Sid-G: thanks, we'd appreciate it [14:00:37] Sid-G: have you seen https://translatewiki.net/wiki/Project:MediaWiki_localisation_in_the_50_most_spoken_languages ? [14:00:44] sumanah:Heh, no problem. :) [14:00:49] sumanah: long time ago, yeah [14:00:55] it's getting better [14:00:56] :-) [14:01:22] *Sid-G goes looking [14:01:29] We definitely need help with Sindhi, Urdu, Kannada, and some other languages, in case you know some folks [14:01:52] sorry, I dont [14:02:08]