[00:00:13] apergos: we can be pretty darn ready with some good profiling data and the knowledge that we don't have to flip the switch on all of them at once [00:00:34] If all but English is switched, it'll be 50-50 [00:00:42] yes, amazingly [00:00:52] Anything disturbing will have manifested itself by then [00:01:23] Actually it won't even be 50-50, it'll be 47.5-52.5 in favor of 1.16 [00:01:38] The fact that enwiki is actually more than 50% amazes me [00:01:42] RoanKattouw: but then you have to take into account the Electoral College ;-) [00:01:48] haha [00:01:51] yeah, we need to do more work on th other language wikis [00:02:33] ok so tomorrow we would roll these over, watch the profilind data, and hope we some little anomaly pointed to by it, that's the upshot right? [00:02:57] Yes [00:02:59] and of course we can compre to the profilign data available now from 12.16 [00:03:01] er 1.16 [00:03:11] There'll be zillions of wikis running 1.16 [00:03:23] do we have enough detail in there in what we capture? [00:03:58] Hmm, I have an idea [00:04:05] I'm gonna add '1.17' as a profiling category [00:05:55] a nice thing about keeping it small is that it gives us time to fix some of the non-performance annoyances before the bigger rollout [00:06:44] I'm all for that [00:06:51] I just want to *also* fix the performance issue [00:07:05] sure [00:07:07] otherwise we're still in the hole we want to dig ourselves out from [00:07:52] bah, I asked for 5000 events and it gave me... [00:08:01] [00:08:06] thanks so muich python [00:08:22] worst case scenario: we're not aggressive enough about the Fri 2/11 deploy, so we don't find the performance problems until after a larger group on Tue 2/15. That just means we might need a third window to finish the job [00:08:52] event??= (u'msg::editnotice-1-Bed\u0159ich_Smetana'... etc. it didn't like the \u0159 in there [00:09:14] ok, as long as we have that in our mental plan [00:13:26] Where does noc.wm.o/cgi-bin live? Looking for report.py specifically [00:13:54] alright....I've dawdled long enough on the blog post. here's what I've got: http://eiximenis.wikimedia.org/ReleaseBlogPost2011-02-09 [00:14:00] /usr/lib/cgi-bin [00:14:10] Reading [00:14:12] Thanks [00:15:42] ok looks fine [00:16:00] Looks good [00:18:21] I need Python help [00:18:26] try: foo [00:18:29] except KeyError: bar [00:18:38] Then script execution needs to stop after bar, instead of continuing [00:18:50] How do I kill the script in Python (equivalent of die() in PHP)? [00:20:44] I guess you could sys.exit or something [00:22:41] sys.exit() ? [00:23:00] Oh there we go, files are root-writable [00:23:36] apergos: It's about /usr/lib/report.py . When it hits the KeyError in the first Except, it goes on and dies with an error immediately after because of an undefined variable [00:24:23] tyipcally we would just raise a sepcific exception... you don't want to do that? [00:24:40] We catch it, then output stuff [00:24:41] (I say as the absolutely non python export in the room. ugh) [00:25:36] ok so [00:25:58] after you output stuff you want to barf? then yeah just ... [00:26:05] Alright, my midnight UTC cron job just finished, that's my cue to go to bed [00:26:20] Yeah I want to barf but barf with the info that it's printing, not because stuff blows up down the line [00:26:33] http://noc.wikimedia.org/cgi-bin/report.py?db=1.17 [00:26:58] Despite the fact that it's probably been printing stuff from the Except branch, that seems to get wiped and replaced by the error message [00:27:16] Or not [00:27:18] Wait [00:27:22] It does get printed [00:27:38] But the error message is ugly [00:27:48] we'll live with ugly for now [00:27:52] OK [00:28:03] go sleep [00:28:11] I'll poke at the profiler more tomorrow [00:28:18] more fun and games tomorrow [00:28:24] Good night folks [00:28:34] geez I can't believe that tomorrow is so soon [00:28:35] night [00:33:20] please tell me he didn't edit in place over there [00:42:33] hrm? [00:42:47] btw: I've posted: http://techblog.wikimedia.org/2011/02/1-17deployment-attempt2/ [00:43:00] some minor edits and additions [03:52:47] robla: Hi Rob [04:09:16] hi nadeesha_calcey, how is it going? [04:09:51] nadeesha_calcey: see http://techblog.wikimedia.org/2011/02/1-17deployment-attempt2/ if you haven't read that yet [04:19:52] robla: Hi Rob, I read the techblog now [04:20:59] robla: so I think we can do a testing on Friday on first window [04:21:09] excellent! [04:21:46] the other thing is that we've also deployed a different test wiki, which is a more accurate test of our infrastructure: [04:21:59] http://test2.wikipedia.org/wiki/Main_Page [04:23:38] robla: we will continue our manual testing on http://test2.wikipedia.org/wiki/Main_Page until friday [04:24:59] thanks [04:25:15] nadeesha_calcey: did you find much yesterday? [04:28:33] robla: Yesterday I checked Special pages and I have entered only one bug yesterday (bug id : 27278). people commented on that as its a results of debugging added on prototype [05:33:41] Can any body grant administrator rights to my user (test) in http://test2.wikipedia.org/wiki/Main_Page ? [05:55:15] nadeesha_calcey: yes [05:56:30] what username? [06:00:24] TimStarling: username is test [06:01:46] done [06:02:12] done what? [06:02:29] *apergos goes to see if we're running 1.17 everywhere :-P [06:02:47] TimStarling: thanks [06:05:13] sadly no... still a 6 hour window ahead of me in a bad time zone tomorrow [06:07:20] What's the bugzilla number for pcache bug? [06:09:59] dunno [06:48:47] what pcache bug? [06:59:12] TimStarling: I thought there was some bug with parser cache delaying deployment of 1.17 [07:00:16] the known issues were fixed by bri on/Platon ides, and by Tim... [07:00:28] there was r81741 [07:00:29] we are left with a CPU load issue for which we don't know the cause [07:00:58] and there was $wgInvalidateCacheOnLocalSettingsChange [07:19:12] Somehow my SUL to test2 didn't log it to newusers. [07:21:14] curious [07:22:30] is your nick here your username there ? [07:22:52] Bsadowski1 [07:22:53] :) [07:23:05] Bsadowski1@test2wiki [07:23:07] Hmm [07:24:05] not in the autocreated list either [07:24:57] Special:Preferences says: "Registration time: 07:21, 10 February 2011 " [07:25:12] "User ID: 68" [07:25:21] Yeah, I think Platonides and Krinkle had the same issue. [07:25:36] Some sort of bug there where some users aren't being logged or something? [07:25:40] must be a buglet someplace [07:26:12] I don't see you on there either, Shirley. [07:26:53] Lame. [07:27:00] apergos: Even CentralAuth knows I'm on there: http://test2.wikipedia.org/wiki/Special:CentralAuth/Bsadowski1 [07:27:13] But how come it didn't log? [07:27:22] It's a software bug. [07:27:46] guess so [07:28:37] https://bugzilla.wikimedia.org/show_bug.cgi?id=27287 [07:29:47] Maybe the upgrade from 1.16 broke it somehow? [07:39:26] Don't think so. [07:45:15] sleep time. goodnight [08:04:33] Anybody know of an extension that's a good example of extending the API? [11:40:43] Is there any bug? I got no CSS on Wikipedia using IE8 @ Win XP. [ crossposting #wikimedia-tech ] [11:41:52] waihorace: which wikipedia? [11:42:20] zhwp, enwp [11:43:44] liangent, normal under Firefox but no CSS under IE [11:43:55] IE8@Win XP [11:44:34] I don't have IE so I cannot test it [11:45:12] anyway I found nothing unusual in HTML [11:45:17] Wow... You don't have IE? You are not using Windows? [11:45:54] I'm not [11:47:03] ...:) [11:53:01] waihorace: no problems for me with IE [11:53:16] dunno which version, default IE on Vista [11:53:21] ehh, Win7 [11:53:25] Bryan, I can view meta normally but not zhwp and enwp [11:53:33] I am using IE8 @ XP [11:53:39] try force-reload [11:53:44] ctrl-r or ctrl-f5 [11:56:29] holy fuck, who is responsible for the software pages of my university... [11:56:43] blinking animated gifs with "new" and "warning" are so 1999 [11:57:07] Still no CSS [11:58:07] OK, I fix it now... [12:03:48] is Gadget::loadList() really 97% of a typical load.php request? [12:08:01] hard to believe [12:08:18] the expensive stuff was cached, last time I looked at it [12:09:35] ah, it calls LocalisationCache::recache [12:44:42] TimStarling: What were those load.php and ResourceLoader.php syncs for? I see the other ones in SVN but it seems you reverted yourself there or something? [12:45:22] just trying to track down the profiling errors [12:45:50] when I found a real profiling error, I committed the fix to svn [12:46:05] Ah OK [12:46:09] it seems to be working now, no more profiling errors [12:46:18] there's a lot of LocalisationCache::recache() [12:46:32] maybe someone is browsing the site with a non-english user language, that would do it [12:46:33] I apologize for missing the $wgCacheDirectory setting [12:46:35] a lot as in 4 in total [12:46:42] I'm surprised that didn't have more impact [12:46:52] it's ok [12:47:14] You'd think it'd have broken things more badly [12:48:12] anyway ResourceLoader::__construct() is profiling high because it calls Gadget::loadList() which is the first thing that uses the localisation cache [12:48:23] it's not really that expensive [12:49:46] Ah rigth [12:50:01] Does our profiler have an "own time" feature? [12:50:15] no [12:50:38] you have to use xdebug for that [12:51:12] I'd better go to bed [12:51:23] Rob will probably send me an SMS in 8 hours [12:51:37] Oh, right, you have to get up early for the meeting [12:52:55] night [14:56:20] if anyone wants to comment on the wording, now is the time: http://meta.wikimedia.org/wiki/CentralNotice/Generic_maintenance_notice [14:59:21] Looks good [14:59:28] <^demon> Nice and generic :) [14:59:56] That's the point :) [14:59:59] thanks [15:01:04] I find that passive a bit strange, but that's better assessed by native speakers :P [15:17:48] The passive seems appropriate to me [18:03:27] can somebody sysop me on test2? [18:09:49] Bryan: What's your username? [18:09:56] Bryan [18:10:25] Done [18:11:40] I'm afraid you'll have many revisions to merge, Roan [18:12:38] lots of wfProfileOut added [18:13:13] Tag em 1.17wfm1 [18:13:34] they are [18:13:58] hehe, good strategy :) [18:14:52] I already tagged some of them :P [18:18:03] Reedy, I saw you tagged one as 1.17wmf1 but not as 1.17 [18:18:45] I think they should be merged on both branches [18:19:02] thanks, Roan [18:38:28] Ryan_Lane: yh? [18:40:27] yeah, sup? [19:49:20] more profiling the better [20:38:51] Ryan_Lane: so I finally got my account set up on your tesla sandbox wiki, but now it tells me I don't have Nova creds. And you said I needed an SSH key. [20:39:06] hmm [20:39:09] says you have no creds? [20:39:12] odd [20:39:14] sec [20:42:24] hexmode: which page is saying no creds? [20:44:58] Ryan_Lane: it was saying it on Special:NovaKey, but now it isn't [20:44:59] hmm I must have something screwed up [20:45:06] Created a regular filter: (=markahershberger) [20:45:13] *that* isn't right :) [20:45:16] should be cn= [20:45:26] heh [20:48:58] hmm [20:49:45] hexmode: log out, and log back in [20:49:50] so, I can see instance lists now. [20:49:51] k [20:49:54] I've seen this error before. [20:49:55] ah ok [20:50:01] then things should be working [20:50:12] :) [20:50:13] there must be something messed up when changing passwords upon login [20:50:21] something that unsets something [20:51:46] hexmode: the ssh key you add needs to be a public ssh key :) [20:51:54] sounds likw you have "something" to fix [20:52:04] yeah. it's likely in the ldap plugin [20:52:14] which is going to be pretty annoying [20:52:23] I got it from my authorize_keys ... Aren't those the public ones? [20:52:31] yep [20:53:17] hi folks: this page is updated: http://www.mediawiki.org/wiki/MediaWiki_roadmap/1.17/Revision_report [20:53:50] it's also possible it's a mediawiki issue. I'm betting the domain doesn't get set when the password is changed [20:53:54] we're up to 20 revs there, which leads me to the question: does the report need tweaking, or are we building up more work that we still need to get done? [20:54:05] or a new session is made, that doesn't include the domain [20:54:10] more work, probably some things dealing with profiling [20:54:20] if it's those they should be easy revs [20:54:39] (I am badly distracted righ tnow, I am listening to Mubarak's speech) [20:55:45] Is he quitting? [20:55:48] I really want to avoid the situation we had on Monday/Tuesday, where we spent most of the rollout window doing prep work [20:56:39] no! he says he will not step down til a govt is elected in September [20:56:47] ...and if we're in denial about how much work we have yet to do, let's call off the deploy window now rather than rationalizing [20:56:59] robla: I agree, they should be reviewed and checked well before that [20:57:36] nope, I'm just reporting what I ... where is Platonides? .. was hearing earlier from someone who made some commits [20:57:49] apergos, I'm here [20:58:19] discussing the new unreviewed revs: are these mostly profiling things? [20:58:24] you were talking about this earlier [20:58:27] Platonides: I see you just changed your password via email, you have credentials problems, right? [20:58:45] (I have debugging enabled) [20:58:46] I couldn't remember the password I set there [20:59:05] 2011-02-10 21:01:15 wikidb: 1.2d User is not using a valid domain. [20:59:05] 2011-02-10 21:01:15 wikidb: 1.2d Setting domain as: invaliddomain [20:59:19] yep.... it's mediawiki's fault alright [20:59:23] is that bad? [20:59:44] it makes a new session after the password is changed and logs you in [20:59:56] but doesn't set the domain [21:00:13] without a domain, none of the settings get picked up [21:00:14] maybe that's why it tells me "There were no Nova credentials found" ? [21:00:18] yep [21:00:28] if you log out, and log back in, it'll work [21:00:53] now I need to track down a mediawiki bug :( [21:00:55] confirmed [21:03:12] I can theoretically ssh to testing10 [21:03:21] yes [21:03:33] not via the DNS name [21:03:34] but it fails strangelly [21:03:36] Robla at least 8 of these are adding profile in/out [21:03:37] but by the floating IP [21:03:39] oh, it does [21:03:43] so those will be fast reviews indeed [21:03:57] I don't actually have the DNS server enabled [21:04:01] I was passing a port as a parameter and ssh thought it was a command :P [21:04:10] ah [21:04:11] heh [21:04:26] if you create an instance from scratch, it'll send you an email when it is done building too :) [21:04:51] testing10 [21:04:56] it as a PEBKAC [21:05:06] eh? [21:05:15] ah [21:05:47] through clever hackery it will send a localized email even ;) [21:06:36] though I haven't tried it. I guess I should test that [21:06:54] a localised email for what? [21:07:22] "The following instance is finished building and is ready to be logged into: " [21:07:32] the instance itself sends it [21:07:36] does instance creation work now? [21:07:45] yeah, you logged into testing10, right? [21:07:57] yes [21:08:11] i'm adding a PS1 [21:08:29] otherwise the shell is too sad [21:08:44] I created that via the interface. nova started an instance, the instance registered with puppetmaster, puppetmaster looked it up in LDAP, and signed the request [21:08:53] then the instance built itself from puppet [21:09:00] scrum starting in a couple of min [21:09:08] which as a last step sends an email to whoever created it [21:09:16] shouldn't user shell be /bin/bash instead of /bin/sh ? [21:09:22] (I assume that's dash) [21:09:34] yeah. I need to fix that [21:09:58] I need to add another interface to the wiki that lets users change their info [21:10:03] like shell, and a few other things [21:10:16] let me set a default shell [21:10:24] maybe I should make that configurable [21:10:41] nova dumped all the 'keys' into .ssh/authorized_keys [21:10:49] yeah [21:10:51] it should [21:10:55] yes, it should [21:11:01] well, nova didn't do it [21:11:07] but it's funny given all the garbage I placed there xD [21:11:12] I have a script that pulls everything from LDAP [21:11:51] in reality I could allow uid/gid changes too, since that script will handle that as well [21:12:03] I'll avoid that though [21:12:15] if we ever need to do that, it can be done manually :) [21:12:40] what's the source for the uname? [21:12:50] the hostname? [21:13:17] is Trevor out today? [21:13:24] Ryan_Lane, yes [21:13:30] the instance id [21:13:53] it's guaranteed to be unique [21:13:57] so it's a good hostname to use [21:14:26] are we having a meeting? [21:14:42] I assume that the server ssh key is automatically created. Is it included in the email notifying it has born ? [21:15:02] nope. that could be a good thing to put in there [21:15:07] the fingerprint, at minimum [21:15:14] yes, the fingerprint [21:15:27] good idea [21:15:31] *Ryan_Lane goes to enter a bug [21:16:28] what are all those [apt-get] ? [21:16:37] mmh, they are puppet children [21:16:43] doesn't it wat for them? [21:16:56] it's puppet, who knows :) [21:17:24] looks like a puppet bug [21:17:25] it took me two or three days to make puppet work for this [21:18:05] so far the most time I've spent on any one part [21:18:43] I'll take that as a no [21:18:50] how does it work? [21:18:52] the best part is, I don't need to make my own images. I'm just re-using the eucalyptus ones [21:19:14] isn't all of that configured in the image? [21:19:26] nova (and ec2) allow you to add "user-data" to the instance's meta-data [21:19:37] this user-data, is a multi-part mime encoded set of data [21:19:57] which can be upstarts, scripts, hooks, and cloud-init config [21:20:13] robla: Did we have a meeting? Is it still going on? [21:20:17] cloud-init will do things like apt-get update/upgrade on boot, configure puppet, etc [21:20:28] so, I set the puppet server information via cloud-init [21:20:33] RoanKattouw: yes, it's still going on [21:20:47] when the instance boots, it starts puppet, which connects to the puppet master [21:20:55] it's pretty hard to see that because it's been buried in Ryan_Lane and Platonides' discussion [21:21:12] the puppet master has a script which sees if the instance is in ldap [21:21:19] Ryan_Lane, let's go to #wikimedia-ops :P [21:21:22] if so, it signs the request, and the instance continues on with building stuff [21:21:23] heh [21:21:26] robla: About the review backlog: a lot of those is additions of wfProfileOut() calls [21:21:28] ok [21:21:30] Let me grab my headset and call in [21:24:47] http://www.mediawiki.org/wiki/MediaWiki_roadmap/1.17/Revision_report [21:30:29] http://noc.wikimedia.org/conf/1.17.dblist [21:30:45] http://eiximenis.wikimedia.org/1-17 [21:32:59] <^demon> http://meta.wikimedia.org/wiki/CentralNotice/Generic_maintenance_notice [21:37:22] TimStarling: Is https://secure.wikimedia.org/wikipedia/mediawiki/wiki/Special:Code/MediaWiki/81729 needed in deployment? [21:37:58] depends how noisy you want the syslog to be [21:38:00] <^demon> It caused a lot of spam in the apache log I believe. [21:38:28] yeah when someone renders an SVG from Adobe Illustrator, without that, it puts about 20 lines in there [21:38:36] 20 long lines [21:39:42] OK I'll just put it on the list then [21:49:16] one thing I forgot to bring up is the dev shifts [21:50:20] RoanKattouw: you're not available Friday 12:00-16:00 UTC, are you? [21:50:52] I will probably leave between 1300 and 1400 [21:51:05] Reedy is the only other logical choice that I can think of [21:51:50] is werdna still in Europe and planning on being around? [21:52:47] <^demon> Other than one class tomorrow from 15:00-16:00 I should be around all weekend. [21:53:20] 12:00-16:00 UTC is business hours in Europe, reasonably early morning in eastern US, grossly early in western US, and pretty late in Australia [21:53:47] <^demon> (that was utc time, btw, not est) [21:54:25] ^demon: that'll be right after the deployment on Friday morning for you. I wouldn't want to sign you up for that unless you were absolutely sure you wanted to do it [21:55:14] oh, and you have a class to go to....d'oh [21:55:21] <^demon> Just one class :) [21:55:23] <^demon> And only for an hour. [21:55:35] I've got a quick meeting to run off to. (it's actually the pending changes meeting [21:56:02] <^demon> Oh are we doing that on mumble today or kicking it because of the deployment? [21:56:28] Aaron is on mumble now, so I'll at least catch up with him [22:38:24] so what was the result? tomorrow deployment? [22:38:40] Yes [22:38:47] Read the techblog [22:39:11] tomorrow? [22:39:14] *jorm invokes krinkle [22:39:24] ah eet [22:39:25] good, i'm home tomorrow [22:40:54] probably not awake at the beginning, but I doubt it will be at 06:00 exactly [22:43:25] Probably not [22:43:37] I'm setting my alarm for 6:30 (CET) so I'll probably just be logging onto IRC at 7