[00:17:08] !log aaron synchronized php-1.20wmf3/includes/filerepo/file/LocalFile.php 'deployed dfa7120f1bcd2c172096caf0ca65a06119e592c3' [00:17:11] Logged the message, Master [00:34:53] is this how Wikimedia still pushes out Nagios?: [00:34:54] http://www.mediawiki.org/wiki/Wikimedia_Labs/Nagios_management_without_exported_puppet_resources [01:38:46] teratoma: what you linked to is labs. (so not prod) [01:39:51] teratoma: i think they use naggen but i'm not sure yet [01:40:53] !log updated payments cluster to Donation Interface 67b40c9307b [01:40:58] Logged the message, Master [01:43:28] teratoma: https://gerrit.wikimedia.org/r/gitweb?p=operations/puppet.git;a=blob;f=manifests/nagios.pp;hb=HEAD#l448 [01:45:01] i want some magic way of provisioning it all in puppet [01:46:00] teratoma: i think naggen is used in some way, but idk how. certainly whatever is actually in use here is neither magical nor perfect [02:26:50] !log LocalisationUpdate completed (1.20wmf3) at Tue May 22 02:26:50 UTC 2012 [02:26:54] Logged the message, Master [02:50:09] !log LocalisationUpdate completed (1.20wmf2) at Tue May 22 02:50:09 UTC 2012 [02:50:16] Logged the message, Master [12:49:10] Reedy: poke when you get a minute [12:55:34] * closedmouth kicks db12 [12:59:21] is anyone familiar with the job table ? [12:59:30] I got a table with 8k job entries [12:59:41] 382 of them are actually distinct jobs [12:59:58] so it looks like we keep redoing stuff over and over [13:00:03] for no purpose [13:09:05] hashar: when a job is picked up, duplicates should be removed [13:09:18] ohh [13:09:32] there must be a bug somewhere so :( [13:09:39] i know tim has fixed those in the past [13:09:40] take a look at JobQueue.php, method pop [13:11:05] thanks Beau_ [13:11:37] looks like whatever query it was finished up [13:16:23] hmm [13:16:51] Beau_: looks like it prevents a job to be inserted if one already exists with the exact same fields [13:16:59] and of course [13:17:11] there is always a "timestamp" field [13:17:26] wich mean we ensure we have no duplicate PER SECOND [13:17:27] WTF [13:17:30] is that mess [13:20:00] 0bab706c includes/job/JobQueue.php (Sam Reed 2012-01-03 15:14:35 +0000 375) 'job_timestamp' => $dbw->timestamp(), [13:20:01] ahah [13:20:21] well it's still dropping (lag) just slowly [13:39:48] apergos: it's still up in the 80s [13:39:56] I see it [13:47:33] the thing is that queries are getting processed over here so I'mnot sure what the deal is [15:08:46] uh, guys, well over 400 seconds of lag... [15:16:19] not good... [15:23:59] Extreme replag Due to high database server lag, changes newer than 533 seconds may not appear in this list. [15:24:10] Up to 554 :| [15:24:47] going up and down [15:34:06] !replag [15:34:55] I'm on a lagging database server. 11 minutes and counting. [15:35:38] we all are [15:35:46] O.O [15:36:52] @replag [15:36:55] Matthew_: [s1] db12: 639s; [s5] db44: 1s [15:38:12] lag approaching critical mass [15:40:53] work the fuck on the replag [15:41:01] enwp replag, not toolserver [15:41:07] that's not helpful [15:42:07] depooling db12 should do the trick [15:42:27] yes [15:42:35] so start pinging the people who can do that [15:46:31] mark, are you around? [15:46:44] yes [15:46:59] are you able to fix the lag on db12? [15:54:51] I'm (hopefully) doing that now [15:54:58] there's a config change going around [15:55:24] doing what? [15:55:36] mark: We've got a huge database lag on db12 [15:56:15] updating the acl to block the crawler who is maybe responsible for the slow queries [15:56:23] oh [15:56:26] I just null routed that [15:56:29] hahaha [15:56:38] oh well :-D [15:57:04] .101 ? [15:57:19] yup [15:57:33] their in the common-acl now [15:57:37] *they're [15:58:02] if you explain how you null routed them I will know how to use that tool the next time though [15:58:23] ip route 255.255.255.255 null0 [15:58:30] that's it? [15:58:34] yes [15:58:35] swet [15:58:38] not that you should generally do that [15:58:38] *sweet [15:58:43] banning in squid is better [15:58:46] ok [15:58:47] but I wanted to see if it'd help first [15:58:55] lemme see what it looks like [15:58:56] so now we've worked in parallel [15:59:18] sofirst off, they are in fact not requeting anything, yay for that [15:59:58] now I get to watch the lag for a minute [16:00:30] i love you mark [16:00:51] Lag is dropping :) [16:01:29] poor apergos does a lot of work [16:01:31] and I get the love [16:01:34] hahaha [16:01:44] jeff green actually solved it [16:02:01] althogh now I have a better idea for next time [16:02:23] let's see if it continues to drop though [16:02:43] time for ssds [16:04:25] om nom nom nom [16:05:44] Still dropping [16:07:26] liking it [16:25:10] hola [16:25:31] Nikerabbit: Hola! Como estas? [16:26:42] Matthew_: preparing to deploy [16:27:03] That's great! [16:36:00] Ohai [16:37:29] hi Reedy [16:37:42] Reedy: want to approve my commits? :) [16:37:59] Sure [16:38:15] and amir's [16:38:23] and deploy it [16:38:34] to fix that annoying bug [16:39:28] Reedy: https://gerrit.wikimedia.org/r/8465 [16:41:18] o_0 [16:41:26] When was interwiki cleared for enabling? :/ [16:43:59] Reedy: I reviewed it, that's not enough? [16:44:58] this discussion has come up a few times... and the answer has usually been fairly vague [16:47:34] Reedy: well, I'm going to enable it on test, any party killers can do it until next window [16:47:47] haha [16:49:06] Reedy: so will you approve it? [16:52:45] robla: Did we ever get quorum with regards to revieing and deploying new extensions? [16:55:15] * Nikerabbit can't wait forever or I will run out of my window [16:58:15] Rarrgh [16:58:17] Computers [16:58:20] I approved it [16:58:27] Also: "file a bug in Bugzilla asking that the extension be reviewed by one of the senior MediaWiki developers and deployed by the release manager." [16:58:32] I guess it's good then :) [16:58:59] Nikerabbit: ^ [16:59:08] Reedy: wonderful! [17:02:14] Reedy: here's another one: https://gerrit.wikimedia.org/r/#/c/8466/ [17:05:22] Done [17:06:16] Danny_B|backup: which commimt? [17:08:39] Reedy: was wmf-config moved to git already? [17:08:46] yup [17:09:03] stage the files you want to use, then just use git push origin [17:09:04] Reedy: see bug 37015 pls [17:09:18] * Danny_B|backup needs to go now, sorry [17:12:05] Reedy: can you check git diff HEAD^..HEAD ? [17:13:47] Looks good to me [17:14:42] Reedy: and then git push origin? [17:15:11] Yup [17:17:17] Interwiki needs adding to make-wmf-branch later too, but that's not urgent [17:17:37] Reedy: well, better to do now to not forget it? [17:17:43] sure [17:17:52] I'm meaning, it doesn't need doing *right* now [17:19:36] oki [17:19:51] Reedy: I'm going to run sync-dir on extensions/TranslationNotifications and them I'm done [17:20:14] Alright [17:20:36] It's probably worth sync-dir'ing out Interwiki too, and obviously the settins files [17:21:26] Reedy: I'm hoping someone will do scap before next week [17:21:44] I'll run it if you want [17:22:21] Reedy: that's fine too [17:22:21] Why does Interwiki_body call $this->showForm but then ignore the returned text? [17:23:04] <^demon> showForm() sounds like a bad name if it returns something. [17:23:11] heh [17:23:21] Nikerabbit, Reedy - do you tihnk you'll be done with your deplay by 17:30UTC or should aaron and I plan to start a little late? [17:23:22] 6 calls [17:23:30] One does $out->addHTML( $this->showForm( $action ) ); [17:23:35] ^demon: can you cr now? [17:23:41] Rest do $this->showForm( $do ); [17:23:54] <^demon> AaronSchulz: Depends on what I'm being asked to review ;-) [17:24:18] Reedy: all of those are in write-paths which are not executed on wmf, but good catch... must have been like that for a long time [17:24:20] ^demon: https://gerrit.wikimedia.org/r/#/c/8031/ [17:24:59] <^demon> INF? Is that a php constant? [17:27:38] Nikerabbit: I'll log a bug for that [17:27:44] Reedy: I can do that too [17:27:51] if you want :) [17:27:55] Reedy: are you syncing code? [17:28:03] Nope [17:28:33] I can do to finish up and let Ben and Aaron get on [17:28:37] And run scap later on today [17:28:53] Reedy: fwiw our deploy will probably be pretty short. [17:30:06] k [17:30:58] !log reedy synchronized php-1.20wmf3/extensions/Interwiki/ 'pushing interwiki code to cluster' [17:31:02] Logged the message, Master [17:31:04] 1 done [17:31:12] Nikerabbit: I'll do the settings files now then [17:32:02] !log reedy synchronized wmf-config/InitialiseSettings.php 'pushing interwiki variables out' [17:32:05] Logged the message, Master [17:32:26] Reedy: oki, TranslationNotifications is the most important one [17:32:33] Sure [17:32:35] !log reedy synchronized wmf-config/CommonSettings.php 'pushing interwiki loading code' [17:32:38] Logged the message, Master [17:32:40] do you want to do it, or shall I? [17:33:01] Reedy: you are in speed already :) [17:35:42] !log reedy synchronized php-1.20wmf3/extensions/TranslationNotifications/ 'Pushing new version of translationnotification out' [17:35:46] Logged the message, Master [17:35:57] ^demon: yes, it is [17:36:02] and it's really cool ;) [17:36:21] Nikerabbit: i think we're done! [17:36:31] <^demon> AaronSchulz: I learned something today :D [17:36:42] whee! [17:36:45] yup [17:36:46] you can also use it to avoid special case code [17:36:49] AaronSchulz: ready to start? [17:37:04] are we doing all wikis at once? [17:37:07] * AaronSchulz doesn't care [17:37:17] yeah3 [17:38:16] <^demon> AaronSchulz: 8031 looks sane, but I don't have swift setup locally to verify it. [17:39:22] AaronSchulz: gerrit/8473 [17:39:51] aka https://gerrit.wikimedia.org/r/#/c/8473/ [17:40:21] loosk ok [17:41:58] AaronSchulz: are you ready with the change on the mediawiki side? [17:43:14] yeah [17:44:21] ok, starting the ball rolling. [17:44:53] !log starting deploy to make mediawiki write thumbnails to swift for all wikis [17:44:57] Logged the message, Master [17:46:47] ok first test passed [17:47:00] AaronSchulz: rock on. [17:47:11] I've confirmed that new thumbs are not being written to the fr container. [17:47:56] !log aaron synchronized wmf-config/swift.php 'Switched all wikis to new Swift thumb copy hook.' [17:48:00] Logged the message, Master [17:48:26] ^demon: well it doesn't explode with the unit tests or my manual testing [17:48:33] <^demon> Good enough for me. [17:48:44] AaronSchulz: I've confirmed thumbs are now being written to the fr container again. [17:49:12] <^demon> AaronSchulz: Rebase and try again :) [17:49:27] <^demon> I really want 2.4. Automagic rebase button <3 [17:49:50] AaronSchulz: is parallel purging enabled for all wikis or is that still in the test wikis only? [17:50:06] just commons and the testwikis [17:50:22] shall we do that one too then? [17:50:29] I think we're done with the writing part. [17:50:39] I'd have to merge some stuff to wmf2 to do that [17:51:01] oh, I thought that was ready to go to all too. [17:52:01] ^demon: profile call conflict ;) [17:56:14] !log done with deploy to swift to make mediawiki write thumbnails for all wikis [17:56:17] Logged the message, Master [17:57:16] ^demon: done [17:57:31] ^demon: what's the release schedule for 2.5 and 2.5 [17:57:45] <^demon> 2.4 is soonish, they're on rc1 already. [17:57:46] <^demon> No clue on 2.5 [17:58:02] <^demon> "Soonish" as in "there is no schedule until it happens" [17:58:22] ah. cool [17:58:26] * Ryan_Lane nods [17:58:30] <^demon> Ah, 6 days ago I got "I would like to release 2.4 within one week from now." [17:58:33] <^demon> When rc1 hit. [17:58:38] that's like a month after 2.3 [17:58:48] so, quick schedule over all [17:59:01] <^demon> Yeah overall, pretty quick. About every ~month or so. [17:59:19] ^demon: https://gerrit.wikimedia.org/r/#/c/8031/ [17:59:35] <^demon> AaronSchulz: Already looking, Ryan distracted me with gerrit-talk :) [18:00:17] <^demon> Done. [18:04:44] gah, conflict on merge [18:05:42] Reedy: did you get to it by any chance? [18:05:53] no [18:06:10] <^demon> Ryan_Lane: It's a bit on the long side, but one of the gerrit devs wrote a nice writeup to the list about their hackathon and the basic plugin/extension design decisions they've made. https://groups.google.com/forum/#!msg/repo-discuss/kEL3FPT2rLo/w39qjsmnrwwJ [18:06:34] <^demon> Might be worth skimming if you're bored on the train or something. [18:08:11] what's a train? [18:08:13] :) [18:08:25] we have those here? [18:08:28] >.> [18:08:30] <^demon> Or MUNI bus :p [18:12:23] remote: Change-Id: I6417bf35ce1974f9f5bffc6565738c58ac6558d9 [18:12:25] remote: Hint: A potential Change-Id was found, but it was not in the footer of the commit message. [18:12:38] I guess I have to move the "Conflicts" bit up [18:12:44] * AaronSchulz sighs tremendously [18:16:31] !log aaron synchronized php-1.20wmf3/includes/filerepo/backend/SwiftFileBackend.php 'deployed 178a8597e32122feeb593219452f26864639d9ad' [18:16:34] Logged the message, Master [18:16:48] ^demon: there we go :) [18:31:58] binasher: 99% load belongs to 1% of queries! [18:32:11] ;) [18:32:20] :D [18:32:43] WWD? [18:32:44] argh [18:32:46] WWDD? [18:33:02] <^demon> WND? [18:33:19] DROP TABLES! [18:33:42] hush, bobby... [18:33:47] <^demon> WND is always a fun read if you don't get enough laughs out of Fox and Friends. [18:35:15] What Would Domas Do? [18:35:50] :) [18:36:13] binasher: btw, re- site_stats and other things, an elegant solution would be providing 'commit hooks' [18:36:22] that is, statements that are supposed to be executed right before commit [18:36:51] binasher: wow! [18:36:54] just wow [18:38:05] domas: what would you do in a site_stats commit hook? [18:38:34] binasher: make sure that widely locking statements are executed right before commit [18:38:34] binasher: where's the query profiling page though? [18:38:42] binasher: with no mediawiki code in between [18:39:00] meh, they are all deferred to their own commit after the main transaction now [18:39:20] so we have BEGIN ... lots-o-stuff COMMIT BEGIN stats update COMMIT [18:40:05] ^demon: https://gerrit.wikimedia.org/r/#/c/8482/ [18:40:08] that will leak stuff, probably [18:40:10] maybe not [18:40:17] hehe [18:43:13] paravoid: see the Analyze: slow | sampled links for each db on the dbtree page [18:43:29] sample 404s [18:43:38] slow doesn't appear to do anything? [18:43:56] hah i broked it [18:44:02] !log aaron synchronized php-1.20wmf3/includes/filerepo/backend/SwiftFileBackend.php [18:44:06] Logged the message, Master [18:44:31] !log aaron synchronized php-1.20wmf3/includes/filerepo/file/LocalFile.php 'deployed 826f82eaccdf2a017a8ddb27829156f7c474db84' [18:44:35] Logged the message, Master [18:45:46] paravoid: and fixed it [18:47:54] Hi, I just tried to create an LDAP user and something failed terribly [18:48:42] Pastebin the error? [18:48:47] I used sudo add-labs-user [18:48:51] yep [18:48:53] willdo [18:49:12] http://p.defau.lt/?Jeyd3WspLqfXlxylsIJpcg [18:49:14] Reedy: ^ [18:50:13] aaron cleared profiling data [18:50:46] I know I used the right username, and this person already has a SVN account so I'm using the command line method instead of Special:Createaccount on labsconsole.wm.o [18:51:06] mwclient.errors.LoginError: (, {u'result': u'WrongPluginPass'}) [18:51:24] maybe related to the new OATH stuff? [18:51:29] OH [18:51:40] yeah, I think this is the first time I've tried this since the OATH rollout [18:51:57] binasher: and where do you see the 73%/58% that you reported? [18:52:15] (I've just been creating accounts via the web interface on Labsconsole because all the requesters are newbies who don't already have SVN accounts) [18:53:50] paravoid: the break down for the last 24 hours is slightly different than yesterday when i pulled that number, but it's the first line of https://ishmael.wikimedia.org/sample/?host=db38&hours=24&sort=count [18:54:46] Reedy: paravoid - so what do you suggest, to help this person get Labs and Gerrit access? [18:55:04] We could just disable the extension temporarily [18:56:00] <^demon> sumanah: Create the account and then add them to labs via command line? [18:56:55] ^demon: So, create a new account, even though this person already has an SVN account? [18:57:07] and then use what command to add them to labs? I just used sudo add-labs-user and that's what choked [18:57:15] <^demon> Wasn't there a command line script for turning an already-existing LDAP account into a labs account? [18:57:35] <^demon> Yeah, add-labs-user if they're already in SVN. [18:57:43] <^demon> Oh :\ [18:57:46] <^demon> Misread. [19:05:33] maplebed: AaronSchulz are you guys done? [19:05:47] Reedy: yes. I tihnk I logged us finishing. [19:05:57] Ah, I missed it then :) [19:05:58] Thanks [19:06:10] yeah. 17:56. hope I didn't keep you waiting. [19:06:15] ok, so, Reedy, do you think it's a good idea to temporarily disable the extension? what effect would that have? [19:06:51] maplebed: Nope you didn't. I just wanted to make sure I'd run scap at some point [19:07:02] excellent. [19:07:04] scap away! [19:07:06] heh [19:08:30] sumanah: probably not a lot. Ryan is most likely the only person using it... Though, as it's optional, I'm wondering why it blocked the account creation [19:08:59] * sumanah does not know these mysteries [19:09:18] Lets see [19:10:15] Also, the error is in mwclient, not actually the script [19:10:54] Let me try something else logging in via the api [19:12:31] !log reedy synchronizing Wikimedia installation... : Rebuilding message files for interwiki extension [19:12:34] Logged the message, Master [19:16:48] sumanah: we might aswell try it [19:16:59] and if that fixes it, log a bug [19:17:40] will need to get opsen to do it [19:17:45] Reedy: ok, let me know when to re-run the command. [19:17:56] Reedy: I'm taking some of today off so you might need to email me if it is more than 15 min from now [19:18:11] paravoid: could you disable OATHAUTH on labsconsole temporarily please? [19:18:20] I have no idea how to do that [19:18:34] // [19:19:20] the hardest part should be finding the fi;le [19:19:23] *files [19:20:33] how do you disable MW extensions? [19:21:06] <^demon> Just comment out where it was included in LocalSettings. [19:21:10] paravoid: Comment them out of your LocalSettings.php [19:21:28] I was looking there [19:21:36] turns out it includes stuff from outside the tree [19:22:07] Reedy: done [19:22:22] sumanah: ^ [19:22:23] it's hard not to feel like an idiot :) [19:23:00] Reedy: yep, the command went through fine just now. [19:23:08] right [19:23:17] so we need to log ryan a bug saying oathauth breaks api login [19:23:19] Reedy: ok, so you already have all the info, can I ask you to log the bug? I need to pay attention in the git office hr.... [19:23:25] sure [19:29:03] sync done. [19:29:18] Thanks Reedy [19:29:50] Reedy: you think we should now re-enable OATHAUTH so that *other* things don't break? that would be my preference [19:30:01] Sure [19:30:10] I don't think anything should break [19:30:52] paravoid: ^ [19:33:34] done [19:34:07] ok, thanks paravoid & Reedy [19:34:18] yvw [19:38:28] Reedy: did you open the bug or should I? [19:38:46] I've done it :) [19:39:16] ♥ :-) [19:39:57] heh [19:40:37] I've got a feeling it might be LDAP and it's ChainAuth hook [19:41:22] $token = $wgRequest->getText( 'wpOATHToken' ); is checked for.. [20:52:14] !log reedy synchronized php-1.20wmf3/extensions/MoodBar/ 'updating to master' [20:52:18] Logged the message, Master [21:53:17] Reedy: Poke. [22:00:56] ? [22:02:09] its a trap Reedy [22:02:16] I see no tarp [23:21:46] !log preilly synchronizing Wikimedia installation... : MobileFrontend Weekly Deployment [23:21:49] Logged the message, Master [23:38:26] preilly: scap-1 is called with -F 5 [23:38:37] and that does the scap-2 work (the bulk of it) [23:38:40] so it will be ass slow [23:39:20] * AaronSchulz wonders how far along Ryan_Lane 's new git based deploy stuff is [23:39:32] was just talking about that [23:39:35] almost finished [23:39:47] I'm writing the last piece of it right now [23:40:50] \o/ [23:41:09] Ryan_Lane: which do you prefer, wmf wiki or airplane wiki? [23:41:33] *wifi [23:41:43] lol [23:43:19] hmmm [23:43:31] the airplane has worse latency, but I don't get dropped [23:49:18] sync done. [23:49:45] i've had good experiences with plane wifi, just... not enough of them exist yet. :P [23:50:29] brion: I'm a huge fan [23:50:34] the only thing missing now is power [23:50:41] all planes with wifi should have power :) [23:51:07] binasher: the invalidation for bug 37030 seems a bit too tricky [23:51:10] * AaronSchulz gives up [23:52:25] my battery is going to die in like 20 mins or so probably :( [23:52:45] i've had like two flights with power, ever [23:52:52] one with usb and one with a US-style plug [23:53:05] yeah, I've had one with usb [23:53:11] which does nothing to help my laptop :( [23:53:16] sometimes i bring my ipad on trips solely to use on the plane since the battery lasts (and the thing fits on your tray table) [23:53:52] yeah. I like to program on planes, though [23:54:09] wifi lets me program in labs, which makes things so, so much easier :) [23:56:00] programming on an ipad still sucks. :) there's ssh clients and syntax-hightlighting text editors but that onscreeen keyboard's a bitch when you need a lot of {}\#|$| etc [23:56:17] it's great for whipping together a first draft of a presentation in keynote though ;) [23:59:19] heh. yeah. that would be useful