[02:26:26] * jeremyb wonders if Ryan_Lane saw the response from Platonides [02:29:10] * jeremyb heads to sleeeeeeep [05:50:49] 08/27/2012 - 05:50:49 - Creating a home directory for solon at /export/keys/solon [05:55:50] 08/27/2012 - 05:55:50 - Updating keys for solon at /export/keys/solon [07:48:07] hello [08:02:19] hi hashy [08:05:26] !log deployment-prep updating MediaWiki core : Updating d47c1e9..1c00630 [08:05:28] Logged the message, Master [08:05:46] !log deployment-prep Updating all extensions to latest master [08:05:48] Logged the message, Master [08:14:43] !log reverted live hacks made to ConfirmEdit extension. Fix saved in file PlatonidesPatch and in git stash [08:14:43] reverted is not a valid project. [08:14:53] !beta reverted live hacks made to ConfirmEdit extension. Fix saved in file PlatonidesPatch and in git stash [08:14:54] !log deployment-prep reverted live hacks made to ConfirmEdit extension. Fix saved in file PlatonidesPatch and in git stash [08:14:55] Logged the message, Master [13:12:42] ^demon: Do you not get "E: Unable to locate package gerrit" ? [13:13:03] <^demon> I haven't tried yet since you put in your new patch [13:13:09] <^demon> But I know we've got gerrit 2.4.2-1 in apt. [13:13:30] Seemingly not :( [13:14:21] In theory I know how to mess with the apt repo, trying to jog my memory. [13:15:35] I'm here btw :) [13:15:56] <^demon> Howdy paravoid. [13:16:00] <^demon> Oh and it's hashar too :) [13:16:11] Oh, it's because it's in the lucid rep but not the precise one. [13:16:27] if the package is installable in both just use reprepro copy [13:16:31] hi :) [13:16:32] finally back from vacations ! [13:16:35] 'morning paravoid! (Or, whatever time of day it is there. Tea time?) [13:16:37] <^demon> Welcome back :) [13:16:42] paravoid: Yep, I'm on it. [13:16:50] <^demon> andrewbogott: That'd be it. We need a precise copy now :) [13:17:09] <^demon> Granted for precise, we should prolly just go ahead and package 2.5rc0 ;-) [13:17:20] <^demon> So we can get on the 2.5.x series. [13:18:30] ^demon: did you get Gerrit in puppetized ? [13:18:39] <^demon> We're close :) [13:19:35] <^demon> I did get Ryan to merge the cron to clean out old logs at least :p [13:22:16] ^demon: OK, much better! There are still some errors… I'll debug a bit more before I go to breakfast. [13:22:38] <^demon> Okie dokie. Lemme finish catching up on my e-mails and then I'll dive in as well. [13:26:26] ^demon: y'think there's an attempt to create user group 'puppet2' buried in here someplace, or is that just missing? [13:27:39] finally! I'm able to access to sube:/data/project/ [13:28:02] <^demon> andrewbogott: Possibly from puppetmaster::self? I can't see any reason the gerrit manifest would be doing that. [13:28:32] Really? It chgroups several things to puppet2. You think that group should just exist everywhere? [13:28:52] um… s/puppet2/gerrt2/ ! [13:28:54] oops [13:28:56] gerrit2 [13:29:07] * paravoid was wondering what puppet2 might be [13:29:08] heh [13:29:08] <^demon> Oh, gerrit2 user is created by the gerrit package. [13:29:11] <^demon> :) [13:29:12] Man, the way I said it before it didn't make any sense at all. [13:29:23] Hm… ok. [13:31:37] !beta rebuilding list of extension messages and rebuilding localization cache [13:31:37] !log deployment-prep rebuilding list of extension messages and rebuilding localization cache [13:31:40] Logged the message, Master [13:33:23] ^demon: welp, I have gerrit installed but no gerrit2 group. So that's the next stumbling block. I'll be back in ~45 or so. [13:33:39] <^demon> Mmk. I'll see how far I get between now and then [13:46:21] <^demon> hashar: We're up to 25 patches now :p [14:11:15] ^demon: How are you getting your patches from /var/lib/operations/puppet into git-review? Are you moving them into a user-owned repo like I am, or is there some way to do it straight from the root-owned repo? [14:21:27] <^demon> andrewbogott: You could add a new remote that points to gerrit with your username instead of the labs-whatever username. [14:21:57] I think I did that… it's getting keys forwarded to root@labsmachine that is stopping me I think. [14:23:07] <^demon> Yeah, that's kind of a pain...you have to do it as your own user and not when you're sudo'd afaik. [14:23:21] <^demon> I've just been copying to my localhost and committing. Probably not the most time-efficient way, but it works :p [14:23:33] yeah, that's what I've been doing too [14:25:57] error-prone though (as you see) [14:26:12] <^demon> Yeah. Such is life :\ [14:30:46] paravoid: Any thoughts on a fancy way to make puppetmaster::self talk to gerrit? It could install its own key, but that would allow people to spoof their ID when comitting... [14:31:00] it does already [14:31:22] there's a labs account explicitly for this purpose and a private key that's not so private [14:31:22] really? Is there a howto, or am I missing the obvious? [14:31:31] I'm not terribly excited with what we did there [14:31:35] By 'talk to gerrit' I mean, 'allow me to use git-review without transplanting my patches' [14:31:37] but there was no other obvious way [14:31:46] oh [14:32:22] can't think of anything proper [14:32:28] you could have a local checkout on your ~ [14:32:45] Yep, that works, it's just sad to have to redo for each project. [14:32:52] with one remote being /var/lib/git/... and another one being ssh://faidon@ [14:35:59] Ah, dammit, that time I messed up the changeid [14:37:41] Also for some reason $HOME doesn't have space for a puppet repo in this project [14:39:21] <^demon> It doesn't? I had gerrit itself cloned in $HOME before. [14:39:31] <^demon> Which is way larger than ops/puppet. [14:40:21] Maybe I was doing something silly; I'll try again soon. [14:40:38] s/Maybe/If history has taught us anything/ [14:45:15] <^demon> Yay, made it past the parsing of the manifest :) [14:45:19] <^demon> Seeing how far we get now. [14:48:55] <^demon> andrewbogott: Ok, got as far as missing gerrit2 group. [14:49:36] I think that the gerrit dpkg creates the user but not the group. Is that what you see as well? [14:49:49] <^demon> Yeah, looks about right. [14:50:07] It's easy to create it explicitly, if that's the case. [14:50:08] <^demon> "err: /Stage[main]/Gerrit::Jetty/File[/var/lib/gerrit2/review_site]/ensure: change from absent to directory failed: Could not set 'directory on ensure: Could not find group gerrit2 at /etc/puppet/manifests/gerrit.pp:197" [14:50:54] Of course both our test machines are in silly, inconsistent states now… I'm trying to clean up to see what a clean dpkg install really does. [14:52:16] Well. When the gerrit package is removed it cleans up user gerrit2. But when installed it does not seem to create it. *confused* [14:55:04] <^demon> Oh yay [14:55:43] So maybe gerrit2 is obsolete. [14:56:22] <^demon> Not afaik...we still run gerrit as gerrit2. And everything in /var/lib/gerrit2/ is owned by gerrit2:gerrit2. [14:56:24] <^demon> At least in prod. [14:56:44] <^demon> I'd assume an incomplete package more than it being obsolete. [15:00:34] Lemme try on a cleaner instance [15:00:54] <^demon> Yeah, I'll do the same. This one's kind of messy [15:05:57] Even on a clean system, the gerrit package is busted in the same way. [15:06:19] So, how to proceed? Work around the breakage or fix the package? Can we work around it in a way that won't immedately break with a new, fixed version? [15:09:13] <^demon> I'd prefer to fix the package, so we can install beginning to end from puppet. [15:09:19] <^demon> Manual steps will bite us in the butt later. [15:12:29] ^demon: Is fixing the package something you know roughly how to do? Any idea if this package is home-made or the standard upstream ubuntu one? [15:12:44] And, is there any change that I just need to get a precise-specific package rather than the copy I just made? [15:12:45] <^demon> This is a home-made one. There's not an upstream package (yet) [15:13:16] <^demon> I know zilch about packaging. Every time I try I get buried in Ubuntu docs. [15:13:29] <^demon> I refuse to believe it's as complicated as the docs say. [15:14:11] paravoid: Any chance we can delegate this to you? [15:14:41] * hashar invokes Faidon to do some .deb rituals [15:26:33] ^demon: I'm wrong, it does create the user; it just doesn't put an entry in /etc/passwd (which causes a warning when the package is removed.) [15:28:20] <^demon> Ah ok. [15:28:27] <^demon|lunch> ^ Time for that [15:29:56] a java package? [15:30:07] that always sucks :) [15:31:59] paravoid: Actually I can probably work around this problem, so never mind (for now) [15:43:20] !beta l10n cache rebuild [15:43:20] !log deployment-prep l10n cache rebuild [15:43:21] Logged the message, Master [16:19:16] ^demon|lunch: I have a couple of useful local changes now; you should bug me to update the patch before you dig in any further. [16:19:45] <^demon> I'm now back from lunch :) [16:21:31] ok then, I will try to get this into gerrit... [16:23:10] Also, btw, the reason I can't have a local puppet repo: labs-nfs1:/export/home/gerrit/andrew 18G 17G 21M 100% /home/andrew [16:23:28] <^demon> Ah, that might do it :p [16:23:33] Which puzzles me because my homedir is empty, but... [16:23:45] Must be shared with others [16:36:08] And, getting the patch into git-review requires moving it through 4 different hosts which means I inevitably screw it up. [16:42:58] ok, at last, ^demon, there is a modern version of the patch up. [16:43:29] * ^demon pokes [16:45:17] I hope it applies, at lesat [16:49:26] <^demon> I think I might have to wait until Ryan's done with his upgrade. labsconsole doesn't want to let me do anything. [16:49:40] upgrade is starting soon [16:49:44] <^demon> I know :) [17:02:00] paravoid: mind taking a quick glance over this? http://etherpad.wikimedia.org/essex-upgrade [17:02:06] rollback plans seem sane? [17:02:11] sec. [17:04:06] for (7), why virt1 services? [17:04:09] do you mean virt0 services? [17:05:15] also, why the naming thing is in step (8)? [17:05:35] ^demon: Do you have any guess why my homedir is full without being full? [17:05:46] <^demon> Not a clue. [17:06:29] Ryan_Lane: labs-nfs1:/export/home/gerrit/andrew 18578176 17614080 20224 100% /home/andrew <- and yet when I 'du' in my homedir I'm only using 108k. What am I missing? [17:06:45] the home dirs are shared by everyone [17:06:51] it's labs-nfs1 [17:07:05] we probably need to clean up some bot's crap [17:07:10] lemme do that before I start the upgrade [17:07:27] thanks! That will make my life much much easier. [17:09:19] paravoid: no, virt1's [17:09:38] technically I need to enable virt0, virt1 and virt2's services [17:09:54] because I'm going to test creating instances [17:10:05] I'm only going to enable compute on virt1, thouhg [17:10:07] *though [17:10:36] paravoid: the naming thing means I need to rename the instance's directories [17:10:51] on each of those hosts before I bring up the openstack services [17:10:53] yeah I know [17:11:03] I can't do it till the instances are down [17:11:07] that's not step (8) though [17:11:12] not related to virt6 I mean [17:11:28] that was the question, but yeah, I guess you meant right after you upgraded all of them [17:11:42] well, I need to do it on 6 and on the others [17:11:44] so, all looks sane [17:11:57] I wanted to test one of the current hosts before doing the others [17:12:10] so that if I need to rollback I only need to do so with a small number of instances [17:12:17] yeah [17:12:22] looks well thought to me [17:12:26] cool [17:12:28] and I can't see any holes in the plan [17:12:31] heh [17:12:35] kudos [17:12:36] we'll see when it's underway :) [17:12:42] and +1 :) [17:12:45] yeah [17:12:46] hahaha [17:12:56] first I'm going to get coffee, though [17:13:13] this is going to be a long day :( [17:13:14] heh [17:17:36] tell me about it... [17:19:36] <^demon> andrewbogott: The gerrit package doesn't seem to be in gerrit yet :\ [17:19:46] <^demon> Yet we have an ops/debs/gerrit repo. [17:21:14] ^demon: I'm not sure we need to do anything with the gerrit package at the moment… unless you've come across a new problem with it. [17:21:33] <^demon> Well I was hoping to move that group addition to the package, rather than the manifest. [17:21:41] <^demon> But for now, it's probably fine. [17:21:48] Ah, sure, that wouldn't hurt. [17:25:38] <^demon> Minor nitpick I fixed in PS29: https://gerrit.wikimedia.org/r/#/c/13484/28..29/manifests/site.pp [17:30:17] a_engels [17:30:24] is he online? [17:38:15] * ^demon whacks labsconsole with a 2x4 [17:42:00] ^demon: let me help you with that. see -operations [18:12:58] * jeremyb pokes ^demon for 26 21:06:53 < jeremyb> acct creation waiting: https://www.mediawiki.org/w/index.php?title=Developer_access&diff=575757&oldid=575514 (has svn already) [18:14:23] not right now [18:14:24] please [18:14:30] try not to make ldap changes if possible [18:14:39] it won't break what I'm doing, but it's a bad idea [18:14:46] <^demon> Was about to say the same thing. [18:14:55] <^demon> Also, he wants a new username anyway, not his old SVN name. [18:14:55] okey [18:15:00] <^demon> So better off making a new user anyway [18:15:01] if I need to recover from backups, you'd lose your work [18:15:02] huh [18:15:08] right [18:15:12] I doubt I'd need to do that with ldap, but still [18:15:20] <^demon> His old svn name was tisane. He's asking for leucosticte. [18:15:27] <^demon> So no need to reuse the old one. [18:16:05] so i'll do it myself sometime later then? [18:16:37] we can rename his old user [18:16:47] <^demon> Or that, I suppose. [18:17:07] k [18:17:20] let's do this stuff tomorrow maybe [18:17:38] sure [18:45:56] woah, the update of openstack must be happening right about now [18:46:03] yes it is. [18:46:08] okay cool! [19:03:05] <^demon> Ryan_Lane: Does the upgrade make "*** /dev/vdb will be checked for errors at next reboot ***" go away? [19:03:17] unlikely [19:03:22] a reboot is going to make that go away [19:03:23] :P [19:03:30] * Ryan_Lane sighs [19:03:31] <^demon> I've rebooted. Twice ;-) [19:03:33] <^demon> Since that showed up [19:03:35] still having mysql issues [19:03:54] Plugin '0' is not loaded [19:04:28] I guess I need to start mysql with skip grants, and run mysql_upgrade? [19:04:36] paravoid: ? [19:05:08] mysql_upgrade should be handled by the package [19:05:12] let me see [19:05:20] I'd surely hope it isn't [19:05:43] also, it failed to start from the package due to apparmor and a config option that isn't supported in 5.5 [19:06:09] looking. [19:06:14] ok [19:07:40] there [19:07:47] (I think) [19:07:58] a lot of warnings though [19:07:58] you ran mysql_upgrade? [19:08:06] no, I ran dpkg-reconfigure mysql-server-5.5 [19:08:06] :) [19:08:19] 120827 19:07:04 [Warning] 'user' entry 'wikiuser@virt0.wikimedia.org' has both a password and an authentication plugin specified. The password will be ignored. [19:08:21] that didn't help [19:08:21] 120827 19:07:04 [Warning] 'user' entry 'wikiuser@virt1.wikimedia.org' has both a password and an authentication plugin specified. The password will be ignored. [19:08:24] etc. [19:08:38] oh, auth fails [19:08:39] how nice [19:08:52] hm [19:09:02] we have an authentication plugin? what for? [19:09:07] we shouldn't [19:12:53] http://bugs.mysql.com/bug.php?id=60432 [19:12:54] sigh [19:13:35] well, can start with skip grants [19:13:45] done that already [19:14:06] how would this even occur, though? [19:17:00] because debian maintainers don't support upgrades between non-consecutive releases and ubuntu does that [19:17:03] :P [19:17:11] heh [19:17:15] good reason [19:17:37] I don't see what I need to fix now though [19:17:39] grr [19:17:58] I can recreate the access, if needed [19:18:15] is there known network pain in eqiad? [19:18:25] in labs? no :) [19:18:40] Ryan_Lane: how about in frack? :-P [19:18:44] heh [19:18:50] not that I know of [19:19:01] but you'll get better answers in the operations channel [19:19:08] k [19:19:13] our networking people probably don't read much here [19:20:22] <^demon> This is where the cool kids hang out though ;-) [19:21:59] * Damianz throws some ice at ^demon [19:22:06] Ryan_Lane: I'm installing a pristine mysql in a precise box to compare [19:22:14] paravoid: you can check virt1000 [19:22:25] wait [19:22:26] actually, no [19:22:30] that's not pristine [19:22:48] though I reinstalled it a number of times, so realistically it is [19:24:35] paravoid: there's no plugin column on virt0 [19:24:53] or authentication_string column [19:25:06] I have the diff [19:25:11] ah. cool [19:25:30] http://pastebin.com/3fL5BvNc [19:25:32] that's just crazy [19:26:51] you'd think mysql_upgrade would handle that [19:29:08] fixed [19:29:11] \o/ [19:29:12] thanks [19:29:21] I re-ran mysql-upgrade [19:29:24] that added the two fields [19:29:31] but didn't drop that other fields [19:29:32] heh [19:29:35] * Ryan_Lane nods [19:29:49] and then because of the mysql bug above order matters so you can't have random fields in mysql.user [19:29:52] so I dropped them by hand [19:30:02] ah. good to know [19:30:06] I think the problem was that mysql-upgrade didn't ran because of the apparmor issue [19:30:09] or something [19:30:09] yep [19:30:11] o.0 wth does order matter!? [19:30:12] very likely [19:30:18] Damianz: it's a bug [19:30:29] ok. now for the scary parts :) [19:30:42] upgrading nova and glance dbs [19:30:49] Also apparmor should diaf, selinux ftw :D [19:31:22] nova db upgrade went through [19:31:35] same with glance [19:31:49] and keystone [19:37:23] \o/ [19:42:16] paravoid: I know you have stuff scheduled for tonight but when you get a chance could you tell me if change 21307 is along the right lines of thought, or if I need to pass the var into the gitclone class also. [19:52:43] ok. virt0 done [19:52:49] hm [19:52:55] I should check the nova services there [19:53:33] scheduler service is up [19:53:46] no errors there [19:53:53] ok. virt2 time [19:53:58] this is going to cause downtime [19:58:32] I should have got lunch before this step [19:58:35] * Ryan_Lane sighs [19:58:44] too late now [19:59:17] Damianz: does this work? [19:59:39] if so, I'm okay with it, although I'm not sure how you would include puppetmaster::self with a parameter [20:00:51] can't from labs [20:01:05] would need to have a role, where the role can pass the variable as a parameter [20:01:17] right [20:01:19] which makes sense [20:01:30] yeah [20:01:45] I don't like putting parameterized classes in node definitons anyway [20:01:55] I seriously can't type today [20:04:30] ok [20:04:31] downtime