[00:02:20] wp === 'wikipedia' || wp === 'wordpress' [00:02:54] preilly: http://codex.wordpress.org/images/9/9e/WP3.0-ERD.png [00:02:58] marktraceur, yeah, it took me too long to figure out tewwy's wp-related code in GuidedTour was for WordPress. :) [00:03:10] to bad the "indexes" widget is not expanded there [00:04:04] preilly: any, that approach could work for some case, though a lot of cases might have "round peg square hole" syndrome [00:04:11] *anyway [00:59:59] awight: ping [01:34:34] binasher: maybe this redis stuff can be looked at now [01:35:01] I fixed the stuff in my todo list [01:35:28] AaronSchulz: the redis jobq changeset? [01:35:32] yeah [01:35:41] * AaronSchulz lifts his -2 [01:42:52] binasher: anyway, it's that or the separate cluster thingy [01:42:59] it's all up to you! [01:43:01] ;) [01:43:21] its separate boxes regardless! [01:43:45] yeah I mean "DB cluster" [01:45:21] binasher: red pill or blue pill? [01:46:14] * binasher swallows them both.. InnoReddisDB! [01:46:30] surely madness lies that way [01:46:55] I'd say "there be dragons there" but even the dragons are too scared [07:49:09] hello [07:52:14] New review: Hashar; "Well we still have some pep8 errors:" [integration/zuul-config] (master); V: 0 C: -1; - https://gerrit.wikimedia.org/r/39694 [07:52:47] hashar: so, I have good new for you :) [07:52:52] ahh [07:53:14] guess what I have installed in beta? [07:53:17] Ryan_Lane: did you get git deploy hacked properly on labs ? :-D Managed to filter minions by instance names instead of the i-0000x [07:53:32] \O/ [07:53:38] well, I'm using the i-ooox names [07:53:46] i-000x names, that is [07:54:01] since we only have 4 instances I guess that is manageable for now :-] [07:54:16] I need to open a bug with the salt folks [07:54:20] grain matches aren't working [07:54:39] otherwise I would have done it a simpler way [07:54:53] I'm deploying slot0 right now [07:55:12] nice [07:55:18] then I'll deploy slot1, and we'll have all the repos deployes [07:55:22] *deployed [07:55:30] not that they have the right stuff in em :) [07:55:41] I will have to look at the apaches confs I guess [07:55:54] it's likely missing the private config files [07:56:02] well, I shouldn't say likely [07:56:04] it *is* missing them [07:56:11] and it's missing localization update stuff [07:56:30] I got the commands for that : mw-update-l10n [07:56:40] though that use the /h/w/c/p path I guess [07:56:42] that'll write into the git-deploy repos? [07:56:50] this works differently [07:56:50] probably not [07:56:58] I also had to remove a class from bastion [07:57:15] I had to remove: misc::beta::autoupdater [07:57:18] it was missing a file [07:57:36] err [07:57:43] that one is important but I can fix it up [07:57:43] I had to remove: role::beta::autoupdater [07:58:59] I will fix it up [07:59:03] cool [07:59:20] Tim did some changes to the scap scripts that might be related [07:59:36] * Ryan_Lane nods [08:00:09] hm. I need to improve the reporting some [08:00:28] when deploying a new repo, it says the repo doesn't exist, because the fetch hasn't finished [08:00:39] when the fetch starts it should write into redis [08:02:26] ahh [08:02:27] misc::deployment::scap_scripts [08:02:29] the new class [08:04:11] !g Ica1f8c182d2b407ef343051f402879392b38cb82 [08:04:11] https://gerrit.wikimedia.org/r/#q,Ica1f8c182d2b407ef343051f402879392b38cb82,n,z [08:05:46] ugh. got damnit [08:05:52] 1.21wmf6 -> ../slot1 [08:05:56] 1.21wmf7 -> ../slot0 [08:06:02] of course I read that backwards [08:06:09] and set the damn repos up incorrectly [08:06:24] well wmf8 will go in slot1 isn't it ? [08:06:30] yes [08:06:46] it's only possible to screw this up the first time you do it [08:07:06] You seem tired. [08:07:30] yeah you should probably head to bed Ryan :-D [08:07:34] meh [08:07:47] past midnight work might have a high errors/productiveness ratio [08:07:58] Diminishing returns. :-) [08:08:00] kudos on git-deploy anyway :-] [08:08:08] nah. this is the last step [08:08:16] I'm going to finish this before I go to bed [08:10:17] I have put back the autoupdater class on -bastion. [08:10:27] Ryan_Lane: beta auto updater should be fixed with https://gerrit.wikimedia.org/r/#/c/43157/ [08:10:39] that brings the class which provides the file { mw-update-l10n } [08:10:58] yeah, that should work [08:10:59] merged [08:11:38] thankfully fixing my screw up with the slots was easy :) [08:12:14] salt -E '^(i-000004ff|i-000004cc|i-0000031b|i-0000031a).pmtpa.wmflabs' cmd.run 'rm -Rf /srv/deployment/mediawiki/slot0' [08:12:15] :) [08:12:24] now I guess we need all the logic in multi version to fetch the /srv/ paths instead of /h/w/c/p don't we? [08:12:37] that's already done [08:12:42] oh [08:12:43] and on beta [08:12:51] where is that logic ? [08:12:56] in the common repo [08:13:06] I have the "newdeploy" remote branch running on beta [08:13:11] ok [08:13:22] note we also need to have wikis to run out of the master branches :) [08:13:38] that can be a different slot [08:13:45] a third slot so ? [08:13:50] yeah [08:13:59] the config is always master anyway [08:14:00] err [08:14:09] common repo is [08:14:10] I was referring to the mediawiki code base [08:14:32] ah, this reporting I'm getting back is way better now [08:14:39] anyway, will make Reedy aware about the mediawiki-config branch has been deployed on beta [08:14:59] this is the one at /src/deployment/mediawiki/common [08:15:02] *srv [08:15:21] I'd imagine he knows since he pushed that branch in :) [08:15:35] ohh yeah true [08:15:43] still have to actually use it now :D [08:16:01] I can point the apache docroot to that path [08:16:09] bleh initial checkouts take a while due to submodules [08:16:21] err: Could not retrieve catalog from remote server: Error 400 on SERVER: Duplicate definition: Class[Misc::Deployment::Vars] is already defined; cannot redefine at /etc/puppet/manifests/misc/deployment.pp:178 on node i-00000390.pmtpa.wmflabs [08:16:21] not that that's a big deal [08:16:22] damn puppet [08:16:30] heh [08:16:35] isn't that annoying? [08:16:50] can't it see that it's the exact same thing and ignore it? [08:16:58] I had that issue yesterday when trying to apply two role::applicationserver::* classes on a host yesterday [08:17:02] conflict with the common class :/ [08:17:57] different parameters I guess [08:18:53] removed misc::deployment::scripts from -bastion, already provided by misc::deployment::scap_scripts [08:19:16] works now :-] [08:20:34] so my usual radio station is on strike. instead of the dumb talks they only broadcast popular french music :D [08:20:48] kind of running a list of the top 500 french songs [08:20:54] * hashar enjoy working at home [08:20:57] :D [08:21:04] I've been working from home all week :) [08:21:11] don't you like it ? :-D [08:21:28] it's good when I need to really concentrate on something for a deadline [08:21:35] I heard complains about how the new SF openspace tends to be more noisy than before. [08:21:46] noise doesn't bother me, really [08:22:06] err: /Stage[main]/Role::Deployment::Deployment_servers::Common/Deployment::Deployment_repo_sync_hook_link[parsoid/config]/File[/var/lib/git-deploy/hooks/sync/parsoid/config.sync]/ensure: change from absent to link failed: Could not set 'link on ensure: No such file or directory - /var/lib/git-deploy/hooks/sync/parsoid at /etc/puppet/modules/deployment/manifests/deployment_repo_sync_hook_link.pp:5 [08:22:12] I have the same problem either way. getting interrupted [08:22:15] hashar: yep [08:22:20] that's a problem I need to fix [08:22:33] when I don't want to be interrupted, I work from home, shutdown mail client AND IRC client :-) [08:22:35] works well [08:22:49] I keep everything open [08:22:56] I need to multitask occasionally or I get bored [08:22:58] then I need to connect to get a change merged in our puppet repo and my day is done cause of all the interrupt ;) [08:28:21] oh repo dependencies [08:28:22] nice [08:28:37] so mediawiki code base is merged with its l10n cache [08:28:42] s/merged/deployed/ [08:32:10] yes [08:32:16] fetched at the same time [08:32:22] then the checkout also occurs at the same time [08:32:33] hashar: run this: deploy-info [08:32:51] when you wake up tomorrow, I got a change to add a master slot : https://gerrit.wikimedia.org/r/43158 ;D [08:32:58] heh [08:32:58] cool [08:33:04] I just copy pasting without knowing what I was actually doing hehe [08:33:07] seems I need to re-clone slot0 [08:33:12] I screwed it up after all [08:33:19] oh [08:33:20] wait [08:33:22] no I didnt [08:33:49] you should relocate to Europe [08:33:56] so we can work together more often :-] [08:34:02] :D [08:34:07] ah deploy-info is nice [08:34:19] slot1-20130110-081127 [08:34:28] ideally we would have the checked out sha1 too :-] [08:34:37] that's a possibility [08:34:47] does it actually ask the servers or is that cached locally ? [08:35:12] when the deploy.checkout function on the minion returns, it write to redis [08:35:20] *writes [08:35:25] deploy-info reads from redis [08:35:29] ah redis as a data backend [08:35:30] nice [08:35:50] I heard a lot of praises regarding redis [08:36:09] it's easy in this case because I don't need to worry about a schema [08:36:21] and I don't really care about the data [08:36:29] if the data goes away, I can regenerate it [08:37:21] that said, the data is snapshotted, and I can replicate it from one deploy host to another [08:37:32] think the master slot could go in or do you wanna sleep ? :D [08:37:42] it can go in [08:37:43] that would let me migrate beta to the slot [08:37:49] lemme merge [08:37:53] https://gerrit.wikimedia.org/r/43158 [08:38:05] and thus get rid of /data/project for the mw instances. \O/ [08:38:16] heh [08:38:24] so, this isn't too hard to configure, eh? :) [08:38:31] I have no idea [08:38:36] well, you just did it [08:38:37] heh [08:39:18] I'll walk you through setting it up on the deployment host [08:39:30] I need to run puppet on virt0 first [08:40:06] well the mw boxes are running out of /data/project/apache/common-local for now [08:40:13] * Ryan_Lane nods [08:40:25] I think it's going to take a little work getting everything going from /srv [08:40:32] which is a very old copy of our Apaches conf with ton of hacks [08:40:35] we have thursday and friday to do it [08:40:44] otherwise we're sticking with scap for eqiad [08:40:58] yeah I talked about with Rob during our 1/1 [08:41:09] I was like: wtf are we sticking git-deploy before eqiad ? :-D [08:41:16] :D [08:41:19] but I guess it is the perfect time to handle this [08:41:22] yes [08:41:38] and since scap is still around, we have a non-blocking fallback anyway [08:41:44] worst case we push it off to next mw release [08:41:46] that just add a lot of pressure on your shoulders :/ [08:42:29] I'm pretty happy with how it's working right now. as long as we get the l10n stuff in, we should be solid [08:43:15] ugh puppet takes ages on virt0 [08:45:57] hashar: ok, ready? [08:46:03] we'll start with the l10n [08:46:09] sure [08:46:13] 1. umask 0002 [08:46:21] permission denied [08:46:28] umask 0002 [08:46:29] (just kidding, set) [08:46:33] hahaha [08:46:37] I was wondering :) [08:46:50] cd /srv/deployment/mediawiki [08:47:02] ack [08:47:21] ? [08:47:37] like : acknowledged, I am in that dir and ready to serve master. [08:47:37] you mean acknowledged? [08:47:39] heh [08:47:40] ok [08:47:57] AHHH [08:47:58] some people also use that expression as kind of like a "wtf, something weird just happened" [08:48:04] the common has symlinks to the slots [08:48:09] yes [08:48:27] so, you should push that into newdeploy [08:48:58] we'll get to do a normal deploy and some new repos :) [08:49:12] so, you'll want to do that in the remote branch [08:49:21] then we'll deploy it [08:49:41] once you push it in and merge it: [08:49:42] but my slot master is not there yet [08:49:43] git deploy start [08:49:48] git deploy sync [08:49:49] err [08:49:53] git deploy start [08:49:54] git pull [08:49:57] git deploy sync [08:50:01] hashar: it doesn't matter [08:50:06] it's just a symlink [08:50:24] when the slot is there it'll point to the right spot [08:51:02] /var/log/git-deploy/git-deploy.log belongs to root:wikidev [08:51:04] I'm going to run puppet on the deployment host to ensure the new repos sync scripts are there [08:51:11] is it complaining? [08:51:18] but there is no wikidev in labs :/ [08:51:19] are you not in wikidev? [08:51:24] $ git deploy start [08:51:25] Can not append to global log file '/var/log/git-deploy/git-deploy.log': Permission denied at /usr/lib/git-deploy/bin/../lib/Git/Deploy/Say.pm line 96. [08:51:26] # FATAL: current working directory is not part of a git repository [08:51:29] ugh [08:51:36] I'm in wikidev [08:51:43] you may not be [08:51:45] hm [08:51:55] we really need to make it project-deployment-prep [08:52:19] did you push the link into the remote repo? [08:52:29] you probably hacked your account to be in wikidev [08:52:30] as in gerrit [08:52:44] you don't commit locally to repos, unless you are doing a live hack [08:53:10] and you never do git commands outside of git deploy start [08:53:14] https://gerrit.wikimedia.org/r/43160 smylink to slotmaster [08:53:18] cool [08:53:21] let me fix the perms [08:53:30] I'm going to need to fix the sudo stuff too [08:53:38] it is in [08:54:03] so I am in /srv/deployment/mediawiki/common and do git deploy start [08:54:13] don't do it yet [08:54:18] I need to fix perms and sudo [08:54:25] that dies anyway [08:54:25] # FATAL: Failed to read '.git/git-deploy/lock-state': No such file or directory [08:54:25] Died at /usr/local/bin/git-deploy line 45. [08:54:42] right. because of group ownership ;) [08:55:12] and the slotmaster I have created is not going to land cause puppet is failing with an unrelated issue [08:55:37] -_- [08:55:46] how's it failing? [08:55:52] err: /Stage[main]/Role::Deployment::Deployment_servers::Common/Deployment::Deployment_repo_sync_hook_link[parsoid/config]/File[/var/lib/git-deploy/hooks/sync/parsoid/config.sync]/ensure: change from absent to link failed: Could not set 'link on ensure: No such file or directory - /var/lib/git-deploy/hooks/sync/parsoid at /etc/puppet/modules/deployment/manifests/deployment_repo_sync_hook_link.pp:5 [08:56:00] but maybe it keep going though [08:56:05] it'll keep going [08:56:44] we can fix that temporarily by adding a directory [08:56:50] but I wanted to fix it properly in puppet [08:59:14] on beta your default group is 500 (wikidev). hashar and reedy have 550 (svn) :-] [08:59:30] ok, perms changed [08:59:35] nice [08:59:42] in common: [08:59:44] so git deploy start, git pull, git deploy sync ? [08:59:48] yep [08:59:49] that [09:00:01] Can not append to global log file '/var/log/git-deploy/git-deploy.log': Permission denied at /usr/lib/git-deploy/bin/../lib/Git/Deploy/Say.pm line 96. [09:00:05] oh [09:00:06] right [09:00:07] sorry [09:00:08] one sec [09:00:08] but I get a nice text now [09:00:20] Deploy procedure has started. You should now be checked out to the latest commit. [09:00:21] \O/ [09:00:32] # NOTE : Rolled back to 'common-20130110-074739' succesfully [09:00:33] nice [09:00:40] AND WE GET FUCKING COLORS IN OUR TERMINAL!!!!!!!!!! [09:00:42] yeahhh [09:00:46] heh [09:01:00] it did a rollback? [09:01:04] yup [09:01:07] ok, restart the deploy [09:01:10] git deploy start [09:01:12] it just can't write to the file [09:01:16] it can now [09:01:18] works now [09:01:18] nice [09:01:27] I changed the group own of that file [09:01:32] Updating 62b8e5c..25701f8 [09:01:32] Fast-forward [09:01:33] master | 1 + [09:01:44] so… when you do sync, it'll bring you to a prompt [09:01:50] Would you like to continue? ([C]heck fetch state,[y]es,[n]o): [09:01:56] hit enter [09:02:00] which is the same as C [09:02:01] or c [09:02:03] which is check [09:02:14] it shows the state of the minions now [09:02:19] here's what you're looking for (I'm going to improve this) [09:02:28] i-0000031b.pmtpa.wmflabs: common-20130110-074739 (fetch: 0 [0 mins], checkout: 0 [74 mins]) [09:02:29] i-000004cc.pmtpa.wmflabs: common-20130110-074739 (fetch: 0 [0 mins], checkout: 0 [74 mins]) [09:02:30] i-000004ff.pmtpa.wmflabs: common-20130110-074739 (fetch: 0 [0 mins], checkout: 0 [74 mins]) [09:02:31] i-0000031a.pmtpa.wmflabs: common-20130110-074739 (fetch: 0 [0 mins], checkout: 0 [74 mins]) [09:02:36] you want to see that the "fetch" state says 0 minutes [09:02:38] or close to it [09:02:44] and that it's state is 0 [09:02:50] that means it succeeded [09:02:56] now you can continue [09:02:59] hit y [09:03:06] well, enter y [09:03:08] did [09:03:10] and now check [09:03:13] yep [09:03:14] 0 minions without matching tag (4 reporting) [09:03:26] that's a better interface :) [09:03:26] oh that failed [09:03:27] no [09:03:29] it succeeded [09:03:35] 0 without matching [09:03:36] it's bad english [09:03:40] damn reverse logic [09:03:51] continueing [09:03:55] and now you're done [09:03:57] # NOTE : Sync ran ok! Everything looks good! Automatically finishing [09:04:00] # NOTE : Looks like you are all done! Have a nice day. [09:04:01] \O/ [09:04:13] ok, let's add the new repos [09:04:20] so git deploy sync is all I need ? [09:04:20] let's start with l10n [09:04:24] yes [09:04:33] no need to git deploy complete / end / YeahIDidItHugeSuccess [09:04:54] when sync ends successfully it finishes for you [09:05:00] and yeah the interface is definitely better than the scap stuff [09:05:08] it can be improved still [09:05:22] ok. so... [09:05:31] cd to /srv/deployment/mediawiki [09:05:53] ah the minion has a tag :-D common-20130110-090135 [09:05:54] nice [09:06:04] git init l10n-slotmaster [09:06:06] cd l10n-slotmaster [09:06:14] vi .gitignore [09:06:18] add .deploy to the list [09:06:24] write/quit [09:06:27] done [09:06:29] git add .gitignores [09:06:31] echo '.deploy' > .gitignore [09:06:35] that works too :) [09:06:40] git add .gitignore [09:06:42] git commit -a [09:06:47] "Initial commit" [09:06:51] git deploy start [09:06:58] git deploy --force sync [09:07:35] you have to force it for two reasons: 1. it's a dependent repo and by default sync won't actually push it out [09:07:54] 2. it's an initial push, so it doesn't see that you made any changes [09:08:32] usually you don't modify l10n-slotmaster, pushing out slotmaster does it for you [09:08:34] # FATAL: It appears you have not specified a tag prefix and it is not configured either. [09:08:34] # FATAL: You can use: [09:08:35] # FATAL: [09:08:36] # FATAL: git config deploy.tag-prefix PREFIX [09:08:37] # FATAL: to configure this repository for deployment, or provide one on the command line. [09:08:37] oh [09:08:38] :) [09:08:38] sorry [09:08:50] git config deploy.tag-prefix l10n-slotmaster [09:09:25] then go on as normal [09:09:30] works fine [09:09:33] I think so [09:09:53] though no file has been generated [09:09:58] no file? [09:10:12] what do you mean? [09:10:14] empty dir /srv/deployment/mediawiki/l10n-slotmaster [09:10:26] is something saying that? [09:10:43] oh [09:10:49] well, of course it's empty :) [09:10:59] l10n stuff isn't working yet [09:11:05] AHHHH [09:11:09] that explains it all :-] [09:11:12] yep [09:11:18] ok, now for the slot [09:11:20] so we need a minion script that will rebuild the l10n cache when asked to sync ? [09:11:26] yep [09:11:38] cd /srv/deployment/mediawiki [09:11:59]