[04:11:32] Ryan_Lane: btw, i tried to create a huge instance yesterday or the day before and it failed (status: "error", no other details). are huge instances verboten / over-allocated? [04:14:28] hm [04:14:35] how huge? [04:14:48] I'd need to see the error, [04:14:55] I'll trigger it tomorrow [04:15:11] it could be a quota in the project or on the hosts [04:20:56] m1.xlarge [04:23:01] is it possible to request an instance that doesn't match any of the predefined machine classes? i could really use >8 gb of ram (ideally around 10gb), but i don't need any of the other stuff -- 8 CPU cores and piles of disk space [04:26:29] i'm setting up a service to help run automated round-tripping tests for parsoid by serving raw wikitext. because it's desirable for the tests to iterate through as large a number of articles as possible (there are lots of pathological edge cases), latency counts. i'm doing a lot of seeking in an 8gb dump of enwiki and would love to be able to copy it to /dev/shm [04:35:10] ^ Ryan_Lane [04:35:56] * Ryan_Lane nods [04:36:01] we can make custom flavors [04:36:23] I believe that's a limited right, though [04:36:35] because it's handled globally [04:47:06] how should i request it? [07:46:58] @infobot-detail ping [07:46:58] Info for ping: this key was created at N/A by N/A, this key was displayed 2 time(s), last time at 9/14/2012 11:02:07 AM (2.20:44:51.5225690 ago) [07:52:40] @infobot-detail petan [07:52:41] Info for petan: this key was created at N/A by N/A, this key was displayed 0 time(s), last time at N/A [08:55:43] 09/17/2012 - 08:55:42 - Created a home directory for danny_b in project(s): incubator [08:59:16] DEFIANCE [09:00:39] 09/17/2012 - 09:00:39 - User danny_b may have been modified in LDAP or locally, updating key in project(s): incubator [09:08:19] wut [09:08:20] :o [09:18:13] somehow adding members to a project is screwed up [10:57:12] Hydriz did you create a ticked for it [10:57:14] * ticket [14:35:44] 09/17/2012 - 14:35:44 - Created a home directory for mf-warburg in project(s): bastion [14:35:55] 09/17/2012 - 14:35:55 - Creating a home directory for mf-warburg at /export/keys/mf-warburg [14:40:41] 09/17/2012 - 14:40:41 - Created a home directory for ajentzsch in project(s): wikidata-dev [14:40:50] 09/17/2012 - 14:40:50 - Updating keys for mf-warburg at /export/keys/mf-warburg [14:45:43] 09/17/2012 - 14:45:43 - User ajentzsch may have been modified in LDAP or locally, updating key in project(s): wikidata-dev [15:41:05] could i get a new project for the devolpment and possible use of a system for Wiki Loves Monuments Judging system? I would also need 1 public IP. [15:42:15] Labs User: Jasonspriggs [17:02:56] * Damianz thinks Ryan_Lane should add 'gluster' to his dictionary :D [17:03:31] also fuck [17:03:41] Ryan_Lane: Did I just reply to you rather than the mailing list? [17:04:07] * Damianz stabs thunderbird [17:04:51] eh? [17:04:58] ah [17:05:11] yep, you did [17:05:16] * Damianz whistles [17:05:35] Now I replied twice heh [17:15:48] hm [17:15:49] autodir [17:15:54] rather than pam_mkhomedir [17:20:44] 09/17/2012 - 17:20:44 - Creating a project directory for wlmjudging [17:20:45] 09/17/2012 - 17:20:44 - Created a home directory for jasonspriggs in project(s): wlmjudging [17:24:40] Ryan_Lane, you around? [17:25:00] yep [17:25:23] seems Wlmjudging's only member is Novaadmin [17:25:31] ah [17:25:38] 09/17/2012 - 17:25:38 - User jasonspriggs may have been modified in LDAP or locally, updating key in project(s): wlmjudging [17:26:50] I agree with your reply, re homedirs yeah - but we still have other stuff like web storage on bots pointing at nfs (which isn't your problem to solve) but still messy. Understandaly prod gets priority atm (due to the improvements swift etc will bring) but if labs is jank then we're back to testing in prod which is bad long term. [17:27:24] (and logins being nfs alive dependant is meh ignoring data) [17:28:40] * Damianz would reply via email but typing is hard while eating curry [17:31:31] Ryan; do i need to relog for the changes to take effect? [17:33:49] shouldn't do [17:37:19] JasonDC: nope. gimme a mi [17:38:02] JasonDC: you are a member [17:38:06] and sysadmin and netadmin [17:38:12] why do you say that only novaadmin is? [17:38:37] heh. I just found a bug in project creation [17:38:43] though it wouldn't affect you [17:40:03] oh; kk; i was a bit confuzzed right there xD, its not showing me as a sysadmin or netadmin tho [17:40:43] 09/17/2012 - 17:40:43 - Created a home directory for laner in project(s): wlmjudging [17:41:45] really? [17:41:47] it is for me [17:41:52] I thought you'd hidden novaadmin on the list? [17:42:13] nope [17:42:18] I need to at some point [17:42:31] oh wtf [17:42:55] there we go [17:43:08] I swear labsconsole plays russian roulette with my session [17:43:10] could be a caching issue in the display [17:43:15] all systems go cap'n :) [17:43:26] Damianz: if you use two browsers, then mediawiki will screw you [17:43:35] Damianz: you can only have a single long-lived token [17:43:50] I havn't logged in on my other pc since I logged into this one because I was away all weekend [17:43:51] if you log in on another browser then it'll log you out of the other [17:44:02] Just went from logged in to not logged in in les than 4 pages [17:44:15] and you are using long-lived tokens? [17:44:22] * Damianz notes to get his phone so he can log back in when he finishes eating [17:44:39] mediawiki has some problem with https [17:44:42] no clue what it is [17:45:45] 09/17/2012 - 17:45:45 - User laner may have been modified in LDAP or locally, updating key in project(s): wlmjudging [17:46:18] I had remember me and logged in, left if for ~3 days, opened it was logged in frontpage was logged in, list instances was out, list projects was in, frontpage was out now it all seems to be out. [17:46:28] I can't tell if it's just really wonkly cached or what [17:46:42] cached, likely [17:46:46] in browser cache [17:46:57] but, it should remember you for at least 7 days [17:47:55] where would the SSL certs for *.wmflabs.org be stored, or do we need to make a CSR? [17:48:19] self signed [17:48:54] ah; i saw something in the puppet config bit about SSL star_wmflabs_org [17:50:02] it exists, but it's no better than self-signed [17:50:13] I'd use self-signed [17:50:21] Everyone has access to it so security wise it's pointless really [17:50:22] I should just kill off those wmflabs certs [17:50:25] yeah [17:52:05] It would be nice to have a real cert when we have a not-user-accessible cluster but then the private repo wouldn't really work... guess if we've got to modules by then with a re-shuffled private repo it could be done with a not-user-accessible submodule repo or such. Still for phishing etc points of attack it would be pointless and if the app isn't taking basic security steps of salting passwords [17:52:12] etc you're screwed [18:25:47] 09/17/2012 - 18:25:47 - User jasonspriggs may have been modified in LDAP or locally, updating key in project(s): bots,bastion,testing,wikibits,wlmjudging [18:25:57] 09/17/2012 - 18:25:57 - Updating keys for jasonspriggs at /export/keys/jasonspriggs [19:42:23] hey Ryan_Lane, to follow up: how should i request additional ram? [19:42:57] ah [19:43:22] which project is this? [19:43:38] ori-l: ^^ ? [19:43:39] visual editor, ve-roundtrip [19:44:03] project, instance = last_msg.split(', ') [19:44:04] :) [19:44:51] ^ Ryan_Lane [19:45:01] xlarge would work? [19:45:41] 09/17/2012 - 19:45:41 - Created a home directory for laner in project(s): visualeditor [19:45:42] needs to have >= 10gb of ram, but cpu and storage don't need to be fancy [19:46:10] xlarge would work, yeah, just worried that it's overkill [19:46:31] no, it's fine [19:46:38] it seems to be building fine [19:46:45] ve-roundtrip2 [19:47:04] * ori-l boggles. [19:47:19] didn't work before. huh. well, thanks, and sorry for the hassle! [19:47:22] listed as active [19:47:27] yayyy [19:47:29] there's a rare race condition [19:47:49] where all of the compute services come back as unavailable to the scheduler [19:48:01] when that happens, it returns an error state. [19:48:11] it's stupid. it should wait till a service comes back [19:49:22] might lead to a cascading failure tho, where the services are unavailable and you accumulate a pile-up of requests in a wait state [19:49:27] yeah [19:49:40] should have a timeout, then [19:49:49] yeah, that seems optimal [19:50:20] paravoid: what are your thoughts on autodir? [19:50:43] 09/17/2012 - 19:50:43 - User laner may have been modified in LDAP or locally, updating key in project(s): visualeditor [19:52:36] Ryan_Lane: hm, i can't ssh in, it's rejecting my key. [19:52:43] did it finish building? [19:53:04] dunno? i though "active" meant it was done [19:53:20] ah, no, console output is still showing puppet stuff [19:53:21] [1;35merr: Could not retrieve catalog from remote server: Error 400 on SERVER: Exported resource Sshhostkey[gerrit-build.pmtpa.wmflabs] cannot override local resource on node i-0000040d.pmtpa.wmflabs[0m [19:53:21] my bad [19:53:24] * Ryan_Lane sighs [19:53:51] next run seemed to go better [19:54:10] ori-l: nah. puppet needs to finish running [19:54:15] active just means the instance is running [19:54:37] it takes absurdly long to get the console log back from the nova api [19:54:46] all it needs to do is read a damn file [19:55:55] but through puppet, which probably means doing some kind of complicated in-flight handshaking that involves passing base64-encoded and cryptographically signed instances of the ruby interpreter [19:56:02] eh [19:56:14] (i'm joking.) [19:56:17] nah, openstack has kvm write the console log out to the filesystem [19:56:23] oh [19:56:29] so, it's really just reading the console log directly off disk [19:57:00] it takes a while because nova-api needs to make an async request to another process [19:57:05] and that process is busy [19:57:21] the problem with a fully async system [19:59:01] ori-l: it should let you in now [19:59:07] yup, was just about to say [19:59:12] it's working! thanks again [19:59:21] basically you need to look for this line in the output: Finished catalog run in 362.18 seconds [19:59:46] what I want to do is have a process on the system that writes to a queue that an ircecho process is listening to [20:00:05] when the system is done building, it would write into it saying "I'm done", then ircecho would spit it into this channel [20:00:41] it could also email the creator (which in the case would be me) or the entire project (which would require us to finish setting up the labs email relay) [20:01:26] and if we ever get echo for MediaWiki, it could also notify the project members in wiki via notifications [20:02:04] hm. I'm installing salt by default on all new instances [20:02:12] I could have salt do all of this [20:02:18] via an event [20:02:59] <3 salt [20:03:27] i've used fabric before, but salt does looks cool.. [20:08:43] yeah [20:08:50] salt can be used in a fabric-like way [20:09:13] but can do way, way more [20:10:27] ugh [20:10:30] homedirs are filled again [20:10:58] really need to move to gluster [20:11:16] * TomDaley doesn [20:11:21] 't use his homedir much [20:11:51] it's a good idea not to [20:11:57] they need to not be full, though [20:15:52] andrewbogott: can you clean up your homedir in the openstack project? [20:16:15] devstack and essextest eat about 250MB [20:16:21] can you move them to project storage? [20:30:38] Ryan_Lane: Yep, I can just delete everything. I'll clean up in a few. [20:31:30] well, no need to delete if you need it for any reason [20:31:34] can just move it to project storage [20:32:40] I don't think I need it, but I'll check before deleting :) [20:34:14] ok [20:35:18] Ryan_Lane: stupid question: where do I find documentation on how to use the new 2-factor authentication for labs console? [20:36:38] Jeff_Green: there is none [20:36:51] in the sidebar, under labs users dropdown... [20:36:51] oic [20:37:12] "Manage two-factor auth" [20:37:21] it'll bring you to a page to enableit [20:37:31] you'll need the google authenticator app on your phone [20:37:53] i see no labs users dropdown [20:39:33] are you logged in? [20:39:46] it's in the sidebar. it's an expandable [20:40:04] I am not logged in. I saw "Token" and came asking. [20:40:11] ah [20:40:15] you need to be logged in [20:40:19] logging in [20:40:29] token is only required if you have tfa enabled [20:40:40] k [20:40:47] I'd stick it in a challenge interface, but mediawiki doesn't support that [20:40:53] and I don't want to refactor core [20:41:11] :-) [20:41:37] I have 5 chained auth plugins. it's hard enough as it is ;) [20:42:10] well, 4. I want to add openid as a provider in there too, but it's broken [20:42:39] * Jeff_Green looks forward to redoing the token foo every time I close my browser [20:42:43] so, I didn't know what this was before, but I'm kind of in love with it now: http://docs.saltstack.org/en/v0.10.0/topics/event/index.html [20:42:45] Jeff_Green: no need [20:42:49] Jeff_Green: use "remember me" [20:42:57] it'll keep you logged in for one week [20:43:08] i am morally opposed to cookies [20:43:13] ah [20:43:18] I see [20:43:21] i've dumped them at browser close since they were invented [20:43:24] it's ok, I can deal [20:43:27] yeah, you're shit out of luck, then :) [20:43:37] mediawiki also handled sessions poorly in https [20:43:46] I haven't been able to track that bug down yet [20:43:57] #mediawikiproblems [20:44:18] TomDaley: 8309285082 mediawiki problems but unicorns ain't one [20:44:36] Unicorns are never the problem :) [20:44:38] heh [20:45:12] Jeff_Green: not sure if remote execution in fundraising scares the hell out of you, but salt has been making my life better so far :) [20:45:48] for code propagation? [20:45:54] and other things [20:46:08] I'm using salt events to have a minion fire an event to the master [20:46:17] so that I can report when an instance is finished building [20:46:21] ah [20:46:35] but, you could use an event from a minion to trigger an event on another minion, through the master [20:46:40] in the past I've just used a web request to do that [20:47:00] yeah. me too [20:47:18] I've found that to be kind of heavy-weight [20:47:22] and it has no authentication [20:47:29] unless you add it [20:47:34] right right [20:48:13] really nothing scares me as much as puppet . . . [20:48:16] heh [20:48:40] I really kind of despise it [20:48:45] but it's what we use [20:49:04] and the alternatives aren't really any better. the ones that are saner aren't totally up to snuff yet [20:49:10] it's true [20:49:28] it's just sort of pesky to secure [20:49:34] oh. yeah. [20:49:39] that part sucks even more [20:49:53] saltstack is working on some of those deficiencies in their implementation [20:49:59] thanks re. 2-factor auth. I'm in [20:50:02] great [20:50:13] you saved the scratch keys, right? [20:50:20] i'll feel a whole lot better when anything even vaguely related to fundraising is behind firewalls [20:50:22] i did [20:50:25] great [20:50:37] Jeff_Green: +1 on firewalls [20:51:10] hm. I think we can use salt for my ircecho + queue idea [20:51:59] ircecho on systems can trigger salt events to the master. the master can write out the events to irc, based on tag [20:52:11] then we only need a single ircecho process [20:52:29] well, a single ircecho user in irc, I mean [20:53:22] I know what you can do with ircecho [20:53:26] fix it so it doesn't die [20:53:27] :P [20:53:36] heh [20:53:39] Damianz: have at it [20:53:47] I'll gladly accept fixes [20:53:51] meh :P [20:53:56] it's netsplits that kill it right now [20:54:26] I should actually replicate the issue first... it uses inotify+twisted right? that should be easy to fix but also easy to put into a loop forever [20:56:55] Awesome, I just saw a picture of someone with a #!/usr/bin/perl tat on their arm [20:58:53] Damianz: not twisted [20:58:57] just regular threads [20:59:24] events trigger actions in the threads [20:59:32] Oh, threads make me sad. I might get bored enough to find which vm has irc setup to break with it later. [20:59:51] well, ircecho is incredibly simple [20:59:56] and it's in the repo [21:00:00] so, you can just apt-get install it [21:00:23] inotify triggers the events [21:00:24] Is it actually in git yet? Last time I found itm it was still in svn [21:00:32] should be in git [21:00:43] 09/17/2012 - 21:00:43 - Created a home directory for pgehres in project(s): fundraising [21:01:02] hm. maybe not [21:01:17] ah. right [21:01:19] adminbot is [21:05:42] 09/17/2012 - 21:05:42 - User pgehres may have been modified in LDAP or locally, updating key in project(s): fundraising [23:03:18] paravoid: what are your feelings on autodir vs pam_mkhomedir? [23:03:28] never used autodir [23:04:02] I really want to avoid fstab mounts [23:04:38] fstab for what? [23:04:44] so, it's possible to use autodir + autofs, where autofs mounts the home share in /data/home, and autodir mounts them locally in /home [23:04:47] /home? [23:05:12] yeah, for home [23:05:23] I like my unix to be simple, i.e. I don't like autofs, bind mounts and such very much [23:05:38] but that's more of a personal preference rather than having good arguments about it [23:05:42] I like my unix to not hang because something isn't available [23:06:02] you mean at boot? [23:06:04] yes [23:06:05] or in general? [23:06:08] or on shutdown [23:06:22] both of which happen with fstab mounts [23:06:31] well, fstab mounts to remote filesystems [23:06:38] I'm sure you know what I meant :) [23:07:27] heh, yes [23:07:36] can't you set timeouts and such with gluster? [23:07:49] timeouts never help [23:08:03] also, it means that /home never gets mounted on boot, even if the timeout works [23:13:41] hm [23:13:52] it's also possible to use an executable map in autofs for this [23:15:15] that may be the simpliest route [23:23:11] Ryan_Lane: php-wise, is apache2/error.log all I get for logging, or is there a more verbose error log someplace on nova-precise1? [23:25:04] you can turn on debugging logs for different parts of mediawiki too [23:25:31] openstackmanager and ldapauthentication are configured to put logs at: /tmp/ldap-s-1-debug.log [23:26:52] paravoid: ignore everything I've said [23:27:00] pam_mkhomedir will work perfectly [23:27:05] using autofs [23:27:21] we can mount /home directly using a direct mount [23:27:25] err [23:27:26] direct map [23:27:36] rather than an indirect, like we currently are [23:27:56] /tmp/ldap-s-1-debug.log, of course! [23:28:00] thanks [23:28:03] yw [23:28:15] I wonder if you can do direct maps via LDAP [23:28:53] sure can! :) [23:31:37] what do you mean by "direct map"? [23:32:34] /- [23:32:46] see: http://www.linux-consulting.com/Amd_AutoFS/autofs-5.html [23:32:49] it has an example [23:33:20] inside of a direct map file, you can say "this specific directory maps against this specific location" [23:33:31] ah [23:33:49] I know very little of autofs (obviously) [23:33:50] so, we can make a /- entry in ldap that points to a set of direct maps [23:33:58] I've used it very extensively [23:34:05] maps under maps, executable maps, etc [23:34:17] this is the first time I've needed a direct map, but it's useful [23:35:59] I just tested it out on an instance. works well [23:36:00] hm [23:36:12] I could probably switch this out in two steps, too [23:36:32] indirect -> direct, then labs-nfs1 -> gluster [23:36:44] lemme see how horribly I'll break things if I do that [23:37:44] -_- [23:38:15] stupid buggy autofs [23:43:09] Damianz: any word on the sudo issue? [23:49:04] Ryan_Lane: btw, I'm getting salt-minion errors when running puppet on labs [23:49:21] paravoid: really? [23:49:23] where? [23:49:24] err: /Stage[main]/Salt::Minion/Service[salt-minion]: Could not evaluate: Could not find init script for 'salt-minion' [23:49:25] everywhere? [23:49:39] the full log is [23:49:39] notice: /Stage[main]/Salt::Minion/Package[salt-minion]/ensure: current_value purged, should be present (noop) [23:49:43] notice: /Stage[main]/Salt::Minion/File[/etc/salt/minion]/ensure: current_value absent, should be file (noop) [23:49:46] info: /Stage[main]/Salt::Minion/File[/etc/salt/minion]: Scheduling refresh of Service[salt-minion] [23:49:49] err: /Stage[main]/Salt::Minion/Service[salt-minion]: Could not evaluate: Could not find init script for 'salt-minion' [23:50:20] is this a box with puppetmaster::self? [23:50:30] kind of, assume yes [23:50:44] does it have salt-minion installed? [23:50:48] no [23:50:54] puppet should be doing that [23:51:00] this is with --noop [23:51:10] it shouldn't err though [23:52:46] paravoid: I don't get this error with puppetd -t --noop [23:53:01] of course, this system already has salt-minion installed [23:53:21] maybe noop is screwed up? the service requires the package to be installed