[00:07:52] PROBLEM Total processes is now: WARNING on bots-4.pmtpa.wmflabs 10.4.0.64 output: PROCS WARNING: 164 processes [00:17:01] opps :P [00:58:24] hi; I'm now getting a "Permission denied (publickey)" when I try to ssh into my instance [00:58:34] I was able to ssh last week [00:58:57] connecting into bastion (with ssh -A) still works [01:00:26] does that have to do anything with /home ? [01:02:52] PROBLEM Total processes is now: CRITICAL on bots-4.pmtpa.wmflabs 10.4.0.64 output: PROCS CRITICAL: 201 processes [01:08:43] PROBLEM Total processes is now: WARNING on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS WARNING: 169 processes [01:09:05] bots-4 is fine btw :) [01:10:01] gribeco, i dont think so [01:10:17] ok [01:10:21] isnt it -f you have to use, not -A? [01:10:40] wait, how are you sshing to bastion? :) [01:10:45] it still says -A in https://labsconsole.wikimedia.org/wiki/Help:Access#Using_agent_forwarding [01:11:02] that's from my local system [01:11:39] and then straight ssh from bastion into bots-salebot [01:11:39] from your local system you should use -f [01:11:39] i think [01:11:39] hmmm, ok, worth a try [01:11:39] and then once in bastion use -A to go to your instance [01:11:39] gribeco: you can get into bastion? [01:11:39] but not into bots-4? [01:11:46] Ryan_Lane: yes, into bastion, but not into bots-salebot [01:11:53] let me try bots-4... [01:11:57] on bastion, type this: ssh-add -l [01:12:07] works into bots-4 [01:12:31] I see why [01:12:41] ah! [01:12:46] well, kind of [01:12:51] /public/keys isn't mounting [01:12:54] eh. [01:13:12] now it is [01:13:12] okay, that's consistent with what ssh -v is saying [01:13:17] autofs needed to be restarted [01:13:43] RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 94 processes [01:13:54] is it working for you now? [01:14:13] I can log in, but the connection gets closed immediately [01:14:17] hm [01:14:18] one sec [01:15:39] weird [01:15:44] there's something up with this instanc [01:17:08] hm. it's shared to it properly [01:19:44] gribeco: I think this instance needs to be rebooted [01:19:58] it seems it was never rebooted after the switch to the new homedirs [01:20:12] and I can't get the mounts to be sane [01:20:17] sure -- is that something I can do myself without logging in? [01:20:25] yes, but I can also do it right no [01:20:26] *now [01:20:29] ok [01:20:34] ok, it's rebooting [01:22:41] yup, I can log in now; thanks! [01:23:11] hm [01:23:16] there's still something wrong [01:23:21] xD [01:23:33] it's still mounting the old homedir [01:23:40] even after a reboot [01:23:42] which makes no sense [01:23:55] yes, I only have r/o access to /home/gribeco [01:25:11] o.O [01:25:23] I can't write to anything in /root or /tmp [01:29:22] I'm so confused [01:30:18] it says no space left on device [01:30:25] but it also says / has 77% left [01:30:27] err [01:30:30] in use [01:30:32] yeah [01:31:04] df is still reporting errors, too [01:31:25] yeah, that's because I disabled autofs [01:31:37] ok [01:31:44] it seems that / was indeed full [01:33:56] now to see what's eating so much space [01:34:17] heh [01:34:22] it's salebot's cache [01:34:35] 5.2GB of space [01:34:52] you should really write that to /mnt, rather than /var/local/salebot/cache [01:34:59] ok [01:35:12] want me to clear it, or move it? [01:35:32] move, please [01:35:34] I can move the salebot directory to /mnt, if you'd like [01:35:34] ok [01:35:44] I'll add a link, too [01:35:53] thanks [01:35:57] yw [01:36:55] it's weird that df reports incorrectly [01:37:12] I wonder if you are going over the filesystem limit for number of files in a directory [01:37:13] I agree =) haven't seen this before [01:37:21] oh, that's plausible [01:37:34] what does it cache? [01:37:38] I thought I was flushing that cache; I'm having doubts now [01:37:43] heh [01:37:52] at 5.2GB I'd imagine you aren't :) [01:38:07] if it's usually a small amount of cache, you may want to consider using memcache or redis [01:38:07] ^^ [01:38:24] which will automatically purge the last used item [01:38:29] ok [01:38:55] looks like you are good to log in now [01:39:08] it's still moving the data [01:39:20] but the mounts are working [01:39:41] yup, df is a lot longer now [01:39:52] labs-nfs1:/export/home/bots/gribeco [01:39:55] that still shows up [01:39:56] which is wrong [01:40:12] your homedir is writable [01:40:28] yes, works for me too [01:41:01] there we go [01:41:06] I lazy unmounted that [01:42:16] Ryan_Lane, should i keep my data/code in /home/addshore or in /data/project/something ? [01:42:30] addshore: /data/project/something [01:42:59] i guess i should fix that shortly :P [01:43:04] whats the benefit of it being there? [01:45:55] in case you leave and someone else wants to take over your bot [01:46:02] or if you want others to help you run it [01:46:07] or to collaborate in some way [01:46:16] i see [= [01:46:57] it's possible to make that happen in your homedir, too, but messing up your homedir permissions could end up with your account getting owned [01:47:11] usually other people are unwilling to go into a homedir, too [01:47:25] also, we quota the homedirs more heavily than the project data store [01:49:29] gribeco: heh. I very much think this is over the limit for files in a directory [01:49:36] or well, over what's a good idea [01:49:42] the copy is going incredibly slow [01:49:45] im guessing there is no special way to make a project folder, i just dive in and do it? [01:49:46] err [01:49:47] move [01:50:00] addshore: hm. it may require sudo [01:50:11] okay =] [01:50:17] Ryan_Lane: ok, it's fine to nuke all the files [01:50:51] it's not a big deal, it can copy as long as you'd like to wait [01:51:07] if the cache is important [01:55:36] Ryan_Lane: you can abort the move for now, I'm deleting older files [01:55:46] I'll resume when I'm back to a smaller amount [01:56:10] and add the symlink once that's done [01:56:50] ok [02:22:52] PROBLEM Total processes is now: WARNING on bots-4.pmtpa.wmflabs 10.4.0.64 output: PROCS WARNING: 197 processes [02:30:53] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [02:35:53] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [02:54:23] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 22% free memory [03:12:22] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 12% free memory [03:57:52] RECOVERY Total processes is now: OK on bots-4.pmtpa.wmflabs 10.4.0.64 output: PROCS OK: 150 processes [04:00:52] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to popen() failed [04:15:52] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [04:42:42] PROBLEM Total processes is now: WARNING on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS WARNING: 152 processes [04:50:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 5.42, 5.30, 5.11 [05:10:33] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 4.81, 4.77, 4.93 [05:17:42] RECOVERY Total processes is now: OK on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS OK: 150 processes [05:20:53] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [05:38:33] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 7.11, 6.38, 5.50 [05:40:52] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [06:30:42] PROBLEM Total processes is now: WARNING on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS WARNING: 155 processes [06:32:52] PROBLEM Total processes is now: WARNING on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: PROCS WARNING: 151 processes [06:35:42] RECOVERY Total processes is now: OK on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS OK: 149 processes [06:37:52] RECOVERY Total processes is now: OK on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: PROCS OK: 148 processes [09:05:54] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [09:10:52] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 10% free memory [11:22:42] PROBLEM Total processes is now: WARNING on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS WARNING: 152 processes [12:07:42] RECOVERY Total processes is now: OK on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS OK: 142 processes [13:05:52] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [13:10:58] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [13:22:22] PROBLEM Total processes is now: WARNING on vumi-metrics.pmtpa.wmflabs 10.4.1.13 output: PROCS WARNING: 151 processes [13:23:43] PROBLEM Free ram is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Critical: 5% free memory [13:27:23] RECOVERY Total processes is now: OK on vumi-metrics.pmtpa.wmflabs 10.4.1.13 output: PROCS OK: 148 processes [13:41:23] !g I78263b8b428a37549db5b59400867ccdc27bb0e7 [13:41:23] https://gerrit.wikimedia.org/r/#q,I78263b8b428a37549db5b59400867ccdc27bb0e7,n,z [13:58:23] Change abandoned: Hashar; "(no reason)" [labs/nagios-builder] (master) - https://gerrit.wikimedia.org/r/39691 [14:32:47] !log Made enwiktionary to use 1.21wmf6 and enwikibooks to use 1.21wmf7 {{gerrit|42951}} [14:32:47] Made is not a valid project. [14:32:54] !beta Made enwiktionary to use 1.21wmf6 and enwikibooks to use 1.21wmf7 {{gerrit|42951}} [14:32:54] 42951}}: !log deployment-prep Made enwiktionary to use 1.21wmf6 and enwikibooks to use 1.21wmf7 {{gerrit [14:33:03] o.0 [14:33:04] STUPID FOGUAC BOT [14:33:17] !log deployment-prep Made enwiktionary to use 1.21wmf6 and enwikibooks to use 1.21wmf7 {{gerrit|42951}} [14:33:19] Logged the message, Master [14:55:34] !beta reloaded udp2log-mw on -dbdump [14:55:34] !log deployment-prep reloaded udp2log-mw on -dbdump [14:55:37] Logged the message, Master [14:58:36] !beta -video05 : trimmed /var/log/glusterfs/data-project.log file [14:58:36] !log deployment-prep -video05 : trimmed /var/log/glusterfs/data-project.log file [14:59:10] !log deployment-prep -video05: running apt-get upgrade [15:00:59] !log deployment-prep -video05 updating puppet local repository and running puppet [15:04:25] stupid wm-bot [15:04:29] seriously pissed up by it [15:04:41] !beta deployment-prep -video05: running apt-get upgrade; updating puppet local repository and running puppet [15:04:41] !log deployment-prep deployment-prep -video05: running apt-get upgrade; updating puppet local repository and running puppet [15:05:03] !beta copy php-master/LocalSettings.php to php-1.21wmf6 and 1.21wmf7 [15:05:03] !log deployment-prep copy php-master/LocalSettings.php to php-1.21wmf6 and 1.21wmf7 [15:32:15] !beta -video05 : restarted puppet and puppetmaster, killed stalled puppet processes. Rerunning puppet manually [15:32:15] !log deployment-prep -video05 : restarted puppet and puppetmaster, killed stalled puppet processes. Rerunning puppet manually [15:32:18] Logged the message, Master [15:46:11] !log deployment-prep Running mw-update-l10n on deployment-bastion in screen 16609.pts-0.i-00000390 [15:46:14] Logged the message, Master [15:50:22] iw bots can run in labs? [15:56:02] !log deployment-prep Refreshing the TrustedXFF cache: cd /home/wikipedia/common/php-master/extensions/TrustedXFF && mwscript extensions/TrustedXFF/generate.php --wiki=aawiki ../../cache/trusted-xff.cdb [15:56:03] Logged the message, Master [15:59:24] !beta cp php-master/cache/trusted-xff.cdb php-1.21wmf6/cache/ [15:59:24] !log deployment-prep cp php-master/cache/trusted-xff.cdb php-1.21wmf6/cache/ [15:59:28] Logged the message, Master [15:59:29] !beta cp php-master/cache/trusted-xff.cdb php-1.21wmf7/cache/ [15:59:29] !log deployment-prep cp php-master/cache/trusted-xff.cdb php-1.21wmf7/cache/ [15:59:32] Logged the message, Master [15:59:45] hashar: please don't use !beta. then we don't know who !log'd it [15:59:50] !beta del [15:59:50] Successfully removed beta [15:59:59] seriously [16:00:11] too many bugs :-] [16:00:12] i is totally serious ;-) [16:00:56] <^demon> too many bots. [16:01:30] 09 15:59:28 < hashar> !beta cp php-master/cache/trusted-xff.cdb php-1.21wmf7/ca\che/ [16:01:44] hashar: also, your last path component there is messed up [16:01:54] some encoding thing? but it should just be ascii [16:02:37] !log deployment-prep enwiktionary beta now running 1.21wmf6 http://en.wiktionary.beta.wmflabs.org/wiki/Special:Version [16:02:38] Logged the message, Master [16:03:06] jeremyb: bug in the bot I guess :-D [16:03:12] no... [16:03:15] jeremyb: looks fine in my scrollback [16:03:23] !beta cp php-master/cache/trusted-xff.cdb php-1.21wmf7/cache/ [16:03:26] copy paste [16:03:40] hashar: it's broken on my screen too [16:03:51] not just on wiki [16:04:12] might be an issue from my terminal thus [16:04:44] !log deployment-prep the recent (today at least, probably most of them) !logs from wm-bot are really from hashar. in case you were looking for the source. [16:04:46] Logged the message, Master [16:05:08] got the job done anyway :-] [16:05:10] now I am out again! [16:05:14] * hashar waves [16:07:59] Alchimista, you could try bots-nr2 [16:07:59] I think that's unused - https://labsconsole.wikimedia.org/wiki/Nova_Resource:Bots [16:08:33] Thehelpfulone: ok, i must finish something here, 2 min.. [16:10:07] sure [16:10:31] jeremyb, did we disable the bot that tells everyone who added whom to which project? [16:10:36] !log bots added Alchimista [16:10:38] Logged the message, Master [16:15:17] hello lovely people [16:15:25] I'm Dan Andreescu, analytics [16:16:05] I just rebooted a labs instance (reportcard2) to fix the read-only home directory [16:16:12] and now I can't ssh into it [16:16:31] *** /dev/vda1 will be checked for errors at next reboot *** [16:16:31] *** /dev/vdb will be checked for errors at next reboot *** [16:16:35] those are two of the errors [16:16:49] and these are two others: [16:16:50] Creating directory '/home/milimetric'. [16:16:56] Unable to create and initialize directory '/home/milimetric'. [16:17:23] milimetric: will be checked for errors is not an error... [16:17:25] that's reportcard2.pmtpa.wmflabs [16:17:26] Thehelpfulone: erm, i never knew about that bot [16:17:40] well, it won't let me ssh into it [16:17:43] sure [16:18:05] let me know if you'd like any other info [16:18:10] drdee can't hit it either [16:18:39] thanks for looking into it :) [16:18:53] it happened at a very bad time - I just took down the reportcard for the metrics meeting [16:19:00] * jeremyb can't look into it... [16:19:06] maybe andrewbogott_afk or paravoid can? [16:19:14] ok, cool [16:19:16] (reportcard2's borked) [16:19:22] milimetric, metrics is tomorrow right? [16:19:26] yep [16:19:31] milimetric: hah, bad time ;) [16:19:36] indeed [16:19:41] i have to deploy the new limn :) [16:20:05] milimetric: well it's all puppetized right, so you can just build a new one? ;-P [16:20:27] server guy I am not :) [16:21:16] so maybe I can get otto to help but do you think that'd be easier than fixing this? [16:22:56] let's first try to fix it [16:23:17] paravoid, are you around? we have a borked labs instance that we need for tomorrows metrics meeting [16:25:33] do you need to do some work on it too drdee? I wouldn't worry too much if not, ~30 mins until the usual SF folk turn up [16:26:17] Thehelpfulone: that early?? ;) [16:43:25] well if we're lucky :P [16:49:54] hey [16:49:55] what's up [16:50:11] reportcard2? [16:50:13] looking [16:50:14] yup [16:50:25] milimetric: ^ [16:50:33] hi! [16:50:35] yes [16:50:40] can't ssh into it after reboot [16:50:58] Creating directory '/home/milimetric'. [16:50:58] Unable to create and initialize directory '/home/milimetric'. [17:00:51] sigh I have no idea how the new glusterfs setup works [17:01:33] and this non very unixy way of doing things (no entry in fstab) doesn't help me understand it :) [17:03:05] paravoid: i think it's an automount? [17:05:15] maybe from /etc/autofs_ldap_auth.conf [17:05:23] * jeremyb isn't really sure of course [17:06:08] sorry paravoid, is there anything I would know that would help? [17:06:16] * milimetric doesn't know almost anything [17:06:17] :) [17:07:03] milimetric: i think the problem is not specific to your instance [17:07:11] gotcha [17:07:49] so the reason I restarted this machine was because /home/milimetric was read-only [17:08:20] I think I'll wait for Andrew or Ryan, they should be waking up about now [17:09:22] paravoid: for future reference, would mike be good to poke about this? [17:09:36] I think so [17:09:56] k [17:10:57] ooh, just read about Cambridge. Congratulations Thehelpfulone!! Best of luck [17:11:28] thanks milimetric :) [17:11:47] yep, congrats :) [17:11:52] Thehelpfulone: where shall i do this reading? [17:12:29] or milimetric ? [17:12:55] * jeremyb also has cambridge, MA on the mind atm. but presumably you don't mean there [17:13:55] Thehelpfulone: i can't conect to bastion, did you add me there too? [17:14:57] it was on wmfall jeremyb https://lists.wikimedia.org/mailman/private/wmfall/ [17:15:10] but even though I'm on that list, I can't authenticate to find the message [17:15:15] Alchimista: you already have the shell right? [17:15:22] no :s [17:15:30] well you need that first then [17:15:34] milimetric: i'm not on that list [17:15:35] i mena, i only have git access [17:15:39] *mean [17:15:45] Alchimista, oops sorry - added you [17:15:49] :D [17:15:55] Thehelpfulone: i wants news! ;) [17:16:03] lol [17:17:27] i still can't connect :s If you are having access problems, please see:bla bla bla \n Connection closed by 208.80.153.207 [17:17:43] do i wait some time, or is it an ssh key problem? [17:18:03] i'm doing ssh -A alchimista@bastion.wmflabs.org by the way [17:18:43] I don't think you need the bastion.wmflabs.org bit? or is that how you're doing the instance? [17:18:59] Alchimista, did you use the correct type of SSH key? [17:19:20] i'm following the manual: https://labsconsole.wikimedia.org/wiki/Access#Accessing_public_and_private_instances [17:20:13] Thehelpfulone: well the ssh key worked for git, so it should work for labs, right? [17:20:15] it needs to be an OpenSSH ormat [17:20:32] yeah it should do I think (presuming you've added it at https://labsconsole.wikimedia.org/wiki/Special:NovaKey ) [17:21:07] yap [17:24:45] do you use windows Alchimista? [17:25:12] nop, ubuntu [17:25:41] hmm "clear any cache settings or deny host settings that may also prevent you from ssh�ing into bastion. [17:25:41] [edit]" -- - jeremyb are you a labs admin? [17:27:33] Thehelpfulone: i isn't [17:27:47] Thehelpfulone: mazel tov! [17:30:37] spaciba! [17:49:42] PROBLEM Total processes is now: WARNING on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS WARNING: 151 processes [17:50:15] Thehelpfulone: i've done better, removed all my keys, and restarted ssh-server, uploaded a new key and nnothing [17:50:21] i still can't conect to bastion [17:57:18] Alchimista, are you going to be around for a bit? Ryan Lane is pretty good at troubleshooting given he's the lead on Labs :) [17:57:34] sure, i'll poke him later then :d [17:58:05] and thanks by the help :D [17:59:44] RECOVERY Total processes is now: OK on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS OK: 149 processes [18:05:16] milimetric: reportcard2's still broke? [18:05:19] * jeremyb spies a Ryan_Lane [18:05:36] jeremyb I will try again but I don't think anyone did anything [18:05:46] yes, still broken [18:05:51] 09 16:50:56 < milimetric> Creating directory '/home/milimetric'. [18:05:51] 09 16:50:57 < milimetric> Unable to create and initialize directory '/home/milimetric'. [18:05:54] 09 17:07:48 < milimetric> so the reason I restarted this machine was because /home/milimetric was read-only [18:06:11] Ryan_Lane ^ we're not able to connect to labs instances via ssh [18:06:13] Ryan_Lane: ^ paravoid looked and deferred to you [18:06:17] thanks jeremyb :) [18:06:52] Ryan_Lane: oh, you're here now. I've lost track of the gluster /home work [18:07:03] how do we mount gluster nowadays? [18:07:12] how is glusterfs spawned? [18:07:29] Alchimista: did you get sorted? [18:08:02] nops, still can't connect [18:08:27] Alchimista: what key are you using? i see 4 in ldap and one of those is formatted differently from the other 3 [18:09:54] jeremyb: the last one, i'll clean all the others. i've got one single key in ./ssh [18:11:50] now there is only on on labs too [18:13:16] Alchimista: so what's the key in ssh then? ssh-keygen -l -f .ssh/foo.pub [18:14:38] i've followed the labs tutorial: https://www.mediawiki.org/wiki/Git/Tutorial#Set_Up_SSH_Keys_in_Gerrit so it's ./ssh/id_rsa.pub [18:15:31] jeremyb: ^ [18:15:56] Alchimista: so run `ssh-keygen -l -f .ssh/id_rsa.pub` [18:16:03] and tell me what it says [18:16:08] ./ssh is the wrong place [18:16:32] 2048 b8:2e:d5:0e:01:d6:e9:f2:74:a6:8e:ca:37:2a:8f:6f alchimistawp@gmail.com (RSA) [18:18:55] Ryan_Lane: got you the patch to have 'beta' application servers to mount /dev/vdb on /srv : https://gerrit.wikimedia.org/r/#/c/42743/ :D [18:19:15] and we also have wmf6 / wmf7 wikis in beta now :-] [18:25:14] jeremyb: 2048 b8:2e:d5:0e:01:d6:e9:f2:74:a6:8e:ca:37:2a:8f:6f alchimistawp@gmail.com (RSA) where should be the key? [18:25:31] hold on, now i'm logged on bastion [18:27:02] and now i'm in \o/ [18:34:57] drdee: uuuuugggghhhhh [18:35:08] reportcard2 is running oneiric? [18:35:36] Alchimista: keys go in .ssh not ./ssh [18:36:55] hashar: cool [18:37:15] milimetric ^^ [18:37:26] i see yea [18:37:31] is reportcard2 puppetized? [18:37:34] no [18:37:39] lets see how it goes [18:37:52] I'm going to see if I can add the proper version of glusterfs on it [18:37:59] ok [18:38:07] if it doesn't work in oneiric, there isn't much I can do [18:38:13] you guys really need to puppetize this [18:38:16] gotcha, we'd have to update it? [18:38:20] and to move it to a precise instance [18:38:26] no. upgrades don't really work [18:38:26] yes, we should [18:38:31] oh ok [18:38:49] in fact, that usually permanently break the instance [18:38:58] so there'd be no way to regain ssh access to it if oneiric doesn't like glusterfs, correct? [18:38:58] most of the time people get locked out half-way through [18:39:15] I can likely add your keys to root's key [18:39:17] but... [18:39:29] that would be just to get stuff off of it [18:39:38] puppet would overwrite it [18:40:59] labs-morebots, feeling ok? [18:40:59] I am a logbot running on i-0000015e. [18:40:59] Messages are logged to labsconsole.wikimedia.org/wiki/Server_Admin_Log. [18:40:59] To log a message, type !log . [18:41:10] ok, hm, well if I could get back in I would just fix up the deployment of the reportcard so it's ok for tomorrow [18:41:17] then we can puppetize it after tomorrow [18:42:35] milimetric: does it use anything out of /home [18:42:36] ? [18:42:55] are you going to blow /home out? [18:43:02] well, /home is gluster [18:43:08] no glusterfs, no /home [18:43:23] to deploy the new version of limn, I'll need a writeable /home or to figure out some work-around [18:43:35] are you guys seriously running stuff out of /home? [18:43:38] nono [18:43:49] it just needs to write to /home/user/.npm when it does npm install [18:43:58] and I think I can configure it to use some other directory [18:44:11]