[04:08:25] hi Ryan_Lane, could I get public IP, I want some users test my extension. Probably I need it just for several days. [04:13:12] sure [04:13:27] what's the project name? [04:14:29] MichaelShavlovsk: ? [04:14:38] BlameMaps [04:15:07] !resource blamemaps [04:15:07] https://labsconsole.wikimedia.org/wiki/Nova_Resource:blamemaps [04:16:21] MichaelShavlovsk: done. you can allocate one now [04:16:30] thanks! [04:16:43] yw [04:25:47] Ryan_Lane: Are you fixing it now? [04:25:55] no [04:26:01] :( [04:26:07] sec [04:26:13] gimme an instance name [04:26:24] dumps-incr [04:26:31] I keep getting: -bash: cd: /data/project: Transport endpoint is not connected [04:27:02] you need to run this on all nodes: apt-get install glusterfs [04:27:04] as root [04:27:24] doing... [04:27:27] then, /etc/init.d/autofs restart [04:27:31] and it'll work [04:27:42] your instances may have been too overloaded to get the automated update [04:27:47] and puppet must not be runnin [04:28:04] last puppet ran 10 min ago :P [04:28:11] ah, great [04:28:18] so the trick is to restart the autofs thingy [04:28:22] thanks! :) [04:28:33] second thing [04:28:37] ah, right, puppet has => present and not => latest now [04:28:53] give me a moment... [04:29:01] ? [04:29:06] okay, dumps-2 is screwed, after I tried to reboot it on the web [04:45:19] was it one of the corrupted ones? [04:45:23] what's the instance id? [04:45:30] you're on labs-l, right? :) [04:45:38] I sent out a list of corrupted instances [04:51:56] hmm [04:52:05] possibly... [09:52:00] does anyone know if wikibugs script is in git? (http://svn.wikimedia.org/viewvc/mediawiki/trunk/tools/wikibugs/) [09:52:12] if so, where? [11:51:10] aude: I don't think those tools are in git yet, could be wrong though [11:57:24] Damianz: poking around, i think you are right [11:57:33] * aude obtained the script from svn :o [14:16:23] Hey folks. I'm having trouble accessing my instance through bastion. I'm online through port 80 for http://minnesota.wmflabs.org/ [14:16:37] But when I log into bastion and try to SSH to it, I fail: [14:16:46] halfak@bastion1:~$ ssh minnesota.pmtpa.wmflabs [14:16:48] ssh: Could not resolve hostname minnesota.pmtpa.wmflabs: Name or service not known [14:17:10] What about ssh minnesota? [14:17:19] dns was a little broken yesterday though I think Ryan fixed it [14:17:27] halfak@bastion1:~$ ssh minnesota [14:17:29] ssh: Could not resolve hostname minnesota: Name or service not known [14:17:38] halfak: what's the instance id? [14:17:50] I also don't see my instance appear in the list on labsconsole. [14:18:13] halfak: when you created, it was i-0000000 or something? [14:18:41] I had ori-l help me create it. I'm not familiar with the systems I'm working with. [14:19:36] Is there some way I can look that up? [14:20:58] which project is it? [14:22:28] Hmmm... I'm not sure what you mean. [14:22:30] for some reason, i had the same problem i think [14:22:35] i was able to do ssh i-0000039b.pmtpa.wmflabs [14:23:07] but not maps-osmmapnik.pmtpa.wmflabs (as of a few days ago, until someone manually fixed the dns) [14:23:15] i created my instance last friday [14:23:24] and don't see it in the list of instances but it is there [14:23:37] paravoid: any idea? [14:23:38] Gotcha. [14:24:00] paravoid might be able to help, but without the i-0000039b (or whatever yours is), i'm not sure how to find it [14:24:58] Thanks for your help :) [14:25:39] otherwise, you might have to wait for Ryan but it's still rather early in SF [14:26:56] Indeed it is . [14:27:07] When you say Ryan, do you mean Ryan Lane? [14:27:49] mhm [14:28:16] SF is only awake at this time when you take bolt cutters to their fiber lines [14:32:57] Maybe if they are still up from the night before. [14:38:30] err: Could not retrieve catalog from remote server: Error 400 on SERVER: Duplicate definition: Package[apache2] is already defined in file /etc/puppet/manifests/webserver.pp at line 91; cannot redefine at /etc/puppet/manifests/webserver.pp:42 [14:49:49] halfak: ryan should be here maybe in ~2 hours or so [14:49:54] ryan lane [14:57:40] Thanks :) [16:43:09] is my instance supposed to renew its dhcp lease every 50 seconds? [16:44:02] Ryan_Lane: ^^^ [16:44:10] yes [16:44:25] why so often? [16:44:38] in case the instance switches network nodes [16:44:59] why does it matter? [16:45:04] I was just curious. [16:45:10] * Ryan_Lane nods [16:45:18] I'm used to setting renewal time much longer. [16:46:37] It's so when the dhcp server goes down everyone notices quickly [16:46:38] :D [16:53:31] Damianz: if it goes down, then updates don't happen [16:53:34] the instances keep the ip [16:53:51] mhm [16:53:58] btw is dns still a little broken? [16:54:41] can you elaborate? [16:55:22] [15:16:46] halfak@bastion1:~$ ssh minnesota.pmtpa.wmflabs [16:55:22] [15:16:48] ssh: Could not resolve hostname minnesota.pmtpa.wmflabs: Name or service not known [16:55:25] From earlier today [16:55:34] Never found out what his instance id was though [16:56:52] I don't see that record in ldap [16:57:23] it isn't in the wiki either [16:57:35] we do have an issue with arecords. it seems my live hack didn't take [16:58:07] Hmm he did mention something about the wiki *looks* [16:58:38] Hey guys. I just came back from AFK. [16:58:41] 15:17:38] halfak: what's the instance id? [16:58:41] [15:17:50] I also don't see my instance appear in the list on labsconsole. [16:58:45] [15:18:12] halfak: when you created, it was i-0000000 or something? [16:58:48] [15:18:41] I had ori-l help me create it. I'm not familiar with the systems I'm working with. [16:58:49] Ryan_Lane: This was an instance that ori-l set up for me. [16:58:52] [15:19:36] Is there some way I can look that up? [16:58:54] [15:20:58] which project is it? [16:58:57] [15:22:28] Hmmm... I'm not sure what you mean. [16:58:59] [15:22:30] for some reason, i had the same problem i think [16:59:02] [15:22:35] i was able to do ssh i-0000039b.pmtpa.wmflabs [16:59:04] Maybe just being a little special, thought it might be related to the issues you where trying to fix yesterday [16:59:08] Oh, hi halfak [16:59:10] Totally could have just not spammed the channel =/ [16:59:11] :) [16:59:28] We have logs somewhere actually, need a nice show from time to time interface though [16:59:31] I appreciate you helping me figure out the issue when I was absent though :) [17:00:59] Ryan_Lane: Do you know a way I could look up my instance ID when my instance isn't listed in labsconsole (should it be listed there?) [17:01:02] Not so much helping, only just got home from work :) Need to pack my climbing gear actually and find some food for this weekend =/ [17:02:42] yo halfak [17:02:46] only got a minute [17:02:58] halfak: that instance's name is: maps-osmmapnik.pmtpa.wmflabs [17:03:20] halfak: i-000003a5.pmtpa.wmflabs [17:03:24] Ryan_Lane: ?! [17:03:36] first time i see that hostname [17:03:39] ah. different instance id [17:04:06] osmmapnik is a lot of fun to say out loud, though [17:04:17] that instance is: loon.pmtpa.wmflabs [17:04:33] I should be "minnesota.wmflabs.org" [17:04:36] halfak: state bird of minnesota [17:04:49] it's that too [17:05:04] what's the problem with it? [17:05:04] Wait... am I also "loon"? [17:05:53] halfak: i'm not a psychiatrist [17:06:05] lol [17:06:05] but yes [17:06:38] an aside, here's why i like erlang: [17:06:40] <#erlang> (MononcQc) Poutine is like the triforce of junk food [17:06:42] OK. So the web address is minnesota.wmflabs.org, but I ssh to loon.pmtpa.wmflabs from bastion.wmflabs.org? [17:07:13] ori-l: ^^ [17:07:34] halfak: if you follow the instructions, you can make the middle step (connecting to bastion) transparent by declaring it as a proxy [17:07:39] I think if you work in IT you have to be part psychiatrist [17:07:41] did you try using the configuration i pastebin'd? [17:07:57] ori-l: I'm aware of the instructions, but I need to know the parameters. [17:08:11] halfak: sec, i'll give you the precise config i used to ssh into it [17:08:41] ori-l: I'm just going to learn the parameters from your config and write my own anyway. [17:10:01] halfak: http://dpaste.com/783904/ [17:10:34] i'm not sure it's fully correct. i hacked around [17:10:54] make sure it works by explicitly sshing into bastion and then sshing into your machine first [17:11:10] also the nickname of the machine is halfak so if you use this config verbatim just "ssh halfak" [17:13:24] Hmm... It looks like loon/minnesota/whatever doesn't want to accept my key. [17:13:50] halfak: add -vvv switch to the ssh command and pastebin somewhere? [17:14:15] No worries... I'm confirming the problem still. I figured a way I might be doing it wrong. [17:15:20] What's the preferred away to change my public key on bastion & minnesota/loon? Via labsconsole? [17:15:36] preferred and only [17:15:49] Hmm... Seems like the authorized_keys file is writeable :P [17:16:12] How fast is the turnaround for the update? Minutes? Seconds? [17:16:21] mins [17:16:28] and writeable, but will get overwritten [17:16:32] Indeed. [17:17:25] Is there any indication for when the key switch will be completed? [17:17:46] wm-bot may yell sruff [17:17:53] Cool. [17:20:18] * Damianz goes to find food so he doesn't starve on the crag [17:23:17] well, if you update in labsconsole and *also* overwrite authorized_keys, you don't have to wait [17:23:38] Sadly, I can't do that in loon/minnesota since I haven't been able to get in. [17:23:45] Assuming he can't login to the box he wants it updated on that might not work so well [17:23:55] Though, as home dirs are on nfs that might work [17:24:01] Should labsconsole be writing to authorized keys on both machines? [17:24:06] * Damianz really doesn't keep track as to where we're at with killing ngs [17:24:09] s/ngs/nfs/ [17:24:17] Home dirs are on NFS! [17:24:22] Awesome. [17:24:25] Yeah [17:24:29] Per project though [17:24:41] ori-l: What "project" am I? [17:24:43] Eventually I believe they will be on gluster and keys handled in pam [17:24:57] halfak: editor-engagement [17:25:04] That makes sense. [17:25:14] I would have guessed "e3" though. [17:28:56] Definitely not getting into loon with the same key that let me into bastion or the new one I wanted to switch to: http://dpaste.com/783930/ [17:29:43] http://dpaste.com/783932/ <-- New key [17:30:58] * halfak has to run to next meeting. Thanks to Damianz, Ori and Ryan for the help! [17:31:52] halfak: ori-l: there's an ldap cache. if you're denied for ldap then something on the instance needs a restart (or you need to wait for it to expire) [17:32:30] jeremyb: anything I can do to give it a kick? [17:32:44] halfak: get someone else on the box to restart it ;) [17:33:07] <\3 nscd [17:33:12] :\ [17:33:25] Any idea how long an expiration might take? [17:33:29] i think it's either nscd or nslcd [17:33:34] halfak: no clue [17:34:06] Gotcha. [17:34:30] nslcd probably for ldap queries though nscd can cache some ldap stuff [17:39:59] ugh [17:40:06] i hope halfak is not using ori-l's config [17:40:16] please everyone just use what's on the wiki. that's right [17:40:23] (ori too!) [17:40:51] jeremyb: It works fine... until you have an overcomplex .ssh/config file for ~8 different envs :) [17:41:23] Damianz: huh? [17:41:36] Damianz: you know i do use it myself? [17:42:05] Hopefully :P [17:42:21] Also, that blue unicorn is no where near as awesome as the pink one :( [17:43:55] > Host bastion1.pmtpa.wmflabs; Hostname bastion.wmflabs.org; ProxyCommand none; ; Host *.pmtpa.wmflabs; ProxyCommand ssh -e none bastion1.pmtpa.wmflabs exec nc -w 40 %h %p; [17:44:11] that's my whole config (at least the part that's not for non-WMF hosts) [17:44:23] just s/; /\n/g [17:44:36] how is that out of hand? [17:44:52] or should i count the non-WMF hosts in determining the complexity???? [17:45:00] That's fine, when you work for multiple different companies, with different un-standard domains it gets really messy. [17:45:42] ... [17:46:22] anyway, halfak: please just use what's on the wiki ;) [17:46:58] * Damianz goes back to racking gear and finding food [20:28:58] hmm [20:28:59] !pin [20:29:00] pong [20:31:51] Not sure, ask petan when he's here [20:33:05] don't think so [20:33:18] !log bot test [20:34:58] !log bot test [20:35:02] FFS [20:35:22] !log bot test [20:35:40] !log bot test [20:36:13] !log bot test [20:36:13] Can't contact LDAP for project list. [20:36:14] bot is not a valid project. [20:36:17] !log bot test [20:36:18] bot is not a valid project. [20:36:23] seriosuly [20:36:28] I fixed one but, I find another [20:37:52] !log bot test [20:37:52] bot is not a valid project. [20:37:59] !log bots test [20:37:59] bots is not a valid project. [20:38:12] !log testlabs test [20:38:12] testlabs is not a valid project. [20:39:19] !log bots test [20:39:20] Can't contact LDAP for project list. [20:39:20] bots is not a valid project. [20:41:20] nope [20:42:13] hmm proxy agent is there [20:43:21] !log testlabs test [20:43:21] Can't contact LDAP for project list. [20:43:21] testlabs is not a valid project. [20:48:17] Hmm [20:48:25] The query is right, it's just not returning it [20:48:32] Too late to debug this atm [20:52:30] For some reason it's failing to pull the groups from ldap so sees no projects [20:59:16] we should have docs for what does each puppet class and how they conflict with each oteher [20:59:24] *other [21:37:29] Damianz: which instance ae you running that bot from? [21:37:32] it shouldn't be bots-2 [21:38:41] 1 and labs/tools/w/e it's called, both throw shit at me [21:38:47] Busy watching defcon videos to care tbh [21:38:49] hm [21:38:51] heh [21:38:55] I'll take a look [21:39:41] It had the usual /var/run/adminbot doesn't exist issue then when loading the cache the ds search returned empty, though running the query in ldapsearch returns the groups as expected. [23:40:41] is there certificate for Issuer: C=US, ST=California, L=San Francisco, O=Wikimedia Foundation, CN=Labs CA