[00:19:30] No worries Ryan_Lane, anyway the identifier is complex enough :] I target using -G option [00:19:45] salt -G 'host:app1' test.ping [00:19:52] * Ryan_Lane nods [00:20:04] renoirb: well, I mean other projects will conflict [00:20:09] other than webplatform [00:20:18] ooh, that. [00:20:36] it was a poor early on decision that causes us problems now :) [00:20:42] and isn't easy to change [00:20:52] but we target by hosts name. I might just create a set of grains per VM type and stop relying on the name. [00:21:09] ok. [00:21:11] it's hard to target on grain unless you inject it when the instance boots [00:21:20] targeting host name is fine [00:21:40] I'd say use something like webp-app1 [00:32:27] ok, i'll do Ryan_Lane [00:38:23] if im running php on tools, to i need to install my own lamp instance? [00:39:39] html seems like it is being served, but php does not seem to run [00:40:21] Coren: ^ [00:41:09] notconfusing: It should be; but the scripts need to be executable (most common mistake) [00:41:32] chmod a+x something.php [00:41:50] Coren, thanks [00:58:23] Coren, i dont have permissions to chmod the files [00:58:42] and the owner doesn't seem to either [00:58:49] of the tool [00:58:52] ... that's odd. [00:58:57] What file is this? [01:00:01] Coren, sorry the owner just said he could [01:00:53] So it did work after all? [01:01:28] Coren http://tools.wmflabs.org/doi-bot/doibot.php [01:02:20] but /data/project/doi-bot/public_html/doibot.php is executable [01:03:01] Ah, I see all the files are now executable; you almost certainly want to have them all owned /by the tool/ (second most common mistake). There is an utility named take you can use for this, from the tool account. [01:03:28] wrought is the owner, just enetered [01:04:03] groovy [01:07:05] Coren, its working a little [01:48:08] !ping [01:48:08] !pong [11:57:55] !ping [11:57:55] !pong [11:57:57] wow [11:58:04] in all the channels I'm in, there's literally nobody speaking [11:58:05] grr [12:45:48] omfg [12:45:53] who keeps removing these channels from topic [12:46:41] I keep forgetting their names [12:46:53] !channels is | #wikimedia-labs-nagios #wikimedia-labs-offtopic #wikimedia-labs-requests [12:46:53] Key was added [12:47:00] !channels [12:47:00] | #wikimedia-labs-nagios #wikimedia-labs-offtopic #wikimedia-labs-requests [12:58:39] petan: who removes it is apparently Coren :) [12:58:55] 17:38 Coren set the topic at: 16-Oct-2013 12:23 am [13:02:47] maybe it wasn't there before [13:27:56] pet an; we are using wm-bot2 in #wikimedia-analytcs, thanks for developing this! but there is one small bug where the bot will just randomly clutter the channel with 'False' [13:27:56] Hey drdee, you are welcome! [13:28:13] do you track bugs somewhere? [13:28:28] petan: ^ [13:28:33] drdee: he loves pings :P [13:28:39] good :) [13:29:00] also a bot that says thank you when you want to file a bug :) [13:29:05] heh [13:29:18] drdee: it looks for the strings wm-bot and 'thank' in the same line [13:29:26] or wm-bot and thanks [13:29:26] Hey YuviPanda, you are welcome! [13:29:28] yeah [13:31:52] drdee lemme check [13:32:14] ohh it only happens when i do @rss-on [13:32:38] so we disabled that for now but i would like to be able to use rss feature [15:02:57] !tunnel [15:02:57] ssh -f user@bastion.wmflabs.org -L :server: -N Example for sftp "ssh chewbacca@bastion.wmflabs.org -L 6000:bots-1:22 -N" will open bots-1:22 as localhost:6000 [15:19:32] !log deleted tools-tyrant and tools-exec-cyberbot (cleanup of obsoleted instances) [15:19:33] deleted is not a valid project. [15:19:40] !log tools deleted tools-tyrant and tools-exec-cyberbot (cleanup of obsoleted instances) [15:19:43] Logged the message, Master [16:06:36] hi guys [16:06:48] i'm new to wm-labs [16:07:49] I created a new tool here http://tools.wmflabs.org/. it's called 'chie-bot'. [16:08:52] on https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help it says 'Every member of the group has the authorization to sudo to the tool account.' [16:09:15] I tested: id $(whoami) [16:09:15] uid=3794(leloiandudu) gid=500(wikidev) groups=50062(project-bastion),50380(project-tools),50813(local-chie-bot),500(wikidev) [16:09:31] so, it says i'm in 50813(local-chie-bot) group [16:09:44] but when i do become chie-bot [16:09:44] sudo: sorry, a password is required to run sudo [16:09:52] what shoud I do? [16:23:39] YuviPanda: There's now python-flask-sqlalchemy_1.0-1_all.deb in /data/project/repo -- want to give it a try, or shall I? [16:26:36] Leloiandudu: That sounds weird, it should just work. Which box are you logged into? [16:27:25] andrewbogott: leloiandudu@tools-login [16:30:35] Leloiandudu: Well… I don't see anything wrong :( Hopefully Coren will arrive soon, he might have ideas. [16:30:45] andrewbogott: thanks [16:32:05] maybe I missed some step? e.g. here: https://wikitech.wikimedia.org/wiki/User:Magnus_Manske/Migrating_from_toolserver it says I needed to create local-TOOL user and add myself to local-TOOL manually. but I found it's already done. is this tutorial outdated somehow? [16:33:58] Leloiandudu: I think if you create the group then you are automatically added [16:35:45] but I didn't. all i did was created tool here http://tools.wmflabs.org/. after that 'id $(whoami)' shows that local-TOOL group is created and I'm already in it. [16:36:09] um… by 'create the group' I mean, 'create the tool'. Same thing, in this context. [16:36:18] oh, ok [16:36:29] How long ago did you create it? [16:36:32] andrewbogott: thanks [16:38:06] did you re-login? there's something with groups that need to be synced on login [16:38:46] around an hour ago [16:39:13] Yep, might be worth logging out and in to tools-login and trying again [16:39:33] valhallasw: yeeaaah... i haven't thought about that... now it works. thanks a lot! [16:39:49] thank you guys [16:41:38] andrewbogott: I'm here (burp - lunch). What be up? [16:41:50] Coren -- resolved now, nevermind :) [17:37:01] renoirb: hm. an app server says you weren't in sudoers... [17:37:04] let me check the sudo policy [17:37:31] hm. it should allow any project member to sudo [17:37:49] on any project instance [17:37:50] Hey. [17:38:08] Yes, I just ran salt on a few of my new boxes and I jailed myself out :/ [17:38:17] oh. the sudoers file [17:38:20] right. salt overwrites that [17:38:25] sure [17:38:30] heh [17:38:35] And since the home is shared across the project [17:38:41] I would overwrite my own stuff [17:38:44] well, that shouldn't matter [17:38:49] oh [17:38:52] you mean your ssh info? [17:38:57] that's managed by LDAP [17:39:04] no, not the ssh info. [17:39:23] you should disable the user management states [17:39:36] I mean on usera@node1 I write in /home/usera… but is the same as node2:/home/usera [17:39:41] yeah [17:39:48] yeah [17:39:53] what's wrong with shared homes? [17:40:03] I'm using this now: https://github.com/saltstack-formulas/users-formula/blob/master/users/init.sls [17:40:18] nothing, just that I did not realize it earlier. [17:40:31] yeah, you should just not use the user state [17:40:33] I will skip the user management in the top state then :) [17:40:37] yep [17:40:48] labs handles auth automatically [17:41:14] yeah, my problem is that I am testing to build from scratch. I'm hitting snags where I should not be hitting because of this. [17:41:17] the sudo state should be disabled too [17:41:36] yeah. it won't be possible to test full states [17:41:44] Sure. Anyway, Users on each nodes should not be possible. [17:41:50] Only useful for debugging, nothing else. [17:42:12] yeah, we don't really do anything with sudo except use it for ops and dev groups [17:42:13] I'll have time for testing later in the new infra, but I want to go as far as i can [17:42:20] * Ryan_Lane nods [17:42:25] indeed [17:42:46] let me know if you run into any issues [17:45:15] ok thanks Ryan_Lane [17:59:18] andrewbogott: sorry, was away [17:59:29] andrewbogott: we've a release today, our last one for a while, so I expect I won't be able to do much [17:59:30] sorry! [17:59:36] andrewbogott: but yeah, trivial enough to test. [17:59:47] andrewbogott: though you should uninstall the current version of flask-sqlalchemy that I installed via pip :D [18:00:11] YuviPanda: no problem. Can you tell me what I need to do to support the older version of redis? [18:00:31] andrewbogott: there's the diff in /home/yuvipanda/invisible-unicorn. +1 -1 [18:00:40] ok, simple enough [18:00:42] yeah [18:00:46] s/StrictRedis/Redis/ [18:00:47] done [18:05:59] Ok, Ryan_Lane I'm cancelling the users management :/ [18:06:00] hahaha [18:06:09] I am abandoning it! [18:06:21] should've listened to you earlier. [18:06:25] Ryan_Lane: are there APIs for 1. adding an SSH key, and 2. creating service groups? [18:06:51] Ryan_Lane: just thinking of creating a heroku-type system for tools [18:11:21] YuviPanda: no APIs for either [18:11:40] should be relatively simple to add it to OpenStack Manager [18:11:58] when you say heroku-like...? [18:12:20] Ryan_Lane: 'tools init', 'tools create', etc [18:12:21] similar to the mediawiki project we were looking at? [18:18:17] Ryan_Lane: yah, but a lot more freeform [18:18:24] * Ryan_Lane nods [18:18:25] Ryan_Lane: and it'll be a lot easier, since it'll just be sshing in to tools and doing things [18:18:39] it would be pushing code to local git repos? [18:19:03] well, it would be sshing to a user. it can't ssh directly to a tool [18:19:05] tools don't have ssh [18:20:18] Ryan_Lane: yeah, local repo [18:20:18] Ryan_Lane: of course. [18:20:22] Ryan_Lane: sshing as user and then doing things, like becoming tools and then doing things :D [18:20:23] * Ryan_Lane nods [18:20:32] we can make that into a gerrit repo at some point in the future if that happens [18:20:56] Ryan_Lane: will also give me a nice excuse to learn Go or something :D [18:20:56] with a gerrit repo it could use git-deploy [18:21:17] though I guess it's not necessary since we're putting everything on a shared filesystem [18:21:19] Ryan_Lane: this should be a lot more interactive, and more geared towards exploratary things than deploymenty-things [18:21:27] * Ryan_Lane nods [18:21:40] Ryan_Lane: have you seen iPython notebook? [18:21:58] I'd like to make that a supported service on tools at some point, in some (secure) way. Expands horizons to a lot of people [18:22:03] and isn't too hard either [18:22:38] heh. that sounds... difficult to do securely [18:23:55] Ryan_Lane: securely *And* easily is hard. Just securely... :P [18:24:19] Ryan_Lane: 1. Run the core process on grid, 2. ssh tunnel from local machine 3. Exports are just simple HTML files that can be put in public_html [18:24:29] just need to have a tool that can say [18:24:33] tools ipython start "name" [18:24:38] and it opens up a browser for you to work on [18:24:46] and you hit save, and there's a public URL you can share! [18:25:08] Ryan_Lane: I've tested it out manually, it works. Just need to automate it [18:25:24] Ryan_Lane: would also require *users* to submit jobs to the grid [18:27:26] Ryan_Lane: have you seen http://zerovm.org/wiki/The_Cloud_Hypervisor [18:27:43] meh, 1.20 [18:32:10] heh [18:33:25] this is a hypervisor that runs within a node engine? :) [18:34:36] some..thing like that, yes [18:35:10] heh [18:35:24] this wiki doesn't have much info ;) [18:44:02] (03PS1) 10Andrew Bogott: Add setup.py and organize for easy install. [labs/invisible-unicorn] - 10https://gerrit.wikimedia.org/r/91429 [18:45:05] (03PS1) 10Andrew Bogott: Yuvi tells me this will make things work with older redis. [labs/invisible-unicorn] - 10https://gerrit.wikimedia.org/r/91430 [18:45:08] YuviPanda: ^ and ^^ [18:46:35] (03CR) 10Yuvipanda: [C: 04-1] "(1 comment)" [labs/invisible-unicorn] - 10https://gerrit.wikimedia.org/r/91430 (owner: 10Andrew Bogott) [18:46:47] andrewbogott: ^ 1 comment, + change commit message? [18:47:50] What should the commit message say? [18:49:45] andrewbogott: 'Compatibility for older python-redis'? [18:50:40] (03PS2) 10Andrew Bogott: Use 'redis' rather than Strictredis. [labs/invisible-unicorn] - 10https://gerrit.wikimedia.org/r/91430 [18:51:26] andrewbogott: wait, I don't have +2 on that repo!!! [18:51:32] huh [18:52:34] Is that because the group I added before doesn't exist anymore? [18:52:46] YuviPanda: hm. I wonder if zerovm will have an openstack driver [18:53:40] ^d, can you give YuviPanda +2 on https://gerrit.wikimedia.org/r/#/admin/projects/labs/invisible-unicorn? I thought I did it before but now it's gone again :( [18:54:07] <^d> No, don't think you did. [18:54:18] <^d> Change owner to ldap/wmf or something instead of Project and Group Creators. [18:55:19] * Coren strongly believes that when Ubuntu reaches U it needs to be 'uncanny unicorn' [18:55:44] what do they do after z? [18:55:46] 'or something'? [18:55:53] <^d> jeremyb: No more ubuntu. [18:55:59] orly! [18:56:31] ^d, sorry, I really don't follow at all. I presume you don't mean that I should literally enter the string 'ldap/wmf' there... [18:56:41] <^d> Yes, I do. [18:56:59] Won't that give /everyone/ +2? [18:57:08] <^d> Everyone in ldap/wmf. [18:57:17] ... [18:57:24] That's ok I guess [18:57:27] YuviPanda: try now? [18:58:49] (03CR) 10Yuvipanda: [C: 032 V: 032] Use 'redis' rather than Strictredis. [labs/invisible-unicorn] - 10https://gerrit.wikimedia.org/r/91430 (owner: 10Andrew Bogott) [18:59:01] andrewbogott: still can't merge / 'publiush' [19:01:16] now? [19:07:18] andrewbogott: done [19:07:49] Regarding the other patch… I'm not sure what else needs doing to turn it into a proper service. Maybe nothing? [19:09:17] (03CR) 10Yuvipanda: [C: 04-1] "(3 comments)" [labs/invisible-unicorn] - 10https://gerrit.wikimedia.org/r/91429 (owner: 10Andrew Bogott) [19:10:53] (03PS2) 10Andrew Bogott: Add setup.py and organize for easy install. [labs/invisible-unicorn] - 10https://gerrit.wikimedia.org/r/91429 [20:10:43] Hello, I wanted to ask if it is possible to manually test some sql queries and output them to a csv file for example [20:11:34] My problem is that I get the 1045 error message [20:16:14] Asaifm, this is a Access denied error [20:16:18] an [20:16:23] yes [20:16:55] your user have access to the database you are querying? [20:16:56] should I adjust a configuration file or reset a certain password [20:17:16] I'm not sure [20:17:58] I can query the database replicas but I cannot store the results in files [20:18:36] you only get this error when you're trying to save the results in files? [20:18:56] yes [20:19:44] weird [20:20:38] could you pastbin it? [20:23:39] Ryan_Lane: do we have to use 'Instances list''s reboot button to reload the VM and let it work Puppet things? [20:23:43] Or a simple sudo reboot can do? [20:23:56] sudo reboot works [20:24:00] ok [20:24:08] unlike hp cloud, we won't destroy your instance with a reboot :) [20:24:10] Lots of things rewrites and break things hahaha [20:24:28] yeah, it's a slightly different environment [20:24:31] I got jailed out because of resolv.conf being rewritten [20:24:32] :/ [20:24:35] :D [20:24:37] right [20:24:47] jailed out as in, salt minion wasn't finding the master anymore [20:24:53] honestly, I'd like us to not have to mess with resolv.conf [20:25:00] indeed [20:25:03] or hosts [20:25:11] I dislike having to do it [20:25:22] the only reason I did is because we don't have access to DNS [20:25:24] i thought of creating a local dns with what you suggested (i have a note for this, with the name you of the package you recommended) [20:25:31] gdnsd? [20:25:35] yes [20:25:37] yep [20:25:39] we have access to dns [20:25:41] I do. [20:25:43] yeah [20:25:56] we may want to point the records to our own DNS at some point [20:26:02] so that we can have automated control of it [20:26:02] I might just create a dns subdomain to relay to it when I have one. [20:26:06] indeed [20:26:46] DNS is the 'nerf de guerre' of a network. It must not fail [20:26:55] (french expression) [20:27:21] @HenriqueCrang Nothing fancy is done here, I tried this simple query http://pastebin.com/yhqJ42Z9 on several replicas [20:27:31] I'm even stumbling on snags such as rewriting /etc/sudoers. I am now ensuring it adds or uncomment lines in it instead. [20:27:48] it's better to not mess with sudoers [20:27:53] and instead use sudoers.d [20:27:54] and there is this autogenerated .cnf file that has a user name but not password [20:28:16] On my local dbs I know I have to give my user a password [20:28:29] should I manipulate this .cnf file? [20:28:37] that's what I am uncommenting, the includefile /etc/sudoers.d/... line [20:29:47] Did you know that you can upgrade all salt stack minion from the master? ('m very happy to hear) [20:30:38] yeah [20:30:44] I'm in the process of doing that right now [20:30:49] on wikimedia's infra [20:30:59] salt '*' pkg.install 'salt-minion' [20:31:08] yeah, that's very nice! [20:31:24] Also, in 0.17 they now have salt-ssh. No need to install minions [20:31:40] we still want minions ;) [20:31:51] pub/sub is way better than ssh [20:31:58] and more reliable [20:32:10] Sure, but on some situations, very useful and besides, people who uses Ansible would use this against us [20:32:16] Asaifm, the user seted in the .my.cnf file has rights to write on the folder that you are working on? [20:32:53] I think it'll be useful for some situations [20:33:03] but the 0mq architecture is way more useful [20:33:51] @HenriqueCrang, no rights just a username and it's password [20:34:23] should I request permission for a certain project? [20:36:16] Asaifm, i think you should grant this user the linux permission to write on the folder that you want this file to be saved [20:36:40] renoirb: I like being able to batch things so that I don't kill my cluster :) salt -b 20 '*' pkg.install 'salt-minion' refresh=True [20:37:01] apt-get update on 400 nodes at once is rough, batches of 20 is way nicer :) [20:38:04] @HenriqueCrang ok thanks I will try and see how it goes [20:39:32] Ryan_Lane: http://i.imgur.com/C4buo.gif [20:39:51] That's so nice! [20:40:52] Is there an automated edit counter anywhere? [20:41:23] :D [20:42:48] Ryan_Lane: Bah. do a puppetd -tv on all nodes at once. The puppet master deserves the pain. :-) [20:43:03] I've done that in the past [20:43:07] the puppet master doesn't care [20:43:11] the compute nodes do [21:32:58] hey [21:33:09] oh labs console [21:35:39] milimetric: ? [21:35:46] I'm confused [21:35:50] I can ssh limn0 [21:35:53] but not ssh wikimetrics [21:36:01] I tried from bastion just to make sure it's not me [21:36:22] and I think console output says that wikimetrics is requesting some type of login? [21:36:36] but it seems to be up [21:36:46] 1 $> ssh root@wikimetrics [21:36:46] ssh_exchange_identification: Connection closed by remote host [21:36:47] like, the website hosted on that instance is running [21:36:54] I bet it OOM'd [21:37:06] ah [21:37:18] so I'll just reboot it, see if it helps [21:38:21] * Ryan_Lane nods [21:44:17] huh, Ryan_Lane, there is some really weird stuff happening with that instance [21:44:35] is there any way to get more information about what's going on with it? [21:45:20] and of course - now I can get into the box [21:45:22] nvm... [21:52:53] hmmm. qsub is not installed in the exec instances. [23:22:50] (03CR) 10Yuvipanda: [C: 032 V: 032] Add setup.py and organize for easy install. [labs/invisible-unicorn] - 10https://gerrit.wikimedia.org/r/91429 (owner: 10Andrew Bogott)