[00:43:52] <Ryan_Lane>	 andrewbogott: reviewed your changes. I had a minor −1 on one of them
[00:44:08] <andrewbogott>	 thanks
[00:44:27] <andrewbogott>	 What's the story with false vs FALSE vs False?
[00:44:37] <Ryan_Lane>	 false is the proper value
[00:44:41] * andrewbogott  hopes never to see 'false'

[00:44:42] <Ryan_Lane>	 not FALSE or False ;)
[00:45:00] <Ryan_Lane>	 err
[00:45:04] <andrewbogott>	 OK… FALSE was documented as the return value for that function.
[00:45:38] <Ryan_Lane>	 php is annoying in lots of ways
[00:46:30] <Ryan_Lane>	 http://stackoverflow.com/questions/3093576/false-false-any-difference#
[00:47:07] <andrewbogott>	 OK, so it's a style thing?
[00:47:11] <Ryan_Lane>	 yep
[00:47:18] <Ryan_Lane>	 and style in mediawiki is false, not FALSE
[00:47:55] * Ryan_Lane  likes when languages only give you one option

[00:48:35] <andrewbogott>	 ok
[00:56:32] <wm-bot>	 [bz] (NEW - created by: Ryan Lane, priority: Unprioritized - normal) [Bug 46817] Upgrade OpenStack to the Folsom release (tracking) - https://bugzilla.wikimedia.org/show_bug.cgi?id=46817
[00:59:02] <wm-bot>	 [bz] (NEW - created by: Ryan Lane, priority: Unprioritized - normal) [Bug 46818] Switch DNS middleware to Moniker - https://bugzilla.wikimedia.org/show_bug.cgi?id=46818
[00:59:32] <Krenair>	 Ryan_Lane, so what's this openstack notification system?
[00:59:48] <Ryan_Lane>	 Krenair: it's how openstack handles plugins
[01:00:24] <Ryan_Lane>	 it's an asynchronous system, so it sends notifications that can be picked up
[01:00:41] <Ryan_Lane>	 the plugin system was mostly written by andrewbogott
[01:01:44] <wm-bot>	 [bz] (RESOLVED - created by: Faidon Liambotis, priority: Unprioritized - normal) [Bug 39782] Switch instances to use virtio network adapters - https://bugzilla.wikimedia.org/show_bug.cgi?id=39782
[01:04:56] <wm-bot>	 [bz] (NEW - created by: Ryan Lane, priority: Unprioritized - normal) [Bug 46819] Change network configuration to network-node per compute-node - https://bugzilla.wikimedia.org/show_bug.cgi?id=46819
[01:06:32] <wm-bot>	 [bz] (NEW - created by: Ryan Lane, priority: Unprioritized - normal) [Bug 46821] Add BGP support to Quantum for floating IP addresses - https://bugzilla.wikimedia.org/show_bug.cgi?id=46821
[01:07:54] <wm-bot>	 Change on 12mediawiki a page Wikimedia Labs/Stability improvement project was modified, changed by Ryan lane link https://www.mediawiki.org/w/index.php?diff=667889 edit summary: [+196] /* OpenStack */
[01:08:23] <wm-bot>	 Change on 12mediawiki a page Wikimedia Labs/Stability improvement project was modified, changed by Ryan lane link https://www.mediawiki.org/w/index.php?diff=667890 edit summary: [+11] /* OpenStack */
[01:10:26] <wm-bot>	 [bz] (NEW - created by: Ryan Lane, priority: Unprioritized - normal) [Bug 46822] Shrink gluster volumes from 4 bricks to 2 bricks - https://bugzilla.wikimedia.org/show_bug.cgi?id=46822
[01:11:59] <wm-bot>	 [bz] (NEW - created by: Ryan Lane, priority: Unprioritized - normal) [Bug 46823] Replace glusterfs - https://bugzilla.wikimedia.org/show_bug.cgi?id=46823
[01:12:41] <wm-bot>	 Change on 12mediawiki a page Wikimedia Labs/Stability improvement project was modified, changed by Ryan lane link https://www.mediawiki.org/w/index.php?diff=667891 edit summary: [+5] /* Gluster */
[01:15:45] <wm-bot>	 [bz] (NEW - created by: Ryan Lane, priority: Unprioritized - normal) [Bug 46824] Switch DNS to gdnsd - https://bugzilla.wikimedia.org/show_bug.cgi?id=46824
[01:18:47] <wm-bot>	 [bz] (NEW - created by: Ryan Lane, priority: Unprioritized - normal) [Bug 46825] Write a Moniker backend for gdnsd - https://bugzilla.wikimedia.org/show_bug.cgi?id=46825
[01:20:09] <wm-bot>	 Change on 12mediawiki a page Wikimedia Labs/Stability improvement project was modified, changed by Ryan lane link https://www.mediawiki.org/w/index.php?diff=667893 edit summary: [+127] /* OpenStack */
[01:20:34] <wm-bot>	 Change on 12mediawiki a page Wikimedia Labs/Stability improvement project was modified, changed by Ryan lane link https://www.mediawiki.org/w/index.php?diff=667894 edit summary: [-1] /* OpenStack */
[01:44:55] <jeremyb_>	 Coren: they're not even imaged yet?
[01:45:08] <jeremyb_>	 just to move to a less busy place :)
[01:45:28] <Coren>	 "they"?  The labstore100x?  Nope.
[01:45:38] <jeremyb_>	 yes, they :)
[01:48:48] <Coren>	 jeremyb_: About to do just that.
[01:49:12] <Coren>	 Once someone +2 and merges the changeset.  (hint, hint)
[01:49:48] <jeremyb_>	 Coren: uhhh, mixing uppercase and lowercase MACs :( (but already was inconsistent when you started)
[01:50:11] <jeremyb_>	 hint: i don't have +2
[01:50:22] <Coren>	 Ah.
[01:50:33] <jeremyb_>	 you do actually
[01:50:41] <jeremyb_>	 but you should get a review first :)
[01:50:49] <Coren>	 I know, but self +2 is not a good thing.  :-)
[01:52:11] <Coren>	 Ryan_Lane: How does that fit in your philosophy.  Self +2 after review on a "small" change?
[01:52:29] <Ryan_Lane>	 it's generally fine
[01:53:15] <Coren>	 (I always use lowercase for hex, personally.
[07:51:31] <liangent>	 any guide available when I want to puppetize my instance configuration?
[07:54:44] <liangent>	 !puppet
[07:54:44] <wm-bot>	 learn: http://docs.puppetlabs.com/learning/  troubleshoot: http://docs.puppetlabs.com/guides/troubleshooting.html
[07:54:54] <liangent>	 !
[07:54:54] <wm-bot>	 There are multiple keys, refine your input: !log, $realm, $site, *, :), access, account, account-questions, accountreq, addresses, addshore, afk, alert, amend, ask, b, bang, bastion, beta, blehlogging, blueprint-dns, bot, botrestart, bots, bots-bsql, botsdocs, broken, bug, bz, cmds, console, cookies, coren, credentials, cs, damianz, damianz's-reset, db, del, demon, deployment-beta-docs-1, deployment-prep, docs, documentation, domain, epad, etherpad, extension, -f, forwarding, gerrit, gerritsearch, gerrit-wm, ghsh, git, git-branches, git-puppet, gitweb, google, gridbots, group, help, hexmode, home, htmllogs, hyperon, icinga, info, initial-login, instance, instance-json, instancelist, instanceproject, keys, labs, labsconf, labsconsole, labsconsole.wiki, labs-home-wm, labs-morebots, labs-nagios-wm, labs-project, labswiki, leslie's-reset, link, linux, load, load-all, logs, mac, magic, mail, manage-projects, meh, mobile-cache, monitor, morebots, msys, msys-git, nagios.wmflabs.org, nagios-fix, newgrp, new-labsuser, new-ldapuser, nova-resource, op_on_duty, openstack-manager, origin/test, os-change, osm-bug, pageant, password, pastebin, pathconflict, petan, ping, pl, pong, port-forwarding, project-access, project-discuss, projects, puppet, puppetmaster::self, puppetmasterself, puppet-variables, putty, pxe, python, q1, queue, quilt, report, requests, resource, revision, rights, rt, Ryan, ryanland, sal, SAL, say, search, security, security-groups, sexytime, single-node-mediawiki, socks-proxy, ssh, sshkey, start, stucked, sudo, sudo-policies, sudo-policy, svn, terminology, test, test2, Thehelpfulone, time-greece, tl, tooldocs, trout, tunnel, unicorn, whatIwant, whitespace, wiki, wikitech, wikiversity-sandbox, windows, wl, wm-bot, wmflabs, 
[07:55:57] <legoktm>	 !addshore
[07:55:57] <wm-bot>	 lolz
[07:56:01] <legoktm>	 ahaha
[07:56:03] <legoktm>	 !:)
[07:56:03] <wm-bot>	 /me laughs
[07:57:42] <liangent>	 !afk
[07:57:42] <wm-bot>	 Away From Keyboard
[08:12:22] <hashar>	 why the hell labs is always having troubles
[08:12:41] <hashar>	 one of my instance now receives a different IP address though the DNS entry has not been updated :/
[08:13:06] <hashar>	 ah no
[08:13:39] <hashar>	 I am ranting 
[12:56:10] <addshore>	 !legoktm 
[12:56:16] <legoktm>	 heheh
[12:56:20] <legoktm>	 btw
[12:56:22] <legoktm>	 i moved a page
[12:56:28] <legoktm>	 and your bot is updating the old one still
[12:57:22] <addshore>	 I noticed :P
[12:57:36] <addshore>	 about 12 hours ago :P
[14:40:53] <jeremyb_>	 hehehe, Coren's having fun :)
[14:44:01] <Darkdadaah>	 What's he doing?
[14:47:06] <Coren>	 The joys of remote installs.
[14:47:22] <Coren>	 There are downsides to not being able to just sit up and walk to the MR.
[14:49:13] <Darkdadaah>	 Coren: Do you have an idea when the tools project will be stable enough to migrate my tools?
[14:50:11] <Coren>	 Darkdadaah: It's been working cleanly without outages for two weeks, but with gluster performance.  If your task isn't too I/O bound, now is a good time to start moving things and testing them.
[14:50:23] <Coren>	 Gluster will be replaced shortly, to the joy of all.
[14:50:49] <Darkdadaah>	 Coren: good news :)
[14:51:12] <Coren>	 Otherwise I already have a number of bots running without a hitch.
[14:51:14] * jeremyb_  wonders what an MR is

[14:51:22] <Coren>	 jeremyb_: Machine Room.
[14:51:31] <jeremyb_>	 aha
[14:52:36] <Coren>	 Darkdadaah: Oh, did we already create tool account(s) for you?
[14:53:04] <Darkdadaah>	 Coren: Yes, I have one (anagrimes). But will need more.
[14:53:07] <Coren>	 Darkdadaah: I'll have to transition them to the final scheme.  Takes about 2 minutes.
[14:53:40] <Darkdadaah>	 "Final scheme"?
[14:54:26] <Coren>	 Same working system, but now implemented through the wikitech management interface; it involves a change in UID, which I can do.  It's functionally identical otherwise.
[14:54:47] <jeremyb_>	 is it all ready?
[14:54:55] <jeremyb_>	 is there a shared or unique primary gid?
[14:55:14] <Coren>	 jeremyb_: No, I'll have to work around that.
[14:55:33] <Coren>	 jeremyb_: Probably manage the ACLs through a script that pulls from LDAP.
[14:55:55] <Coren>	 Darkdadaah: Can you log off a minute?
[14:56:06] <Darkdadaah>	 sure
[14:56:20] <Darkdadaah>	 There.
[14:57:00] <Krenair>	 Trying to work out why I can't push to a git repository on nova-precise2... http://pastebin.com/69VH2SPJ 
[14:57:45] <Coren>	 {{doing}}
[14:58:09] <Krenair>	 Ryan_Lane, ^
[14:59:48] <Coren>	 {{done}}
[15:00:27] <Coren>	 Darkdadaah: All good.  You can manage maintainers to your tools through https://wikitech.wikimedia.org/wiki/Special:NovaProject now, by the way
[15:01:35] <Darkdadaah>	 Oh nice :)
[15:02:50] <liangent>	 petan: hi
[15:03:01] <petan>	 hi
[15:03:36] <liangent>	 petan: I assume you know about labs instance management
[15:03:43] <petan>	 of course
[15:03:43] <liangent>	 can you point to me to any docs about puppet on labs?
[15:04:05] <liangent>	 s/to//
[15:04:11] <petan>	 I don't there is docs related to labs only we have just a link to generic help
[15:04:12] <petan>	 !puppet
[15:04:12] <wm-bot>	 learn: http://docs.puppetlabs.com/learning/  troubleshoot: http://docs.puppetlabs.com/guides/troubleshooting.html
[15:04:23] <petan>	 * don't think
[15:05:07] <petan>	 try looking here: https://wikitech.wikimedia.org/wiki/Help:Contents
[15:05:45] <liangent>	 I can create some "names" on Special:NovaPuppetGroup, but how are they used?
[15:06:06] <Darkdadaah>	 Coren: I'd like to request other tools to be created, can you do it?
[15:06:33] <Coren>	 Darkdadaah: Sure thing.
[15:07:23] <Coren>	 liangent: That's for you to pick; the idea is that you create classes for "your" stuff, then add them in PuppetGroup so you can include them in your node definitions.
[15:08:06] <Darkdadaah>	 Coren: I need 2 more tools: "Daahbot" and "Wiktioutils"
[15:09:30] <Darkdadaah>	 (I'm not really good with names)
[15:10:31] <liangent>	 Coren: and where can I put stuff created myself?
[15:10:32] <Coren>	 Darkdadaah: done.  You need to log off and back on to have access to them though.
[15:12:34] <Coren>	 liangent: you want clone operations/puppet
[15:12:54] <Coren>	 liangent: You can add your classes, there, then git review for a merge
[15:12:56] <Darkdadaah>	 Coren: "wiktioutils" does not seem to work (but daahbot is ok).
[15:13:13] <Coren>	 Ah, crap, I named it 'wikioutils'
[15:13:31] <Coren>	 Give me a minute.
[15:13:41] <Darkdadaah>	 Ah.
[15:13:53] <Coren>	 My bad.
[15:14:17] <Darkdadaah>	 Nah, the name itself is bad :P
[15:14:49] <liangent>	 Coren: so there's no place I can do some "local" tests?
[15:16:04] <Coren>	 liangent: Yes, but it needs a bit of setup.  Add the puppetmaster::self class to an instance, and that will give you a self-puppetized box you can test changes on.
[15:16:23] <Coren>	 liangent: I created an instance just for that in tools, for instance.
[15:17:39] <liangent>	 Coren: but what if I want to apply it on multiple instances in my project without poking other people?
[15:18:07] <liangent>	 ah the same question is appearing in the mailing list 
[15:18:19] <Coren>	 liangent: The "right" way, then, is to submit through gerrit.  It's a little heavy, but it works.
[15:18:59] <liangent>	 Coren: maybe you want to answer otto in the mailing list with the same answer now...
[15:19:12] <andrewbogott>	 which list?
[15:19:19] <liangent>	 labs-l
[15:19:21] <andrewbogott>	 oh, nmm, I see it.
[15:19:24] <andrewbogott>	 Coren, I can respond.
[15:19:53] <Coren>	 andrewbogott: ga, then, because I was going to respond something along the lines of "I'm not the one who makes the rules, but..."  :-)
[15:21:19] <liangent>	 by searching "puppetmaster" I got
[15:21:30] <liangent>	 https://wikitech.wikimedia.org/wiki/Help:Self-hosted_puppetmaster   is this the (only) documentation about this? 
[15:25:26] <petan>	 liangent I am back
[15:25:54] <Coren>	 liangent: I'm not sure if it's the only one, but that's the one I used.
[15:26:14] <Coren>	 Darkdadaah: Should work now
[15:27:08] * andrewbogott  hopes that makes sense

[15:27:20] <Darkdadaah>	 Coren: perfect :) Thank you.
[15:27:44] <andrewbogott>	 liangent:  Yep, that's the only doc page.  Revisions welcome!
[15:28:01] <Darkdadaah>	 I have to go, have a good day.
[15:35:51] <andrewbogott>	 liangent, you made instance-proxy.wmflabs.org didn't you?
[15:39:00] <liangent>	 andrewbogott: yeah I made it
[15:39:35] <andrewbogott>	 liangent, I want to puppetize it so that we can have multiple proxies and such… I just added you as a reviewer.
[15:39:35] <andrewbogott>	 https://gerrit.wikimedia.org/r/#/c/43886/
[15:39:53] <liangent>	 andrewbogott: that's what I tried to do
[15:40:00] <liangent>	 so I asked that :)
[15:40:10] <andrewbogott>	 It's a bit of a can o' worms since I'm trying to make code that can be reused for other nginx applications in house.
[15:40:16] <andrewbogott>	 Oh, do you have a patch as well, someplace?
[15:41:08] <liangent>	 nope yet
[15:41:44] <andrewbogott>	 ok -- well, I welcome your comments on the above patchset.
[15:42:49] <liangent>	 I had an idea to also configure a mail proxy on it but I'm not sure whether it's somehow useful
[15:43:40] <andrewbogott>	 If you want to adopt 43886 and push it through review that'd be great… I haven't been giving it the love it needs.
[15:44:03] <andrewbogott>	 The mail proxy might be useful… probably best to write to labs-l for thoughts beforehand.  there might be security/anti-spam reasons why people would oppose it.
[15:47:48] <Coren>	 andrewbogott: I know legal has very strong reservations; I already raised the subject with them in re Tool Labs.
[15:48:14] <andrewbogott>	 as I feared
[15:49:28] <liangent>	 why legal?
[15:50:55] <Coren>	 liangent: Because mail would be, because of domain names, associated with "WMF as a body".
[15:51:20] <Coren>	 liangent: Labs access is open to non-staff is the bigger issue.
[15:51:59] <legoktm>	 so we arent going to get emails like how we had with toolserver?
[15:52:17] <liangent>	 Coren: I assume you're talking about mails sent from labs
[15:52:22] <liangent>	 but I'm considering a mail recv proxy
[15:52:26] <Coren>	 legoktm: We're working on finding a way to do so.  :-)
[15:52:31] <legoktm>	 ok :)
[15:52:40] <liangent>	 currently people are already able to send mails from labs addresses
[15:52:43] <Coren>	 liangent: No, incoming mail too; because the address would be @foo.wmflabs.org
[15:53:33] <Coren>	 liangent: Yes, and with much displeasure from legal -- though they understand that anyone can but anything in "From:"; the actual Received headers give it credence.
[15:54:09] <liangent>	 but ... sending to @foo.wmflabs.org means nothing
[15:55:08] <liangent>	 what can someone "prove" if they sent a mail to @foo.wmflabs.org ?
[15:55:47] <Coren>	 liangent: To geeks, not lawyers.  The "Received: from foo.wmflabs.org" doesn't help.
[15:56:51] <Coren>	 liangent: I think it's about liability.  IANAL, so I'm not the one with the cold chills.  :-)  But I'm working with them to find "a way" to do email in a way that doesn't make them cringe.
[16:17:32] <Ryan_Lane>	 Krenair: hm. maybe I pulled something in as root
[16:17:58] <Krenair>	 lots of stuff under there is owned by root
[16:18:06] <Ryan_Lane>	 whoops
[16:18:09] <Krenair>	 however the group is project-openstack
[16:18:15] <Ryan_Lane>	 right
[16:18:20] <Ryan_Lane>	 the permissions are probably bad
[16:18:21] <Krenair>	 Which I... Should be in? I think?
[16:18:56] <Ryan_Lane>	 yeah, but if someone does something git related as root it'll cause issues
[16:18:58] <Ryan_Lane>	 due to umask
[16:19:03] <Ryan_Lane>	 no group write
[16:20:46] <Ryan_Lane>	 Krenair: should work now
[16:20:50] <Ryan_Lane>	 I just did a chmod g+w -R
[16:22:42] <Krenair>	 it worked, thanks Ryan_Lane 
[16:22:46] <Ryan_Lane>	 yw
[16:24:59] <Krenair>	 -.-
[16:25:15] <Krenair>	 uncommitted changes
[16:26:38] <Ryan_Lane>	 heh
[16:26:45] <Ryan_Lane>	 andrewbogott: ^^
[16:26:55] <Ryan_Lane>	 were you working on something?
[16:27:15] <Ryan_Lane>	 seems it's the service group and sudo stuff
[16:27:38] <Ryan_Lane>	 we should really use separate wikis on nova-precise2
[16:27:53] <Krenair>	 We have 3
[16:28:44] <Ryan_Lane>	 yeah. we're only really using one :)
[16:29:07] <Krenair>	 I set up the third because I gave up waiting for someone to clear the uncommitted changes
[16:29:54] <Krenair>	 modified files are owned by reedy, Apr  3 03:47
[16:30:04] <Krenair>	 created ones have the same timestamp but are owned by root
[16:30:26] <Krenair>	 the timestamp... that's last modified I guess
[16:31:31] <andrewbogott>	 Ryan_Lane, Krenair, you can reset those changes… it's all in gerrit now.
[16:32:18] <andrewbogott>	 Also… I have some pending patches that will get us closer to being able to create more labsconsole test boxes.
[16:32:27] <andrewbogott>	 Needs work, though, and probably help from ryan to get the wiki set up properly.
[17:20:56] <^demon>	 Ryan_Lane: So, legoktm has svn access for pywikipedia (svn group in ldap shows him), but he's getting errors about not being able to write to /svnroot/pywikipedia/db/*, even though everything there is owned by *:svn
[17:20:57] <^demon>	 Thoughts?
[17:21:50] * legoktm  waves

[17:22:12] <legoktm>	 probably worth mentioning that this is a recent issue, ive committed plenty of times in the past with no issues
[17:22:32] <^demon>	 Indeed.
[17:26:14] <Ryan_Lane>	 ^demon: is he in the svn group?
[17:26:27] <^demon>	 ldaplist -l group svn says yes.
[17:26:37] <^demon>	 (Although there's way less people in that group than I'd expect...)
[17:51:15] <Ryan_Lane>	 ^demon: hm
[17:51:25] <Ryan_Lane>	 ^demon: is he included in the allowed people in svn's auth file?
[17:51:40] <^demon>	 pywikipedia doesn't have an auth file, all users can commit to it.
[17:52:04] <^demon>	 Again, it worked up until a couple of days ago. And I've not made any changes.
[18:12:47] <Ryan_Lane>	 ^demon: I haven't either...
[18:12:52] <Ryan_Lane>	 let me look at formey
[18:13:11] <Ryan_Lane>	 god damn it
[18:13:21] <Ryan_Lane>	 I know why its broken
[18:17:56] <^demon>	 Hrm?
[18:23:00] <Ryan_Lane>	 https://gerrit.wikimedia.org/r/#/c/57324/
[18:23:16] <Ryan_Lane>	 ^demon: should work now
[18:23:19] <Ryan_Lane>	 legoktm: ^^
[18:24:05] <legoktm>	 great :)
[18:24:12] <legoktm>	 lemme find a commit to make to test it
[18:30:56] <Ryan_Lane>	 so. how brave am I feeling today?
[18:31:07] <andrewbogott>	 ?
[18:31:11] <Ryan_Lane>	 I need to upgrade LDAP, and I have a package built
[18:31:17] <Ryan_Lane>	 I tested it in Labs
[18:34:38] <^demon>	 Wanna be doubly brave?
[18:34:48] <legoktm>	 yay
[18:34:50] <legoktm>	 it worked
[18:34:51] <legoktm>	 https://www.mediawiki.org/wiki/Special:Code/pywikipedia/11325
[18:34:54] <legoktm>	 thanks Ryan_Lane :)
[18:35:56] <andrewbogott>	 Ryan_Lane:  If you can't think of any other way to test, then it's probably time to be brave...
[18:36:44] <Ryan_Lane>	 :)
[18:37:04] <Ryan_Lane>	 well, I can upgrade virt1000, and let it run for a while
[18:37:06] <Ryan_Lane>	 to be safe
[18:38:17] <Ryan_Lane>	 I'll need to disable puppet on virt1000 and virt0 until the upgrade is complete
[18:38:27] <Ryan_Lane>	 the upgrade doesn't take long, though
[19:17:33] <jeremyb_>	 https://wikimediafoundation.org/w/index.php?diff=91190&oldid=91159
[19:18:33] <legoktm>	 finally :P
[19:34:36] <Damianz>	 I hate that page
[19:34:39] <Damianz>	 it's so badly designed
[19:34:57] * Damianz  swears 8 people work for wmf

[19:37:41] <Coren>	 Hey, I exist!
[19:37:47] <petan>	 :o
[19:39:19] <hashar>	 Damianz: oh did WMF hire one more person?
[19:39:51] <Damianz>	 I can never tell, I just know your designer dude has an awesome beard and tat
[20:05:19] <ori-l>	 andrewbogott: can i pounce on you with another project request? :P i got a volunteer dev interested in hacking on eventlogging stuff and it's a good excuse to finally do the work of replicating the setup on labs
[20:06:17] <andrewbogott>	 ori-l:  Sure.  Lemme make sure there isn't an existing project for that
[20:07:09] <andrewbogott>	 …looks like not.  Project name and initial user?
[20:07:26] <andrewbogott>	 And, also:  do you want instances to have shared homedirs and/or shared project storage in /data/project?
[20:07:26] <ori-l>	 'eventlogging', ori.livneh
[20:07:49] <ori-l>	 yes to both, unless there is some horrible bad side to that
[20:09:10] <andrewbogott>	 Not especially horrible.  Lots of volumes degrades gluster performance, so we try not to have shared volumes unless they're actively used.
[20:09:35] <ori-l>	 oh, if the preference is against, then let's not
[20:09:47] <ori-l>	 i don't actually have a concrete preference, was just going along
[20:10:08] <andrewbogott>	 OK.  You can turn it on later if you want (although turning on shared homedirs will cause… undefined things to happen to existing homedirs :)  )
[20:10:33] <Damianz>	 http://www.pinaraf.info/2013/03/my-post-engine-a-postgres-engine-for-mysql/ ohmywtfbbq
[20:10:33] <andrewbogott>	 Should be all set.
[20:10:47] <ori-l>	 i'm imagining something like "the nothing" from the neverending story
[20:10:51] <ori-l>	 andrewbogott: <3 thanks!
[20:15:10] <ori-l>	 andrewbogott: https://wikitech.wikimedia.org/wiki/Shell_Request/Ryansb has the proverbial +2 from me
[20:15:27] <ori-l>	 seems like a smart egg, hoping to bring him on board
[20:17:38] <Damianz>	 We should just have 1 entry question everyone must pass 'Should I install php my admin'
[20:18:53] <wm-bot>	 [bz] (RESOLVED - created by: Antoine "hashar" Musso, priority: High - normal) [Bug 45084] autoupdate the databases! - https://bugzilla.wikimedia.org/show_bug.cgi?id=45084
[20:18:53] <mutante>	 heh, isnt that blocked already?:)
[20:19:04] <mutante>	 using a fake package
[20:19:45] <Krenair>	 someone will find a different way to install it
[20:19:59] <wm-bot>	 [bz] (NEW - created by: Chris McMahon, priority: High - normal) [Bug 46166] Add automated browser tests to beta labs (tracking) - https://bugzilla.wikimedia.org/show_bug.cgi?id=46166
[20:20:00] <Damianz>	 One should never underestimate the stupidity of humanity
[20:20:09] <mutante>	 can always just remove it from all using salt i guess
[20:20:35] <mutante>	 afaik Leslie started compiling a list of evil packages already for other servers
[20:21:10] <Damianz>	 It takes someone truly evil to determine evil packages
[20:21:33] <mutante>	 haha
[20:22:03] <mutante>	 adds plesk and  webmin
[20:22:06] <Krenair>	 (bug 123456) Rename bad package list to disliked package list!
[20:22:19] <mutante>	 lol
[20:22:34] <Damianz>	 oh god... webmin
[20:22:41] <ori-l>	 Damianz: that's not how i look it
[20:22:42] <Damianz>	 we blocked that port on the core routers at my last place
[20:23:10] <ori-l>	 the way i look at it is: if people are constantly reaching for these solutions, there must be something about them that is useful and enabling
[20:23:46] <ori-l>	 and the security policies that these users are (wittingly or not) circumventing are probably difficult to understand or work with and could be made more transparent
[20:23:49] <Damianz>	 chmod 777 -R / is useful and enabling if you're a pleb
[20:24:05] <ori-l>	 yes, unix file permissions are opaque too :)
[20:24:27] <Damianz>	 The amount of sloppy code around is just disturbing... large part of the reason I dislike php as it encourages sloppyness
[20:24:33] <Krenair>	 Do we have any clear security policies for what you can/can't install on labs? (apart from phpmyadmin)
[20:24:34] <mutante>	 plesk is the biggest PITA ever, if you're the one expected to install and upgrade it
[20:24:57] <mutante>	 and then if people think it's a good idea to click that "upgrade" button, because it's there and they can.. it breaks $everything and is unfixable :p
[20:25:00] <ori-l>	 mutante: yeah, i'm not saying that these things should be permitted, the flaws are very real
[20:25:13] <ori-l>	 but that we shouldn't be glib about dismissing the fact that users reach for these solutions as idiocy
[20:25:23] <Damianz>	 Every time someone's hit upgrade plesk I've had to go foook about with the database to get it working again
[20:26:05] <ori-l>	 andrewbogott: thanks again, btw
[20:26:19] <mutante>	 ori-l: yea, i got the point, just replace idiocy with a nicer word
[20:26:19] <andrewbogott>	 np
[20:26:57] <ori-l>	 mutante: well, transfer some of the criticism from users to infrastructure and its usability
[20:27:44] <ori-l>	 Damianz's example (chmod 77 -R /) is pretty good
[20:27:47] <ori-l>	 777
[20:28:40] <ori-l>	 base 8 number system!
[20:28:51] <ori-l>	 for representing file permissions!
[20:28:56] <ori-l>	 maybe not the most suitable system for 2013
[20:28:59] <ori-l>	 but hard to replace
[20:29:27] <mutante>	 eh.. go ahead and rewrite Unix?:)
[20:29:37] <mutante>	 i mean, GNU of course
[20:29:41] <Damianz>	 Enable acl support and use setfacl if you'd rather
[20:30:00] <Damianz>	 chmod supports u/g/o r/w/x also... which is more human friendly
[20:30:29] <Damianz>	 Though really, make everythign 777 and confine them to types properly in selinux :D
[20:30:41] <ori-l>	 <mutante> eh.. go ahead and rewrite Unix?:)
[20:30:55] <ori-l>	 we'll have to eventually, i don't think it'll be adequate for the next 100 years
[20:31:06] <ori-l>	 some of the plan 9 filesystems had some neat ideas about security models
[20:31:22] <mutante>	 can we first get rid of Linux and finish GNU/Hurd?:)
[20:32:09] <mutante>	 but seriously, the point of all those "web admin" tools is to hide complexity from people
[20:32:46] <mutante>	 and i'm not sure it's possible to have a complex systems but make it "easy" by hiding complexity without also having someone who supports it
[20:33:33] <ori-l>	 well, reducing complexity is good, hiding complexity is bad
[20:33:33] <ori-l>	 the crappy tools hide complexity, yeah
[20:33:57] <ori-l>	 i'm not saying their use should be tolerated or that everything should be migrated to GNU Hurd
[20:34:10] <mutante>	 the unix way is already to have many small tools that just do one thing, but do it right
[20:34:27] <Damianz>	 The tools are part of the issue - the other side is people use them to replace skills... which they don't have
[20:34:28] <ori-l>	 but there's a difference between "yeah, sorry this complicated, but it represents the best and sanest way to run this kind of infrastructure"
[20:34:30] <mutante>	 those tools are trying to be the opposite.. fix all in one giant tool
[20:34:33] <ori-l>	 and "lol idiot"
[20:35:08] <mutante>	 oh, i totally agree to that, newbies should be treated friendly and not called idiots ever
[20:35:39] <Damianz>	 People who are unwilling to learn and want it handed to them on a plate should be called idiots
[20:35:52] <ori-l>	 okay, so we basically agree
[20:35:58] <ori-l>	 well, with mutante, i mean
[20:36:12] <ori-l>	 calling people idiots is bad
[20:37:48] * Damianz  much prefers 'pleb'

[20:38:16] <mutante>	 doesn't that just mean people
[20:38:43] * ori-l  chmod -x's Damianz

[20:38:50] * Damianz  runs in memory

[20:39:03] <Damianz>	 Humanity is stupid - this was never a debate
[20:39:14] <mutante>	 that's what my grandma said too
[20:39:15] <Damianz>	 Look at everything the world does and it's just bizzare
[20:40:25] <mutante>	 not fixed by:  having access to the sum of all knowledge  ?:p
[20:42:59] <Damianz>	 all knowledge would drive you crazy, learning is fun
[21:44:06] <wm-bot>	 [bz] (REOPENED - created by: Antoine "hashar" Musso, priority: High - major) [Bug 41104] glusterfs log files are not rotated - https://bugzilla.wikimedia.org/show_bug.cgi?id=41104
[21:50:53] <bblack>	 oh maybe this is a better forum for this: http://pastebin.com/04naC5Wz
[21:51:23] <bblack>	 is it normal for the gluster-mounted homedirs to behave like that? (directories created locally not showing up in ls)
[21:58:21] <bblack>	 for that matter, also: http://pastebin.com/ht35D5Mq
[21:58:28] <bblack>	 the filesystem just doesn't seem…. sane
[22:01:08] <Ryan_Lane>	 bblack: which project is this?
[22:01:19] <Ryan_Lane>	 gluster is full of bugs. we're moving away from it
[22:01:28] <bblack>	 varnish, my new node zero-vmod
[22:01:44] <Ryan_Lane>	 is this in /data/project or /home ?
[22:02:29] <bblack>	 it's in /home/bblack , which was set up as a gluster automount or something of that sort
[22:02:43] <bblack>	 maybe I'll just kill that and use the local fs for my homedir
[22:02:51] <Ryan_Lane>	 no. please don't
[22:03:10] <bblack>	 ok :)
[22:03:23] <Ryan_Lane>	 it's about to go away for a minute
[22:03:27] <bblack>	 ok
[22:04:29] <Ryan_Lane>	 bblack: try now
[22:04:37] <Ryan_Lane>	 one of the bricks was hung
[22:04:50] <Ryan_Lane>	 I'm also going to shrink this volume now, which should make it slightly more stable
[22:05:42] <Ryan_Lane>	 hopefully in a couple weeks we'll no longer be using this
[22:07:48] <bblack>	 Ryan_Lane: seems sane now, thanks
[22:07:55] <Ryan_Lane>	 cool
[22:08:10] <Ryan_Lane>	 I'm about to commit the shrink
[22:08:17] <Ryan_Lane>	 let me know if anything breaks or freaks out
[22:08:37] <Ryan_Lane>	 committed
[22:09:07] <bblack>	 yeah it kinda freaked out on me now :)
[22:09:57] <bblack>	 seems like maybe it just rewound in time, new stuff seems to be working ok
[22:10:07] <bblack>	 but some existing unimportant stuff poofed
[22:13:37] <Ryan_Lane>	 bblack: files disappeared?
[22:14:40] <bblack>	 yeah a whole directory tree disappeared, while my $PWD was inside there and I had a file open in vim.  ZZ -> couldn't save swapfile, then $PWD doesn't exist, etc.
[22:14:49] <Ryan_Lane>	 ewww
[22:14:52] <Ryan_Lane>	 fucking gluster
[22:15:27] <bblack>	 network sync of data is not for the faint of heart :)
[22:15:59] <Ryan_Lane>	 well, gluster is supposed to redirect all writes to the other bricks, then move the files off the ones being removed
[22:17:27] <bblack>	 it's probably that my previous writes were going somewhere bad.  the directory tree that pooled was initially created while everything was acting funny
[22:17:38] <bblack>	 *poofed
[22:17:56] <bblack>	 I really need to go make Colloquy stop incorrectly autocorrecting words as I type :P
[22:30:12] <Ryan_Lane>	 bblack: oh. that's likely
[22:30:23] <Ryan_Lane>	 exit the directory structure and let's umount it
[22:30:32] <Ryan_Lane>	 and see if it's on the underlying filesystem
[22:30:49] <Ryan_Lane>	 oh. I may need to do that with you logged out
[22:42:00] <bblack>	 I have stuff running now anyways and things seems stable.  I'm using a different directory name so there won't be any conflict in any case
[22:43:03] * Ryan_Lane  nods

[22:43:08] <Ryan_Lane>	 I wanted to check more for my sanity :)
[22:43:27] <Ryan_Lane>	 I'm shrinking a shitload of volumes and data loss worries me
[23:08:25] <Coren>	 bblack: Thankfully, gluster is going tfa
[23:26:11] <Coren>	 @notify petan
[23:26:11] <wm-bot>	 This user is now online in #huggle so I will let you know when they show some activity (talk etc)
[23:51:58] <Ryan_Lane>	 andrewbogott: can we change single instance mediawiki to use instance-proxy by default?
[23:52:44] <andrewbogott>	 Sure.
[23:53:42] <ori-l>	 andrewbogott: I don't think the project you added (eventlogging) has an IP allotted to it -- I can't seem to allocate one, anyhow
[23:53:45] <Ryan_Lane>	 it would also be nice to set $wgServer conditionally, based on the host passed in
[23:54:11] <andrewbogott>	 ori-l:  Yeah, you don't get IPs by default.  Do you need one?
[23:54:45] <ori-l>	 yeah. we need to simulate some HTTP analytic events coming from outside the cluster
[23:54:53] <andrewbogott>	 ok, stay tuned...
[23:56:04] <andrewbogott>	 ori-l:  OK, try it now?
[23:56:47] <ori-l>	 trying
[23:57:06] <ori-l>	 yes! thanks