[00:12:27] Change abandoned: awjrichards; "This is no longer a necessary change." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/10203 [01:40:45] New patchset: Jeremyb; "make ircecho config sane (not just very long strings)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/8344 [01:41:07] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/8344 [01:41:13] New review: Jeremyb; "rebased on top of Ib8c0ea3e5bbb54fd" [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/8344 [01:46:54] some people have said that Teredo (2001::/32) should not be blocked b/c Windows Vista/7 uses it, but I disagree. [01:48:18] isn't that for the stewards to decide? [01:48:56] a steward told me to talk to ops [01:54:04] ? [01:54:34] that means you, Snowolf [01:54:42] Yes but I didn't tell you to talk to ops [01:54:50] I told you that I wrote to one of the wmf ops people :P [01:55:01] I took that as a request to discuss with ops [01:55:19] i should've said [01:55:31] "I was advised that the ops opinion was needed" [01:56:13] In any case I'm not touching or listening to any IPv6 stuff until it's live [07:49:45] New patchset: Jeremyb; "admins.pp: comment includes of disabled accounts" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/10226 [07:50:06] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/10226 [07:56:27] morning asher [07:56:47] heya [08:11:41] wtf [08:11:54] if I send PyBal a TERM signal, it kills its BGP connection, not itself [08:21:39] New patchset: Pyoungmeister; "adding a key that ben gave me. at his request someone else must +2" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/10227 [08:22:01] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/10227 [08:31:44] !log reimaging searchidx2 [08:31:49] Logged the message, notpeter [08:45:11] !log restarted mysql on es1004 with default innodb file format as barracuda [08:45:16] Logged the message, Master [09:02:25] yay [09:02:27] nailed that bug [09:05:48] what was it? [09:06:04] are you familiar with Twisted (python)? [09:06:53] yep [09:07:01] basically I was installing a signal handler to run before twisted shutdown [09:07:09] the signal handler was returning a Deferred [09:07:25] the deferred was coming from my twisted BGP library, which was supposed to fire the callbacks on it [09:07:35] the signal handler was meant to stop/release BGP connections [09:07:44] but because the actual code was synchronous, no callbacks were ever fired [09:07:49] and the signal handler did never finish [09:07:56] so pybal's own terminate handler never got called, either [09:08:29] the synchronous code made a new fresh Deferred, but nothing did anything with it [09:08:42] event based code can be fun to debug ;) [09:09:44] New patchset: Mark Bergsma; "Fix erroneous use of str.find()" [operations/debs/pybal] (master) - https://gerrit.wikimedia.org/r/10228 [09:09:45] New patchset: Mark Bergsma; "Catch and handle server list load errors" [operations/debs/pybal] (master) - https://gerrit.wikimedia.org/r/10229 [09:09:45] i looked at the pybal code after you mentioned the bug an hour ago and couldn't see anything wrong :) [09:09:45] New patchset: Mark Bergsma; "Set umask 022 for the daemon" [operations/debs/pybal] (master) - https://gerrit.wikimedia.org/r/10230 [09:09:46] New patchset: Mark Bergsma; "Simplify subCommandServer" [operations/debs/pybal] (master) - https://gerrit.wikimedia.org/r/10231 [09:09:47] New patchset: Mark Bergsma; "Make sure we don't pass any IPv6 service IP addresses to BGP at the moment" [operations/debs/pybal] (master) - https://gerrit.wikimedia.org/r/10232 [09:09:47] New patchset: Mark Bergsma; "Replace some too generic exception handlers by more specific exceptions in bgp.py" [operations/debs/pybal] (master) - https://gerrit.wikimedia.org/r/10233 [09:09:48] New patchset: Mark Bergsma; "Fix bug where BGPPeering.manualStop() would not fire callbacks" [operations/debs/pybal] (master) - https://gerrit.wikimedia.org/r/10234 [09:09:49] New patchset: Mark Bergsma; "pybal (1.01) precise; urgency=low" [operations/debs/pybal] (master) - https://gerrit.wikimedia.org/r/10235 [09:09:55] ^ last change :) [09:10:01] well, one but last [09:10:31] it's not really a critical bug, just has been annoying me for some time [09:10:39] and since i'm rolling a new release now anyway, I just wanted to fix it ;-) [09:10:44] but now back to installing LVS servers! [09:14:35] !log Built PyBal 1.01 for precise, and included it in the precise-wikimedia APT repository [09:14:35] bot is currently down [09:14:40] Logged the message, Master [09:14:52] * mark kicks wm-bot in the nuts [09:16:05] New review: Mark Bergsma; "(no comment)" [operations/debs/pybal] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/10228 [09:16:07] Change merged: Mark Bergsma; [operations/debs/pybal] (master) - https://gerrit.wikimedia.org/r/10228 [09:16:41] New review: Mark Bergsma; "(no comment)" [operations/debs/pybal] (master); V: 0 C: 2; - https://gerrit.wikimedia.org/r/10229 [09:16:49] New review: Mark Bergsma; "(no comment)" [operations/debs/pybal] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/10229 [09:16:50] Change merged: Mark Bergsma; [operations/debs/pybal] (master) - https://gerrit.wikimedia.org/r/10229 [09:17:18] New review: Mark Bergsma; "(no comment)" [operations/debs/pybal] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/10230 [09:17:20] Change merged: Mark Bergsma; [operations/debs/pybal] (master) - https://gerrit.wikimedia.org/r/10230 [09:18:01] New review: Mark Bergsma; "(no comment)" [operations/debs/pybal] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/10231 [09:18:03] Change merged: Mark Bergsma; [operations/debs/pybal] (master) - https://gerrit.wikimedia.org/r/10231 [09:18:29] New review: Mark Bergsma; "(no comment)" [operations/debs/pybal] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/10232 [09:18:31] Change merged: Mark Bergsma; [operations/debs/pybal] (master) - https://gerrit.wikimedia.org/r/10232 [09:19:13] New review: Mark Bergsma; "(no comment)" [operations/debs/pybal] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/10233 [09:19:15] Change merged: Mark Bergsma; [operations/debs/pybal] (master) - https://gerrit.wikimedia.org/r/10233 [09:20:13] New review: Mark Bergsma; "(no comment)" [operations/debs/pybal] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/10234 [09:20:14] Change merged: Mark Bergsma; [operations/debs/pybal] (master) - https://gerrit.wikimedia.org/r/10234 [09:20:38] New review: Mark Bergsma; "(no comment)" [operations/debs/pybal] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/10235 [09:20:40] Change merged: Mark Bergsma; [operations/debs/pybal] (master) - https://gerrit.wikimedia.org/r/10235 [09:21:59] or, perhaps, breakfast first... [09:35:01] New patchset: Pyoungmeister; "giving searchdix2 new indexer class" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/10237 [09:35:23] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/10237 [09:35:28] !log rebooting es1 for kernel+mysql upgrade. dont need to pull from db.php because it was never correctly added or queried? [09:35:32] Logged the message, Master [09:35:56] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/10237 [09:35:59] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/10237 [09:42:17] mark: ! [09:52:18] !log stopping indexing on searchidx1001 to rsync to searchidx2 [09:52:22] Logged the message, notpeter [09:53:02] !log cancel that, it's mid-cron. will do later [09:53:06] Logged the message, notpeter [09:54:40] innodb compression will not help with our blobs scheme too much [09:57:34] paravoid? [09:58:26] i don't think it will either, i'm trying dynamic though since we have some amount of blobs that should fit in the pkey page in that case [09:59:50] mark: hi [09:59:55] what to do about lvs servers? [10:00:00] you told me not to reinstall [10:00:04] but dunno why [10:00:19] i'm comparing the size of compact vs. dynamic for one of the enwiki blobs tables.. i will probably try compressed after, just for the hell of it. maybe the page size change will result in better space utilization. innodb compact uses around 30% more space than myisam in this case [10:03:04] paravoid: you can reinstall lvs1 until it works fully automatically [10:03:11] but I want to check it completely before we do the others [10:03:15] a few changes to the manifests too [10:03:18] no more pdns recursors [10:03:28] oh? [10:03:37] no longer needed, pybal now does its own dns resolution [10:03:46] no timeout on every check anymore [10:03:50] that was the issue [10:03:59] if primary resolver was down, checks would timeout and depool everything [10:04:09] even before the 2nd resolver could help out [10:04:17] right [10:04:37] now it resolves everything on startup and then it's fine [10:04:48] in a future release I'll make it resolve when the ttl expires, but I think it's fine now [10:04:49] you guys need any help with anything? [10:05:10] feel free to setup 6to4/miredo relays ;-) [10:05:19] first we need to fix the lvs server installs [10:05:22] heh [10:05:27] then we could use some help in reinstalling the boxes perhaps [10:05:29] but right now, not much [10:05:36] where should I stick the relays? [10:05:36] try again later today ;-) [10:05:42] on new misc servers [10:05:43] low perf [10:06:01] this of course assumes I know how to do that :D [10:06:07] hehe [10:06:08] I mostly know how to do that [10:06:08] better time than any to learn [10:06:10] it can also wait on paravoid [10:06:18] so, feel free to let it for me or do it and ask me [10:06:30] * Ryan_Lane nods [10:06:34] I've setup 6to4 in ciscos before and read about miredo [10:06:35] i'm having breakfast now [10:06:38] just had shower :D [10:06:42] will be back soon [10:06:54] well, I don't like having infrastructure around that I don't understand, so I don't mind learning. don't wait on me, though [10:07:00] I do my best software releases before breakfast [10:07:06] like the spirit :) [10:07:31] Ryan_Lane: 6to4 is easy, miredo setup is easy too but you first have to understand Teredo [10:07:37] which is a huge mess [10:07:37] * Ryan_Lane nods [10:07:45] start reading the Teredo article on wikipedia [10:07:52] I dislike when things break and I need someone else to fix them [10:07:58] I needed to read it like 4 times before I understood what's going on [10:08:01] :D [10:08:13] what will we be using this for? [10:08:18] a replacement for doing it on nginx? [10:08:22] no no [10:08:29] a lot of people use tunneling [10:08:30] ah [10:08:31] I see [10:08:37] mostly automatically [10:08:38] this is so that we can use ipv4 internally [10:08:44] nope [10:08:59] so, 6to4 for example [10:09:06] some OS like Windows set it up automatically [10:09:08] (teredo too) [10:09:16] 6to4 connects to an anycast relay [10:09:28] both on the way forward and on the way back (different relays) [10:09:37] so you hit random tunneling servers on the internet [10:09:40] some of them may be buggy [10:09:48] or have packet loss [10:09:54] or be very far way increasing latency [10:10:10] by setting up our own relays, we minimize the hop count and can ensure performance [10:10:30] we're sure people will use a relay that works and their experience won't be very crappy [10:10:40] oh, this is nasty [10:10:49] I'm not sure how much tunneled traffic we will really see [10:10:56] but I guess we'd have to set it up and check :) [10:11:00] yeah [10:11:06] (we could also check analytics for the address spaces, but...) [10:11:12] I get the reasoning, this is dirty as hell, though :( [10:11:36] the way those protocols work you mean? [10:11:39] yes [10:11:42] yes. [10:12:23] so, [10:12:27] teredo is symmetric [10:12:46] as a client, you always use the relay which is closer to the destination (e.g. website that you visit) [10:13:10] right, so they'd use us, if we have one available [10:13:16] right [10:13:22] and we'd use that for the replies too [10:13:34] so all of a client's traffic would pass through our teredo relay [10:13:40] both ways [10:13:50] 6to4 otoh, is asymmetric [10:14:04] so, we need at least two. can this be load balanced, or set up in a failover way? [10:14:14] we'd handle it via bgp? [10:14:24] nah, I think that's an overkill [10:14:36] well, we don't want a SPOF [10:14:40] if that fails, another relay would be used on the internet [10:14:44] it's all dynamic [10:14:57] ah [10:15:07] that's easy enough, then [10:15:18] it's anycast [10:15:24] we'll have one per dc [10:15:28] eqiad and tampa [10:15:30] that's enough ;) [10:15:32] you just announce 2001:0::/32 [10:15:32] and one in esams [10:15:36] over bgp [10:15:44] ah, was gonna say, no esams? :D [10:15:53] esams is less necessary [10:15:56] tons of relays in amsterdam ;) [10:16:00] so, if the one in eqiad dies, they'll hit pmtpa? [10:16:04] yeah [10:17:03] oh wow [10:17:15] mark: the relay I'm hitting from pmtpa is... SURFnet!!! [10:17:20] haha [10:17:23] why am I not surprised [10:17:26] that sucks [10:17:36] Ryan_Lane: SURFnet is the GRNET of the Netherlands [10:17:42] so a Teredo user in the U.S. [10:17:51] :D [10:17:56] would go back to the netherlands, to be decapsulated [10:17:57] we could announce it to everyone [10:18:00] and then back to the u.s. [10:18:10] hence the need of running our own relays [10:18:10] but it might hit a lot of traffic [10:18:14] so let's wait with that ;) [10:18:17] * apergos starts reading the teredo article [10:18:19] don't want to have that tomorrow [10:18:57] also the prefix gets filtered by our transits now anyway ;) [10:19:07] yeah, I thought of that... [10:19:15] 6to4 is easier and doesn't have that problems [10:19:32] we just route 2002::/16 internally for our network to our relay [10:19:34] and poof [10:19:46] yep [10:20:02] how do you want to do bgp injection? [10:20:18] we could do a simple quagga [10:20:23] just in case the box goes down [10:20:26] yeah [10:20:29] that should be enough [10:20:30] 6to4 is being done on kernel space [10:20:35] so, no other daemons to watch for [10:20:39] indeed [10:20:46] tcp over udp? uurgghh [10:20:59] apergos: it's better than tcp over tcp :P [10:21:04] hahahaha [10:23:51] Ryan_Lane: http://www.getipv6.info/index.php/Linux_or_BSD_6to4_Relays [10:23:54] this is getting a little network-y for me abilities [10:23:59] *my [10:24:10] you did juniper lesson 1, didn't you [10:24:13] :D [10:24:14] get on it already! ;-) [10:24:25] hey, at least I can set up my own ports and such now [10:25:01] I really need to tell my old employer that it's a bad idea to split the network group from the ops group [10:25:09] well, if you have free time, this is nice practice [10:25:13] true [10:25:18] we can help out :) [10:25:27] just play with it on a scratch box [10:25:31] then when you have it working, puppetize [10:25:38] i'll handle the router config for you ;) [10:25:43] * Ryan_Lane nods [10:25:51] but if you don't have time, fine too [10:25:54] I do [10:26:00] I have nothing better to do right now [10:26:16] you have a day off or something? ;D [10:26:30] no, but the things I need to work on, I was going to work on with paravoid ;) [10:26:38] ah ;) [10:27:08] so, helping frees him up faster anyway. heh [10:27:15] start with 6to4 i'd say [10:28:22] so, the router will be setup to advertise a route to the relay? [10:28:45] or will the relay use quagga to advertise its own route? [10:29:23] oh? you're waiting for me? [10:29:32] for labs things [10:29:37] didn't realize, sorry [10:29:40] it's no rush, though [10:29:57] ipv6 obviously has higher priority right now :) [10:30:23] (and seeing as that I wanted to do this last year, I'd like to see it happen ;) ) [10:31:07] Ryan_Lane: the relay will use quagga to announce its route [10:31:14] * Ryan_Lane nods [10:31:16] and the router will accept that and route it on the network [10:31:26] the routers will choose the closest relay automatically (anycast) [10:31:47] it sees the same route in multiple places, it picks the shortest/fastest [10:32:01] wow, this is seriously sketchy... (still reading article) [10:32:03] heh. the good thing is the juniper class actually made it so that I understand the concepts of this at least ;) [10:32:11] that the route actually goes to different destinations is not a problem here [10:32:25] the same we'll do with dns soon [10:32:29] yep [10:32:50] though we're doing anycast everywhere will this, rather than just in the US, like we'll do with DNS [10:32:55] *with [10:32:58] well [10:33:03] it's actually the same [10:33:08] the anycast will not leave our network (for now) [10:33:11] and esams is a different network [10:33:28] aren't we going to also have a relay in esams, though? [10:33:31] so esams will always hit either its own relay, or it will use some external one from the internet [10:33:39] ahhh ok [10:33:39] and pmtpa and eqiad will share their two relays [10:33:49] esams will never use the pmtpa/eqiad relays [10:33:57] (unless we make that work) [10:34:58] paravoid: so... lvs1 [10:35:03] yes, on it [10:35:07] it needs a reinstall, i used it as a test box now ;) [10:35:14] yep [10:35:15] let's make that work fully and then we'll move on with the others [10:35:17] btw [10:35:26] something to note [10:35:36] is that the precise installer does indeed IPv6 [10:35:43] yeah [10:35:46] we should do something with that soon [10:35:50] just... perhaps not now ;) [10:35:50] and at some point (probably not today) [10:35:53] indeed [10:36:03] heh, glad we're on the same page :-) [10:36:29] also, if people are now gonna add AAAA records to each and every service on our network i will shoot them [10:36:44] because the way most apache vhosts are now handled, that's a massive pain, just like https [10:36:57] that should be redone, and only after we've had this done properly, I'd like to add ipv6 everywhere [10:37:06] so mostly only core services right now [10:40:02] do you guys know molly-guard? [10:40:30] i've seen it [10:40:33] not used it [10:40:43] it's interesting [10:41:02] it prompts you to type the hostname before rebooting, as to not reboot by mistake [10:41:05] but [10:41:08] you can also write custom hooks [10:41:16] what would you like to use it for then? [10:41:20] so I had one that refused to reboot if you had kvm running processes [10:41:25] H [10:41:26] ah [10:41:29] and we could use it to refuse to reboot if it's an active LVS [10:41:35] well [10:41:41] it would automatically failover anyway [10:41:53] bgp session closes, router switches over [10:42:01] however, tcp sessions break [10:42:04] but all the connections [10:42:06] right [10:42:06] (lvs state is not shared currently) [10:42:07] yeah [10:42:15] currently? is that possible? [10:42:17] yes [10:42:20] oh! [10:42:23] lvs has replication for v4 [10:42:25] (lvs is new for me) [10:42:32] it uses multicast [10:42:37] and sends connection tuples over udp [10:42:54] we don't use it, except manually sometimes when I feel the need [10:42:59] but failovers are very rare [10:43:07] and mostly people don't notice when they happen [10:43:22] (obviously some people will be hitting reload ;-) [10:44:52] i'm gonna add static routes on the router now [10:45:00] routers [10:45:14] static of what? [10:45:19] what routes? [10:45:22] lvs routes [10:45:25] instead of bgp [10:45:32] ah, right [10:45:34] perhaps i'll implement bgp soon, but not today ;) [10:45:40] hahaha [10:45:45] we always have them anyway, as backup routes [10:45:51] i.e. if all pybal hosts die, there's still a route to one lvs box [10:46:11] if all the hosts or daemons? [10:46:14] daemons [10:46:21] right, that makes more sense :) [10:46:26] or if the bgp implementation would have a bug or so [10:46:29] I just changed it, for example [10:46:34] yep [10:46:39] I love how it's all redundant! [10:46:57] yeah [10:47:05] people often ask me why we don't just use VRRP/CARP/etc [10:47:10] but I think this is so much better [10:47:10] ew [10:47:14] that feels clumsy ;) [10:47:53] here's a static route for upload: [10:47:54] static { [10:47:54] route 2620:0:860:ed1a::b/128 { [10:47:54] next-hop 2620:0:860:1:208:80:152:120; [10:47:54] readvertise; [10:47:54] no-resolve; [10:47:55] preference 250; [10:47:55] } [10:47:56] } [10:48:27] yay [10:49:18] I think we can also have juniper do a simple ping check on a static route, if we don't wanna do quagga on those relays [10:49:32] iirc that was possible [10:50:29] iface eth0 inet dhcp [10:50:30] argh. [10:50:32] still. [10:50:34] :( [10:52:48] oh I know [10:55:25] mark: oh, you're saying we should have a misc-services address for https and ipv6, rather than having a billion records? [10:56:14] I'd love that, as I'd really like to get the https certificates off the misc servers [10:56:42] Ryan_Lane: rather, I want to migrate to a new apache manifest/module in puppet first [10:56:46] though if we move to stud, that becomes harder [10:56:50] so we're not rewriting the same vhost stuff each and every time ;-) [10:56:51] I understood that he said that we should use Include in Apaches [10:57:02] ah [10:57:14] I despise our apache config in puppet [10:57:17] yeah [10:57:24] just wanna say "enable ipv6!" and be done. [10:57:31] like Include commonSSLEngine on [...] Include common and so on [10:57:33] like we did for nginx? :) [10:57:36] yeah [10:57:45] yeah or some template [10:57:50] i started a new style system a while ago [10:57:51] but tbh [10:57:57] many people must have written good apache puppet modules ;-) [10:58:02] yes [10:58:17] if it's sort of similar to what I started and offers the same flexibility we need, then i'm fine with using that too [10:58:22] as long as we don't continue the way we do it now [10:58:26] hate hate hat [10:58:31] see, can't even type [10:58:35] agreed. we have 5 different ways to do it [10:58:46] and all of them suck [10:58:51] so unlike with https, I will shoot when people add v6 everywhere now [10:58:55] before we tackle this [10:59:06] even though v6 is a bit easier than https and new :443 vhosts ;) [11:02:47] junos needs a 'sort' command [11:03:05] I hate it when my routes and stuff are not in order :P [11:05:35] aaaaand reinstalling again [11:06:00] any smart ideas on how to fix the partitioning while I'm at it? [11:06:35] what's currently wrong about it? [11:07:11] there's no partioning for lvs servers [11:07:14]