[08:10:11] New patchset: Hashar; "update several live hacks:" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/12841 [08:11:12] New review: Hashar; "This change submit to git several changes which have only be available in the local subversion repos..." [operations/apache-config] (master) C: 0; - https://gerrit.wikimedia.org/r/12841 [08:12:40] New review: Dzahn; "ack, these rules are either live already or have been tested on srv193 by us before" [operations/apache-config] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/12841 [08:12:42] Change merged: Dzahn; [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/12841 [08:43:21] New patchset: Hashar; "unduplicate sync-apache-simulated and sync-apache" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/12842 [08:43:53] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/12842 [09:15:26] New review: Dzahn; "yea, the sync-apache-simulated was a temp. thing to test with rsync -n and not even meant to be here..." [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/12842 [09:15:29] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/12842 [09:20:52] !log creating dsh group "testwikipedia" with just srv193, creating sync-apache-test to just sync there...testing sync [09:20:59] Logged the message, Master [09:23:14] !log looking good. running sync-apache [09:23:19] Logged the message, Master [09:30:46] !log apache-graceful-all to push out needed redirects for education [09:30:52] Logged the message, Master [09:38:37] !log so the several redirects for education->outreach requested to work by today look good now. RT-3138 [09:38:43] Logged the message, Master [10:34:56] Ryan_Lane: when do you want to do the labs network node stuff? [10:35:23] we can probably do it now, if you'd like [10:35:41] ok [10:35:47] i also need to do reviews today ;) [10:35:51] * Ryan_Lane nods [10:36:00] labs stuff sounds better! [10:36:19] hahaha [10:36:24] not often I hear that [10:36:32] especially not from me [10:37:02] I'm trying to set up a swift backend for a test wiki and am failing miserably [10:37:14] there's absolutely no documentation on this\ [10:37:21] it's really annoying [10:37:24] anyway.... [10:37:28] let's see [10:37:32] we need to give each node an IP [10:37:52] I think the routing rules are probably already partially set up [10:38:21] on what? [10:38:44] I think we're sending traffic to the public IPs to all compute nodes already [10:38:57] unless that changed to only be virt2 [10:39:17] i'm pretty sure we don't [10:39:22] route 208.80.155.0/24 { [10:39:22] next-hop 10.4.16.3; [10:39:22] readvertise; [10:39:22] no-resolve; [10:39:22] } [10:39:31] seems we don't then :) [10:39:51] 155? [10:40:10] look for 153 [10:40:11] not 155 [10:40:19] 155 is what leslie set up [10:40:20] oh right [10:40:29] /* Route virt test server public IPs to the virt gw */ [10:40:29] route 208.80.153.192/26 { [10:40:29] next-hop 10.4.16.3; [10:40:29] readvertise; [10:40:29] no-resolve; [10:40:30] } [10:40:34] that comment is a bit misleading [10:40:34] heh [10:40:47] btw [10:40:51] we're out of IPs on 153 [10:41:06] I need to get a reverse proxy going soon [10:41:08] just continue with 153.257 [10:41:17] 257? [10:41:24] who knows, perhaps it'll work [10:41:27] :D [10:41:36] so which nexthops should I add? [10:41:47] sec [10:42:11] 2-9 [10:42:17] 9?! [10:42:20] oh [10:42:22] ciscos [10:42:24] yes [10:42:34] 8 systems total [10:42:41] not sure our routers support that many next hops ;) [10:42:43] we'll see [10:43:28] are you ready? [10:43:29] hm. I'm pretty sure we can do multi-node network node without having them on all nodes [10:43:31] it may breaqk everything ;) [10:43:35] heh [10:43:35] yeah [10:43:45] route 208.80.153.192/26 { [10:43:45] next-hop [ 10.4.16.2 10.4.16.3 10.4.16.4 10.4.16.5 10.4.16.6 10.4.16.7 10.4.16.8 10.4.16.9 ]; [10:43:45] readvertise; [10:43:45] no-resolve; [10:43:46] } [10:44:06] ok [10:44:07] there we go [10:44:15] still have network [10:44:19] still? [10:44:24] no [10:44:25] not now [10:44:26] haha [10:44:29] heh [10:44:30] rollback? [10:44:33] yes, please [10:44:45] done [10:44:48] back [10:45:29] let's test with just two boxes first ;) [10:45:37] sure [10:45:41] .2 and .3? [10:45:49] yep [10:45:54] we need to give it an IP [10:46:04] you didn't do that yet? [10:46:28] no. I need to figure out which one we gave to virt2 as well [10:46:37] what do you mean here? [10:46:37] users report some DNS issues on -wikitech [10:46:38] what kind of ip? [10:46:41] public [10:47:07] 208.80.153.192 [10:47:14] * Ryan_Lane nods [10:47:18] that's for outgoing dynamic NAT [10:47:21] yeah [10:47:44] each of the servers will need one [10:47:48] yep [10:47:52] I'm going to have to deallocate some public Ips [10:47:58] err. floating ips [10:48:34] thankfully I saved IPs for just this purpose [10:48:43] damn it [10:48:49] I used one in the sequence for bastion [10:48:54] hehe [10:49:04] it would be handy if they were all in one range yeah [10:49:06] for acls [10:49:22] since that's where labs traffic is coming from, from the pov of the internet [10:49:31] yeah. would be nice [10:49:35] so that would be 208.80.153.192/29 then [10:49:36] going to be hard to change bastion's IP [10:50:06] hm. maybe not [10:50:19] it won't throw key errors [10:50:25] it won't? [10:50:26] it'll just be unavailable temporarily [10:50:35] right [10:50:37] new IPs are fine. reusing old IPs is a problem [10:50:38] new ips will just be added [10:50:39] yeah [10:50:49] and noone's gonna ssh to the old ip [10:50:53] yep [10:51:05] lemme do that really quick [10:51:19] this is going to be a pain in the ass [10:52:29] hm via ldap it is [10:56:27] 1H ttl [10:56:44] damn [10:57:06] I'll have to keep the old one allocated at least that long for it [10:57:11] yes [10:59:05] stupid nova doesn't tell me which project has some of these ips allocated [10:59:12] dear mysql [11:01:28] ah. juju [11:01:37] is that still active? [11:01:44] nope [11:01:47] taking that ip back [11:02:46] we can use 208.80.153.193 right now [11:02:49] I already took it back [11:03:01] I need to remove it from the floating list, so no one else can use it, though [11:04:10] lemme start fixing bastion now [11:10:48] well… I caused an nxdomain temporarily [11:10:53] hopefully no one was doing a lookup [11:14:19] * Ryan_Lane groans [11:14:37] of course, the old IP was used for something. it was in testlabs, so no one should notice, but I still need to send an email [11:14:55] it's in my file, but shouldn't be in anyone else's [11:17:55] ok.... [11:20:15] I'm really glad I stashed a range of IPs [11:23:30] New patchset: Ryan Lane; "Moving ip address addition to the nodes, rather than the network service" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/12848 [11:23:35] this openstack puppet config really badly needs to be fixed [11:23:48] maybe after we do this, and I upgrade gluster, I'll rewrite all of this [11:24:02] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/12848 [11:24:33] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/12848 [11:24:35] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/12848 [11:24:39] New patchset: Matthias Mullie; "Bug 37616 - Article Feedback - Increase Test Sample to 10% of English Wikipedia (using article ID code)" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/12849 [11:28:01] New patchset: Ryan Lane; "Another temporary fix for network nodes" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/12850 [11:28:05] oh yeah, this is all fucked up [11:28:35] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/12850 [11:28:53] * mark is writing reviews in the mean time [11:29:13] * Ryan_Lane nods [11:29:51] New patchset: Ryan Lane; "Another temporary fix for network nodes" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/12850 [11:29:53] should be done soon [11:30:23] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/12850 [11:30:29] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/12850 [11:30:32] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/12850 [11:36:55] Stderr: "iptables-restore v1.4.4: host/network `208.80.153.0/22,10.0.0.0' not found\n [11:36:57] hm [11:38:23] oh [11:38:24] right [11:38:33] nova-network is patched [11:38:35] let me document that [11:39:52] actually, let me puppetize that [11:41:28] that indeed solved that [11:41:50] * mark thinks there's a cheese cake in the making here [11:42:00] oh? your gf making it? [11:42:04] yeah [11:42:11] getting hungry now [11:44:23] how long are you still in berlin for? [11:44:30] till friday [11:44:46] enjoying it? [11:44:52] yep. great place [11:45:02] heya [11:45:07] howdy [11:45:09] hi [11:45:15] we're doing network node per compute node [11:45:28] great [11:45:56] I'm adding my patched code into puppet [11:46:03] I need to do so for the compute code too [11:46:17] *patched nova-network code [11:46:54] the router's dropping packets from virt6-8->stafford [11:46:59] or the other way around [11:47:29] New patchset: Ryan Lane; "Adding in nova-network patch" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/12851 [11:47:39] I really need to upstream this nova-network change [11:48:01] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/12851 [11:48:04] (I reinstalled all three of them during the weekend) [11:48:58] New patchset: Ryan Lane; "Adding in nova-network patch" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/12851 [11:49:01] cool [11:49:07] could it be the access list? [11:49:30] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/12851 [11:49:37] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/12851 [11:49:39] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/12851 [11:50:28] I really want to be done with virt6-8 :) [11:51:19] mark: access list? [11:51:31] New patchset: Ryan Lane; "Fix typo" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/12852 [11:51:41] paravoid: ssh into them from one of the virt nodes [11:51:58] we occasionally need to copy new_install off to other machines temporarily [11:52:03] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/12852 [11:52:07] puppet access works [11:52:15] just not ping or ssh [11:52:32] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/12852 [11:52:34] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/12852 [11:52:51] paravoid: what are their ips? [11:52:56] virt6.pmtpa.wmnet doesn't resolve [11:53:09] oh nm [11:53:13] eh? [11:53:16] getting lunch. back in a little bit [11:53:19] was on my own colo [11:53:28] Ryan_Lane: I've ssh'ed fine [11:53:36] also run the first puppet run with sockpuppet, that worked [11:53:47] ahhh ok [11:53:53] and puppetca -s [11:54:02] term sockpuppet { [11:54:02] from { [11:54:02] source-address { [11:54:02] 10.4.16.0/24; [11:54:02] } [11:54:03] destination-address { [11:54:03] 10.0.0.245/32; [11:54:04] 10.0.0.24/32; [11:54:04] } [11:54:05] protocol tcp; [11:54:05] destination-port 8140; [11:54:06] } [11:54:07] then accept; [11:54:45] what about on the other way around? [11:54:54] there's no filter on that [11:55:03] except usual uRPF [11:55:15] you're right about the drop vs reject though [11:55:17] because it does this: [11:55:46] term default { [11:55:46] filter deny-private-subnets-in4; [11:55:46] } [11:55:57] and that filter does a DROP because it's mostly for external traffic coming in [11:56:02] where we don't want to do reject per se [11:56:05] hmm let's see [11:56:31] hmm, now it works [11:56:38] i didn't change it [11:56:40] so, yesterday, I was running puppet off of these 3 nodes [11:56:52] and it timed out on connect [11:57:16] handshake worked but then it couldn't pass traffic; even tried with openssl s_client [11:57:19] that's weird [11:57:22] mtu issue? [11:57:51] thought about it but why? it's mtu 1500 as everything else on our network, right? [11:58:06] yeah [11:58:14] perhaps some network node issue [11:58:22] on the networking side it's just another vlan [11:59:03] Protocol inet, MTU: 9170 [11:59:14] it's jumbo like any other, but no servers are configured for > 1500 mtu [11:59:57] anyway there's no filtering that should affect virt6-8 specifically [12:00:06] okay [12:00:08] sorry for the trouble [12:00:15] don't know what happened there [12:00:19] and I just assumed filtering :) [12:00:20] it is strange [12:01:53] puppet runs now on all three of them, I'll just be happy about that [12:02:02] okay [12:02:05] btw, since you're here, I got pages on Saturday [12:02:13] so did I [12:02:14] and from reading irc backlog, it was the same issue again [12:02:17] apergos handled it [12:02:20] that's the third time [12:02:23] yeah [12:02:32] we should find a more permanent solution… [12:02:38] yes [12:02:41] i wonder why it started happening [12:02:49] that would make people happy on the weekends [12:03:27] well, one thing is why boxes keep getting killed by load [12:03:46] but the other architectural issue is how a malfunctioning jobrunner can get the whole cluster down [12:04:26] more like, why does a single malfunctioning memcached kill the cluster [12:04:39] but, I haven't looked at it at all yet [12:04:45] I saw there were like 3 of you doing a good job ;-) [12:05:19] well, it's just incident response so far [12:05:31] so, I was thinking that maybe we should split apache/jobrunners from memcache [12:05:40] that's happening soon anyway [12:05:55] how soon? [12:05:59] asher has the boxes [12:06:28] yeah getting the memcaches onto their own box will be a huge win [12:06:32] boxes [12:18:40] I wonder if setting memory pressure per process makes sense for memcached [12:19:15] hmm, or is that just for ooms [12:24:52] New patchset: Hashar; "(bug 37699) Change logo on uz.wikipedia" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/11977 [12:25:37] New review: Hashar; "(no comment)" [operations/mediawiki-config] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/11977 [12:25:39] mutante: are the puppet configs still in SVN? [12:25:39] Change merged: Hashar; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/11977 [12:26:40] New review: Hashar; "deployed live" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/11977 [12:27:21] rebooting [12:28:00] Platonides: http://lists.wikimedia.org/pipermail/mediawiki-distributors/2012-June/000018.html [12:41:21] mark: ok. back [12:41:28] mark: and I have network nodes on virt1/2 [12:41:32] ok [12:41:37] so you wanna have those two active now [12:41:48] yes, please [12:42:08] done [12:42:17] still have network [12:42:38] of course, I doubt anything is routing through virt1 [12:44:11] oh gosh [12:44:14] i'm ordering cables in this web shop [12:44:14] ? [12:44:20] and they have a quick link to your shopping cart [12:44:23] "to show others" [12:44:27] so I test that link [12:44:29] and the way it works [12:44:30] hm. planet.wmflabs.org is timing out [12:44:34] they encode all articles in the url [12:44:38] so I loaded that link [12:44:46] and it doubled my shopping cart effectively [12:44:52] because it just added everything again in my shopping cart [12:45:10] heh [12:45:11] nice [12:46:51] mark: can you revert the change for now? [12:46:56] ok [12:47:12] done [12:47:27] I want to make sure it isn't the change that's affecting this public IP [12:47:34] well, still can't ping it. [12:49:21] well, the scheduler already reassigned the addresses [12:49:26] they're on the network node [12:49:29] the new one [12:50:50] mark: ok, can you re-add it? [12:50:55] it isn't working either way [12:50:56] ok [12:51:09] but, now that it switched the instances to virt1... [12:51:10] done [12:51:21] it actually moved instances? [12:51:28] only the NAT rules [12:51:39] because those instances are on virt1 [12:51:43] ah yeah [12:51:51] but they need to be on all [12:51:52] I can't ping it, though [12:51:53] not just virt1 [12:52:48] hm [12:52:49] lemme add on the other router as well [12:53:25] done [12:53:50] keep in mind that we can't scale the number of next-hops indefinitely [12:53:54] yeah [12:54:02] I don't know what the limit on juniper MX is, but I wouldn't be surprised if it were no more than 32 or so [12:54:05] or maybe less [12:54:08] hm. still isn't pinging [12:54:25] also, often a number that's not a power of 2 gives odd distribution [12:54:31] ok, what are you pinging? [12:54:37] 208.80.153.223 [12:54:58] that's the public ip of an instance? [12:55:08] yep [12:55:15] and do both network nodes have NAT rules for it? [12:55:21] ah [12:55:22] I see [12:55:29] why isn't forwarding enabled? [12:55:36] it's disabled by default [12:55:52] yeah, but it should be enabled by puppet [12:56:08] unless puppet is wrong [12:57:12] weird... [12:57:17] I'm including it via puppet [12:57:22] but it isn't there [12:57:47] if $lsbdistid == "Ubuntu" and versioncmp($lsbdistrelease, "12.04") >= 0 { [12:57:50] 12.04? [12:58:01] what is that for? [12:58:07] class generic::sysctl::advanced-routing($ensure="present") { [12:58:07] if $lsbdistid == "Ubuntu" and versioncmp($lsbdistrelease, "12.04") >= 0 { [12:58:12] it wraps the file [12:58:18] hmm that seems wrong [12:58:18] why is that being done? [12:58:24] 10.04 has /etc/sysctl.d/ [12:58:27] yes [12:58:28] 8.04 doesn't [12:58:49] fixing [12:58:54] New patchset: Hashar; "(bug 37852) enable WikimaniaShopLink on labs" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/12863 [13:00:04] apparently, the default max of next-hops on MX is 16, but can be raised [13:00:10] so we'll be good for a while [13:00:12] yeah [13:01:01] New patchset: Ryan Lane; "Fix sysctl.d inclusions" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/12864 [13:01:05] likely 64 max [13:01:28] if they don't all have the static rules, then next-hop will occasionally reject traffic, eh? [13:01:33] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/12864 [13:02:30] Ryan_Lane: so... [13:02:41] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/12864 [13:02:43] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/12864 [13:02:53] paravoid: ? [13:03:02] the nodes are up, puppet has run [13:03:15] puppet was complaining about nova-common before but I guess that was you [13:03:23] yes [13:03:29] now it rightfully complains that it can't find the private key for virt6.pmtpa.wmnet [13:03:31] I broke something temporarily [13:03:35] yep [13:03:38] it needs to be created [13:03:42] do you have the steps somewhere? [13:03:55] or should I trial-ask-and-error? :) [13:03:57] you generate them on sockpuppet, and yep, I do :) [13:04:11] * paravoid looks [13:04:14] http://wikitech.wikimedia.org/view/Using_the_local_certificate_authority [13:04:19]