[08:04:31] paravoid: do you have time to do some puppet reviews today? [08:11:09] matanya: I'm afraid not [08:11:47] paravoid: ok, thank you. I would like to have a short personal talk with why you have 5 minutes [08:11:54] *when [08:12:03] let's do it now :) [08:14:39] (03PS2) 10Matanya: ganglia_new: lint clean [operations/puppet] - 10https://gerrit.wikimedia.org/r/107128 [08:15:57] (03PS7) 10Matanya: etherpad: convert into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/107567 [08:29:06] (03PS6) 10Matanya: beta: convert into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/108289 [08:30:46] (03PS4) 10Matanya: coredb_mysql: puppet 3 compatibility fix: fully qualify variable [operations/puppet] - 10https://gerrit.wikimedia.org/r/108488 [08:33:21] (03PS3) 10Matanya: Torrus: move from manutius to netmon1001 [operations/puppet] - 10https://gerrit.wikimedia.org/r/108314 [08:36:13] (03PS3) 10Matanya: coredb_mysql: puppet 3 compatibility fix: fully qualify variable [operations/puppet] - 10https://gerrit.wikimedia.org/r/108313 [08:37:08] (03PS3) 10Matanya: torrus: move into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/108498 [08:38:51] (03PS2) 10Matanya: realm: lint clean [operations/puppet] - 10https://gerrit.wikimedia.org/r/109074 [08:44:33] (03PS1) 10Ori.livneh: LVS: add Icinga checks for critically important sysctl params [operations/puppet] - 10https://gerrit.wikimedia.org/r/111163 [08:48:37] good morning [08:51:47] (03PS3) 10Matanya: nfs: lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/109081 [08:51:48] hi hashar [08:52:03] ori: hello [08:52:22] (03PS3) 10Matanya: gerrit: lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/109088 [08:52:38] yesterday summary score for player [ori] : -1 on easter egg removal +1 on migrating scap to python [08:52:39] :D [08:53:05] it's just one script of several [08:54:13] we should probably think of working toward integration with salt [08:54:32] i'll reply on the patch [08:54:37] or Fab if it better suit our purposes [08:56:46] (03PS20) 10Matanya: site: lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/109507 [08:59:26] PROBLEM - Puppet freshness on cp3019 is CRITICAL: Last successful Puppet run was Mon 03 Feb 2014 08:55:41 PM UTC [09:01:42] ori: integrate scap with salt? [09:02:24] Ryan_Lane1: with $DEPLOYMENT_SYSTEM [09:02:30] heh [09:02:44] why not just move to the new system? [09:02:59] I'll likely be replacing the perl code with trigger this week [09:03:05] (03PS7) 10Hashar: beta: convert into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/108289 (owner: 10Matanya) [09:03:20] I'm in the last stages of testing that. it's packaged and everything [09:03:35] if it works, sure [09:03:40] (03CR) 10Hashar: "Misc fix:" [operations/puppet] - 10https://gerrit.wikimedia.org/r/108289 (owner: 10Matanya) [09:03:55] after doing so I can also add the web interface for reporting [09:04:22] ah Ryan_Lane1 good morning :-] [09:04:29] hashar: howdy [09:04:33] it's like 1am here ;) [09:04:58] was wondering how hard it is to create a labs project which isolate instances from the network [09:05:13] would the security group be enough for that or does it need something on openstack side? [09:05:19] should be easier with neutron [09:05:28] which andrewbogott is working on for eqiad [09:05:31] would have to be done manually or maybe provisioned via LDAP ? [09:05:50] well, if we expose access to neutron, you could create your own network [09:05:55] but I wouldn't recommend that [09:06:26] What kind of isolation do you need that isn't accomplished via firewall? [09:06:30] security groups are only somewhat helpful there [09:06:49] andrewbogott: this would be for running jenkins slaves in lavs [09:06:50] *labs [09:06:51] my use case is to isolate an instance from the rest of our network but still let bastion/contint server to ssh to it and grant web access to the instance [09:07:06] Ah, I see. Isolated in both directions :) [09:07:10] good afternoon Singapore :-] [09:07:18] will write a blueprint somewhere on the wiki [09:07:45] it may not be a big deal. if someone wants to attack labs, they just need to get an account for the same level of privilege I guess [09:08:03] another question Ryan_Lane1, is it possible to get access to OpenStack REST API with a user that would be able to create instances bypassing OpenStackManager ? [09:08:17] yes. we've been hesitant to do it previously [09:08:21] (03PS3) 10Nemo bis: Relative path in varnish error message: remove excess / [operations/puppet] - 10https://gerrit.wikimedia.org/r/102945 [09:08:27] use case is OpenStack created a small daemon that can maintain a pool of instances to be consumed by Jenkins/Zuul [09:08:29] because it wouldn't add DNS or puppet configuration for the instance [09:08:46] Who touches varnish stuff apart from m.ark? https://gerrit.wikimedia.org/r/#/c/102945/ [09:08:52] if we use designate the dns issue goes away [09:09:01] the instances can likely live without puppet [09:09:27] sounds good [09:09:35] gotta wait for eqiad to be ready anyway [09:09:37] it would be ideal to have both, though :D [09:10:03] ah puppet I would probably need it to provision the instance properly :( [09:10:17] maybe not [09:10:31] you could create an image that was private to the project [09:10:40] using vmbuilder [09:10:49] it could be pre-loaded with everything you need [09:10:58] yeah that is another point, I would need to be able to refresh the image on a daily basis [09:11:03] daily? [09:11:16] cron @daily glance .... [09:11:20] I'd say write some puppet modules that can be run via puppet apply [09:11:33] put the puppet module into the image, and have it git pull on update [09:11:36] err [09:11:39] on instance start [09:11:44] then run puppet apply [09:11:59] then you'd only need to update the image occasionally for speed [09:12:11] daily image updates would eat a ton of space [09:12:17] still mean the instance will take a bunch of time to build [09:12:26] depends on how much it changes [09:12:51] well the instances are going to be deleted anyway, so that would free up space? [09:12:57] nope [09:13:08] * hashar is a newbie [09:13:11] because the images would still be in glance [09:13:16] you'd have to delete the images too [09:13:25] it's doable, but it's quite a bit of work [09:13:43] ok ok [09:13:45] so, it's possible to upload your own custom images yourself, if we give access to do so [09:13:49] and to delete them [09:14:09] you could have jenkins build them and upload them [09:14:22] and maybe have a cron to delete any older than 7 days or so [09:16:22] yeah [09:16:30] I guess that is enough for today. Sleep week Ryan_Lane1 :-] [09:16:52] :) [09:16:55] * Ryan_Lane1 waves [09:22:40] (03PS1) 10Andrew Bogott: Add a few more neutron packages, adjust sysctl settings. [operations/puppet] - 10https://gerrit.wikimedia.org/r/111165 [09:25:17] (03PS8) 10Hashar: beta: convert into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/108289 (owner: 10Matanya) [09:25:32] (03CR) 10Hashar: "limited number of roles" [operations/puppet] - 10https://gerrit.wikimedia.org/r/108289 (owner: 10Matanya) [09:25:45] Ryan_Lane1: speaking of images, any advice about how to get our glance images from virt0 to virt1000? [09:25:57] I presume that I can't just rsync stuff since the db needs to know what's in there. [09:26:13] (03CR) 10Alexandros Kosiaris: [C: 04-1] "You are removing all usages of nrpe_check_disk_6_3. Why not remove the nrpe definitions as well?" (036 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/110844 (owner: 10Matanya) [09:26:21] matanya: ^ there you go. [09:26:44] btw this is going to need an nrpe change on my part. I am evaluating it. [09:26:48] akosiaris: good morning :-D could you merge in https://gerrit.wikimedia.org/r/#/c/108289/ please that makes beta a module :-] [09:27:09] thanks akosiaris [09:27:14] akosiaris: will apply it right away and tweak the classes applied on instances [09:28:24] (03CR) 10Ori.livneh: "There are plusses and minuses that come with using Fabric." [operations/puppet] - 10https://gerrit.wikimedia.org/r/110904 (owner: 10Ori.livneh) [09:29:27] hashar: it has no reviews ... [09:29:47] andrewbogott: why bother? [09:29:56] make new ones on virt1000 with the same names [09:30:13] akosiaris or apergos, do you know? That's a pretty ugly error: Nemo_bis> Who touches varnish stuff apart from m.ark? https://gerrit.wikimedia.org/r/#/c/102945/ [09:30:16] akosiaris: the modularization has been made by matanya so I reviewed it already :d Did a few tweaks [09:30:21] oh. hm. we're migrating instances [09:30:24] Ryan_Lane1: well, when we transfer... [09:30:25] yeah [09:30:26] PROBLEM - Puppet freshness on cp3021 is CRITICAL: Last successful Puppet run was Mon 03 Feb 2014 09:26:27 PM UTC [09:30:33] Nemo_bis: MaxSem has some varnish knowledge [09:30:48] andrewbogott: dump the database [09:30:52] copy the files over [09:30:53] thanks, added [09:30:58] upgrade glance [09:31:02] akosiaris: are you planning to add the critical, or implement some other mechanism? [09:31:02] (the schema) [09:31:11] (03CR) 10Nikerabbit: "This is a followup to Ie5e46a9feb I assume?" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/110970 (owner: 10Reedy) [09:31:11] matanya: add the critical [09:31:15] that should likely work ok [09:31:28] Ryan_Lane1: hm… yeah, that'll probably work. [09:31:40] I'll give it a try [09:31:46] I'm glad the new version has a sync for glance [09:32:25] Yeah, it's a good idea, just too late to help me right now. [09:32:29] heh. yep [09:32:42] well, it'll help when we set up a region in the new dc [09:33:01] yep. [09:33:26] Do I even need to dump the database? Can I just copy those files too? This page, surprisingly, seems to think that that works. https://dev.mysql.com/doc/refman/5.0/en/copying-databases.html [09:34:02] hm [09:34:07] I'd do a dump [09:34:14] in fact, dumps already exist [09:34:22] Oh, sure, in backup [09:34:23] in /a/backups [09:35:21] ok, off to sleep [09:35:23] * Ryan_Lane1 waves [09:36:06] 'night! [09:38:36] (03PS2) 10Matanya: mail :lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/109514 [09:39:13] (03CR) 10jenkins-bot: [V: 04-1] mail :lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/109514 (owner: 10Matanya) [09:39:53] andrewbogott: if you create new images with glance, I could use one with ubuntu 14.04 :-D [09:40:09] (filled a RT about it iirc) [09:40:09] andrewbogott: +1 [09:40:16] ok -- I haven't done it before. [09:40:23] let me find the doc [09:41:18] https://wikitech.wikimedia.org/wiki/OpenStack#Building_new_images [09:41:51] it doesn't say where to get .img from though :D [09:42:27] I'm in the middle of something right now, sorry. I'll try to get to that soon. [09:42:33] yeah no hurry [09:42:40] just one more thing on your stack [09:42:46] until it is full like Faidon one :-D [09:44:39] Nemo_bis: I am failing to understand the problem. It adds indeed an extra slash (which is correct because it wants a protocol relative url). Also on any javascript aware browser will just reload due to the onclick. [09:46:12] andrewbogott: for labs do you prefer bugs in Wikimedia Labs > Infrastructure or RT ticket to ops-requests ? [09:46:25] bugzilla mostly. [09:46:54] akosiaris: it's not a protocol relative URL. Have you tried looking at the link? [09:47:03] (03CR) 10Ori.livneh: "...basically, I think we should stick to shelling out to dsh, even though it's horrible and gross, until we've finished Pythonizing the re" [operations/puppet] - 10https://gerrit.wikimedia.org/r/110904 (owner: 10Ori.livneh) [09:47:30] (03PS3) 10Matanya: mail :lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/109514 [09:47:32] akosiaris: andrewbogott: the request for Ubuntu 14.04 is in bugzilla https://bugzilla.wikimedia.org/show_bug.cgi?id=60684 [09:48:08] (03PS4) 10Nemo bis: Relative path in varnish error message: remove excess / [operations/puppet] - 10https://gerrit.wikimedia.org/r/102945 [09:48:09] (03CR) 10jenkins-bot: [V: 04-1] mail :lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/109514 (owner: 10Matanya) [09:48:10] Nemo_bis: it is [09:48:28] Yes, which is obviously wrong. :) [09:48:39] "wiki" is not a domain [09:48:55] obviously [09:49:03] but why does it matter ? [09:49:39] hmmm now I see what you mean [09:50:26] (03PS4) 10Matanya: mail :lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/109514 [09:50:36] Firefox even refuses to do anything when I click such a malformed link. [09:51:13] heh ? doesn't javascript kick in ? [09:51:25] or do you have it disabled ? [09:51:42] It's enabled on that domain. [09:52:54] (03CR) 10Mark Bergsma: [C: 031] "Awesome." [operations/puppet] - 10https://gerrit.wikimedia.org/r/111163 (owner: 10Ori.livneh) [09:53:37] mark: i felt pretty awful reading your e-mail.. sorry about that. not quire sure how we missed it. [09:53:42] *quite [09:54:16] Nemo_bis: still, removing the / won't solve anything. [09:54:22] don't worry about it... in dutch we have a saying: "waar gehakt wordt vallen spaanders" ;) [09:54:31] let's see if I can find an english translation of it hehe [09:54:54] an omelette without breaking eggs? [09:55:01] the dutch one is better ;) [09:55:25] we have that saying too [09:55:30] akosiaris: why not? [09:56:30] heh [09:59:02] mark, http://www.slate.com/blogs/lexicon_valley/2013/12/30/english_idioms_it_may_be_true_that_you_can_t_make_an_omelet_without_breaking.html [09:59:31] Nemo_bis: cause it is not my day today... It will obviously solve it. [09:59:34] (I do not share that writer's opinion about banning the phrase, just funny that many languages claim it for their own) [09:59:43] i told you the dutch one is better ;) [10:00:34] mark: mind if I merge https://gerrit.wikimedia.org/r/#/c/102945 ? [10:01:12] (03CR) 10Andrew Bogott: [C: 032] Add a few more neutron packages, adjust sysctl settings. [operations/puppet] - 10https://gerrit.wikimedia.org/r/111165 (owner: 10Andrew Bogott) [10:01:22] akosiaris: go ahead [10:01:45] Nemo_bis: btw my firefox does try to reload despite the malformed link. [10:02:16] akosiaris: mine shows some spinning but doesn't actually do anything (nothing visible or logged). [10:02:51] then i'd say it works as expected. [10:03:53] (03CR) 10Alexandros Kosiaris: [C: 032] Relative path in varnish error message: remove excess / [operations/puppet] - 10https://gerrit.wikimedia.org/r/102945 (owner: 10Nemo bis) [10:06:10] (03PS4) 10Matanya: mysql: change nrpe monitoring to use nrpe::monitor [operations/puppet] - 10https://gerrit.wikimedia.org/r/110844 [10:07:00] (03PS2) 10Matanya: swift: lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/109625 [10:07:09] mark, I already have an appointment with Leslie to sort this out, but in case there's an easy answer: How can assign additional IPs to additional nics on a server? I know how to add them to the dns repo, I think, but it's unclear to me what happens if multiple fixed IPs are assigned to one box. [10:07:55] so for multiple ips to the same interface you can use interface::ip [10:08:06] if it's for a different interface, we don't currently have puppet config for that [10:08:19] that's really a matter of configuring it in /etc/network/interfaces [10:08:36] In this case it's different interfaces -- three IPs, three interfaces. [10:08:51] Ok, but the server will pick up those IPs from dhcp, right? So they have to be assigned externally first? [10:09:06] well... [10:09:12] dhcp is only used during the installation phase [10:09:16] and should not be used after [10:09:24] oh, ok. [10:09:33] the installer converts the dhcp ip into a static ip [10:09:40] so you should only do that for eth0, the "system" ip [10:09:49] other nics normally have different functions [10:09:56] have a look at site.pp for lvs1001 etc [10:10:02] those configure ips on additional nics [10:10:06] I think you could use that [10:10:34] hmm it uses interface::tagged though [10:10:39] which I think is what we used anyway [10:10:46] do you know what a "tagged" interface is? [10:10:49] 802.1Q tagging [10:10:54] Where do I document the fact that these additional IPs are 'taken'? [10:11:01] in reverse dns [10:11:02] (I don't know what a tagged interface is, yet.) [10:11:17] ok, 802.1Q tagging is vlan tagging, basically multiplexing of vlans [10:11:17] ok. [10:11:33] so using tagging you can have traffic for multiple virtual lans across the same link/interface [10:11:46] a tag is nothing more than a prefix to the message saying "this packet belongs to subnet X" [10:12:02] I don't need that, though, do I? Since I have multiple interfaces anyway? [10:12:04] so e.g. the LVS servers sit in all vlans we have realservers in [10:12:09] yeah but you do want that anyway [10:12:12] since you may want it later [10:12:14] and it's "cleaner" [10:12:19] we did it in tampa that way [10:12:24] even though I think only one vlan was used so far [10:12:45] so, iirc, the tampa hosts have eth1 configured with tagging [10:12:47] but in any case don't use untagged vlans [10:12:55] ? [10:13:21] if this is andrewbogott's setup, an untagged vlan won't work [10:13:39] i'm not sure what you mean [10:14:22] andrewbogott is trying to have multiple interfaces having muliple source traffic, correct? [10:14:58] For reference, I'm trying to set up what is specified in the first 'note' on this page: http://docs.openstack.org/trunk/install-guide/install/apt/content/neutron-install.dedicated-network-node.html right up top [10:16:36] oh, then ignore me [10:16:46] ok so [10:16:53] linux treats a tagged interface as a separate nic [10:16:55] sorry for interupet [10:17:07] so eth1.2 is "all packets arriving/sending on eth1 which have vlan tag 2" [10:17:36] so you can treat eth1.2 as if it were a dedicated nic (eth1) that is just connected to a subnet vlan 2 as normal [10:18:39] mark: That means I'm doubling up the traffic on eth1 though. Isn't that worse than using the separate nics? [10:18:48] Maybe I'm misunderstanding [10:18:53] why would it double up traffic? [10:19:31] I feel like you're describing a setup that maps multiple channels onto a single nic, leaving the box's other nics idle. [10:19:42] no [10:19:50] it's also exactly the same as in tampa [10:19:51] i.e. [10:20:00] eth0 is used as it always is [10:20:17] and eth1 is used for inter-host communication (e.g. between instances on different nodes) [10:20:25] the fact that a vlan tag is added doesn't matter [10:20:31] it just provides additional flexibility later [10:20:40] should we need to expand labs to multiple vlans, or between multiple data center rows [10:20:43] we haven't needed that in tampa yet [10:20:50] but it's not complex to add now, and is hard to change later [10:21:45] Ah, ok -- you're talking about a /different/ nic with a tag. Sorry, was distracted by the tag, forgot you were now talking about eth1 instead of eth0 [10:22:59] yeah [10:23:09] so if we set this up, which is pretty easy with puppet [10:23:18] you can just think of eth1. as if it were eth1 [10:23:26] (03PS1) 10Andrew Bogott: Grab a couple more IPs for labnet1001 [operations/dns] - 10https://gerrit.wikimedia.org/r/111169 [10:23:26] So, my attempt to designate additional IPs: [10:23:33] Is that the right idea? [10:24:00] i'm not sure what you're trying to do there [10:24:30] ok :) [10:24:32] why do you need multiple ips in the same subnet? [10:25:32] Sorry, I'm lost. Is your point that they should be in different subnets? Or that that's not the place to do this at all? [10:25:59] so, normally the only reason you need multiple ips [10:26:18] is because you need to do different things on the same port (say port 80) on an ip [10:26:23] if they are in the same subnet [10:26:33] if they are in different subnets, it is to communicate across subnets [10:26:43] i think i'm not being clear, sorry ;) [10:26:55] perhaps I should ask [10:26:59] why do you think you need multiple ips? [10:27:30] Ah! Because the docs say so :) Here's that link again…. http://docs.openstack.org/trunk/install-guide/install/apt/content/neutron-install.dedicated-network-node.html [10:27:50] A few lines down (after saying that I need three nics) it says "All nics need static IPs" [10:28:02] So, I'm just blindly complying, so far. [10:28:08] hehe [10:28:16] I think I should find some time to look over this with you [10:28:16] (03CR) 10Hashar: "I have copy pasted ori comments on wikitech at https://wikitech.wikimedia.org/wiki/Fabric for later references." [operations/puppet] - 10https://gerrit.wikimedia.org/r/110904 (owner: 10Ori.livneh) [10:28:36] because it's already complex enough without the openstack specific terminology [10:29:01] but right now I can say that allocating multiple ips within the _same_ subnet is probably not what you need [10:29:10] Later on there are steps like "Configure the EXTERNAL_INTERFACE without an IP address and in promiscuous mode. Additionally, you must set the newly created br-ex interface to have the IP address that formerly belonged to EXTERNAL_INTERFACE." [10:29:16] yes [10:29:23] So, further refs to each nic having an IP. [10:29:23] i'm not sure yet what EXTERNAL_INTERFACE means in openstack speak [10:29:26] do you know? [10:29:32] we should definitely draw some diagrams [10:30:10] It's awkward because there's clearly an editing mistake here. But it says "The management network handles communication among nodes. The data network handles communication coming to and from VMs. The external NIC connects the network node, and optionally to the controller node, so your VMs can connect to the outside world." [10:30:18] Which makes modest amounts of sense to me. [10:30:30] The 'management' network is what I would call the normal connection, eth0 [10:30:49] 'data' and 'external' are the additional OpenStack-specific bits that I need to add. [10:30:53] yeah it's confusing [10:31:17] (03PS2) 10Matanya: dns: lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/109632 [10:31:50] external nic is almost certainly eth0 then [10:32:12] and the data network is almost certainly eth1. [10:32:15] (03PS2) 10Matanya: varnish: puppet 3 compatibility fix: correct variable [operations/puppet] - 10https://gerrit.wikimedia.org/r/109869 [10:32:16] but i'll have to check [10:32:35] (03PS2) 10Matanya: facilities: lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/110339 [10:33:00] (03PS2) 10Matanya: certs: lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/110366 [10:34:01] i would say "management network" and "external nic" are the same, eth0 [10:34:15] we use it both to manage the hosts (with puppet and all) [10:34:30] and right now it's also used for sending off traffic from the network node to the internet [10:34:40] there's hardly any reason to separate them, especially with 10G [10:36:11] ok, I thought I saw a reason here, let me find the note... [10:36:59] Ah, ok, this: "The host must have an IP address associated with an interface other than EXTERNAL_INTERFACE, and your remote terminal session must be associated with this other IP address." http://docs.openstack.org/trunk/install-guide/install/apt/content/install-neutron.install-plug-in.ovs.html [10:37:00] (03PS2) 10Matanya: emery: RT #6143 move two logs to erbium [operations/puppet] - 10https://gerrit.wikimedia.org/r/110382 [10:37:12] That suggests that they can't be the same, at least to follow this guide. [10:37:41] (03PS5) 10Matanya: webserver: lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/110454 [10:37:46] (03PS1) 10Ori.livneh: package-builder.pp: add admonition to not refactor or lint this module [operations/puppet] - 10https://gerrit.wikimedia.org/r/111170 [10:38:07] so [10:38:19] the network node(s) will have at least one additional interface ip [10:38:20] for NAT [10:38:32] so any instance that doesn't have its own public ip [10:38:38] will share in the SNAT ip for outgoing traffic [10:39:16] sure. That sounds like 'External' to me. [10:39:30] (03CR) 10Ori.livneh: [C: 032] "No-op." [operations/puppet] - 10https://gerrit.wikimedia.org/r/111170 (owner: 10Ori.livneh) [10:40:00] yes [10:40:26] but that NAT ip will be in a very different ip subnet/prefix [10:40:49] ok [10:41:07] so, have a look at how tampa is configured currently [10:41:15] even though it's different openstack component [10:41:21] i think the setup will be very similar [10:41:39] what is the network node in tampa currently? [10:42:11] virt2 [10:43:01] right [10:43:05] have a look at role::nova::network [10:43:19] it sets up the bonding, which we won't need with 10G [10:43:23] and it sets up the tagged interface [10:44:02] not quite correctly I think, we should just use interface::tagged now [10:44:03] but anyway [10:44:11] you see there that it sets up the "SNAT" ip for the network node [10:44:26] ah no [10:44:35] it looks like the interface is actually managed by openstack itself [10:44:43] so openstack currently configures the ip to that interface [10:44:51] and puppet only assigns the SNAT ip to the loopback interface for other reasons [10:45:22] which seems to be configured in role::nova::config [10:45:26] that will be different now i'm sure [10:47:46] So, are you still thinking that we don't need to use three interfaces? [10:47:55] absolutely [10:48:02] and if we do need 3 [10:48:06] we'll use tagging [10:48:28] so we might end up with say, eth0, eth1.1101, eth1.1102 or something [10:48:41] that's 3 interfaces right there, even though it's only 2 physical ones [10:48:45] Ah, ok, so that's effectively three. [10:48:47] yes [10:48:56] but i don't think we'll want that [10:49:04] That's fine, I'm not bothered by whether or not there are three /physical/ networks. [10:49:05] i'll try to find some time to read these docs too [10:49:07] Well... [10:49:09] so I understand the openstack terms [10:49:21] ok. Because it's going to be very hard for me to follow the setup guide if at step one we're already not doing what it says :) [10:49:33] i know what the network should look like, but it's difficult to know what openstack expects and what it manages itself [10:49:40] yes i understand [10:51:11] (03PS2) 10Matanya: nrpe: remove hard coded disk checks [operations/puppet] - 10https://gerrit.wikimedia.org/r/110880 [11:05:39] (03PS1) 10Springle: repool db1027 in s3, warm up [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/111173 [11:06:15] (03CR) 10Springle: [C: 032] repool db1027 in s3, warm up [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/111173 (owner: 10Springle) [11:06:23] (03Merged) 10jenkins-bot: repool db1027 in s3, warm up [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/111173 (owner: 10Springle) [11:08:04] !log springle synchronized wmf-config/db-eqiad.php 'repool db1027 in s3, warm up' [11:08:12] Logged the message, Master [11:15:27] (03PS1) 10Springle: increase db1059 load (96G ram compared to 64G for s4 siblings) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/111175 [11:15:50] (03CR) 10Springle: [C: 032] increase db1059 load (96G ram compared to 64G for s4 siblings) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/111175 (owner: 10Springle) [11:15:56] (03Merged) 10jenkins-bot: increase db1059 load (96G ram compared to 64G for s4 siblings) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/111175 (owner: 10Springle) [11:17:18] !log springle synchronized wmf-config/db-eqiad.php 'db1059 LB increase' [11:17:26] Logged the message, Master [11:28:42] (03PS9) 10Alexandros Kosiaris: beta: convert into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/108289 (owner: 10Matanya) [11:38:57] (03CR) 10Alexandros Kosiaris: "avoid the include class1,\n class2,\n class3 pattern" [operations/puppet] - 10https://gerrit.wikimedia.org/r/108289 (owner: 10Matanya) [11:39:11] (03CR) 10Alexandros Kosiaris: [C: 032] beta: convert into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/108289 (owner: 10Matanya) [11:53:04] (03CR) 10Byfserag: [C: 031] "Per TTO" [operations/apache-config] - 10https://gerrit.wikimedia.org/r/110155 (owner: 10TTO) [12:00:26] PROBLEM - Puppet freshness on cp3019 is CRITICAL: Last successful Puppet run was Mon 03 Feb 2014 08:55:41 PM UTC [12:05:06] PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 70.433334 [12:18:06] RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [12:23:46] RECOVERY - puppetmaster https on virt1000 is OK: HTTP OK: Status line output matched 400 - 336 bytes in 1.223 second response time [12:27:06] PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 243.366669 [12:29:06] RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [12:31:26] PROBLEM - Puppet freshness on cp3021 is CRITICAL: Last successful Puppet run was Mon 03 Feb 2014 09:26:27 PM UTC [12:35:17] mark: This should be an easier one… do you know what (if anything) is blocking http and https on virt1000? I have apache up but still no access. [12:35:59] there's no firewalling by the network equipment if that's what you mean [12:36:13] or hm [12:36:16] for labs maybe there is [12:36:18] let me check [12:36:35] Ryan said that the website was 'disabled' and I thought that just meant he had stopped apache… until just now. [12:36:37] thanks [12:38:59] so right now there is no filtering even though there should be [12:39:22] just tcpdump port 80 and see if you get packets when you telnet to it? [12:40:55] Oh, yep, packets are getting through. So, not a filtering issue. [12:43:55] (03PS4) 10Ori.livneh: Rewrite 'scap' script in Python [operations/puppet] - 10https://gerrit.wikimedia.org/r/110904 [12:44:06] PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 86.866669 [12:44:33] (03CR) 10Ori.livneh: [C: 04-1] "Needs more work" [operations/puppet] - 10https://gerrit.wikimedia.org/r/110904 (owner: 10Ori.livneh) [12:45:06] RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [12:50:46] PROBLEM - puppetmaster https on virt1000 is CRITICAL: CRITICAL - Cannot make SSL connection [12:53:34] (03PS1) 10Matanya: bots: minor lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/111181 [13:02:43]