[07:54:05] Krinkle, AaronSchulz https://grafana.wikimedia.org/d/000000316/memcache?viewPanel=37&orgId=1&from=now-60d&to=now :) [07:54:22] (kudos to Effie for the big work of reimaging to Buster) [08:21:18] <_joe_> we had a broken tunnel though earlier... did you fix it? [09:26:11] sigh 1024 :( [09:26:49] I will remove this redis shard, but I can't remove it from the memcached pool [09:27:10] if my memory serves me well, this will trigger a resharding [09:27:29] as what matters in mcrouter is the order of the servers in a pool [09:27:46] so if for instance we replace a server IP with another, that is just fine [09:27:55] if we remove an ip, not fine [09:28:58] my opinion is we keep the gutter pool serving while we wait for a replacement, that is what we did with a dead codfw server in August [09:29:13] or july, I don't remember [09:30:32] elukey: sigh, too many channels [09:30:34] sorry [09:32:54] another alternative would be to temporarily replace mc1024 with one mc-gp host [09:33:34] akosiaris: how hard is it to have a public IP on a ganeti instance? [09:33:47] we had a rolling reboot scheduled (cc moritzm) for those hosts, which is an additional problem [09:34:00] kormat: How much are you ready to pay for it ? :P [09:34:05] kormat: we all have a price [09:34:12] akosiaris: i'm ready to have marostegui pay for it [09:34:13] it's free for v6 [09:34:21] rotfl [09:34:25] deal! [09:34:29] \o/ [09:35:19] kormat: Really easy btw. Just create it with a public IP in the first place. https://wikitech.wikimedia.org/wiki/Ganeti#Create_the_VM [09:35:21] taking a step back: i have a service running on a ganeti VM that needs to be publically accessible, behind IDP [09:36:17] --network public being the key term here. [09:36:45] kormat: can't it be behind a LVS? [09:36:46] akosiaris: can a VM be on both public and private networks at the same time? [09:36:51] But if you mean adding a second IP to the VM, it might be easier down the road to just create a new VM with a public IP [09:37:07] kormat: why? [09:37:09] It can. But we have no example of that around [09:37:09] XioNoX: i don't know much about LVS, so i'm going to say 'probably?' [09:37:24] so, you might not want to create a snowflake [09:37:29] akosiaris: indeed [09:37:50] XioNoX: is ganeti behind LVS supported? [09:38:13] create the host with a private IP and front it with LVS is the preferred approach, but then it probably depend on the service, etc [09:38:46] kormat: Btw you can just front it behind the edge caches. We got history with that, plenty of it [09:39:12] XioNoX: the service is orchestrator - it's a single instance web app [09:39:41] Depends on what the service is of course. The general rule up to now has been that monitoring services that we might rely on while edge caches are dead, might be best to not be behind the edge caches [09:40:13] akosiaris: what does "behind the edge caches" mean? [09:40:52] kormat: Going the ats+varnish to reach the service. As in User -> local POP -> ats -> varnish -> service. e.g. etherpad is like that [09:41:00] or for that matter most services [09:41:14] but e.g. icinga is not. That one has a public IP and is not behind the edge caches [09:41:50] orchestrator will be used to manage our database clusters, so it probably falls into the same category as icinga [09:42:12] in that you might Really need it in an emergency [09:42:29] In that case, better create a new VM with a public IP then. [09:42:54] agreed. for comparison, dbmonitor/tendril is also on public IP currently [09:44:08] moritzm: i had originally been thinking of running the idp auth on dbmonitor, and reverse proxying the access through to the ganeti vm. but i'm not very comfortable with that [09:46:52] yeah, no need for that, we can use a similar mod_cas setup as currently on tendril (and several other services) [09:54:13] what is dyna.wm.o? [09:54:46] dbtree is CNAME'd to it, but tendril is CNAME'd to dbmonitor1001. both are _served_ by the same machine [09:56:16] (nothing on wikitech) [09:57:22] kormat, no idea but there are some results on phab: https://phabricator.wikimedia.org/T208263 [09:58:00] https://phabricator.wikimedia.org/T270664 [09:58:19] kormat: dyna is GEOIP [09:59:33] XioNoX: meaning it resolves to edge caches? [09:59:43] kormat: yep [10:00:15] alright. the lack of docs is a bit.. unhelpful. oh well. [10:00:23] for your usecase I think you only need a CNAME [10:00:46] like librenms [10:01:04] gotcha, thanks. [10:01:17] XioNoX: is there a procedure for acquiring a public IP? [10:01:28] effie: the main issue is also that TTL for keys in the gutter is lower, I wouldn't keep this status for a long time [10:01:43] kormat: netbox :) let me check [10:01:49] effie: we have budget for the new nodes, we can get a new one sooner than Q4 [10:02:16] kormat: what you create the VM with the sre.ganeti.makevm cookbook, the DNS name gets automatically created via Netbox [10:02:19] effie: and we should find out how mcrouter behaves if a shard is removed, in theory it should use consistent hashing [10:02:22] kormat: theissue with having automation is that it's harder to know what's automated and what's not [10:02:34] see [10:02:36] XioNoX: hah, indeed [10:02:38] only the CNAME needs to be manually created via the dns.git repo [10:02:53] moritzm: ah ok, cool. [10:03:57] elukey: having a look [10:03:58] https://github.com/facebook/mcrouter/wiki/Pools [10:04:10] it says "The routing part of the key is hashed and the request is set to the resulting index in the pool." [10:04:21] if I am interpreting this right [10:04:38] some keys will be moved to other hosts, since the index number will change [10:05:19] in theory only the keys related to mc1024 will be re-hashed [10:05:39] can we test it in deployment prep or similar? [10:08:08] <_joe_> elukey: that's not strictly accurate, it's not just the keys assigned to that host, but it should limit the amount of reshuffling, yes [10:08:36] _joe_: ah okok [10:09:13] <_joe_> https://github.com/facebook/mcrouter/wiki/Pools#hash-functions [10:09:48] https://github.com/facebook/mcrouter/issues/112#issuecomment-193420864 [10:10:36] if ^ is true and this is not something that has changed since [10:10:51] thanks for the links [10:10:52] I think our best option is to keep this failed over to the gutter pool [10:11:03] on a brighter side [10:11:11] those are 10G hosts! [10:11:22] we lose something we win something ! [10:11:43] <_joe_> effie: mc1024 is not the first or the last server in the pool, so yeah there will be reshuffling around, but we should be able to withstand it [10:12:01] <_joe_> is the server dead? [10:12:04] yep [10:12:15] and OOW [10:12:15] <_joe_> like forever? [10:12:18] yep yep [10:12:19] <_joe_> ok [10:12:21] yes looks like it is forever [10:12:33] <_joe_> so let's remove it from the cconfiguration [10:12:41] but we have a complete refresh scheduled for Q4, can't we just order a new node? [10:12:42] <_joe_> if you prefer to wait for monday, that's ok too [10:12:52] <_joe_> elukey: sure, but how long will that take? [10:12:55] since this is something we have not done before [10:13:00] _joe_: we wait from rob [10:13:02] <_joe_> effie: incorrect [10:13:08] <_joe_> we did this before [10:13:13] ah! when ? [10:13:14] <_joe_> you just weren't here :D [10:13:22] lol, low blow :p [10:13:34] <_joe_> not really, I'm making myself look old [10:13:40] last time when I was here, we left the gutter pool take over [10:13:44] <_joe_> but yes, we had to remove a mc server from mcrouter here [10:13:46] and it was also fine as well [10:13:54] <_joe_> effie: yeah I'm sure that's fine too [10:14:00] <_joe_> but it really depends on lead times [10:14:03] <_joe_> and btw [10:14:20] <_joe_> if we buy one server, it's better if we buy all of them I think so we get some bulk discount [10:14:23] ok so I will make the call to review this on Tuesday (I may be unavail on monday) [10:14:28] <_joe_> but that's a discussion for later [10:14:36] <_joe_> effie: 👍 [10:14:39] if we have an estimate of when a server will be available [10:14:55] <_joe_> yeah let's talk with dcops later in the day [10:14:57] if it will be added in the q4 order, then yes it makes sense to remove it [10:15:11] also there is the problem of no 10g space in eqiad, if we want to get 10g nodes it will be a problem to rack them [10:15:58] (and we should avoid 4/5 shards in the same rack like we do now :P) [10:16:36] elukey: what's the timeline? [10:16:48] we're ordering 2 more 10G switches this Q [10:18:44] XioNoX: so what we have now is [10:18:50] refresh eqiad in Q4 [10:18:51] XioNoX: in theory Q4, 18 servers in eqiad (mc10xx) [10:19:01] and buy 10G NICs for codfw [10:19:21] that looks fun [10:21:43] elukey: 18 new servers, or replacing 1G ? [10:22:45] XioNoX: new servers, the old ones (1G) will be decommed [10:26:00] elukey: did you check with DCops where they will fit? [10:27:35] XioNoX: I am biased since I am already waiting for 6 10g spots for hadoop :P [10:28:06] but no jokes aside I didn't really ask anything [10:29:12] XioNoX: there is also the main question mark about the links switch-routers, memcached could potentially raise the traffic level a lot (so 100G links would be really nice :D) [10:29:40] (hopefully with on-host memcached on the mw servers we should reduce the traffic I know, but..) [10:31:22] elukey: cross row traffic, right? [10:31:53] XioNoX: exactly yes, mediawiki <-> memcached [10:32:37] elukey: is it rack aware? [10:33:34] XioNoX: not really, keys are sharded using consistent hashing by mcrouter on every mediawiki node, but there is no topology awareness afaik [10:34:06] if mediawiki needs a specific key it will be on one shard only, that might be in the same row or not [10:34:18] (not sure if I have answered, trying to understand the question :D) [10:34:53] unless we move towards a cluster per row... [10:35:05] * volans|off hides [10:36:41] elukey: yeah that was the question [10:36:53] there are many other questions but not for today :) [15:03:40] <_joe_> yeah if we bought new memcached servers with 1G network cards it would really be unfortunate [15:03:47] <_joe_> (I missed the discussion earlier) [15:04:15] <_joe_> and frankly, it should take precedence on stuff that's not involved with serving the live site [15:04:25] <_joe_> if we have to pick what gets 10G nics [16:04:37] 20th celebration party now! [16:05:32] https://www.youtube.com/watch?v=PzGAGfSObOw [16:07:44] re: 10G, our budget included projections for 10G ports and upgrades to accommodate [16:08:04] if you have not budgeted for that in the past, happy to talk and try to accommodate (but no promises) [16:08:45] I'd also like to ask to not think in divides like "serving the live site" -- we try to accomodate a number of diverse use cases based on a number of different criteria [16:09:10] yeah, you may not like backup recovery sites be 10x slower on an incident :-) [16:09:34] but as paravoid says, there is indeed a case to be done for prioritization [16:11:26] So Im just now starting for the day and i see a backlog regbarding mc spec on replacement server [16:11:32] was the above discussion copied to the task [16:11:33] ? [16:11:54] <_joe_> robh: not yet [16:14:21] so yesterday i was under impression this was normal priority and i was goiong to get quotes later today. if this is higher UBN let me know [16:14:36] since the gutter pool took over and it was just one mc system. [16:20:23] <_joe_> robh: yeah but we're working in a degraded status, so it might make sense to make the purchase originally programmed for next quarter during this one [16:21:01] duly noted, seems sensible to me [16:21:21] so on those, you guys expected them to have 10G capability? if we add a 10g nic to these they still have onboard 1g [16:21:33] so should allow for use in either rack for near term future [16:21:41] i saw discussion above [16:21:53] current mc are 1g, but adding in the nic adds about 300usd. [16:22:06] can we not figure this out like this please? [16:22:11] esp. during the birthday party [16:22:14] ? [16:22:22] i was just doing normal work, how was i supposed to figure it out? [16:22:29] on a task please :) [16:22:31] or is today party friday no working like the working day? [16:22:43] paravoid: i find it a lot easier to discuss in irc and then copy to task after discussion [16:22:54] but if this is different from other procurement and is required to be all on task ill change [16:22:56] and after looking at what we have in the procurement sheet please [16:23:07] I'm going back to the livestream ttyl :) [16:23:11] ... i thought that is what this discussion is? [16:23:15] like, this is how i determine specs.... [16:23:38] <_joe_> robh: we'll update the task once I've seen what was added to the sheet, I wasn't directly involved in preparing our procurement this year, and update the task on monday [16:24:01] Do I need to treat this order like special and no irc? I dont get how it differs from my other tasks and I dont get that im not allowed to chat about it? [18:19:44] hnowlan: o/ - not urgent but I am wondering if maps2007 could have puppet enabled (the netbox reports are alerting :) [18:21:33] elukey: yep! sorry about that again, reenabled. [18:24:09] hnowlan: <3 [19:32:58] https://www.elastic.co/blog/licensing-change looks like ES is going SSPL [20:24:24] I guess they've decided the rapid growth /startup period is over and the existing enterprise customers + some organic growth will suffice to keep shareholders happy [20:35:40] yeah, they might also want to hold a knife to the throat of AWS :) [20:41:25] <_joe_> mszabo: I think the situation is the other way around, but yes [20:41:50] <_joe_> we saw that, i think a call will need to be made eventually. [20:42:59] ah, I see [20:46:56] <_joe_> I mean I think AWS has a kinfe to the throat of elastic, and this is elastic trying to turn the table [20:47:05] Yeah.