[08:03:47] jayme: should he start the lvs/dns push-not int a bit? [08:03:53] hi! [08:22:38] hi o/ [08:29:15] :D [09:00:38] backup LVSs are lvs2010 and lvs1016 [09:00:53] primary lvs2009 and lvs1015 [09:28:39] volans: are you around? [09:28:41] help [09:32:01] effie: sure [09:32:33] :D [09:32:43] so we added a new IP address in [09:33:06] https://wikitech.wikimedia.org/wiki/DNS/Netbox?veaction=edit§ion=18#How_to_manually_allocate_a_special_purpose_IP_address_in_Netbox [09:33:33] ok [09:33:40] https://netbox.wikimedia.org/ipam/ip-addresses/6906/changelog/ [09:33:42] and we got that [09:34:10] was there something else we should have chosen ? [09:35:31] mmmh, checking [09:35:56] ah arzhel is not here, let me ping him to double check one thign [09:36:03] you can join is [09:36:09] https://meet.google.com/umn-rigz-udw [09:36:33] the prefix is marked as a pool, I'm not entirely sure if that's correct [09:36:36] <_joe_> effie: it seems you added an entry for the /24 [09:36:39] that's why you got that address [09:37:11] _joe_: no, netbox has all teh addresses with the parent prefix netmask as a standard thing [09:37:24] also they didn't choose one, it got assigned by netbox, following the procedure [09:37:42] we followed the docs [09:37:46] <_joe_> volans: I'm just looking at https://netbox.wikimedia.org/ipam/ip-addresses/6906/ vs https://netbox.wikimedia.org/ipam/ip-addresses/4904/ [09:37:50] because we had never done that before [09:38:10] <_joe_> they attached the A record to 10.2.2.0/24 [09:38:20] <_joe_> instead than to whatever IP they wanted to allocate [09:38:34] <_joe_> effie: I have no doubts, I'm just pointing out the result is incorrect [09:38:45] _joe_: the discussion if about the .0 for now [09:38:57] the netmask I can explain, and we can decide what's the best action there [09:39:07] <_joe_> volans: they wanted I suppose to add a new IP to that list [09:39:12] and they did [09:39:16] just got the .0 [09:39:18] <_joe_> for what IP? [09:39:27] <_joe_> oh I see, why on earth? [09:39:41] because the prefix is marked as pool, see in -foundations [09:39:41] <_joe_> wait, you cannot choose the IP in the range of the free ones? [09:39:43] I'm asking arzhel [09:40:04] yes you can pick if you want, but auto-assign is easier and picks the first one [09:40:07] free [09:40:45] <_joe_> please don't [09:40:53] what? [09:40:59] <_joe_> if that's what the docs suggest for load balancers, it should be changed [09:41:06] don't what? [09:41:10] <_joe_> we have free IPs in that range for a reason [09:41:16] <_joe_> auto-assign will mess with it [09:41:38] I sent the link to the docs for review here more than a week ago [09:41:56] https://wikitech.wikimedia.org/wiki/DNS/Netbox#How_to_manually_allocate_a_special_purpose_IP_address_in_Netbox [09:42:01] <_joe_> sorry if this wasn't immediately clear to me. I'm telling you now [09:42:20] <_joe_> I did read it and that didn't click as a red flag in my mind [09:42:26] <_joe_> I realized it now [09:42:48] in the new IP page you can change it in the UI [09:42:50] as you want [09:43:38] _joe_: is there a documentation on how to pick one? based on what? [09:43:44] <_joe_> the alternative is I reserve with fake hostnames [09:43:57] there is a status reserved in netbox [09:44:00] <_joe_> volans: it was assumed you'd add your services to the end of the used IP lists [09:44:00] no need for fake names [09:44:27] <_joe_> so yes, we need to reserve the IPs between 0 and 8 that are unused, basically [09:44:29] we can adopt the docs a bit to pick a "right" one. IIRC we usually avoid everything from .0 to .9 [09:44:39] to clarify, the svc part is a totally manual one, you can assign whatever you want [09:44:49] there are other holes in the subnet [09:44:56] are those on purpose? [09:45:18] I think they are jsut because something got removed [09:45:29] <_joe_> yes, those are just removed stuff [09:45:36] can they be reused? should those be reserved too? [09:45:39] <_joe_> but let me add a reservation on those IPs [09:45:45] <_joe_> volans: they definitely can be reused [09:46:06] but can we just remove the 10.2.2.0/24 entry again volans? Or do we need to do anything special? [09:46:16] <_joe_> I just want to keep 0-9 free for MediaWiki [09:46:49] jayme: you can delete it sure, no harm [09:46:56] volans: because looking at this https://netbox.wikimedia.org/ipam/ip-addresses/6906/ I'm getting scared of clicking delete because of all the related IPs :D [09:47:34] jayme: let me check there isn't a bad netbox bug [09:47:55] volans: thanks. Feel free to delete :P [09:48:05] damn netbox! [09:48:21] Are you sure you want to delete prefix 10.2.2.0/24? [09:48:24] no, I'm not! [09:49:26] jayme: what IP do you want? [09:49:50] 56 [09:49:55] _joe_: for all service prefixes? both high/med/low? [09:50:15] <_joe_> ? [09:50:39] <_joe_> only for the 10.2.[12].0/24s [09:50:42] jayme: https://netbox.wikimedia.org/ipam/ip-addresses/6906/ [09:50:46] went for edit [09:51:47] great :D - funny that you can change it from being a subnet to being an IP [09:51:47] so it is happy now? [09:51:58] _joe_: ack, will reserve [09:52:12] <_joe_> oh also [09:52:20] <_joe_> we want the same IP in both datacenters [09:52:22] btw .8 is kubemaster [09:53:05] <_joe_> effie/jayme, curiosity - how did you find the VLAN? [09:53:22] oh jayme searched for 10.2.2 [09:53:31] * jayme runns away [09:53:33] and we searched a bit more [09:53:34] https://netbox.wikimedia.org/search/?q=10.2 [09:53:35] and we foun it [09:53:41] https://netbox.wikimedia.org/ipam/prefixes/?expand=on&page=1 [09:53:44] <_joe_> lol [09:53:46] go to prefixes and expand [09:53:53] look for 'services' [09:53:56] or LVS [09:54:00] <_joe_> volans: so not how it's suggested in the docs [09:54:33] there are multiple ways to find it [09:54:49] I think traverse from the vlan is less error prone [09:54:53] <_joe_> and they all suck [09:54:55] <_joe_> :P [09:55:00] <_joe_> ok thanks [09:55:49] is this better? [09:55:49] https://netbox.wikimedia.org/ipam/prefixes/?q=%28internal%29+services [09:55:54] <_joe_> effie / jayme please add the missing pieces of docs to the "add a new lvs service" docs, some details are less obvious probably now that we use netbox and not a test file [09:56:18] _joe_: sure thing [09:56:23] to be fair, the docs are thorough enough, to truly check if the docs are ok, is to actually perform the steps [09:56:27] <_joe_> *text file [09:56:34] this was a bit of a corner case too [09:56:48] <_joe_> why? [09:57:37] because .0 was presented as available [09:57:59] at least that was one of the problems [09:59:07] in https://wikitech.wikimedia.org/wiki/LVS#DNS_changes one is led to https://wikitech.wikimedia.org/wiki/DNS/Netbox?veaction=edit§ion=18#How_to_manually_allocate_a_special_purpose_IP_address_in_Netbox [09:59:07] <_joe_> still, there are things that were obvious before [09:59:12] <_joe_> that are not anymore [09:59:35] we can convert all the logic in a simple netbox script [09:59:35] <_joe_> like that you should ensure you have the same last octect in all datacenters [10:00:01] <_joe_> volans: sure, but for now just pointing it out in the docs for LVS is enough IMHO [10:00:35] <_joe_> "assign automatically in one dc, use the same last octect in the other" [10:01:24] it's quick and easy to create a script, we can do it this week, really [10:05:23] _joe_: I guess I should fix any inconsistencies between the two subnets too [10:05:27] things that are eqiad only [10:05:39] <_joe_> err, no :P [10:05:53] <_joe_> that's the other issue, we do have services that are only in one DC [10:05:59] <_joe_> that we like it or not [10:06:08] yes, I'll make them reserved though [10:06:14] <_joe_> but yes, probably the same IP in the other dc should be reserved [10:06:14] like they are in the file [10:06:19] <_joe_> +1 [10:06:19] with a comment [10:06:23] that's why was not auto-imported [10:06:47] <_joe_> adding a name that points to nothing isn';t great, it's better to just reserve the same slot [10:07:13] <_joe_> yeah reality is asymeetric [10:07:43] yep, reserved with a comment for what it's reserved [10:07:57] like I just did [10:07:57] https://netbox.wikimedia.org/ipam/ip-addresses/6921/ [10:09:43] <_joe_> +1 [10:10:42] we could also add a report that ensure those are always in sync [10:11:17] jayme, effie: are you creating the codfw record too? [10:11:21] <_joe_> so that I can troll everyone by not fixing it and letting icinga alert for months too? cool! [10:11:27] volans: sure [10:11:37] <_joe_> volans: jokes aside, that would be great :D [10:11:37] please use the /32 as netmask [10:11:43] I can update the docs if not already in it [10:11:59] _joe_: ack, let me open a task for chaomodus and me [10:13:53] volans: will do. Is that a bug then? Should we add like "use /32" to the docs? [10:14:59] yes, or I will after you've done editing it :) [10:17:27] volans: done [10:18:00] thx a lot! will review in a sec [10:21:34] _joe_: for the script, should it allow to allocate one of the reserved if one marks a checkbox that says "use reserved for MediaWiki address" or those should left for manual allocation because so rare [10:21:55] <_joe_> probably the latter :) [10:22:01] ok, even easier :D [10:28:02] it's fair to assume we have eqiad-only serviced but we *do not* have codfw-only services? [10:28:09] (at least from the looks of it so far) [10:30:24] *services [10:31:32] I went with this assumption for now, please comment on T263429 if that's not the case [10:32:47] <_joe_> uhmm ok [10:35:35] we can easily convert to something else that allow to specify in which dc it should exist [10:35:48] but I'd like to keep it very simple for the standard use cae [10:35:50] *case [10:36:04] <_joe_> yes agreed [10:56:55] jayme: wikitech looks good for now, I'll update once we have the script so that we can route standard VIPs to the script and leave that manual part for other special addresses that will most likely remain manual and don't have those specific rules [10:57:29] volans: ack - effie ^^ [10:59:06] and thanks for pinging, we all new things it's all a matter to find the best way to do it :) [10:59:27] volans: we were paving the way for greatness [11:58:51] akosiaris: we have an minor issuye [11:59:11] so we made a typo in https://gerrit.wikimedia.org/r/c/operations/puppet/+/623631/11/hieradata/role/common/kubernetes/worker.yaml#56 [11:59:28] so instead of push-notifiactions we got push_notifications [11:59:46] so the vip is not attached to the kubernetes* hosts [12:00:24] if we casualy just fix this typo, will everything be magically fixed? [12:04:39] effie: yes [12:04:42] lol [12:04:55] - vs _ :) [12:07:13] I am sure this is the first time someone mixes - and _ [12:12:16] nope, it definitely is not [12:12:28] but it's a pretty difficult error to catch [12:12:42] you can add push_notifications in the "typos" file of the puppet repo btw [12:12:51] it will consider it as a mistake after that. [12:13:09] but it won't help you with other instances of - vs _ [12:20:01] or you could validate that the keys of profile::lvs::realserver::pools are valid hostnames [13:05:15] 10serviceops, 10Operations, 10Product-Infrastructure-Team-Backlog, 10Push-Notification-Service, and 2 others: Deploy push-notifications service to Kubernetes - https://phabricator.wikimedia.org/T256973 (10jijiki) @JMeybohm and I finished up, you can reach the production environments by using `curl -k https... [13:05:44] volans, akosiaris thank you for the help [13:06:21] yw. Thanks for doing this service deployment! [13:06:23] 10serviceops, 10Operations, 10Product-Infrastructure-Team-Backlog, 10Push-Notification-Service, and 2 others: Deploy push-notifications service to Kubernetes - https://phabricator.wikimedia.org/T256973 (10Mholloway) Thank you, @jijiki! [13:09:56] 10serviceops, 10Operations: validate that profile::lvs::realserver::pools has valid hostnames - https://phabricator.wikimedia.org/T263454 (10jijiki) [13:10:05] 10serviceops, 10Operations: validate that profile::lvs::realserver::pools has valid hostnames - https://phabricator.wikimedia.org/T263454 (10jijiki) p:05Triage→03Medium [14:33:53] effie jayme: one last step missing for the above [14:34:26] run the sre.dns.netbox cookbook (it's a noop right now because the related zonefile has not yet been migrated, and this particular one will be the last one, if ever) but to not leave pending changes in netbox [14:34:37] that will confuse the next person running it [14:34:42] you should run it [14:40:47] volans: that should be done right after adding IPs to netbox? [14:43:58] wkandek: I might be slightly late [15:01:37] jayme: sorry was in a meeting, yes, at the same time you do the manual dns patch today [15:01:51] once migrated you will not do the dns patch and will need that to propagate the change [15:09:25] rats am I completely missing the subteam meeting? ugh [15:10:58] 10serviceops, 10Operations, 10Product-Infrastructure-Team-Backlog, 10Push-Notification-Service, and 2 others: Deploy push-notifications service to Kubernetes - https://phabricator.wikimedia.org/T256973 (10Mholloway) Looking at other service definitions in mediawiki-config [[ https://github.com/wikimedia/op... [15:29:49] 10serviceops, 10Operations, 10Product-Infrastructure-Team-Backlog, 10Push-Notification-Service, and 2 others: Deploy push-notifications service to Kubernetes - https://phabricator.wikimedia.org/T256973 (10Joe) >>! In T256973#6480180, @Mholloway wrote: > Looking at other service definitions in mediawiki-con... [15:50:33] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Update to kernel 4.19 on kubernetes nodes - https://phabricator.wikimedia.org/T262527 (10akosiaris) >>! In T262527#6474874, @JMeybohm wrote: > Interestingly the majority of throttling comes from statsd_exporters (https://grafana.wikimedia.org/d/tn6gBadMz/jayme-c... [15:56:04] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Update to kernel 4.19 on kubernetes nodes - https://phabricator.wikimedia.org/T262527 (10JMeybohm) >>! In T262527#6480462, @akosiaris wrote: > This does look pretty nice indeed. That drop there is actually quite telling. I am inclined to say we may have solved 1... [15:56:12] btw, the Phab search issues i mentioned turned into "WTFPHP": https://phabricator.wikimedia.org/T263063#6473840 [15:58:24] effie: we still need to merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/623790 (https://phabricator.wikimedia.org/T256973#6480180) [15:59:46] mutante: lol [16:00:21] oh ! jayme ! [16:00:49] mutante: excellent material for /r/lolphp [16:01:23] jayme: should I merge now? do I need to do anything else? [16:01:41] volans: added the "run cookbook" part to https://wikitech.wikimedia.org/wiki/LVS#DNS_changes [16:02:08] jayme: great, but I didn't see it running :) [16:02:21] effie: I *think* there is nothing to be done. _joe_? [16:02:32] volans: I did not say I ran it :P [16:02:47] <_joe_> ? [16:02:58] you should, before dcops gets to run it for new changes :) [16:03:12] I'm working on an alert too [16:03:23] _joe_: is ther anything to to rather than https://gerrit.wikimedia.org/r/c/operations/puppet/+/623790 for adding something to the service-proxy [16:03:28] <_joe_> unrelated: can someone add the notes from our meetings to the SRE docs [16:08:55] volans: effie: running sre.dns.netbox [16:09:05] noticed, see -ops :D [16:09:07] thanks a lot [16:09:18] ;-) [16:23:10] For monday 28th: https://phabricator.wikimedia.org/T196487 - there are 4 mc10xx in D4, mcrouter will complain for sure about not being able to replicate [16:23:23] effie: --^ [16:29:20] elukey: won't the gutter pool take care of it ? [16:30:52] effie: right, it should be handled by the gutter but we'll basically have one hour of things not replicated when the failback happens, and plus tko metrics will be high. Just raised as "let's keep an eye on it" [16:31:20] :) [16:32:53] on a separate note - not sure if we have time from dcops but maybe trying to spread some mc10xx nodes to more racks could be interesting [16:37:26] we are one person down in eqiad :/ [17:02:15] _joe_: can I merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/623790 [17:02:17] ? [17:02:23] does it need anything else from me? [17:03:05] because push-notifications will go to prod soon [17:17:26] <_joe_> effie: I was about to say LGTM [17:17:58] <_joe_> but also, that will not add the service to any proxy for now [17:18:11] <_joe_> which services will call push-notifications? [17:19:16] we decided it's safe and will add push-not to the listeners for appservers [17:19:32] if I got it right, only mediawiki will call push-not [17:19:56] <_joe_> that's also my understanding [17:20:00] <_joe_> only jobrunners, even [17:20:30] but we don't seperate envoy listener config for those, right? [17:26:33] <_joe_> no [17:26:39] <_joe_> and we shall not [17:26:46] <_joe_> mediawiki has the same config everywhere [17:31:31] ack [17:34:58] 10serviceops, 10Operations, 10ops-eqiad: mw1360's NIC is faulty - https://phabricator.wikimedia.org/T262151 (10RobH) a:05Jclark-ctr→03None [17:47:11] 10serviceops, 10Operations, 10Product-Infrastructure-Team-Backlog, 10Push-Notification-Service, and 2 others: Deploy push-notifications service to Kubernetes - https://phabricator.wikimedia.org/T256973 (10JMeybohm) >>! In T256973#6480180, @Mholloway wrote: > Looking at other service definitions in mediawik... [17:51:38] 10serviceops, 10DC-Ops, 10Operations, 10ops-codfw: mw2256 - CPU/board hardware issue - https://phabricator.wikimedia.org/T263065 (10Papaul) Drained the power, replaced both PSU's same issue. It has to be the CPU or main-board , pushing the power button on the sever doesn't power on the server. @wiki_will... [17:52:30] 10serviceops, 10Operations, 10Product-Infrastructure-Team-Backlog, 10Push-Notification-Service, and 2 others: Deploy push-notifications service to Kubernetes - https://phabricator.wikimedia.org/T256973 (10Mholloway) Excellent! Thank you! [18:09:28] 10serviceops, 10Push-Notification-Service, 10Product-Infrastructure-Team-Backlog (Kanban), 10User-jijiki: Set up secrets for Token clean-up - https://phabricator.wikimedia.org/T262957 (10MSantos) @jijiki just FYI, I made the credentials available for you, ping me when you are ready for this task. [18:15:38] 10serviceops, 10DC-Ops, 10Operations, 10ops-codfw: mw2256 - CPU/board hardware issue - https://phabricator.wikimedia.org/T263065 (10wiki_willy) Thanks for the initial diagnosis @Papaul Hi @Dzahn - per my earlier comment, are you guys ok moving forward without this server for a couple quarters, until the s... [18:42:58] 10serviceops, 10DC-Ops, 10Operations, 10ops-codfw: mw2256 - CPU/board hardware issue - https://phabricator.wikimedia.org/T263065 (10Dzahn) @wiki_willy Yes, one server more or less does not hurt us. We can just keep it offline and wait for the refresh for now (unless more would start breaking). [18:44:58] 10serviceops, 10DC-Ops, 10Operations, 10ops-codfw: mw2256 - CPU/board hardware issue - https://phabricator.wikimedia.org/T263065 (10Dzahn) We might as well just turn this into a decom task for it. I would be ok to upload and merge patches to remove it from puppet completely and then give it back to you an... [18:45:38] 10serviceops, 10DC-Ops, 10Operations, 10ops-codfw: mw2256 - CPU/board hardware issue - https://phabricator.wikimedia.org/T263065 (10wiki_willy) Thanks @Dzahn , works for me. [18:49:56] mw2256 died - but as i see now it's not just a random appserver.. also mcrouter proxy .. ugh [18:50:19] will try and find other one in B3 now [19:00:03] 10serviceops, 10WMF-JobQueue, 10Patch-For-Review, 10Platform Team Workboards (Clinic Duty Team), and 2 others: Could not enqueue jobs: "Unable to deliver all events: 503: Service Unavailable" - https://phabricator.wikimedia.org/T249745 (10Pchelolo) I've increased the timeout on MW side to match the timeout... [19:05:12] 10serviceops, 10Operations, 10ops-codfw: decommission mw2135-mw2147, mw2187-mw2214 - physical / datacenter part - https://phabricator.wikimedia.org/T261524 (10Papaul) [19:08:22] 10serviceops, 10DC-Ops, 10Operations, 10ops-codfw, 10Patch-For-Review: mw2256 - CPU/board hardware issue - https://phabricator.wikimedia.org/T263065 (10Dzahn) a:05Papaul→03Dzahn [19:08:46] 10serviceops, 10DC-Ops, 10Operations, 10ops-codfw, 10Patch-For-Review: decom mw2256 (was: mw2256 - CPU/board hardware issue - https://phabricator.wikimedia.org/T263065 (10Dzahn) [19:09:07] 10serviceops, 10DC-Ops, 10Operations, 10ops-codfw, 10Patch-For-Review: decom mw2256 (was: mw2256 - CPU/board hardware issue) - https://phabricator.wikimedia.org/T263065 (10Dzahn) [19:10:03] 10serviceops, 10DC-Ops, 10Operations, 10ops-codfw, 10Patch-For-Review: decom mw2256 (was: mw2256 - CPU/board hardware issue) - https://phabricator.wikimedia.org/T263065 (10Dzahn) Just one thing here was a bit critical, this happened to also be an mcrouter proxy. (some appservers are, most are not) and th... [19:47:56] 10serviceops, 10Performance-Team, 10Developer Productivity: Evaluate use of Gerrit dashboard for code review - https://phabricator.wikimedia.org/T263494 (10Krinkle) [20:52:30] mdholloway: all going well? [20:53:14] yes, so far so good! testing now. [20:53:33] nice [20:53:52] we decided to wait to promote to all wikis for now, just to make sure we've given the communities a sufficient heads-up about on-wiki changes [20:54:29] we have deployed on group0 for now? [20:56:11] just testwiki [20:57:12] oh, i was wondering, is there an easy way to view the service logs? [21:05:12] kube_env push-notifications codfw [21:05:21] kubctl get pods [21:05:23] choose a pod [21:05:48] and then kubectl logs --follow -c push-notifications-main [21:06:29] on deploy1001 [21:08:42] <_joe_> or logstash => filter over kubernetes_namespace [21:24:41] thanks! [21:29:01] 10serviceops, 10DC-Ops, 10Operations, 10ops-codfw, 10Patch-For-Review: decom mw2256 (was: mw2256 - CPU/board hardware issue) - https://phabricator.wikimedia.org/T263065 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: `mw2256.codfw.wmnet` - mw2256.codfw.wmnet... [21:55:45] 10serviceops, 10DC-Ops, 10Operations, 10ops-codfw, 10Patch-For-Review: decom mw2256 (was: mw2256 - CPU/board hardware issue) - https://phabricator.wikimedia.org/T263065 (10Dzahn) @wiki_willy @Papaul Removed from everything and ran the decom cookbook on it. It's now in state decom in netbox. https://net... [21:56:05] 10serviceops, 10DC-Ops, 10Operations, 10ops-codfw, 10Patch-For-Review: decom mw2256 (was: mw2256 - CPU/board hardware issue) - https://phabricator.wikimedia.org/T263065 (10Dzahn) a:05Dzahn→03None [22:11:59] 10serviceops, 10Operations, 10ops-codfw: decommission mw2135-mw2147, mw2187-mw2214 - physical / datacenter part - https://phabricator.wikimedia.org/T261524 (10Papaul) [22:18:52] releases.wikimedia.org is now in service.yaml and an active-active service in both DCs [22:19:14] first one I added to service.yaml and geodns, glad it seems to have all been smooth [23:24:06] 10serviceops, 10Patch-For-Review: convert misc services with multiple backends to active-active where possible - https://phabricator.wikimedia.org/T263506 (10Dzahn) [23:24:08] 10serviceops, 10Patch-For-Review: convert misc services with multiple backends to active-active where possible - https://phabricator.wikimedia.org/T263506 (10Dzahn) This was done for https://releases.wikimedia.org in: https://gerrit.wikimedia.org/r/c/operations/puppet/+/623464 https://gerrit.wikimedia.org/r/c... [23:26:22] 10serviceops, 10Patch-For-Review: convert misc services with multiple backends to active-active where possible - https://phabricator.wikimedia.org/T263506 (10Dzahn) [23:26:37] 10serviceops, 10Patch-For-Review: convert misc services with multiple backends to active-active where possible - https://phabricator.wikimedia.org/T263506 (10Dzahn) p:05Triage→03Medium [23:52:21] 10serviceops, 10DC-Ops, 10Operations, 10ops-codfw: decom mw2256 (was: mw2256 - CPU/board hardware issue) - https://phabricator.wikimedia.org/T263065 (10Papaul) @Dzahn Thanks