[04:24:36] 10serviceops, 10Operations, 10Phabricator, 10Traffic, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Phabricator downtime due to aphlict and websockets (aphlict current disabled) - https://phabricator.wikimedia.org/T238593 (10mmodell) @Dzahn perhaps we should try just re-enabling aphlict as i... [05:11:40] 10serviceops, 10Operations, 10Phabricator, 10Traffic, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Phabricator downtime due to aphlict and websockets (aphlict current disabled) - https://phabricator.wikimedia.org/T238593 (10mmodell) So it's worth pointing out that there are two types of con... [09:30:51] running `helm repo index .`: seems really noisy. Seems that's what everyone's doing but it makes the diffs fairly hard to follow. Is there some kind of --merge strategy I should be using to just target newly packaged charts? [09:47:18] 10serviceops, 10Operations: Migrate debug proxies to Stretch/Buster - https://phabricator.wikimedia.org/T224567 (10MoritzMuehlenhoff) a:03MoritzMuehlenhoff @ema, @Vgutierrez, @BBlack With ATS handling the X-Wikimedia-Header and the ATS backend work completed these should be good to go. I've checked the NGIN... [09:52:22] 10serviceops, 10Operations: Migrate debug proxies to Stretch/Buster - https://phabricator.wikimedia.org/T224567 (10ema) >>! In T224567#5792079, @MoritzMuehlenhoff wrote: > Can you confirm there's no other further pending work/tests which would make debug proxies needed again? Then I'd drop them from our enviro... [11:03:08] tarrow: nope. Just keep doing it for now. The goal is not even have to do it manually but rather have CI do it and not even have it in git [11:05:00] akosiaris: Cool. More questions about helmfile: Am I right in thinking that because helmfile.yaml doesn't have a chart version in next time we "apply" or even "sync" the latest chart will be used? [11:06:56] if so how do we handle "breaking changes" in our chart? Do we have to make sure that an helmfile values.yaml is shipped at the same time as the "releasing" of the new index.yaml? [11:08:04] is it not possible to have chart version n+1 on staging and and chart n on production? [11:10:44] tarrow: you can add a version: parameter [11:10:56] and target whatever chart version you want [11:11:17] https://github.com/roboll/helmfile look at the example [11:11:35] chart: roboll/vault-secret-manager # the chart being installed to create this release, referenced by `repository/chart` syntax [11:11:35] version: ~1.24.1 # the semver of the chart. range constraint is supported [11:12:32] effie: _joe_ https://grafana.wikimedia.org/d/JTnxOdEZk/xxx-mathoid-canary?orgId=1 [11:12:33] cool; basically I worried that right now merging https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/563410 would break production as soon as someone touches it? Is that right? [11:12:37] canary support! [11:13:01] <_joe_> akosiaris: oh great, you did it just for mathoid? [11:13:09] _joe_: yeah, it's my PoC [11:13:10] <_joe_> I guess you changed something in the chart? [11:13:11] and just codfw [11:13:30] _joe_: https://gerrit.wikimedia.org/r/#/c/operations/deployment-charts/+/469662/ [11:13:38] and the following helmfile.d change [11:14:13] <_joe_> akosiaris: let's adopt the same technique I used for tls when adding new things to charts [11:15:00] tarrow: yeah if someone deploys after that is merged, it will indeed break things. But you can add a version: 0.0.3 in the helmfile stanzas to make sure you don't inadvertedly upgrade. As long as you have a proper process in place to remember to bump that setting in helmfile.yaml as well when you release a new version of that chart, you should be ok [11:15:34] _joe_: yeah that makes sense. I 'll let it be for a couple of days and then amend all charts to support it [11:15:45] but it's nice that in the end it work so easily [11:15:51] cool; I'll make sure we pin now before we even think about updating. I'll add a comment about that to the README patch as well [11:23:56] akosiaris: nice nice [11:35:24] rlazarus: _joe_: I left a few comments around the switchover OKRs [11:36:23] so I had two KRs, one demonstrating the ability of codfw to serve the site, and another that has a target for the switchover speed [11:36:31] you two focus on the ability to switchover [11:36:40] let's unify that, as your objective becomes part of mine [11:36:47] (I wonder how that works when you both have it, actually...) [12:02:23] <_joe_> mark: sure, but I was thinking I'd prefer the metric to be on the number of code commits rather than speed. A lot of things can go wrong and make the procedure slower [12:02:36] <_joe_> like a large replication lag [12:04:58] yes, but it is the important metric to users of course [12:05:08] if we have replication lag then that is a problem we should try to fix for next time [12:05:57] perhaps we shouldn't be meeting our KR/objective if we've fixed one problem but 5 new ones have appeared that delay things [12:06:10] because then things didn't really get better, even though we did what we set out to do [12:40:05] 10serviceops, 10Operations, 10Phabricator, 10Traffic, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Phabricator downtime due to aphlict and websockets (aphlict current disabled) - https://phabricator.wikimedia.org/T238593 (10akosiaris) > I really can't think of any reason for it to be using... [13:19:24] _joe_: do you agree? [13:43:24] <_joe_> uhm, I cited replication lag because it's one of the typical issues that are transient and quite unpredictable [13:44:01] <_joe_> I mean, we could decide to wait it out before starting the switchover, it usually lasts a few minutes [13:44:39] <_joe_> I'm saying that with such short time as a goal, perturbations might interfere [13:45:01] <_joe_> but anyways, I think if all is ok the switchover should take well under 2 minutes [13:45:24] <_joe_> I'm also a bit worried about making things too rushed for rlazarus driving the process for the first time [13:45:46] <_joe_> but indeed, the read-only time is now basically managed by changing a few values in etcd [13:46:13] <_joe_> and the db reconfigurations [13:46:19] <_joe_> so it's surely pretty fast [13:53:20] yeah should ensure to not put the process under pressure causing stress and unreliability [13:54:03] i wonder if having another KR that focuses on reliability during switchover would help or rather make it worse :) [13:54:22] OKR book I have recommends "pairing" KRs in that situation [13:54:30] so not wrong sacrifices are made to optimize one KR [13:54:37] but yeah, it might just up the pressure :/ [14:19:31] akosiaris: o/ [14:19:35] want to teach me how to add namespaces? [14:19:40] or is that something I shouldn't do [14:19:41] ? [14:25:54] ottomata: it's a messy process we want to automate eventually. Not sure it's worth it to invest in it. It involves changes in 3 different repos and 3 deploys :-( [14:26:07] ok [14:26:14] if it will change in future and get easier I will wait :) [14:26:24] 3 different repos? [14:26:43] yes. puppet, puppet-private and deployment-charts [14:26:52] the first two are about the tokens [14:27:01] and the latter about the creation of the namespaces [14:27:19] ah [14:27:30] oh 4 repos. labs/private as well if you want to test with PCC [14:27:39] ee ya [14:27:48] the cumbersome parts are mostly the tokens (puppet, puppet-private) [14:27:49] i guess i can include the deployment-charts change in that patch [14:28:13] I wait a bit to see if apereo can help in some way with that [14:28:33] cause it's getting cumbersome to manually manage the tokens [14:30:14] k [14:33:55] oh cool akosiaris [14:33:56] https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/563409/1/helmfile.d/services/codfw/mathoid/canary.yaml [14:33:56] so [14:34:20] ohhh [14:34:20] no [14:34:26] ? [14:34:27] what is the value of canaryof? [14:34:37] in mathoid, production is the release name? [14:34:41] yes [14:34:45] for eventgate it'll be dilfferent for each one [14:34:56] since we use release name to do multi deployments of the same chart, right? [14:35:17] for a second i thought that file would be the same for all helmfile servies [14:35:19] yup. so you can say thins like canaryof: analytics or canaryof: logging-external [14:35:20] services* [14:35:23] ok greaet. [14:35:26] awesome [14:35:46] i'll make some patches but will wait a bit for it to simmer before merging etc. [14:35:50] thank you! [14:36:11] also the name of the file "canary.yaml" can be anything. It's referenced in helmfile.yaml [14:36:50] note btw that there is no checking that what you supplied as canaryof value is indeed deployed [14:37:10] so you might end up with a canary release that works perfectly fine but is not addressed by anything [14:37:43] ottomata: yeah, wait a bit. I discovered an issue already when deleting releases [14:40:53] ah k [14:41:32] akosiaris: is the only way to figure out which pod is the canary one to look at labels? [14:41:43] via kubectl get ... [14:41:47] ? [14:42:31] ottomata: it will also be in the pod name [14:42:51] but yeah, the labels is probably a safer way [14:42:56] ah great [14:43:01] it'll be easier to see if in pod name [14:43:08] will be nice to be able to get logs from it, etc [14:43:12] you can do kubectl get pods -l canary= [14:43:27] and kubectl logs -l canary= will work fine [14:43:38] value being the name of the release, probably "canary"? [14:43:42] so canary=canary [14:43:55] I am not overly fond of that label name btw, but enough naming bikeshedding already [14:49:28] hm [14:49:44] maybe the label name shouldn't be canary? [14:50:50] i guess its fine, can even to multiple canaries like that? (if added to the helmfile.yaml) [14:50:50] hm [14:51:01] do* [14:52:23] yeah, in fact you can have canaries of canaries of canaries [14:52:24] akosiaris: maybe it would be good to keep the labels the same on all the pods? [14:52:31] not the best idea, but still [14:52:43] maybe canary=false or canary=no on the main one? [14:52:54] no that wouldn't work [14:53:04] no? [14:53:12] the labels are there for the deployment, service and netpol objects to match the pods [14:53:29] so you want canarydeployment and canarynetpol to match just their pods [14:53:37] while the main service objects to match all pods [14:53:50] main service object* [14:54:03] right so [14:54:04] else [14:54:08] canary: false [14:54:13] e.g in https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/469662/9/charts/mathoid/templates/networkpolicy.yaml [14:54:14] so they can't be shared. You need the canary release to have something extra that is unique to it [14:54:25] right, it would have the named canary value [14:54:28] otherwise 2 canary releases would both have canary=false [14:54:34] ... [14:54:35] and you are in trouble then [14:54:57] which is why I am not loving that canary label name [14:55:08] but I can't think of anything better, if you do please let me know [14:55:17] but it's not the value that is problematic, it's the name of the label [14:55:21] right [14:55:43] not understanding why 2 canary releases woudl have canary=false, brain is churning stay tuned... [14:56:01] with the current chart? they wouldn't [14:56:13] but if we changed it to be canary=false/true it would happen [14:56:14] right [14:56:17] non [14:56:18] ah [14:56:22] not canary=false/true [14:56:22] ok [14:56:28] canary=no [14:56:31] canary=canaryA [14:56:34] canary=canaryB [14:56:40] only the main non canary would have canary=no [14:57:02] ah you mean adding a canary label to the main releases as well [14:57:03] this way every deployment has a canary label, but each one with a different valuue [14:57:05] yes [14:57:18] i thought of this after looking at your https://grafana.wikimedia.org/d/JTnxOdEZk/xxx-mathoid-canary?orgId=1&fullscreen&edit&panelId=2 [14:57:54] we could do that. I thought about it for a bit. It felt like the path of least change was to not add it [14:58:03] aye less modification to main release [14:58:04] hm [14:58:18] would be even nicer if we could think of a better name for the label [14:58:24] you know i love naming... [14:58:29] I am all ears [14:59:13] on the other hand ... [14:59:26] maybe I should just do the changes to the main release objects [14:59:31] hmm [14:59:34] * akosiaris thinking aloud [14:59:37] ? [15:00:52] niah scratch that. I am trying the idea of having a release be able to address pods of all releases [15:01:01] but now that I remember about it, it's a bad idea [15:01:12] it implicitly becomes a parent of all releases as far as services go [15:01:32] hm ah that does sound complex [15:01:35] names: [15:01:38] phase [15:01:40] rollout [15:02:20] rollout_group [15:02:33] stage [15:02:37] stage=canary [15:02:39] stage=main [15:02:54] and instead of canaryof... [15:03:21] you are just trying to indicate which is the parent stage? [15:03:24] ? [15:03:26] sort of? [15:03:27] hm [15:03:56] more or less yes [15:04:07] trying to have a relationship between 2 helm releases [15:04:22] without however actively making the "parent" overly broad [15:04:28] target [15:04:46] stage=canary [15:04:46] target_xxxx=main [15:04:51] target_stage? [15:04:52] hm [15:04:59] it isn't the stage name you are targeting... [15:05:14] it's a bit weird indeed [15:05:31] so the idea is that release "canary" targets release "production" [15:05:41] feel free to switch to A and B for all intents and purposes [15:05:47] however the actual logic is the inverse [15:06:12] it's that "canary" sets the exact same release: label in its pods [15:06:21] as the "production" release [15:06:43] essentially intrustring the "production" service to also address "canary"'s pods [15:06:54] oh yes i see that is weird. from the deployment perspective the releasex is 'canary', but from k8s it is still production [15:07:05] right? [15:07:26] then it needs to somehow differentiate itself uniquely from both the "production" release and other "canaries" to avoid messing with each others resources [15:07:36] hence that "canary" label there being the release name [15:07:50] is 'release' the label that ties all the pods together then? [15:07:55] yes [15:07:59] it isn't matching on all the labels, it is just release? [15:07:59] ok [15:08:00] that and app label [15:08:05] aye ok [15:08:06] but the app label is already shared [15:08:38] release_stage, release_target [15:08:39] in main [15:09:12] hmm [15:09:13] no [15:09:14] messing with the main release to point to optionally and potentially existing canaries (I don't expect everyone will rush to adopt this) [15:09:22] aye [15:09:24] is not a great idea ... [15:09:41] to be honest the best name for that label would be "release" [15:09:45] but that's already used :P [15:10:05] and the entire idea is to differentiate it from the actual release label [15:10:06] sigh [15:10:32] _joe_, mark: re switchover timing, my usual rule of thumb is to think about how I'd feel if I missed the OKR -- in this case, if we completed the switchover as fast as we safely could but the process took longer than 3 minutes, that wouldn't feel like I had failed, so I left it out on purpose [15:10:47] similar to _joe_'s point about pressure but viewed from a different angle, I suppose [15:11:04] ok so your current canary label [15:11:07] doesn't really do anything active [15:11:14] it is just for IDing the pods in metrics [15:11:19] or debugging or whatever [15:11:20] more or less [15:11:25] not just metrics though [15:11:30] also internally in kubernetes objects [15:11:41] but yeah you got the gist of it [15:12:36] for example if I omitted it, network policies from the main and other canaries would also match that releases pods [15:12:37] ya [15:12:42] which is implicit and confusing [15:13:02] you 'd end up wondering what the heck when debugging ACLs for quite some time [15:13:24] ok still working on names [15:13:31] but maybe closer: [15:13:34] # production stage main [15:13:34] release: production [15:13:34] stage: main [15:13:34] target_stage: null # or main? [15:13:34] # production stage canary [15:13:34] release: production [15:13:35] stage: canaryA [15:13:35] target_stage: main [15:14:15] (or let's use stage: canary in this example) [15:14:22] then in canary.yaml [15:14:38] service: [15:14:39] stage: canary [15:14:39] target_stage: main [15:14:57] it's s/canaryof/target_stage/ in this example ? [15:15:00] yes [15:15:36] hmm may stage: and addressedby: then fits this better [15:15:39] hmm [15:15:41] hmm [15:15:54] hmm [15:15:56] member? [15:16:14] memberOf: [15:16:17] niah, it's literally just being addressed by a different releases service [15:16:27] member has way more meaning [15:16:30] haha thiis is really hard to reason about! :p [15:16:30] ok [15:17:04] https://etherpad.wikimedia.org/p/k8s-canary [15:17:09] perhaps we shouldn't chase semantics for now? may identify issues with the current approach? [15:17:22] for you to notice it, it already bugged you somehow, right? [15:17:28] I know it bugged me in grafana already [15:17:41] for example the legend couldn't be {{ method }} - {{canary} [15:17:50] yeah [15:17:56] because then it very easily ended up being the exact same text [15:18:02] i don't love the canary as a label, when the concept isn't realy 'canary' specific [15:18:12] cause grafana when it can't evaluate {{ lala }} it just prints out "lala" [15:18:18] ah yeah [15:19:23] akosiaris: where does .Release.Name come from? [15:19:26] tillerNamespace? [15:19:34] oh [15:19:39] releases name [15:19:42] in hemlfile.yalj [15:19:43] duh [15:19:46] :-) [15:19:48] ok so in eventgate it is main [15:19:55] or analytics [15:19:56] et.c [15:20:03] in eventgate it will actually work better I think [15:20:19] ok going to go with that for my brainstorming might be easier [15:20:24] release: production is confusing :) [15:20:40] sure, but I am open to ideas [15:20:49] yeah am brainstorming [15:20:50] :) [15:20:52] the helm standard approach is to just autogenerate one [15:22:41] yeah i find that hard, i'm glad we aren't doing that [15:24:21] akosiaris: is service.deployment just use for conditoinals in the template between dev envs and 'production' deployment [15:24:21] ? [15:24:51] ottomata: that's a flawed approach we are killing [15:25:00] it's barely used in NOTES.txt IIRC [15:25:04] ok, cool i think that is a source of confustion too [15:29:19] heh, release: analytics is bit confusing in eventgate too [15:29:23] eventgate-analtyics release analytics [15:29:23] ? [15:29:43] may I should lift the constraint of not messing with the "main" service. Allow people to pass a value and allow it to be less constrained [15:29:56] like "address_all_pods" [15:29:57] :P [15:30:02] and default it to false [15:30:31] not sure I understand that one [15:32:26] well up to now I 've been trying to not mess much with the "main" service [15:33:03] but maybe it's fine to add a conditional e.g. "address_all_pods" that removes that "release" label from the k8s service that main release instantiates [15:33:20] that would allow it to address all pods that have app: mathoid label [15:33:25] in that namespace ofc [15:33:52] with that, the entire patch becomes moot then. [15:34:01] ohhh [15:34:02] it's inversion of control essentially [15:34:13] it's also much simpler? [15:34:14] does that get weird during a release? [15:34:19] deployment? [15:34:44] mildly. There are a few questions that then make sense [15:34:56] for a short time during deploy, would pods will be addressed from previous and deployment and next deployment be equally addressed? [15:34:57] if the main release is going to address all pods [15:35:11] should the "other" releases have Service resources? [15:35:36] no there is no such race [15:35:55] cause during deployments the pods first need to be health checked before accepting traffic [15:36:09] 10serviceops, 10Operations, 10ops-eqiad: (No Need By Date Provided) rack/setup/install kubernetes10[07-14].eqiad.wmnet - https://phabricator.wikimedia.org/T241850 (10jijiki) [15:36:10] akosiaris: before going to far that way (might be the way to go) [15:36:12] https://etherpad.wikimedia.org/p/k8s-canary [15:36:24] main line 40-44 [15:45:36] I 've left some comments, I need to be on the move now [15:45:41] I 'll think more about it [15:45:59] I am happy we proved it worked to start with, so that's something. [15:46:56] :) [15:47:23] for anyone interested in having another look regarding racking our new servers [15:47:31] https://phabricator.wikimedia.org/T231255#5789930 [15:47:37] https://phabricator.wikimedia.org/T231255#5792300 [15:47:39] ya there is comment on line 25 about omitting those for main case [15:47:40] but ya [15:47:43] https://phabricator.wikimedia.org/T231255#5792880 [15:48:05] https://phabricator.wikimedia.org/T241850 [15:48:11] https://phabricator.wikimedia.org/T241849 [16:05:52] rlazarus: point taken :) [16:06:05] although I'm trying to incorporate something showing that we've also made some improvements to the process compared to last time [16:06:20] yeah, that makes sense too [16:06:45] but i agree that we should still be fairly happy if the switch went well and without downtime, just slightly slower [16:06:48] (if not toooo slow) [16:06:59] i'll ponder this a bit more [16:07:12] in practice I think it's almost impossible for improvements _not_ to happen, looking at the list of stuff we expect to delete from the scripts because we don't need it anymore [16:07:16] and maybe that should just be MY kr [16:07:23] considering that part is not actually your work [16:07:29] so we could pretty safely add "with fewer steps" but -- right, exactly [16:07:38] I'm happy to take credit for it but I haven't done anything :D [16:07:44] heh [16:07:54] it's just that that works awkwardly in betterworks unfortunately [16:07:57] but i'll try [16:08:06] i also haven't done any of the work [16:08:07] to be fair ;) [16:08:14] other than being the manager [16:08:31] ...manager of the manager in this case [16:10:56] what about something like "switchover process reflects infrastructure improvements made since the 2018 exercise"? [16:11:23] since bringing the procedure up-to-date _is_ something I'll be doing, and the effect of all that is to make it shorter [16:11:48] (highlighting _joe_ again on this too) [16:11:50] it's a problem that that is needed, tbh ;) [16:11:58] it should be done as those changes are being made [16:12:01] but yeah that's tricky [16:12:14] haha if we're deleting all the OKRs that reflect problems... [16:12:27] sure ;) [16:13:24] <_joe_> the procedure needs updating on the traffic part for sure [16:18:30] how about a combination of "fewer 'manual' steps" and "not longer than last time"? [16:18:42] tbh, less than 5 mins is already pretty acceptable :) [16:18:53] there's not a strong need to make it much faster [16:19:04] but yeah just a single outlier can fail that [16:19:07] as we're only doing it... twice [16:20:41] <_joe_> my real goal is that the procedure becomes simple enough this time that we can really do it every 3/6 months [16:21:09] == _joe_ [16:21:37] I'm not sure I know how to express that in an OKR yet, though -- I think I'll have a lot more useful things to say after I've done it, unfortunately [16:21:48] some of this could become a Q4 OKR, maybe [16:22:06] I'm sure there'll be followup work after the switchover anyway, like "oh next time we should do X instead" [16:24:03] yeah [16:24:09] so read-only time is a factor in that but no longer a large factor [16:24:17] e.g. DBAs do read-only migrations too [16:33:14] <_joe_> I think we can consider 5 minutes of read-only per quarter an acceptable penalty for being sure we can do a switchover in an emergency [16:33:21] nod [16:52:36] agreed [16:52:50] and if we do better, we can still report on that and be happy [17:00:22] so is there anything we want to add to the Q3 draft? or should we come back and look at that for Q4? [17:01:04] let's revisit on monday [17:01:10] i think i'll add "with fewer steps" or something [17:01:13] and "within 5 mins" [17:01:25] i.e. not really slower than previous times [17:01:41] but i'm signing off now, family is waiting [17:02:03] "fewer steps" sounds good, I'm still nervous about "within 5 minutes" -- I think we'll hit it fine, but I don't want to be watching the clock when we're working on it [17:02:09] sounds good :) have a nice weekend [17:02:26] either way you shouldn't worry about that [17:02:34] but yeah, how to make you feel like that :) [17:02:40] enjoy the weekend [18:34:26] 10serviceops, 10Core Platform Team, 10Performance-Team, 10Scap, and 5 others: Define variant Wikimedia production config in compiled, static files - https://phabricator.wikimedia.org/T223602 (10Jdforrester-WMF) 05Open→03Stalled Stalled for the next month or two. [20:02:11] 10serviceops, 10Operations: Migrate debug proxies to Stretch/Buster - https://phabricator.wikimedia.org/T224567 (10Dzahn) [20:02:26] 10serviceops, 10Operations: decom debug proxies (was: Migrate debug proxies to Stretch/Buster) - https://phabricator.wikimedia.org/T224567 (10Dzahn) [20:03:22] 10serviceops, 10Operations: decom debug proxies (was: Migrate debug proxies to Stretch/Buster) - https://phabricator.wikimedia.org/T224567 (10Dzahn) created decom subtasks via [[ https://wikitech.wikimedia.org/wiki/Server_Lifecycle#Reclaim_to_Spares_OR_Decommission | Lifecycle decom form ]] [20:39:01] 10serviceops, 10Core Platform Team Workboards (Clinic Duty Team): restrouter.svc.{eqiad,codfw}.wmnet in a failed state - https://phabricator.wikimedia.org/T242461 (10Eevans) p:05Triage→03Normal It's not clear to me what the status of this is. Do we need to deploy the latest code here? Since (long-term) w... [20:42:38] 10serviceops, 10Core Platform Team Workboards (Clinic Duty Team): restrouter.svc.{eqiad,codfw}.wmnet in a failed state - https://phabricator.wikimedia.org/T242461 (10Pchelolo) > Since (long-term) we aim to replace all of this, is abandoning it entirely an option? Is it possible to take it out for now until we... [23:43:23] 10serviceops, 10Operations: package requirements for upgrading deployment_servers to buster - https://phabricator.wikimedia.org/T242480 (10Dzahn) [23:58:59] 10serviceops, 10Operations: package requirements for upgrading deployment_servers to buster - https://phabricator.wikimedia.org/T242480 (10Jdforrester-WMF)