[00:07:30] 10serviceops, 10Operations, 10Release-Engineering-Team-TODO, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services): Upgrade MediaWiki appservers to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Dzahn) [00:17:08] 10serviceops, 10Operations, 10Release-Engineering-Team-TODO, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services): Upgrade MediaWiki appservers to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Dzahn) >>! In T245757#6642525, @jijiki wrote: > If that is the case,... [01:10:13] 10serviceops, 10Graphoid, 10Operations, 10MW-1.35-notes (1.35.0-wmf.34; 2020-05-26), and 2 others: Undeploy graphoid - https://phabricator.wikimedia.org/T242855 (10Jdforrester-WMF) [08:31:57] 10serviceops, 10Operations, 10Release-Engineering-Team-TODO, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services): Upgrade MediaWiki appservers to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10MoritzMuehlenhoff) With all Puppet patches and debs landed, mwdebug10... [08:41:51] 10serviceops, 10Operations, 10Release-Engineering-Team-TODO, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services): Upgrade MediaWiki appservers to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10MoritzMuehlenhoff) >>! In T245757#6644078, @MoritzMuehlenhoff wrote:... [08:45:07] <_joe_> moritzm: merging that patch [08:46:26] ack, sounds good [10:03:50] 10serviceops, 10Operations, 10docker-pkg: Docker image on the build host seem to ignore apt priority for wikimedia packages - https://phabricator.wikimedia.org/T268612 (10Joe) [10:26:17] 10serviceops, 10Operations, 10docker-pkg: Docker image on the build host seem to ignore apt priority for wikimedia packages - https://phabricator.wikimedia.org/T268612 (10Joe) ok, found the problem, and it's slightly embarassing: `docker-registry.wikimedia.org/wikimedia-buster:latest` was 12 months old on d... [10:37:39] hello serviceops! I've a topic to discuss with you, it might be straightforward or controversial. If the latter I'll open a task to discuss more in depth there. [10:38:01] basically if/when we should migrate the SVC records for eqiad/codfw to the Netbox auto-generated files [10:54:11] <_joe_> what is the advantage of moving them to netbox? [10:54:46] they are already there, the files are already generated, it's our IPAM and so they should be tracked there anyway [10:55:02] <_joe_> ok so the advantage is "not adding them twice" [10:55:06] <_joe_> correct? [10:55:23] <_joe_> do we also manage SRV records via netbox? I didn't check [10:56:04] yes for the twice part, tecnically speaking even if we forget to add them they might be auto-imported from puppetdb when the first host that has it assigned will be reimaged, but that's a by-product, and not ideal [10:56:13] do we don't, only A/AAAA/PTR [10:56:21] netbox is not designed to fully manage DNS records [10:56:38] <_joe_> ok perfect [10:57:41] <_joe_> not sure what you mean with "even if we forget" [10:58:29] <_joe_> why would an host with that IP assigned cause it to show up on netbox, and only if reimaged? [10:59:10] because we import data from the puppetdb networking facts upone reimage to get the host interfaces [10:59:29] <_joe_> only on reimage? [10:59:41] <_joe_> so if we add a new ip to the LVSes you wouldn't catch it [11:00:02] <_joe_> also, do we only gather data from the public interfaces or also from loopback? [11:00:55] loopback and other dynamical interfaces are discarded on purpose because netbox should track only the physical ones, as the other will go out of sync very easyly with just a pupept change [11:01:10] <_joe_> ok so [11:01:29] <_joe_> I doubt we'd import these IPs on reimage of a lvs server [11:01:41] <_joe_> or any other server [11:02:00] <_joe_> it would be nice if we had a way to auto-import them from puppetdb into netbox, and from there to dns [11:02:06] <_joe_> but that's for another moment, really [11:02:21] <_joe_> akosiaris: do you see any reason why we could not manage the svc records via netbox? [11:02:35] <_joe_> maybe staging having multiple "A" records can't be managed? [11:03:42] teh current data is in https://netbox.wikimedia.org/ipam/prefixes/92/ip-addresses/ and https://netbox.wikimedia.org/ipam/prefixes/93/ip-addresses/ if you want to check or just clone the auto-generated repo and look at the zonefiles: [11:03:51] 1.2.10.in-addr.arpa, 2.2.10.in-addr.arpa, svc.codfw.wmnet, svc.eqiad.wmnet [11:04:37] <_joe_> volans: yeah I don't see "staging" there [11:04:49] some .svc. are CNAMEs that's 1 reason [11:05:05] CNAMEs will be kept manual after the include [11:05:06] <_joe_> staging 1H IN A 10.64.0.247 ; kubestage1001 [11:05:28] the multiple A RRs are also a reason [11:05:30] <_joe_> and staging 1H IN A 10.64.16.92 ; kubestage1002 [11:05:48] <_joe_> volans: would we keep those as manual after the include? [11:06:41] if we do keep them manual we end up with 2 sources of truth for .svc..wmnet [11:08:53] (sorry to burge into the convo, I would like a +1 for https://gerrit.wikimedia.org/r/c/operations/puppet/+/643229/ ) [11:08:57] (thank you) [11:08:57] let me check one thing, I think we did some effort to prevent duplication but we might be able to manage that given they are 2 different IPs [11:09:14] I'll get back to you in few minutes [11:09:47] the other advantage is that we can add a script to netbox to assign a new SVC IP that will also take care of reserving/assigning it for both DCs with the same last octet [11:09:57] I think this was discussed in a task already [11:16:25] akosiaris: 10.64.0.247 is not an SVC IP, but the primary IP of kubestage1001 [11:16:56] no, multiple DNS records for the same IP are not manageable via netbox. [11:17:12] but also AFAIK we should try to not have them, and use proper service IPs for services [11:19:15] not in this case, we went for that approach to avoid 1 more LVS service [11:19:42] sorry, not 1 more. X more [11:19:47] I'm not talking LVS, just from the network PoV, not use a host IP for a service DNS [11:19:55] *with [11:21:03] other similar exceptions (although different) are gerrit, lists and those too should be "solved" [11:21:12] I am not sure it's worth the complication of adding 1 more IP to hosts for this [11:22:12] fwiw the $INCLUDEs from the netbox data are in place includes, so we can add any record after that. The includes never have any $ORIGIN, those are all in the manual repo [11:22:31] so total flexibility there, it's up to us though to prevent duplicates when mixing manual and automatic data [11:23:54] example: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/dns/+/refs/heads/master/templates/153.80.208.in-addr.arpa#27 [11:24:04] it's not about the duplicates I am worried about [11:24:10] it's about the no single source of truth [11:26:49] IMHO it's wrong to have a 10.64 with a .svc. dns record in the first place [11:28:20] <_joe_> volans: i disagree [11:28:20] we can argue about that, but there is also the CNAME .svc. thing. That would again mean 2 source of truth [11:28:32] it's not just 1 complication there [11:29:34] _joe_: my reasoning is that host IPs should be pure infra, but YMMV [11:29:52] any non-A/AAAA/PTR record is still managed manually, yes [11:29:57] <_joe_> so my original proposal was to instead re-generate the svc records from puppet, that has all the info [11:30:34] <_joe_> volans: having multiple A records as a load-balancing technique when you don't need LVS is not something out of the ordinary [11:31:00] sure, that's a fair use case, not arguing on that [11:31:35] <_joe_> which is what we're doing there [11:31:54] it's not just that we don't need LVS, it's that LVS would make it very problematic here [11:32:06] cause we would need an LVS service for each service [11:32:09] <_joe_> akosiaris: yeah, I was making a more general point [11:32:18] <_joe_> but ofc that's also true [11:34:12] so let's summarize, beside the use of what I would call "infra" IPs vs "service" IPs [11:34:59] - you need to support DNS load balancing with multiple IPs per DNS name and that's not currently manageable with Netbox unless we build something on top of it via a plugin [11:35:32] - the fact of having some manual record in the dns repo and the bulk of standard records in netbox is a concern [11:35:50] for not having a single place as a source of truth [11:36:26] - Netbox is though our IPAM and as such it should track all the allocated IPs [11:36:50] - the only place where we currently have all the data that could be a potential single source of truth is Puppet [11:37:06] is this a decent summary? [11:38:38] not sure this last part is true [11:38:40] e.g. [11:38:47] swift 1H IN CNAME ms-fe.svc.eqiad.wmnet. [11:38:52] that's definitely not in puppet [11:39:07] perhaps we are looking this from different PoVs [11:39:39] <_joe_> akosiaris: I'm saying it's possible to move those small pieces in puppet [11:39:41] I am mostly concerned about the .svc. -> destination PoV, you seem to look at it from the IP angle [11:39:46] <_joe_> and have a single source of truth [11:40:17] <_joe_> and it would mean we only have to define the IPs in one place [11:40:31] ehm, how would you move it to puppet? [11:40:37] I mean that CNAME above ^ ? [11:40:54] I am wondering a bit what puppet has to do with all of this tbh [11:41:01] <_joe_> akosiaris: you have one template with a for over the records in service::catalog [11:41:09] <_joe_> where you have fqdns and IPs [11:41:29] <_joe_> and then those "manual' records [11:41:53] could in theory that data come from netbox instead for example? [11:41:56] <_joe_> puppet just has 99.9% of the info already recorded, and we can't not have it [11:42:23] <_joe_> volans: you mean the SVC IPs data? possibly, yes [11:42:31] <_joe_> we'd have to understand how to do that [11:43:34] yes I meant that [11:46:12] the current problem that I see is that netbox needs some data to be an IPAM, puppet needs more or less the same data but decorated with additional metadata, and the dns needs the same data in netbox plus some additional data for the use cases not currently manageable via netbox for lack of functionalities [11:47:04] and would be nice to find a tetris where we avoid duplicating data or split it in multiple places [11:48:35] 10serviceops, 10Operations, 10docker-pkg: Docker image on the build host seem to ignore apt priority for wikimedia packages - https://phabricator.wikimedia.org/T268612 (10JMeybohm) Ouch. Could we normalize everything to use the public image reference? That would also make local test more easy or straight fo... [11:49:28] I think we can be flexible on the flow of the data as long as each part gets the bits that it needs :) [11:50:04] 10serviceops, 10Operations, 10docker-pkg: Docker image on the build host seem to ignore apt priority for wikimedia packages - https://phabricator.wikimedia.org/T268612 (10Joe) >>! In T268612#6644659, @JMeybohm wrote: > Ouch. > > Could we normalize everything to use the public image reference? That would als... [11:53:38] I can open a task and summarize the above and see if we can find a solutio [11:53:44] sounds good to you? [12:10:41] akosiaris: I've uploaded the "version-number-fixed" helm3 package to apt and also packaged the current helmfile version (as that needed patching again ofc. :D) https://gerrit.wikimedia.org/r/c/operations/debs/helmfile/+/643243 [12:16:28] 10serviceops, 10Operations, 10docker-pkg: Docker image on the build host seem to ignore apt priority for wikimedia packages - https://phabricator.wikimedia.org/T268612 (10JMeybohm) >>! In T268612#6644662, @Joe wrote: >>>! In T268612#6644659, @JMeybohm wrote: >> Ouch. >> >> Could we normalize everything to u... [12:41:13] 10serviceops, 10Operations, 10docker-pkg: Docker image on the build host seem to ignore apt priority for wikimedia packages - https://phabricator.wikimedia.org/T268612 (10akosiaris) >>! In T268612#6644689, @JMeybohm wrote: >>>! In T268612#6644662, @Joe wrote: >>>>! In T268612#6644659, @JMeybohm wrote: >>> Ou... [13:27:21] 10serviceops, 10Operations, 10docker-pkg: Docker image on the build host seem to ignore apt priority for wikimedia packages - https://phabricator.wikimedia.org/T268612 (10JMeybohm) >>! In T268612#6644782, @akosiaris wrote: > Couple of reasons: Okay, thanks. So it's more like we should not use the external... [14:32:51] akosiaris: is there anything else that needs to happen to get eventstreams to talk to schema.discovery.wmnet? [14:34:35] 10serviceops, 10Release-Engineering-Team-TODO, 10Scap: Deploy Scap version 3.16.0-1 - https://phabricator.wikimedia.org/T268634 (10LarsWirzenius) [14:35:12] ottomata: already done! [14:36:13] thanks! just haven't been able to get staging to do it yet [14:36:26] it is still timing out trying [14:39:02] * akosiaris doublechecking [14:39:31] it is able to get to https://api-ro.discovery.wmne fine [14:39:41] times out on for example [14:39:42] https://schema.discovery.wmnet/repositories/primary/jsonschema/mediawiki/revision/create/latest [14:43:14] ottomata: PEBKAC on my part. did helmfile apply instead of helmfile sync [14:43:19] fixing [14:44:51] ah phew! :) [14:45:05] try now [14:45:17] 10serviceops, 10Operations, 10Kubernetes, 10Patch-For-Review: Migrate to helm v3 - https://phabricator.wikimedia.org/T251305 (10JMeybohm) a:03JMeybohm [14:46:54] very cooool! it works! [14:52:00] akosiaris: I did update that one https://gerrit.wikimedia.org/r/c/operations/debs/kubernetes/+/642011 - but needs puppet patches ofc [14:54:24] jayme: yeah saw it. Testing the helm3 admin stuff and will get to it after that [14:54:50] akosiaris: cool! Up to date helm3 and helmfile packages should be available from apt [14:56:33] | $HELM_DRIVER | set the backend storage driver. Values are: configmap, secret, memory, postgres [14:56:34] postgres? [14:56:56] I think I don't want to know. configmap looks ok for now [14:57:38] I did look away pretty fast tbh :D [14:58:18] by default is secrets [14:59:00] I guess because the stuff stored there might as well contain...secrets... [14:59:52] yeah that works fine too. I am guessing memory is not persistent ofc and postgres... well it's postgres, not much more to tell [15:00:31] postgres is also in beta state, IIRC [15:02:51] eheh, calico released 3.17 yesterday [15:07:59] they added MTU auto-detection for pod networks. But apart from that the release also contains a bunch of bugfixes - so we might want to import it instead of 3.16.5 [15:11:58] MTU auto-detection? [15:12:28] "Calico will now automatically determine the optimal MTU for new pods based on the underlying network MTU and enabled encapsulation methods" [15:12:28] ah, for the ipip encapsulation I guess [15:12:38] yeah, think so... [15:12:55] yeah we don't care, we don't encapsulate on purpose to avoid those issues. But +1 on the 3.17 front. Better now than later [15:13:05] ack [15:13:08] to both :) [15:19:02] 10serviceops, 10Operations, 10Release-Engineering-Team-TODO, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services): Upgrade MediaWiki appservers to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10jijiki) [15:19:31] 10serviceops, 10Operations, 10Release-Engineering-Team-TODO, 10Patch-For-Review, and 2 others: Upgrade MediaWiki appservers to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10jijiki) [15:19:49] 10serviceops, 10Release-Engineering-Team-TODO, 10Scap, 10User-jijiki: Deploy Scap version 3.16.0-1 - https://phabricator.wikimedia.org/T268634 (10jijiki) [15:26:31] 10serviceops, 10Operations, 10Release-Engineering-Team-TODO, 10Patch-For-Review, and 2 others: Upgrade MediaWiki appservers to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10MoritzMuehlenhoff) [15:30:54] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Upgrade kubernetes clusters to a security supported (LTS) version - https://phabricator.wikimedia.org/T244335 (10JMeybohm) [15:30:56] 10serviceops, 10Operations, 10Prod-Kubernetes, 10Kubernetes, 10User-fsero: Upgrade Calico - https://phabricator.wikimedia.org/T207804 (10JMeybohm) [15:31:01] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Build calico 3.16 - https://phabricator.wikimedia.org/T266893 (10JMeybohm) 05Open→03Resolved * imported calico 3.17.0 packages into component/calico-future for stretch-wikimedia ** calico-cni ** calicoctl ** calico-images * pushed docke... [15:31:16] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Build calico 3.17.0 - https://phabricator.wikimedia.org/T266893 (10JMeybohm) [15:35:56] 10serviceops, 10Operations, 10Release-Engineering-Team-TODO, 10Patch-For-Review, and 2 others: Upgrade MediaWiki appservers to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10jijiki) >>! In T245757#6614669, @jijiki wrote: > There are a couple of things that we will need to keep in... [15:38:50] 10serviceops, 10Operations, 10Release-Engineering-Team-TODO, 10Patch-For-Review, and 2 others: Upgrade MediaWiki appservers to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10jijiki) [15:39:25] 10serviceops, 10Operations, 10Release-Engineering-Team-TODO, 10Patch-For-Review, and 2 others: Upgrade MediaWiki appservers to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10jijiki) [15:47:54] akosiaris: would a cleaner version of this be useful to have for wider use? [15:47:54] https://gist.github.com/ottomata/f089acd146ac87c0703d88ec85e99a5c [15:48:03] right now its just in my home dir on deploy1001 [15:50:15] ottomata: I guess you don't rely at all on logstash? cause all of that should also be in logstash [15:50:23] btw, echo "Tailing logs for all pods in $SERVICE $CLUSTER" set -x for pod in $(kubectl get pods -o wide | grep $SERVICE | awk '{print $1}'); do [15:50:29] no need for that [15:50:39] kubectl logs -l app= [15:50:43] yes but then i have to rely on logstash [15:50:45] unless you really need to tail -f all the pods [15:50:46] which i do sometimes [15:51:25] i use [15:51:25] https://logstash-next.wikimedia.org/goto/1d35b2aeedb4a6bf7a63e9c2f4a14df1 [15:51:56] when troubleshooting after a release [15:52:03] i really j ust want to make sure stuff is ok [15:52:08] -l app= eh...? [15:52:39] yeah, it will match based on the app label [15:52:47] e.g. kubectl logs -l app=eventstreams [15:52:53] oh but it doesn't let me follow? [15:52:54] -f [15:52:55] ? [15:52:59] no, it doesn't allow -f [15:53:20] weird i wonder why [15:53:40] too many websocket connection I guess? [15:53:48] all of those are web socket connections to the api server [15:53:54] aye [15:55:00] 10serviceops, 10Release-Engineering-Team-TODO, 10Scap, 10User-jijiki: Deploy Scap version 3.16.0-1 - https://phabricator.wikimedia.org/T268634 (10LarsWirzenius) [15:55:48] ottomata: anyway, to answer the question, I haven't had to do that up to now, nor have I been asked. [15:55:58] ok, i'll keep it for me then :) [15:56:02] I guess maybe ask service deployers? [15:56:16] I could be a bad corner case [15:59:19] how do I best ask them? :) [16:09:08] best idea I have is an email to ops-l to be honest, which is meh... it will catch quite a few of them of course. [16:09:57] jayme: is it me or is helm3 faster? [16:10:19] akosiaris: I bet it's you being fast :P [16:10:27] lol [16:10:45] Actually I think it is because it does not need the tiller-hopp [16:10:57] yeah makes sense. I just felt so snappier every time [16:11:23] so, mostly works, got a couple of comments here and there, but 1 thing I am not sure why it doesn't work [16:11:34] in ./helmfile.yaml: failed processing release eventrouter: command "/home/alex/bin/helm3" exited with non-zero status: [16:11:40] Error: failed to download "wmf-stable/eventrouter" (hint: running `helm repo update` may help) [16:11:43] plus (for helmfile.d/admin) we have like 4 or 5 times less releases then with the old structure. So that's faster as well [16:11:47] but I 've definitely refresh the repo [16:12:02] jayme: oh, that's true! good point [16:12:32] akosiaris: did you "helm3 repo update"? :D [16:12:38] yup [16:12:42] and "helm3 repo add ofc..." [16:12:44] hmm... [16:12:45] helm3 search repo eventrouter [16:12:45] NAME CHART VERSION APP VERSION DESCRIPTION [16:12:45] wmf-stable/eventrouter 0.3.2 0.3 A Helm chart for eventrouter (https://github.co... [16:13:35] weird [16:14:00] does "helm3 pull wmf-stable/eventrouter" fail as well? [16:14:09] or just when run via helmfile [16:14:14] oh... [16:14:43] I probably left some broken local filesystem references for the raw chart [16:14:43] and it's pull now? memory muscles did fetch and wondered [16:14:52] yeah, I fixed those already [16:14:57] part of the comments :-) [16:15:04] ok. sorry .. forgot about those [16:17:26] weird though ... [16:17:31] I need to run, ttyl and thanks for the review o/ [16:17:40] yw, have a nice time! [16:23:44] interestingly running the command helmfile outputs it works... [16:27:46] Error: repo wmf-stable not found [16:27:46] lol... [16:28:45] ah, I think I know what this is [16:48:59] 10serviceops, 10Operations, 10CommRel-Specialists-Support (Oct-Dec-2020), 10User-notice: CommRel support for ICU 63 upgrade - https://phabricator.wikimedia.org/T267145 (10RLazarus) This is finished on all wikis, a little bit slower than our original schedule but not stretching into the holiday weekend as I... [17:04:45] 10serviceops, 10Beta-Cluster-Infrastructure, 10DBA, 10Operations, 10Patch-For-Review: Upgrade the MediaWiki servers to ICU 63 - https://phabricator.wikimedia.org/T264991 (10Trizek-WMF) [17:05:15] 10serviceops, 10Operations, 10CommRel-Specialists-Support (Oct-Dec-2020), 10User-notice: CommRel support for ICU 63 upgrade - https://phabricator.wikimedia.org/T267145 (10Trizek-WMF) 05Open→03Resolved Thank you! You mentioned that the next migration would be with a different system not creating any d... [18:22:57] 10serviceops, 10Push-Notification-Service, 10Patch-For-Review, 10Product-Infrastructure-Team-Backlog (Kanban): Push notification service should make deletion requests to MediaWiki for invalid or expired subscriptions - https://phabricator.wikimedia.org/T260247 (10MSantos) [18:39:19] 10serviceops, 10Operations, 10docker-pkg: Docker image on the build host seem to ignore apt priority for wikimedia packages - https://phabricator.wikimedia.org/T268612 (10Dzahn) p:05Triage→03Medium [18:47:00] 10serviceops, 10Push-Notification-Service, 10Product-Infrastructure-Team-Backlog (Kanban), 10User-jijiki: Set up secrets for Token clean-up - https://phabricator.wikimedia.org/T262957 (10MSantos) @jijiki we are ready for this task, the needed work for this is done [1] and deployed [2]. [1] https://gerrit.... [18:50:59] 10serviceops, 10Push-Notification-Service, 10Product-Infrastructure-Team-Backlog (Kanban), 10User-jijiki: Set up secrets for Token clean-up - https://phabricator.wikimedia.org/T262957 (10MSantos) [19:46:34] 10serviceops, 10DBA, 10MediaWiki-Parser, 10Parsoid, 10Platform Team Workboards (Green): CAPEX for ParserCache for Parsoid - https://phabricator.wikimedia.org/T263587 (10WDoranWMF) 05Open→03Declined This work will be handled by #parsoid team [19:54:25] 10serviceops, 10DBA, 10MediaWiki-Parser, 10Parsoid, 10Platform Team Workboards (Green): CAPEX for ParserCache for Parsoid - https://phabricator.wikimedia.org/T263587 (10ssastry) 05Declined→03Open a:05WDoranWMF→03None @WDoranWMF. I am going to reopen this and unassign you, but feel free to change... [20:04:47] 10serviceops, 10Performance-Team, 10WikimediaDebug: Fetch mwdebug backend server list from noc.wikimedia.org - https://phabricator.wikimedia.org/T268167 (10Gilles) p:05Triage→03Low a:03Gilles [20:07:56] 10serviceops, 10DBA, 10MediaWiki-Parser, 10Parsoid: CAPEX for ParserCache for Parsoid - https://phabricator.wikimedia.org/T263587 (10WDoranWMF) @ssastry That works for me, I didn't know if there was an existing ticket elsewhere but I was hoping that closing and tagging would at least assure that your team... [20:11:47] 10serviceops, 10Operations, 10Performance-Team, 10User-jijiki: MediaWiki to route specific keys to /*/mw-with-onhost-tier/ - https://phabricator.wikimedia.org/T264604 (10jijiki) @aaron @Krinkle We have successfully rolled this out on all app and api servers, and I am planning to continue with the jobrunner... [20:23:12] 10serviceops, 10Operations, 10Performance-Team, 10User-jijiki: MediaWiki to route specific keys to /*/mw-with-onhost-tier/ - https://phabricator.wikimedia.org/T264604 (10Krinkle) Can you point me to where (in Puppet?) the ttl enforcement and route/command filter for on-host reside? I'd like to link it from... [20:39:49] 10serviceops, 10Operations, 10Performance-Team, 10User-jijiki: MediaWiki to route specific keys to /*/mw-with-onhost-tier/ - https://phabricator.wikimedia.org/T264604 (10jijiki) @Krinkle would it help if I paste the generated config ? The puppet part is here: [[ https://github.com/wikimedia/puppet/blob/3... [21:32:14] 10serviceops, 10DBA, 10MediaWiki-Parser, 10Parsoid: CAPEX for ParserCache for Parsoid - https://phabricator.wikimedia.org/T263587 (10ssastry) Okay, so I read through the ticket and from what I can tell, there are three different pieces to this in this order: 1. Parser Cache backing "storage" design questi...