[01:34:37] 10serviceops, 10Operations, 10Patch-For-Review: dropped packets to phab1003 22280/tcp - https://phabricator.wikimedia.org/T238781 (10Vgutierrez) @dzahn that's right, Removing the wss:// -> ws:// from ats-tls doesn't stop the requests from reaching varnish-fe as they are accepted as part of the catch-all rema... [08:11:51] 10serviceops, 10Operations, 10Release Pipeline, 10Kubernetes: Identify which parts of the "Add a wiki" procedure can be integrated with the deployment pipeline - https://phabricator.wikimedia.org/T238158 (10akosiaris) p:05Triage→03Low [09:13:01] 10serviceops, 10Operations, 10observability: dropped packets to conf1004/5/6 2379/tcp - https://phabricator.wikimedia.org/T238791 (10fgiunchedi) Indeed looks like prometheus is trying to fetch `conf1004.eqiad.wmnet:2379/metrics` with no success. Locally on conf1004 even past the firewall the endpoint doesn't... [09:15:51] 10serviceops, 10Product-Infrastructure-Team-Backlog, 10Wikifeeds: Wikifeeds deployment failed in staging - https://phabricator.wikimedia.org/T238792 (10akosiaris) 05Open→03Resolved a:03akosiaris >>! In T238792#5679882, @Mholloway wrote: > Looks like there are currently issues accessing docker-registry.... [09:25:51] 10serviceops, 10Operations, 10observability: dropped packets to conf1004/5/6 2379/tcp - https://phabricator.wikimedia.org/T238791 (10Joe) @fgiunchedi you need to use https, and it works locally. If you want to access metrics remotely, either you switch to port 4001 which is publically exposed. We have diffe... [09:52:03] 10serviceops, 10Operations, 10observability: dropped packets to conf1004/5/6 2379/tcp - https://phabricator.wikimedia.org/T238791 (10fgiunchedi) >>! In T238791#5680931, @Joe wrote: > @fgiunchedi you need to use https, and it works locally. > > If you want to access metrics remotely, either you switch to por... [09:54:37] 10serviceops, 10Operations, 10observability: dropped packets to conf1004/5/6 2379/tcp - https://phabricator.wikimedia.org/T238791 (10Joe) >>! In T238791#5681024, @fgiunchedi wrote: >>>! In T238791#5680931, @Joe wrote: >> @fgiunchedi you need to use https, and it works locally. >> >> If you want to access me... [10:07:59] 10serviceops, 10Operations, 10observability, 10Patch-For-Review: Gather metrics on request status codes, latencies from the MediaWiki appservers - https://phabricator.wikimedia.org/T226815 (10Joe) 05Open→03Resolved a:03Joe [13:27:47] 10serviceops, 10Operations: dropped packets to echostore.svc.eqiad 8082/tcp - https://phabricator.wikimedia.org/T238789 (10akosiaris) More investigation in T238823. It's quite possibly expected. [14:09:44] _joe_: hellllooooooooo [14:11:20] <_joe_> ottomata: I got alex to review all your patches :D [14:11:46] <_joe_> I'm in a meeting, but if you need some help I will be around in 1 hour or so [14:12:56] ok, cool, i can do the k8s ones, i'm scared about merging lvs and discovery ones though [14:13:02] almost broke stuff once doing that [14:14:54] <_joe_> yeah I can assist with those [14:16:15] <_joe_> but you need to do the k8s deploy first [14:25:01] 10serviceops, 10Core Platform Team Workboards, 10Proton, 10Product-Infrastructure-Team-Backlog (Kanban): Profile proton memory usage for Helm chart - https://phabricator.wikimedia.org/T238830 (10MSantos) [14:25:34] 10serviceops, 10Core Platform Team Workboards, 10Proton, 10Product-Infrastructure-Team-Backlog (Kanban): Profile proton memory usage for Helm chart - https://phabricator.wikimedia.org/T238830 (10MSantos) [14:27:47] akosiaris: can you check this one out again? [14:27:48] https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/551610 [14:27:58] did your comments, but wanted to see if you liked the way I did it [14:38:36] ottomata: will do, but in a meeting right now [14:52:30] is there some local cachign for helmfile commands? [14:52:42] i ran helmfile diff when my new package file wasn't on releases yet [14:52:45] now it is there [14:52:47] but i still get 404 [14:58:25] weird, i get 404 from curl on deploy1001, but not from my laptop [14:58:28] https://releases.wikimedia.org/charts/eventgate-0.0.13.tgz [14:59:34] the 404 is cached [15:00:13] https://releases.wikimedia.org/charts/eventgate-0.0.13.tgz?asdf returns a 200 heh [15:02:14] curl has a cache too? [15:02:23] < x-cache: cp1085 hit/2, cp1077 miss [15:02:26] * akosiaris back [15:02:27] AHHHH [15:02:31] ha [15:02:42] well that's annoying! :p [15:02:43] ah, ATS ? [15:02:45] i don't know how our traffic layer is configured wrt: caching negative responses [15:02:59] should I summon ema ? [15:03:08] today ATS was introduced into this [15:03:11] cp1077 is ATS IIRC [15:03:21] am trying to get the envoy proxy stuff out before I also have to go to metings [15:03:30] so hopeully jo e can merge the lvs and discovery stuff [15:03:46] but it look slike i got the eventgate tgz cached negatively [15:03:56] and now helmfile on deploy1001 can't get it [15:05:00] cache-control: no-cache,must-revalidate [15:05:04] it shouldn't [15:05:20] btw from esams I have no problems, /me trying eqiad specifically now [15:05:27] from eqiad it is still cached as a 404 [15:05:51] 10serviceops, 10Core Platform Team, 10Product-Infrastructure-Team-Backlog: PCS internal request rates tripled on 2019-11-19 - https://phabricator.wikimedia.org/T238832 (10Mholloway) [15:06:38] heh, i just hacked /etc/hosts for one sec and got helm to dl it from releases.discovery.wmnet, reset etc hosts, still problems tho [15:06:38] curl -i --resolve releases.wikimedia.org:443:208.80.154.224 https://releases.wikimedia.org/charts/eventgate-0.0.13.tgz [15:06:38] says 200 [15:06:58] I go via x-cache: cp1085 pass, cp1081 pass [15:06:58] though [15:07:01] no ATS [15:07:05] I 'll ping ema [15:07:10] ok [15:08:05] o/ [15:08:21] ema: ottomata has issues fetching https://releases.wikimedia.org/charts/eventgate-0.0.13.tgz from deploy1001 [15:08:30] gets back a 404 [15:08:43] and cdanis: < x-cache: cp1085 hit/2, cp1077 miss [15:08:50] can replicate it (I still can't) [15:09:48] my curl -i --resolve releases.wikimedia.org:443:208.80.154.224 https://releases.wikimedia.org/charts/eventgate-0.0.13.tgz [15:09:48] works [15:10:00] x-cache: cp1085 pass, cp1081 pass [15:10:00] plus I 've setup apache to do a [15:10:05] right, so apparently the 404 was cached at cp1085 [15:10:06] cache-control: no-cache,must-revalidate [15:10:16] perhaps something/somebody tried to fetch the file before it existed? [15:10:27] we can purge it [15:10:31] yes, that happened indeed [15:10:45] https://wikitech.wikimedia.org/wiki/Multicast_HTCP_purging#One-off_purge [15:11:04] ah so, the cache control headers were never set for the 404... [15:11:10] ok that explains it [15:11:32] lemme try that ema [15:12:32] ottomata: try now? [15:14:05] server: envoye [15:14:07] lol [15:14:12] _joe_: ^ [15:14:24] I am pretty sure it's not mean to have an 'e' at the end? [15:14:28] <_joe_> nope [15:15:37] ottomata: I think ema solved your problem. I can now consistently get 200 and content-length: 12775 for that URL [15:15:46] I 'll leave the envoye part to _joe_ :P [15:16:03] ema: thanks! [15:16:06] <_joe_> where is that problem akosiaris? [15:16:24] deploy1001:~$ watch -n1 curl -I https://releases.wikimedia.org/charts/eventgate-0.0.13.tgz [15:16:27] akosiaris: yw! We should make sure that the origin sets CC: private on 404s if we don't want them to be cached [15:16:42] everynow and then (when ATS is used) you will notice it [15:16:50] ema: good point. Lemme see what I can do about it [15:17:33] akosiaris: well actually I see that we do it in Lua for ATS regardless of what the origin says ... [15:17:37] probably not optimal [15:17:56] s/do it/set CC: maxage=600/ [15:18:06] <_joe_> akosiaris: I mean envoye [15:18:45] _joe_: it's a header in that response [15:18:59] why is it added again ? [15:19:07] <_joe_> akosiaris: not really [15:19:08] I expected server: apache tbh [15:25:27] 10serviceops, 10Product-Infrastructure-Team-Backlog, 10Wikifeeds: Wikifeeds deployment failed in staging - https://phabricator.wikimedia.org/T238792 (10Mholloway) Deployed on eqiad and codfw. Thank you, @akosiaris! [15:26:39] <_joe_> akosiaris: I get server: apache from eqiad [15:27:06] <_joe_> the server:envoy you get from outside [15:27:42] <_joe_> comes from ats and it's a bug that we will resolve as soon as I finish to build envoy 1.12.1 [15:27:55] sigh [15:27:57] yeah ignore me [15:28:05] it's a by product of watch -n1 it seems? [15:28:48] "envoye" sounded like someone from Sardinia tried to say envoy [15:28:55] I quite liked it [15:30:58] envoye is clearly how you connect to echotore [15:31:56] ahahahaha [15:32:16] funny how gnu watch tricked me [15:32:25] I should have seen it from the other artifacts [15:32:26] like [15:32:27] we're going to wind up with a whole shadow infrastructure [15:32:33] x-cache-status: passu [15:32:39] ahhahahah [15:32:41] age: 0e [15:33:04] not to mention the beautiful server-timing: cache;desc="pass"o [15:33:52] x-cache-status: passé [15:33:58] hahahaha [15:34:54] 10serviceops, 10Mobile-Content-Service, 10Page Content Service, 10Product-Infrastructure-Team-Backlog, and 2 others: Resolve service instability due to excessive event loop blockage since starting PCS response pregeneration - https://phabricator.wikimedia.org/T229286 (10Mholloway) [15:45:08] <_joe_> rlazarus: I wanted to add http headers that should denote the degree of confidence of people running the infrastructure in that specific resource [15:45:18] hahaha [15:45:23] <_joe_> X-Rotten-Tomatoes: putrid [15:45:40] make it an emoji standard [15:45:55] <_joe_> 🤔 [15:45:57] ranging from 😊 through 😬 to 😅 to 😱 [15:46:02] 💩 [15:46:10] <_joe_> I was waiting for it cdanis [15:46:19] it seemed practically obligatory [15:49:44] <_joe_> akosiaris: I don't remember anymore, are ingress controllers accessed via kube-proxy? [15:50:02] <_joe_> as any other service on a node? [15:53:06] _joe_: I have to say I don't remember [15:53:13] if anything I would expect the inverse [15:53:18] thanks all, sorry was afk due to complicated seatbelt induced battery dead car borrowing situtation [15:53:31] however IIRC there are ingresses that talk directly to pods? [15:53:39] <_joe_> ottomata: that must be one of the best BOFH excuses ever [15:53:41] but take that with a huge grain of salt, I 'll have to revisit this [15:53:47] <_joe_> akosiaris: ingress => pod is direct [15:53:50] but I think that linkerd does exact that [15:54:03] <_joe_> but client => kube-proxy => ingress is what I remember [15:54:15] that depends on where the ingress is located [15:54:21] if it's in the cluster, then yes [15:54:30] if not, then no (surprisingly :P) [15:54:41] hmm helmfile diff shows no changes... [15:54:51] /srv/deployment-charts/helmfile.d/services/staging/eventgate-logging-external [15:55:33] ottomata: I have to tell you that your dead battery BOFH excuse managed to coincide with the 1h I had before my next 2 meetings [15:55:42] i know [15:55:45] i have meetings now too [15:55:53] been a bit stressed about it [15:55:57] hi all im trying to update the opserations/puppet docker image and also trying to document it a bit. this is what i have re: documentation https://wikitech.wikimedia.org/wiki/Docker#Updateing_an_image if anyone has a chance could please take a look and let me know if im going of in the wrong direction [15:55:58] I 'll try and have a look, no promises [15:56:23] my gf is driving 7 hours to niagra falls to interview a crazy lady about hermit crabs [15:56:27] if that makes the excuse any better [15:56:40] jbond42: niah, not really [15:57:05] oh wait [15:57:12] releng/operations-puppet ? [15:57:22] hmm best ask hashar, but IIRC it's docker-pkg [15:57:24] lemme fine the repo [15:58:28] akosiaris: thanks [16:00:21] thanks for review on kafka tls alex [16:00:33] will do that stuff, i think we can move the rest in a bit [16:34:28] jbond42: took me a while (sorry about that, but found it) https://www.mediawiki.org/wiki/Continuous_integration/Docker [16:34:55] https://gerrit.wikimedia.org/r/integration/config is the repo where most of those things exist in [16:37:46] thanks akosiaris although that repo just refrences the docker base image:version to pull in right. i want to update the Gemfile with the puppet 5 require and i belive that means producing another version of the bas image [16:43:33] yeah, I think run.sh does a bundle update [16:43:44] so during the building of that image ? [16:44:24] I am not very sure tbh. Haven't really dug into it much [16:44:33] hashar is the person you want for that [16:44:49] ack thanks ill catch up with them tomorrow/monday [17:36:43] 10serviceops, 10MediaWiki-REST-API, 10Operations, 10wikidiff2, and 2 others: Prod compare endpoint missing offset object (with from & to keys) on diff items - https://phabricator.wikimedia.org/T238846 (10Tsevener) [17:38:22] 10serviceops, 10MediaWiki-REST-API, 10Operations, 10wikidiff2, and 2 others: Prod compare endpoint missing offset object (with from & to keys) on diff items - https://phabricator.wikimedia.org/T238846 (10Tsevener) [17:39:17] akosiaris: if you have time, the thing you can help unblock is this helmfile diff and apply on eventgate-logging-external staging [17:39:41] if i can get the envoyproxy enabled there [17:39:48] we can do the lvs and discovery anytime [17:45:28] 10serviceops, 10MediaWiki-REST-API, 10Operations, 10wikidiff2, and 2 others: Prod compare endpoint missing offset object (with from & to keys) on diff items - https://phabricator.wikimedia.org/T238846 (10Pchelolo) No surprise. @jijiki has deployed 1.10 to labs machines as indicated in the parent task, but... [17:48:28] <_joe_> ottomata: what's your problem? [17:48:38] helmfile diff shows no diff [17:48:49] so apply does nothing [17:48:52] i tried a sync too [17:48:59] but nothing changed..., but it certainly has diffs... [17:49:59] <_joe_> LAST DEPLOYED: Thu Nov 21 16:54:29 2019 [17:50:02] <_joe_> uhm [17:50:26] <_joe_> that looks like one hour ago? [17:50:35] yeah, but [17:50:36] pod age [17:50:38] 6d22h [17:50:39] <_joe_> yep [17:50:45] <_joe_> that's what I am seeing [17:50:45] i tried to deploy [17:50:48] but nothing happened [17:51:09] <_joe_> what did change in your deploy? [17:51:28] values tls.enabled: true and chart version 0.0.13 [17:51:39] that's what i'm trying to deploy [17:52:23] <_joe_> logging-external 2 Thu Nov 21 16:54:29 2019 DEPLOYED eventgate-0.0.13 eventgate-logging-external [17:52:27] <_joe_> sigh. [17:53:03] how could 0.0.13 be deploeyd if pod age is 6d? [17:53:21] <_joe_> well if the pod didn't change [17:53:21] i mean, i'm sure i did somethign wrong here [17:53:38] maybe by forgetting to add the 0.0.13.tgz in one of my commits, and running helmfile diff [17:53:39] ? [17:53:40] i dunno [17:53:47] <_joe_> ottomata: I *think* the values have not been applied? [17:53:55] <_joe_> heh I have no idea [17:54:06] <_joe_> I think we ran into this with alex already [17:54:08] oh? [17:57:05] <_joe_> ottomata: helm get logging-external | less [17:57:17] huh [17:57:19] <_joe_> you will see tls.enabled: true as one of your values [17:57:34] hmm, so maybe somethign is wrong with the chart? [17:57:35] <_joe_> so yeah something went definitely very wrong somewhere [17:57:44] tls.enabled not actually changing anything [17:58:10] <_joe_> you can try to delete the chart and recreate it, but I think there is something I'm missing [17:58:17] <_joe_> alex is off by now too [17:59:21] 10serviceops, 10Operations, 10Patch-For-Review: dropped packets to phab1003 22280/tcp - https://phabricator.wikimedia.org/T238781 (10Dzahn) @ayounsi After the merge above this is expected to stop soon, now. [17:59:21] yeah def not working, since name: {{ template "wmf.releasename" . }}-tls-service is not in service.yaml [17:59:21] <_joe_> there must be something we're missing that doesn't allow to apply the change to tls.enabled: true to be registered [17:59:22] hmmm [17:59:26] yeah [17:59:44] <_joe_> ottomata: can you try to release a new version of the chart? [17:59:52] <_joe_> just recreate the index now [17:59:59] ok [18:00:01] good idea [18:00:10] <_joe_> just to make sure we're not missing something else [18:02:42] oo helmfile diff now is doing stuff! [18:02:47] applying [18:03:13] <_joe_> so we just shrug and assume it was a L8 issue? [18:05:22] L8 ? [18:05:38] oo 43192:43192/TCP [18:16:04] hm _joe_ must be missing somethjing else too? [18:16:05] telnet kubestage1001.eqiad.wmnet 43192 [18:16:27] <_joe_> ottomata: maybe the networkpolicy? [18:17:49] <_joe_> tcp6 0 0 [::]:43192 [::]:* LISTEN 887/kube-proxy [18:18:09] in the chart or in calico stuff? [18:18:19] <_joe_> this is netstat [18:18:30] <_joe_> on kubestage1001 [18:18:46] <_joe_> so kube-proxy is definitely listening on that port [18:18:49] ya [18:20:03] the ingress policy has [18:20:05] - port: 43192 [18:20:05] protocol: TCP [18:20:58] and the tls proxy container has containerPort: 43192 [18:21:33] hm [18:21:57] i can get to it from localhost on kubestage1001 [18:22:00] some firewall rule? [18:22:20] <_joe_> can you reach the non-tls port? [18:22:28] yes [18:22:36] <_joe_> oh ofc there are firewall routes :P [18:22:42] <_joe_> rules [18:22:48] in puppet? [18:23:12] <_joe_> yeah I mean I don't think you can access those hosts from everywhere in prod [18:23:32] <_joe_> every port at least [18:23:37] hmm, i can get to the http port from deploy1001 on 33192 [18:24:10] <_joe_> in fact it's a nodeport [18:24:13] <_joe_> so it should be opened [18:24:18] <_joe_> sorry, I was brainfarting [18:24:29] that's done by k8s automatically? [18:24:36] <_joe_> let's take a look at the chart [18:24:38] <_joe_> yes [18:24:57] <_joe_> if you have the courage to look at what iptables does on a k8s node [18:25:14] i'm looking at iptables-save and grepping on kubestage1001 [18:25:26] i see -A KUBE-NODEPORTS chain for 33192 [18:25:28] but not 43192 [18:25:41] <_joe_> uhm [18:25:46] <_joe_> so let's see what helm says [18:25:55] https://www.irccloud.com/pastebin/mdI03axa/ [18:28:00] <_joe_> oh? [18:29:46] <_joe_> ottomata: hah found it [18:29:49] <_joe_> selector: [18:29:51] <_joe_> app: eventgate [18:29:53] <_joe_> vs [18:29:59] <_joe_> selector: [18:30:01] <_joe_> app: eventgate-logging-external [18:30:04] <_joe_> that's the error [18:30:22] <_joe_> kubectl get service -o yaml is your friend [18:30:42] <_joe_> ok I really need to go off now. cya [18:31:24] ok looking ty! [18:32:23] <_joe_> basically it's a classic copypasta [18:32:32] <_joe_> blubberoid is a simple type [18:32:46] <_joe_> while you're posh and you have multiple releases for the same software [18:32:54] <_joe_> so you need to adapt your selectors [18:32:59] ah _joe_ yeah go tit [18:33:06] but probably blubberoid shoudl use releasename too [18:33:08] just in case? [18:33:31] <_joe_> pretty sure it's consistent across the two services [18:33:32] and also dunno what create_chart script does but should use releasename there? [18:33:36] <_joe_> I did copypasta too [18:33:49] ya it has chartname in blubberoid [18:33:50] <_joe_> possibly [18:34:00] which is only working because there chartname and release name happen to match [18:34:02] <_joe_> but it's a tomorrow question [18:34:05] :) [18:35:18] hmm, the main http port also uses chartname for selecgtor [18:38:06] hm, weird, am not sure why it says app: eventgate there..... [18:38:42] ahhh no sorry [18:38:48] haha looking at wrong file in editor ok ok [18:43:21] heheh [18:43:21] curl: (51) SSL: no alternative certificate subject name matches target host name 'kubestage1001.eqiad.wmnet' [18:43:26] better! [18:44:38] it works though! [18:44:39] yeehaw [18:45:53] 10serviceops, 10Analytics-Kanban, 10Better Use Of Data, 10Event-Platform, and 8 others: Set up eventgate-logging-external in production - https://phabricator.wikimedia.org/T236386 (10Ottomata) [18:49:57] 10serviceops, 10Analytics-Kanban, 10Better Use Of Data, 10Event-Platform, and 8 others: Set up eventgate-logging-external in production - https://phabricator.wikimedia.org/T236386 (10Ottomata) Ok thanks for the help today @akosiaris and @Joe, HTTPS via envoyproxy is finally working! I will be off tomorrow... [20:00:44] 10serviceops, 10Performance-Team: make xhgui::app role support stretch/buster and deploy on new xhgui machines - https://phabricator.wikimedia.org/T238788 (10Dzahn) [20:01:38] 10serviceops, 10Performance-Team: make xhgui::app role support stretch/buster and deploy on new xhgui machines - https://phabricator.wikimedia.org/T238788 (10Dzahn) support for stretch and PHP 7.0: https://gerrit.wikimedia.org/r/c/operations/puppet/+/552325 [20:10:08] _joe_: let's find some time to talk Go packaging with rlazarus tomorrow -- I need to do it as well [20:10:37] <_joe_> cdanis: heh sure [20:11:00] <_joe_> we can try look into the poolcounter exporter packaging [20:11:16] yeah sounds good [20:11:30] 👍 [20:11:34] _joe_ gave me some pointers already but I haven't followed em yet, would happily pair on it [20:13:44] <_joe_> I want to underline i will only teach evil and definitely un-debiany techniques. For white magic you've got plenty of debian devs on the team :P [20:14:19] <_joe_> they will gladly point you to subsection 124.3.1.1 of the debian policy whenever appropriate [20:17:13] <_joe_> also, have you both read the new maintainer guide? I would like you to get to the session properly confused. [20:17:34] <_joe_> jokes aside, if you pick the correct sections, it's a very good starting point [20:23:32] I'll give it a look on the plane [20:29:02] _joe_: once upon a time i had an @debian.org email address [20:29:23] it's been about 15 years though [20:46:22] <_joe_> cdanis: you're one of them heh [20:46:36] once upon a time [20:47:35] <_joe_> in other news, I just realized that what we consider a "deployment on kubernetes" is a badly glued ball of 1k lines of yaml, some 10 bash scripts badly weaved together in a chroot, plus some kernel fun [20:48:24] <_joe_> I might want puppet back at some point. [20:51:01] is the 1k lines of yaml mostly generated by helm? [20:53:55] Alice Goldfuss has a great talk about this [20:56:52] oh? [21:01:54] about containers rather than kubernetes in particular, but the main thrust of the first part of the talk is just, "containers aren't... a Thing. they're just processes, created from tarballs, and attached to namespaces and cgroups. they have some interesting properties but they're not Special" [21:03:15] I might have seen that talk online [21:03:22] "The Container Operator’s Manual" is the name of the talk, it's worth a watch -- unfortunately your best bet is to watch a recording of it, now that she's left GitHub I don't know if she has speaking plans for the foreseeable :/ [21:04:47] yeah a recording is what I would have seen, this sounds very familiar and i know i've watched some of her stuff [21:10:14] <_joe_> rlazarus: that quote is taken from a third party [21:10:24] <_joe_> and unattributed [21:10:28] <_joe_> but oh well :P [21:11:06] <_joe_> also that talk becoming popular partly ruined one of my interview questions [21:11:40] <_joe_> but it's a good introduction indeed [21:12:11] <_joe_> cdanis: the 1k lines of yaml are generated by helm from a template [21:12:25] <_joe_> which is already something that should've looked like a bad idea from day one [21:12:28] lol I think I know which interview question [21:12:31] <_joe_> but I wasn't the wiser [21:12:51] <_joe_> it's complex config, you want a DSL, not a templating engine [21:12:58] we could use jsonnet ;) [21:13:00] <_joe_> and not exactly the most friendly one, either [21:13:31] <_joe_> cdanis: helm is a huge advantage for deployers IMO [21:13:42] <_joe_> else I would consider even jsonnet :P [21:14:44] <_joe_> although that's not really a DSL dedicated to kube [21:14:56] there was ksonnet, but it is no more [21:19:00] <_joe_> but seriously, we need to rethink how we write helm charts to make them look more like code and less like 1999-style php copypasta [21:19:27] <_joe_> the result on kubernetes will be as awful of course [21:45:06] You can add kustomize to make it better.. [21:45:19] ... or worse sometimes :-)