[09:06:57] _joe_: are you upgrading envoy today? if not i can do it [09:14:04] <_joe_> yeah i wanted to [09:14:18] <_joe_> but i have a lot of meetings in the afternoon [09:14:24] i can do it [09:14:29] do you mind if i do it? [09:14:30] <_joe_> so I might prepare a patch to the debian dir only [09:14:42] <_joe_> not at all [09:14:57] great then ill start now [09:15:11] <_joe_> the tldr is I'd like to include the hot-restarter.py file in /usr/share/envoyproxy [09:15:23] i can do that as well [09:15:53] <_joe_> thanks :) [09:34:55] _joe_: we're still thinking about smoke testing; the locust venv is fairly chunky and so far we didn't find it portable. Would it be ok to just ssh tunnel traffic from locust on my laptop to our service? [09:35:26] <_joe_> tarrow: yeah, but that would make the results unreliable [09:35:34] <_joe_> if you're planning on testing latency [09:36:49] Yeah, we realise it's not ideal. But it should give us upper bounds for latency and load shouldnt change much. As far as I can tell [09:46:28] <_joe_> I'm sorry I'm basically alone for a week, I really can't help you right now, but I think we should have a better solution [09:47:47] <_joe_> can you open a task and tag it serviceops? [09:56:23] _joe_: can do, but would it still be ok to try the tunneled test? we're only going to have about 2 reqs/s, so the difference in bandwidth and latency should not matter much [09:56:44] <_joe_> sure you can go on [09:56:58] thanks! [09:57:31] <_joe_> sorry, I thought it was implicit given I didn't scream NO :P [10:07:11] all good! :) [10:28:17] Cool, were starting now :) [10:35:05] <_joe_> I can see it happening https://grafana.wikimedia.org/d/AJf0z_7Wz/termbox?refresh=1m&orgId=1&from=now-15m&to=now [10:46:34] :) yep, we just turned it off again since we saw it climbing higher than we expected; probably us using locust wrong [10:52:06] and obviously we're starting again [11:20:32] And we're done. Errors are from bad synthetic data (trying to get redirected pages) so we wouldn't expect that to happen when MW is making the requests [12:51:09] if I wanted to (temporarily) disable TLS encryption of sessionstore in k8s staging, would it be enough to comment out cert & key in values.yaml (https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/master/helmfile.d/services/staging/sessionstore/values.yaml#38)? [12:52:17] simply omitting them is enough for Kask, but it's not clear to me how this stuff is stitched together for deployment [12:54:04] (i.e. would omitting them values.yaml cause them to be omitted in kask's configuration?) [12:59:53] urandom: while we using scap-helm, it was enough to override these values in the appropriate file in /srv/scap-helm/ on deploy1001, but i dunno if that has changed [13:00:11] it doesn't really answer your q, but it does give you an easy way to try it out [13:00:25] I don't want to override, I want to remove [13:00:43] and yeah, smoke testing is my next option [13:03:23] ah these are the certs [13:04:27] afaik alex is the person to ask, but he's out [13:04:44] perhaps fsero can chip in? [13:05:44] <_joe_> urandom: now you need a commit, yes [13:05:47] <_joe_> or well [13:06:03] <_joe_> you can modify the file locally on deploy1001 if you revert quickly [13:08:30] Oh, even better [13:08:57] <_joe_> but I'm in the techcom meeting, so yeah fsero can help you [13:09:13] <_joe_> although I suspect he's showing his love to envoy :P [13:10:00] key is already commented out, and is somehow injected from secrets, but I'll give it a shot [13:10:08] <_joe_> oh ok [13:10:32] The key part is under the private repo puppet [13:11:35] The cert part you can do a commit in the deployment-chart repo, alternatively you can modify your chart to have a flag and ignore those data in configmap [13:11:53] That would be imho more correct since you don't want to avoid tls in production I guess [13:11:59] urandom: [13:12:48] 10serviceops, 10Operations, 10Traffic, 10Patch-For-Review: Applayer services without TLS - https://phabricator.wikimedia.org/T210411 (10ema) [13:13:58] 10serviceops, 10Operations, 10Traffic, 10Patch-For-Review: Applayer services without TLS - https://phabricator.wikimedia.org/T210411 (10ema) [13:16:08] <_joe_> urandom: so you want to run on tls conditionally, it should be a specific key in your chart, instead of relying on a secret being empty [13:18:17] This a rare exception [13:18:48] 10serviceops, 10Operations, 10Traffic, 10Patch-For-Review: Applayer services without TLS - https://phabricator.wikimedia.org/T210411 (10ema) [13:18:56] As a rule I don't need this (I hope) [13:20:40] I need to collect some profile data using go tool pprof, and the version installed on deploy1001 doesn't support ca-signed certs [13:21:44] urandom: you can rollback the change later, but it would be easier to track and understand if you add a debug: disableTLS flag in your chart [13:22:03] while if you want to remove the secret it also involves one of us changing the private repo :) [13:22:19] K [13:27:42] fsero: anything similar I can use as an example? [13:28:56] mmm not that i am aware right now [13:29:03] if i find something ill send it your way [13:29:08] k [13:29:11] thanks [13:34:58] 10serviceops, 10Operations, 10Traffic, 10Wikidata, and 4 others: [Task] move wikiba.se webhosting to wikimedia cluster - https://phabricator.wikimedia.org/T99531 (10abian) [15:10:48] 10serviceops, 10Operations, 10Traffic, 10Wikidata, and 4 others: [Task] move wikiba.se webhosting to wikimedia cluster - https://phabricator.wikimedia.org/T99531 (10BBlack) As noted in T155359 - WMDE has moved the hosting of this to some other platform, including the DNS hosting (and we never had the whois... [20:49:18] anyone know why https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/530158 wouldn't result in differences for helmfile? [20:49:50] https://www.irccloud.com/pastebin/89xIdAla/