[01:39:15] 10serviceops, 10observability, 10Developer Productivity, 10Patch-For-Review, 10Performance-Team (Radar): The "Server: mw•" response header is missing on mwmaint/noc.wm.o - https://phabricator.wikimedia.org/T255629 (10Dzahn) @Krinkle This works now: ` [mwmaint1002:/] $ curl -s --head localhost | grep... [06:52:28] <_joe_> jayme, akosiaris confirmed, the problem is helm diff [06:53:17] _joe_: so we need an update there as well? [06:53:23] <_joe_> yes, definitely [06:53:38] <_joe_> in theory, we should have the latest helm 2.x, helm-diff and helmfile [06:54:09] helm is prepared ;-) https://gerrit.wikimedia.org/r/c/operations/debs/helm/+/616065 [06:54:26] <_joe_> I'm not sure if helm-diff 3.x works with helm 2.x [06:55:21] my guess was no (https://phabricator.wikimedia.org/T251305) [06:55:34] <_joe_> guess or you tried? [06:58:30] <_joe_> because I just ran "helm plugin install" from helm 2.15 and it downloaded helm-diff 3.1.2 :P [06:58:37] <_joe_> and it works :P [07:01:09] nice. I was just guessing because of the versioning scheme they use [07:02:03] So happy packaging day, then :-) [07:03:45] <_joe_> yes, have fun! [07:03:53] eheh [07:03:54] <_joe_> :P [07:04:10] <_joe_> I mean, if you prefer to build mcrouter and envoy [07:04:22] <_joe_> I'm happy to build helm-diff [07:07:27] Yeah, those are easy to build ofc. Unfortunately helm-diff and helmfile don't use the "vendor-in-upstream-import" approach of envoy, helm2 and helm3. I think they both do vendoring in debian patches instead [07:12:35] <_joe_> ewww [07:12:43] <_joe_> let's fix that I guess [07:12:59] <_joe_> or not, up to you [07:15:34] good morning ;) [07:16:39] _joe_ and akosiaris: I have a couple pending changes to improve the python build base images, the aim is to reuse previously build wheels when they are matching the requirements. That is to avoid rebuilding them entirely ;) [07:16:47] https://gerrit.wikimedia.org/r/c/operations/docker-images/production-images/+/605653 [07:16:58] I wrote that for the deployment of zuul and it seems to work [07:17:04] I mean, it definitely work [07:17:25] eventually I would like to avoid us having to copy paste the same Makefile in each of the /deploy repositories but that will be for later [07:17:49] <_joe_> eventually I would like to avoid us having to use scap3 [07:18:02] <_joe_> but yes, sadly right now I have no time to look at it [07:18:53] 10serviceops, 10Prod-Kubernetes, 10Release Pipeline, 10Patch-For-Review: Refactor our helmfile.d dir structure for services - https://phabricator.wikimedia.org/T258572 (10JMeybohm) [07:18:55] 10serviceops, 10Prod-Kubernetes, 10Patch-For-Review: helm2 version string breaks recent helmfile versions - https://phabricator.wikimedia.org/T258773 (10JMeybohm) [07:25:20] _joe_: yeah...I was thinking to change those to to use the "vendor-in-upstream-import" approach as well (to have them all use the same workflow). What do you think akosiaris (you touched helmfile last, I recall) [07:25:54] <_joe_> I'm confident alex doesn't mind as long as he doesn't have to touch it himself [07:25:56] <_joe_> :P [07:25:57] ahahaha [07:26:15] that's true [07:26:31] jayme: if it suits you better, fine by me. [07:26:42] I think we should anyways aim for something more than one person is okay with :P [07:27:16] I am ok with both, hence the "fine by me" [07:27:38] ack [07:31:43] _joe_: it is really just about adding '-f /wheels-cache' to the pip wheel command :] [08:12:17] <_joe_> jayme: you know I do agree with that approach [08:19:58] <_joe_> ok, in the meantime, I'll add the service proxy to termbox I think. [08:30:40] * jayme excited [09:21:57] 10serviceops, 10Patch-For-Review: Investigate outgoing discarded packets in the codfw kubernetes cluster - https://phabricator.wikimedia.org/T226237 (10akosiaris) 05Open→03Stalled Stalled in the hope that I 'll have some time in the next few weeks to git into the kernel source again and figure out why thos... [10:25:30] is cluster-helmfile.sh working at the moment? i'm following https://wikitech.wikimedia.org/wiki/Pipeline_admin#Adding_a_new_namespace_and_service. and it looks like helmfile/helm is just trying to connect to k8s on localhost [10:40:03] <_joe_> akosiaris: ^^ [10:40:37] <_joe_> hnowlan: not sure that sets kubeconfig correctly [10:41:44] yeah, the .hfenv files are invalid it seems: `kube_env "admin" "staging"` [10:41:59] <_joe_> hnowlan: uh? [10:42:08] <_joe_> that should work [10:42:46] <_joe_> oh maybe you can't read /etc/kubernetes/admin-staging.config [10:43:00] <_joe_> yeah you need to use sudo for that [10:43:12] ahh, it's because I'm doing sudo -s [10:43:18] my bad [10:44:48] also needed sudo but that's a given [10:52:43] my helmfile diff in admin for staging is showing increases in quota for all services (related to https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/610800/1/helmfile.d/admin/staging/namespace/cluster.yaml cc akosiaris). Is that safe to apply? [10:57:09] <_joe_> I'd wait for alex, he's probably at lunch now [11:00:00] will do [12:01:43] 10serviceops, 10Prod-Kubernetes, 10Patch-For-Review: helm2 version string breaks recent helmfile versions - https://phabricator.wikimedia.org/T258773 (10JMeybohm) 05Open→03Resolved helm was updated: 2.16.7-2 -> 2.16.9-1 on chartmuseum*, contint*, deploy [12:01:45] 10serviceops, 10Prod-Kubernetes, 10Release Pipeline, 10Patch-For-Review: Refactor our helmfile.d dir structure for services - https://phabricator.wikimedia.org/T258572 (10JMeybohm) [13:55:37] hnowlan: ah yes, I 've been meaning to fix that, thanks for the ping. Lemme do so [14:02:22] <_joe_> jayme: so, I need to bump a chart, should I do the repo index stuff or just commit the new chart number? [14:03:02] _joe_: just increment the version in Chart.yaml and you will be fine [14:04:08] you didn't read the announcement :-P [14:05:32] <_joe_> no, I didn't read wikitech-l today still :) [14:09:53] _joe_: maybe you want to include the tls-sha256sum fix-thing as well when bumping? [14:10:06] <_joe_> jayme: good idea, let's merge it first [14:10:21] _joe_: will rebase in a sec [14:11:27] <_joe_> jayme: I can take care of it, I have more text-template awesome to fix [14:12:04] _joe_: okay, thanks [14:31:00] 10serviceops, 10Operations, 10Platform Engineering, 10Release Pipeline, and 6 others: Kask functional testing with Cassandra via the Deployment Pipeline - https://phabricator.wikimedia.org/T224041 (10akosiaris) >>! In T224041#6353152, @jeena wrote: > We attempted to run the tests using CI, but ran into err... [14:37:43] 10serviceops, 10Operations, 10Platform Engineering, 10Release Pipeline, and 6 others: Kask functional testing with Cassandra via the Deployment Pipeline - https://phabricator.wikimedia.org/T224041 (10akosiaris) >>! In T224041#6353283, @jeena wrote: > So I re-tried installing the chart on the ci cluster tod... [14:39:14] hnowlan: I 've synced all clusters and that also did create the api-gateway namespace so you should be good to go [14:41:05] akosiaris: thanks! [21:50:00] 10serviceops, 10Mobile-Content-Service, 10Page Content Service, 10Product-Infrastructure-Team-Backlog, 10Patch-For-Review: Migrate mobileapps to k8s and node 10 - https://phabricator.wikimedia.org/T218733 (10Mholloway) This is now tracking for Product Infrastructure. [21:50:33] 10serviceops, 10Mobile-Content-Service, 10Page Content Service, 10Product-Infrastructure-Team-Backlog, 10Patch-For-Review: Migrate mobileapps to k8s and node 10 - https://phabricator.wikimedia.org/T218733 (10Mholloway) a:05Mholloway→03None [21:59:53] 10serviceops, 10Operations: httpbb: Mapping between tests and hosts - https://phabricator.wikimedia.org/T259665 (10RLazarus) p:05Triage→03Medium