[01:41:11] 10serviceops, 10Parsoid: Parsoid deb: Error with apt-get update - https://phabricator.wikimedia.org/T242757 (10ssastry) >>! In T242757#5802816, @Dzahn wrote: > I'll assume it was literally the "Mirror sync in progress" (103.102.166.224 is eqsin). Indeed. Looks like it was some transient issue. Works for me no... [08:02:33] 10serviceops, 10Operations, 10Scap: Make canary wait time configurable - https://phabricator.wikimedia.org/T217924 (10jijiki) [08:03:35] 10serviceops, 10Operations, 10Release-Engineering-Team: Hundreds of tags for `wikimedia/mediawiki-core` image - https://phabricator.wikimedia.org/T242775 (10Joe) a:03Joe [08:04:14] 10serviceops, 10Operations, 10Release-Engineering-Team: Hundreds of tags for `wikimedia/mediawiki-core` image - https://phabricator.wikimedia.org/T242775 (10Joe) The total number of images present on the registry is 1003. I'm going to slowly remove most of the old ones in the coming week. [10:36:18] 10serviceops, 10Analytics, 10Product-Analytics (Kanban): Clarify multi-service instance concepts in helm charts and enable canary releases - https://phabricator.wikimedia.org/T242861 (10akosiaris) > Our new Helm chart templates were not originally developed to handle multi-service deployment charts On the c... [11:20:49] 10serviceops, 10Patch-For-Review: setup new, buster based, kubernetes etcd servers for staging/codfw/eqiad cluster - https://phabricator.wikimedia.org/T239835 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by akosiaris@cumin1001 for hosts: `etcd[1004-1006].eqiad.wmnet` - etcd1004.eqiad.wmnet... [12:47:55] 10serviceops, 10Operations, 10Patch-For-Review: No mw canary servers in codfw - https://phabricator.wikimedia.org/T242606 (10jijiki) Yeah, we need at least a total of 4 api and 4 app canary servers in codfw. In eqiad our canary app (5) and api (4) servers are in the same rack actually, we can spread them a b... [13:02:00] 10serviceops, 10Operations, 10Patch-For-Review: No mw canary servers in codfw - https://phabricator.wikimedia.org/T242606 (10MoritzMuehlenhoff) Agreed, I think for our uses of the canaries, rack redundancy is not a must, but would still be nice to have when re-adding canaries to codfw. [14:38:56] 10serviceops, 10Analytics, 10Product-Analytics (Kanban): Clarify multi-service instance concepts in helm charts and enable canary releases - https://phabricator.wikimedia.org/T242861 (10Ottomata) > Why mess with wmf.releasename Because wmf.releasename doesn't currently consider the service's name, only the... [14:39:52] akosiaris: ^ :) [15:55:34] heads up to serviceopsen since we don't have a meeting today -- I'm out tomorrow Friday for vacation, and Monday for a US holiday [16:02:14] nice! [16:28:17] 10serviceops, 10Analytics: Clarify multi-service instance concepts in helm charts and enable canary releases - https://phabricator.wikimedia.org/T242861 (10Neil_P._Quinn_WMF) [16:33:07] yup, in the tech dept meeting now [16:33:38] 10serviceops, 10Analytics, 10Analytics-Kanban: Clarify multi-service instance concepts in helm charts and enable canary releases - https://phabricator.wikimedia.org/T242861 (10Ottomata) [16:53:53] FYI Jakob and I are now bumping all the termboxen to use the new charts and image layouts [17:00:43] akosiaris: Hey; just about to helm file apply on eqiad and noticed there were un-applied changes: specifically we noticed that docker-registry as changed address to an internal one? [17:00:59] that doesn't seem to be the case on eqiad [17:01:27] tarrow: lemme have a look. eqiad cluster you say? [17:01:32] yep [17:02:21] like everything is not using docker-registry.discovery.wmnet instead of docker-registry.wikimedia.org [17:02:24] now* [17:02:33] yeah that's my change [17:02:37] cool [17:02:38] it's fine you can go on with it [17:02:54] cool! we're applying it now [17:03:02] it's so we can switch between the 2 DCs registries in case a local one breaks [17:03:11] ah! [17:03:13] gotcha! [17:06:19] change was in https://gerrit.wikimedia.org/r/#/c/operations/deployment-charts/+/552207/ fwiw. Didn't really deploy every instance of every service cause it's a noop [17:12:10] Great, I guess the change happened to have already been applied on codfw. We just noticed the difference and wanted to check we weren't mid some reigstry move or something [17:21:55] 10serviceops, 10Analytics, 10Analytics-Kanban: Clarify multi-service instance concepts in helm charts and enable canary releases - https://phabricator.wikimedia.org/T242861 (10akosiaris) >>! In T242861#5809439, @Ottomata wrote: >> Why mess with wmf.releasename > > Because wmf.releasename doesn't currently c... [17:44:36] 10serviceops, 10Proton, 10Patch-For-Review, 10Product-Infrastructure-Team-Backlog (Kanban): Profile proton memory usage for Helm chart - https://phabricator.wikimedia.org/T238830 (10akosiaris) @MSantos where were the tests run ? And how was the draft chart deployed? My reason for asking what that if this i... [18:20:11] 10serviceops, 10Performance-Team: Stack for shutdown/destruct fatals missing from php7-fatal-error.php logs - https://phabricator.wikimedia.org/T241097 (10Krinkle) 05Stalled→03Resolved [18:38:24] 10serviceops, 10Graphoid, 10Operations, 10Core Platform Team Workboards (Clinic Duty Team): Undeploy graphoid - https://phabricator.wikimedia.org/T242855 (10CCicalese_WMF) @WDoranWMF Is there work for CPT here past changing the RESTBase configuration? [18:39:05] 10serviceops, 10Analytics, 10Analytics-Kanban: Clarify multi-service instance concepts in helm charts and enable canary releases - https://phabricator.wikimedia.org/T242861 (10Ottomata) > My question is more on the line of why use wmf.releasename to identify the service to begin with. We can just use service... [18:40:01] 10serviceops, 10Proton, 10Patch-For-Review, 10Product-Infrastructure-Team-Backlog (Kanban): Profile proton memory usage for Helm chart - https://phabricator.wikimedia.org/T238830 (10MSantos) >>! In T238830#5810194, @akosiaris wrote: > @MSantos where were the tests run ? And how was the draft chart deployed... [20:17:59] 10serviceops, 10Analytics, 10Analytics-Kanban: Clarify multi-service instance concepts in helm charts and enable canary releases - https://phabricator.wikimedia.org/T242861 (10Ottomata) Too bad set based selectors [[ https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#service-and-replic... [21:55:39] 10serviceops, 10Analytics, 10Analytics-Kanban: Clarify multi-service instance concepts in helm charts and enable canary releases - https://phabricator.wikimedia.org/T242861 (10Ottomata) I just updated [[ https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/564052 | my patch ]]; I'll explain my new...