[00:30:41] 10serviceops: "envoy-tls-local-proxy" image on docker-registry.wikimedia.org has no tags - https://phabricator.wikimedia.org/T271381 (10Legoktm) [00:35:16] 10serviceops: "envoy-tls-local-proxy" image on docker-registry.wikimedia.org has no tags - https://phabricator.wikimedia.org/T271381 (10Legoktm) "wikimedia-jessie" has the same issue, I know that image was purposefully removed though. [07:30:46] 10serviceops: "envoy-tls-local-proxy" image on docker-registry.wikimedia.org has no tags - https://phabricator.wikimedia.org/T271381 (10Joe) This is a problem of the docker registry itself - when you delete images and tags, a general reference to the image name remains, even if you removed all the tags - as is t... [08:48:17] 10serviceops: "envoy-tls-local-proxy" image on docker-registry.wikimedia.org has no tags - https://phabricator.wikimedia.org/T271381 (10JMeybohm) Yeah, the image was removed (as far as thats easily possible) as of T253396 See T242604 and https://wikitech.wikimedia.org/wiki/Docker#Deleting_an_image_(from_regist... [08:55:27] jayme: thanks a lot for the email! [08:57:21] sure! [09:27:00] <_joe_> jayme: I was thinking, we should probably use something like docker-registry.wikimedia.org/debian:stable as our 'seed image' [09:27:12] <_joe_> so that we can make the transition seamless for most base images [09:27:28] <_joe_> and manage manually only the ones we want to pin to a specific debian distribution [09:32:10] that sounds like a good idea to me [09:35:49] but we should probably only do so for leaf images, I guess [09:42:28] <_joe_> what do you mean? [09:43:57] that we should use this "moving-tag" only for images where we are kind-of-sure that they are not used as base image somewhere else [09:45:01] <_joe_> well I was proposing to do so for the base image, instead [09:45:37] <_joe_> the idea being - there are some images that depend on a specific distro (like - the node or python ones) while others don't, and can use this seed image [09:50:12] <_joe_> anyways, I'll proceed to update the seed image to buster, but I'm thinking the alternative would be to improve docker-pkg to admit more than one seed image [09:52:41] if I get that right that matches my expectation. So we set "seed_image: docker-registry.wikimedia.org/debian:stable", tag buster as debian:stable and every production-image that uses "FROM {{ seed_image }}" may be auto upgaded by publishing a new "debian:stable". Right? [09:53:55] So if a production-image does not want to be auto-updated at some point/does want to have a static base it needs to specify a tag instead of "{{ seed_image }}" [10:04:18] <_joe_> yes [10:05:20] <_joe_> but maybe we have better options, I see downsides of that approach [10:06:02] <_joe_> so I might just go in another direction and allow multiple seed images in docker-pkg, which btw I thought we already did. [10:24:36] _joe_: do we have some kind of "process" to deal with changes/replacements to/of grafana dashboards? [10:24:59] <_joe_> uh not that I'm aware of, but ask the observability folks :) [10:26:36] but aren't we the ower of particular dashboards? In the actual case I would like to propose to replace a specific k8s dashboard with a (IMHO :)) improved version [10:27:10] I would have expected thats our business then and o11y just provides infrastructure [10:41:53] <_joe_> yes [10:42:21] <_joe_> so the process is you send an email to serviceops and ask for silent assent :) [10:42:42] <_joe_> sorry, I thought you had a deeper issue than that [10:43:39] eheh, not yet :) [10:43:41] thanks [11:12:45] 10serviceops, 10SRE, 10Traffic: Upgrade envoyproxy to 1.16.2 - https://phabricator.wikimedia.org/T271407 (10Vgutierrez) [11:13:36] 10serviceops, 10SRE, 10Traffic: Upgrade envoyproxy to 1.16.2 - https://phabricator.wikimedia.org/T271407 (10Vgutierrez) p:05Triage→03Medium [13:53:30] 10serviceops, 10SRE, 10Traffic: Upgrade envoyproxy to 1.16.2 - https://phabricator.wikimedia.org/T271407 (10Vgutierrez) [14:01:40] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Calico 3.17.1 kube-controllers fail to reach apiserver at startup - https://phabricator.wikimedia.org/T271422 (10JMeybohm) [14:01:53] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Calico 3.17.1 kube-controllers fail to reach apiserver at startup - https://phabricator.wikimedia.org/T271422 (10JMeybohm) p:05Triage→03Low [15:21:40] <_joe_> jayme/rzl: anything specific we must let platform know about? [15:22:13] _joe_: I'm not aware of anything in perticular [15:37:00] _joe_: if they're looking at Q3 OKRs we would still very much like to get rid of the mc redis cluster, and still can't without their help [15:37:26] but there's nothing new afaik [16:08:04] ahoy! We're planning on adding a new k8s service for the sockpuppet API. It's fairly simple stuff and the service itself is quite lightweight. Could someone review this please? https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/643721 [16:11:22] hnowlan: I can take a look tomorrow [16:14:09] thanks jayme! [17:14:01] <_joe_> yeah I'll take a look too [17:50:00] 10serviceops, 10Platform Engineering, 10SRE, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10jijiki) [18:31:30] 10serviceops, 10Platform Engineering, 10SRE, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` mc2027.codfw.wmnet ` The log can be found i... [18:31:44] 10serviceops, 10Platform Engineering, 10SRE, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` mc1027.eqiad.wmnet ` The log can be found i... [19:17:43] 10serviceops, 10Platform Engineering, 10SRE, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mc2027.codfw.wmnet'] ` and were **ALL** successful. [19:22:08] _joe_: we can finally remove the monogdb PHP extension from appservers now:) planning to merge a patch to absent it. and in another matter, ServerAdmin is dropped from prod apache and a second change to drop it from the docker-images repo as well is in Gerrit as requested [19:25:15] 10serviceops, 10Platform Engineering, 10SRE, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mc1027.eqiad.wmnet'] ` and were **ALL** successful. [19:39:38] <_joe_> oh great [19:40:02] <_joe_> yeah I've seen the production-images patch, didn't review it though, sorry [19:44:07] ok, cool. I was mostly not self-merging it to make sure there isn't some other step after merge in that repo. [19:47:59] re: mongodb module.. interesting, puppet compiler says it is only a change on mwdebug but not other mw. must be due to $profiling_ensure already absenting it everywhere else. ok, fine. cleanup got easier [19:48:41] 10serviceops, 10Platform Engineering, 10SRE, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10jijiki) [20:00:19] done. mwdebug hosts cleaned up and now this is empty: https://debmonitor.wikimedia.org/packages/php-mongodb [20:18:38] 10serviceops, 10Add-Link, 10GrowthExperiments-NewcomerTasks, 10Product-Infrastructure-Team-Backlog, and 2 others: Service operations setup for Add a Link project - https://phabricator.wikimedia.org/T258978 (10kostajh) @akosiaris picking up the thread on this from before the holiday break; IIRC there was so... [23:28:07] 10serviceops, 10Release-Engineering-Team-TODO, 10SRE, 10Patch-For-Review, and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` mw1266.eqiad... [23:40:29] 10serviceops, 10Graphoid, 10Platform Engineering, 10SRE: Final undeploy for graphoid - en.wiki - https://phabricator.wikimedia.org/T271495 (10Jseddon) [23:49:48] 10serviceops, 10Release-Engineering-Team-TODO, 10SRE, 10Patch-For-Review, and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1266.eqiad.wmnet'] ` Of which those **FAILED**: ` ['mw12... [23:50:23] 10serviceops, 10Release-Engineering-Team-TODO, 10SRE, 10Patch-For-Review, and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` mw1266.eqiad... [23:50:27] 10serviceops, 10Release-Engineering-Team-TODO, 10SRE, 10Patch-For-Review, and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1266.eqiad.wmnet'] ` Of which those **FAILED**: ` ['mw12... [23:51:13] 10serviceops, 10Release-Engineering-Team-TODO, 10SRE, 10Patch-For-Review, and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` mw1266.eqiad... [23:53:10] 10serviceops, 10Release-Engineering-Team-TODO, 10SRE, 10Patch-For-Review, and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` mw1276.eqiad... [23:54:53] 10serviceops, 10Release-Engineering-Team-TODO, 10SRE, 10Patch-For-Review, and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` mw1267.eqiad... [23:55:27] 10serviceops, 10Release-Engineering-Team-TODO, 10SRE, 10Patch-For-Review, and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` mw1277.eqiad...