[00:08:10] mutante: if I have to upgrade a single package on just a few hosts, should I do it with plain apt or is debdeploy still recommended? [00:08:45] I uploaded python3-docker-report, now I need it on the 4 registry hosts [00:09:48] legoktm: hmm. 4 hosts is still within "manually do it" for me [00:10:17] debdeploy means you would have to write some yaml [00:10:48] i'd do that if it was for more than 4 [00:11:31] another option is to run apt-get install via cumin .. but with -y so that it's not interactive [00:12:29] * legoktm nods [00:12:31] thanks [00:12:39] but yea.. 4 is kind of my limit for doing stuff directly.. i would split my terminal into 4 but not more than that [00:12:59] just like reimaging 4 at once to still be able to watch it [00:49:11] 10serviceops, 10MW-on-K8s, 10Release Pipeline, 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)): Request volume for Docker images and container filesystems on releases machines - https://phabricator.wikimedia.org/T272092 (10Dzahn) a:05dduvall→03Dza... [00:51:49] 10serviceops, 10MediaWiki-Containers, 10SRE, 10Patch-For-Review: Homepage for https://docker-registry.wikimedia.org - https://phabricator.wikimedia.org/T179696 (10Legoktm) a:03Legoktm [01:06:54] legoktm: re your comment about connecting to https://localhost. theoretically you could add all the registry individual host names to that certificate. then you could connect to https://registry1001.eqiad.wmnet from registry1001, to registry1002 from registry1002 and so on but then we are back to having to touch it each time a backend changes. also then which one would be served to the [01:07:00] public in ATS? So I guess it can be https://docker-registry.discovery.wmnet and doesn't matter if it's not the same one serving the site... as far as I understand the isue [01:07:55] at least "make it allow http" seems wrong with all the "switch internal stuff to https" effort [01:08:03] ok [01:08:19] but then it's also just within the host.. so let's hear j.oe :) [01:08:26] I think using https://docker-registry.discovery.wmnet is fine, in practice it should be identical since the registries should replicate [01:08:51] er not replicate, but be in sync [01:08:54] yea, I am thinking "well if there are 4 differen registries in theory.. but we have to pick one of them to serve to the public anyways" [01:09:17] maybe that is more something for monitoring..to check they are really the same [11:16:16] 10serviceops, 10MediaWiki-Containers, 10SRE, 10Patch-For-Review: Homepage for https://docker-registry.wikimedia.org - https://phabricator.wikimedia.org/T179696 (10akosiaris) >>! In T179696#6760378, @Legoktm wrote: > Got pretty close, one last sticking point is that `docker_report` hardcodes connecting to t... [11:16:51] 10serviceops, 10MediaWiki-Containers, 10SRE, 10Patch-For-Review: Homepage for https://docker-registry.wikimedia.org - https://phabricator.wikimedia.org/T179696 (10Joe) >>! In T179696#6760378, @Legoktm wrote: > Got pretty close, one last sticking point is that `docker_report` hardcodes connecting to the reg... [14:02:24] 10serviceops, 10MediaWiki-Containers, 10SRE, 10Patch-For-Review: Homepage for https://docker-registry.wikimedia.org - https://phabricator.wikimedia.org/T179696 (10Joe) For the record, I'm running the script as follows on registry1001: ` /usr/local/bin/registry-homepage-builder docker-registry.wikimedia.or... [15:20:38] jayme: heya, just merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/656253 [15:20:55] but realized something...i can't include these in values.yaml files, can I? [15:20:57] I [15:21:10] I'd have to modify the chart templates to somehow get them dynamically? [15:21:13] right? [15:22:12] in a meeting currently. But no, you can only include those in helmfile.yaml (should be already) [15:56:00] hm but how do I include/refer the value in values .yaml [15:56:30] e.g. [15:56:30] https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/refs/heads/master/helmfile.d/services/eventstreams/values-eqiad.yaml [15:56:36] i want to set that to the value of [15:56:47] kafka_clusters.main-eqiad.brokers_list.ssl [16:01:12] jayme: ^ [16:06:14] <_joe_> ottomata: helmfile does deep merge [16:07:33] right, i understand that the values in the defaults general-* files will be present, but my chart currently doesn't know anything about e.g. kafka_clusters [16:07:55] it accepts the kafka config from values.yaml as e.g. metadata.broker.,list [16:07:59] so i want to do somethin glike [16:08:49] metadata.broker.list: {{ .Values.kafka_clusters.main-eqiad.broker_list.ssl }} [16:08:55] but, I can't do that in the values.yaml files, right? [16:09:03] I'd have to modify the chart templates to know what to do? [16:09:24] 10serviceops, 10MediaWiki-Containers, 10SRE, 10Patch-For-Review: Homepage for https://docker-registry.wikimedia.org - https://phabricator.wikimedia.org/T179696 (10Joe) For the record, I got the script running, by using ` /usr/local/bin/registry-homepage-builder docker-registry.wikimedia.org /root/homepag... [16:10:40] _joe_: ^^ ? [16:12:46] <_joe_> ottomata: it's possible to also use templated value files in helmfile, but it's kinda thorny and non-obvious and I'd prefer to let any logic live in the templates in the chart [16:13:19] <_joe_> it's already complicated enough to reconstruct the whole hierarchy of yaml values without templating in the way [16:13:43] agree [16:13:43] ok [16:13:59] I'll make the charts take a kafka cluster name parameter then [16:14:24] kinda annoying because now the charts refer to a value set in defaults rendered only on deployment host... hm [16:14:29] i'd guess I'd need some dummy values... [16:14:33] lke we do for tls [16:15:30] sorry - yeah. In that case (different key) you will have to ammend your charts to consume that and provide a sensible default in the charts values.yaml for it to pass ci/render properly [16:15:37] right [16:15:52] jayme: is this a good idea? or soudl I abandon this approach. sorry I didn't think about this before [16:15:56] should* [16:16:15] the other idea was to somehow get LVS or round robin DNS or something for kafka brokers [16:17:05] I think it's as you just change a bit of your chart (and you're the only user of it I guess :)). [16:18:09] but ofc. having that handeled automatically by infrastructure (load balancer or alike) would be better because in that case you will not have to re-deploy your services when kafka typoloy changes [16:18:30] if your service is capable of handling that transparently [16:23:05] the kafka client will actually only use the this list on init [16:23:14] it just uses the list to ask kafka about its topology [16:23:19] then the clients use the response to actually do real work [16:23:50] so it only really matters on service restart anyway.... [16:24:11] actually that's not true for eventstreams, that one has new kafka clients created and connecting during runtime [16:24:30] ok jayme i think i'll revisit the lvs idea [16:24:42] probably will revert that puppet patch eventtually if that works out [17:06:26] 10serviceops, 10MediaWiki-Containers, 10SRE, 10Patch-For-Review: Homepage for https://docker-registry.wikimedia.org - https://phabricator.wikimedia.org/T179696 (10Legoktm) >>! In T179696#6761286, @Joe wrote: > Given the banner page we're creating is for use by the public, I think it can simply run against... [17:29:41] 10serviceops, 10Analytics, 10Analytics-Kanban, 10Event-Platform, and 5 others: Set up internal eventstreams instance exposing all streams declared in stream config (and in kafka jumbo) - https://phabricator.wikimedia.org/T269160 (10elukey) Get up to deploying the service in staging, it seems working! Updat... [17:49:39] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [17:50:23] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [17:51:15] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [17:51:51] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [18:33:14] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2325.codfw.wmnet'] ` an... [18:33:48] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2316.codfw.wmnet'] ` an... [18:35:31] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2327.codfw.wmnet'] ` an... [18:36:07] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2329.codfw.wmnet'] ` an... [18:57:10] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [18:58:09] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [18:59:14] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [19:00:01] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [19:00:07] 10serviceops: mc1024 broke - replace it or remove it from configs - https://phabricator.wikimedia.org/T272078 (10jijiki) @Krinkle @aaron the gutter pool sets a max TTL of 600s to any key with a TTL over 600s, do you think it is fine to keep the gutter-pool substitute the missing server? [19:23:01] 10serviceops, 10Graphoid, 10Platform Engineering, 10SRE: Final undeploy for graphoid - en.wiki - https://phabricator.wikimedia.org/T271495 (10Jdlrobson) Opened T272530 with suggested high priority [19:39:02] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2331.codfw.wmnet'] ` an... [19:40:12] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2333.codfw.wmnet'] ` an... [19:40:45] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2335.codfw.wmnet'] ` an... [19:41:35] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2337.codfw.wmnet'] ` an... [20:41:43] 10serviceops, 10MW-on-K8s, 10Release Pipeline, 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)): Request volume for Docker images and container filesystems on releases machines - https://phabricator.wikimedia.org/T272092 (10Dzahn) New disks have been... [20:55:53] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [20:56:15] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [20:57:09] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [20:57:27] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [21:37:23] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2338.codfw.wmnet'] ` an... [21:37:57] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2339.codfw.wmnet'] ` an... [21:38:29] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2351.codfw.wmnet'] ` an... [21:39:02] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2353.codfw.wmnet'] ` an... [22:02:44] so much buster! :) [22:08:47] :) [22:22:14] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [22:23:55] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [22:24:11] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2355.codfw.wmnet'] ` Of... [22:24:23] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [22:24:49] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [22:25:05] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [22:27:03] 10serviceops, 10SRE: Upgrade docker-registry servers to Debian Buster - https://phabricator.wikimedia.org/T272550 (10Legoktm) [22:28:41] 10serviceops, 10SRE: Upgrade docker-registry servers to Debian Buster - https://phabricator.wikimedia.org/T272550 (10Dzahn) [23:03:28] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2355.codfw.wmnet'] ` an... [23:06:20] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2361.codfw.wmnet'] ` an... [23:06:59] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2359.codfw.wmnet'] ` an... [23:07:38] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2357.codfw.wmnet'] ` an... [23:08:53] 10serviceops, 10MediaWiki-Containers, 10SRE, 10Patch-For-Review: Homepage for https://docker-registry.wikimedia.org - https://phabricator.wikimedia.org/T179696 (10Legoktm) https://docker-registry.wikimedia.org/ ta-da Tested by: * `docker pull docker-registry.discovery.wmnet/wikimedia-buster:latest` on a... [23:26:52] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [23:27:28] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [23:28:07] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [23:28:36] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [23:51:18] 10serviceops, 10SRE: Upgrade docker-registry servers to Debian Buster - https://phabricator.wikimedia.org/T272550 (10Legoktm) Skimming the puppet role, there's: ` # this could be removed when buster or next debian includes a 2.7+ version apt::pin { 'strech_wikimedia_docker_registry_27': packag...