[09:47:31] on boron the docker-report-releng-images service failed for docker-registry.wikimedia.org/releng/bazel:0.4.0, the rest are fine, is that a known issue, worth opening a task? [09:59:15] haven't heard of it before [10:01:04] moritzm: we use that image to build Gerrit iirc [10:03:15] I can open a task, but unsure which projects to add? [10:04:01] moritzm: #continuous-integration-infrastructure [10:04:19] ack [10:04:25] we should also garbage collect the containers we no more need [10:04:38] but it is a bit of a pain to even list the ones we still rely on :-( [10:05:45] created https://phabricator.wikimedia.org/T248963 [10:11:38] moritzm: you might want details as to what that "docker-report-releng-images" is? I have never heard of it ;D [10:13:19] It updates the docker registry, probably something changed with the image so that the reporting now fails, not sure [10:18:09] moritzm: updates? is that attempting to rebuild all images and report on failling to build one? [10:19:46] it scrapes the docker registry for existing and reports all current images to debmonitor [13:01:04] back, lunch with kids at home is epic ;) [13:02:07] moritzm: for https://phabricator.wikimedia.org/T248963 , I have no idea what is going there. I would need more details, I have no clue what to act on [13:37:48] on another topic, I need a container to build wheels using python2 (that is to migrate Zuul to scap) https://gerrit.wikimedia.org/r/#/c/operations/docker-images/production-images/+/580128/ [13:46:52] 10serviceops, 10MediaWiki-General, 10Operations, 10Patch-For-Review, 10Service-Architecture: Use envoy for TLS termination on the appservers - https://phabricator.wikimedia.org/T247389 (10Joe) Both appserver and api canaries now use TLS termination. [13:59:08] _joe_: that's cool! ^ (also i'm about to do the decom of those 5 old eqiad appservers now) [14:17:43] I guess that now (they) use envoy for TLS termination? [14:20:24] 10serviceops, 10Operations, 10Patch-For-Review: decom old appservers in eqiad - https://phabricator.wikimedia.org/T247780 (10ops-monitoring-bot) Icinga downtime for 2:00:00 set by dzahn@cumin1001 on 5 host(s) and their services with reason: decom ` mw[1254-1258].eqiad.wmnet ` [14:21:04] vgutierrez: yes, envoy [14:21:19] yup, the comment has been edited :) [15:21:14] 10serviceops: buster-nodejs10-devel has an npm/node version mismatch - https://phabricator.wikimedia.org/T248928 (10Mholloway) The node/npm version combo packaged for Buster causes npm to throw this warning. Practically speaking, I have never seen an actual negative consequence of using them together. [15:29:44] 10serviceops, 10Operations, 10Patch-For-Review: decom old appservers in eqiad - https://phabricator.wikimedia.org/T247780 (10ops-monitoring-bot) Icinga downtime for 1 day, 0:00:00 set by dzahn@cumin1001 on 5 host(s) and their services with reason: decom ` mw[1254-1258].eqiad.wmnet ` [15:36:23] 10serviceops, 10Operations, 10Patch-For-Review: decom old appservers in eqiad - https://phabricator.wikimedia.org/T247780 (10Dzahn) 05Stalled→03Open [15:36:41] 10serviceops, 10Operations, 10Patch-For-Review: decom old appservers in eqiad - https://phabricator.wikimedia.org/T247780 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: `mw[1254-1258].eqiad.wmnet` - mw1254.eqiad.wmnet (**PASS**) - Downtimed host on Icinga... [15:53:07] akosiaris: _joe_: rlazarus: would it be useful to setup a meeting this week to look at capex figures together? [15:53:59] I think this cycle I'm mostly just following along to learn how we do that -- so, no scheduling input from me, but I'll join whatever meeting we have [15:59:13] 10serviceops, 10Operations, 10Patch-For-Review: decom old appservers in eqiad - https://phabricator.wikimedia.org/T247780 (10Dzahn) All mw servers in the [https://netbox.wikimedia.org/dcim/racks/39/ rack D5] are now decom'ed. There are a few non-mw servers in that rack that were unaffected but besides those... [19:00:30] 10serviceops: Grant Access to Logstash to Peter(peter.ovchyn@speedandfunction.com) - https://phabricator.wikimedia.org/T249037 (10AMooney) [21:05:17] 10serviceops, 10LDAP-Access-Requests, 10Operations: Grant Access to Logstash to Peter(peter.ovchyn@speedandfunction.com) - https://phabricator.wikimedia.org/T249037 (10Reedy)