[08:45:23] 10serviceops, 10SRE, 10Patch-For-Review: decom 44 eqiad appservers purchased on 2016-04-12/13 (mw1261 through mw1301) - https://phabricator.wikimedia.org/T280203 (10fgiunchedi) p:05Triage→03Medium [09:05:38] 10serviceops, 10SRE, 10Patch-For-Review: upgrade conf2* servers to stretch - https://phabricator.wikimedia.org/T271573 (10JMeybohm) a:03JMeybohm etcd cluster is set up now on conf200[4,5,6] although I had some trouble setting it up and I do not yet know why: After the initial puppet runs, the ectd's rejec... [09:12:39] 10serviceops, 10SRE, 10Patch-For-Review: upgrade conf2* servers to stretch - https://phabricator.wikimedia.org/T271573 (10jcrespo) Do you think, with the work done, we could drop support of jessie bacula backups (only etcd cluster was pending with jessie)? [09:16:13] 10serviceops, 10SRE, 10Patch-For-Review: upgrade conf2* servers to stretch - https://phabricator.wikimedia.org/T271573 (10JMeybohm) >>! In T271573#7008274, @jcrespo wrote: > Do you think, with the work done, we could drop support of jessie bacula backups (only etcd cluster was pending with jessie)? If those... [13:56:31] 10serviceops, 10SRE, 10Patch-For-Review: upgrade conf2* servers to stretch - https://phabricator.wikimedia.org/T271573 (10JMeybohm) The tlsproxy currently serves a certificate not valid for conf200[4,5,6] (Prometheus errors with: `Get https://conf2004:4001/metrics: x509: certificate is valid for conf2001.cod... [15:37:18] o/ [15:37:38] i get occaisional 503 errors from eventgate-logging-external.svc.eqiad.wmnet [15:38:09] oh wait hang on, just realized i was looking at staging logs [15:57:37] 10serviceops, 10Analytics, 10Event-Platform, 10SRE, 10Patch-For-Review: DRY kafka broker declaration in helmfiles - https://phabricator.wikimedia.org/T253058 (10Ottomata) [16:37:47] 10serviceops, 10Analytics, 10Event-Platform, 10SRE, 10Patch-For-Review: DRY kafka broker declaration in helmfiles - https://phabricator.wikimedia.org/T253058 (10Ottomata) Actually, I'm not sure even just doing LVS would help here. The helmfiles networkpolicy explicitly lists IP addresses that the servic... [16:56:34] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Jdforrester-WMF) [23:34:18] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Dzahn) [23:39:11] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for host... [23:39:16] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for host... [23:41:01] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for host... [23:52:09] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Dzahn) [23:53:28] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Dzahn) Remaining 3 special cases kept on stretch now reimaged to buster as well. Decom'ed mwdebug1... [23:58:19] 10serviceops, 10Performance-Team, 10WikimediaDebug: create mwdebug1003 - ganeti VM with buster and appserver role - https://phabricator.wikimedia.org/T267248 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: `mwdebug1003.eqiad.wmnet` - mwdebug1003.eqiad.wmnet (**P...