[08:17:33] 10serviceops, 10Dumps-Generation, 10Platform Engineering, 10SRE, and 2 others: Upgrade snapshot hosts to Buster - https://phabricator.wikimedia.org/T269377 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by ariel on cumin1001.eqiad.wmnet for hosts: ` snapshot1005.eqiad.wmnet ` The log can be fo... [08:32:38] 10serviceops, 10Prod-Kubernetes, 10Patch-For-Review: Check/Rebuild all docker-pkg build docker images running on kubernetes - https://phabricator.wikimedia.org/T274254 (10JMeybohm) [08:53:54] 10serviceops, 10Dumps-Generation, 10Platform Engineering, 10SRE, and 2 others: Upgrade snapshot hosts to Buster - https://phabricator.wikimedia.org/T269377 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['snapshot1005.eqiad.wmnet'] ` and were **ALL** successful. [09:12:50] 10serviceops, 10Dumps-Generation, 10Platform Engineering, 10SRE, and 2 others: Upgrade snapshot hosts to Buster - https://phabricator.wikimedia.org/T269377 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by ariel on cumin1001.eqiad.wmnet for hosts: ` snapshot1006.eqiad.wmnet ` The log can be fo... [09:18:24] 10serviceops, 10Prod-Kubernetes, 10Patch-For-Review: Rebuild all blubber build docker images running on kubernetes - https://phabricator.wikimedia.org/T274262 (10JMeybohm) [09:50:42] 10serviceops, 10Dumps-Generation, 10Platform Engineering, 10SRE, and 2 others: Upgrade snapshot hosts to Buster - https://phabricator.wikimedia.org/T269377 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['snapshot1006.eqiad.wmnet'] ` and were **ALL** successful. [09:57:14] 10serviceops, 10Dumps-Generation, 10Platform Engineering, 10SRE, and 2 others: Upgrade snapshot hosts to Buster - https://phabricator.wikimedia.org/T269377 (10ArielGlenn) I was not going to re-image snapshot1005 and 6 because their replacements were due to have come in, but the boxes have not arrived yet a... [12:24:39] 10serviceops, 10Analytics, 10SRE, 10observability, 10cloud-services-team (Kanban): hosts failing puppet compile due to missing secrets - https://phabricator.wikimedia.org/T274392 (10MoritzMuehlenhoff) [12:36:18] 10serviceops, 10WikimediaDebug, 10Patch-For-Review, 10Performance-Team (Radar): Convert mwdebug VMs to debian buster - https://phabricator.wikimedia.org/T274023 (10MoritzMuehlenhoff) I've set mwdebug1002 to "inactive" in conftool to unblock deployments. [12:38:40] 10serviceops, 10SRE, 10observability, 10Patch-For-Review, 10cloud-services-team (Kanban): hosts failing puppet compile due to missing secrets - https://phabricator.wikimedia.org/T274392 (10elukey) [13:09:04] 10serviceops, 10Prod-Kubernetes, 10Patch-For-Review: Rebuild all blubber build docker images running on kubernetes - https://phabricator.wikimedia.org/T274262 (10kostajh) [13:10:03] 10serviceops, 10Prod-Kubernetes, 10Patch-For-Review: Rebuild all blubber build docker images running on kubernetes - https://phabricator.wikimedia.org/T274262 (10kostajh) linkrecommendation / research-mwaddlink was handled via https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/660394 [17:41:17] 10serviceops, 10SRE, 10Patch-For-Review: upgrade conf2* servers to stretch - https://phabricator.wikimedia.org/T271573 (10jcrespo) I've sent: https://gerrit.wikimedia.org/r/c/operations/puppet/+/664313 Independently of the pace of upgrading, we should give some priority to generating fresh backups from the... [18:14:54] 10serviceops, 10DBA, 10SRE, 10Goal, 10Patch-For-Review: Strengthen backup infrastructure and support - https://phabricator.wikimedia.org/T229209 (10jcrespo)