[00:16:05] 10serviceops, 10Graphoid, 10Platform Engineering, 10SRE: Final undeploy for graphoid - en.wiki - https://phabricator.wikimedia.org/T271495 (10Jdforrester-WMF) 05Open→03Resolved [00:16:10] 10serviceops, 10Graphoid, 10SRE, 10MW-1.35-notes (1.35.0-wmf.34; 2020-05-26), 10Platform Engineering (Icebox): Undeploy graphoid - https://phabricator.wikimedia.org/T242855 (10Jdforrester-WMF) [00:25:40] 10serviceops, 10Graphoid, 10SRE, 10MW-1.35-notes (1.35.0-wmf.34; 2020-05-26), and 2 others: Undeploy graphoid - https://phabricator.wikimedia.org/T242855 (10Jdforrester-WMF) [01:02:34] 10serviceops, 10Release-Engineering-Team-TODO, 10SRE, 10Patch-For-Review, and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1266.eqiad.wmnet'] ` and were **ALL** successful. [01:08:00] 10serviceops, 10Release-Engineering-Team-TODO, 10SRE, 10Patch-For-Review, and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1276.eqiad.wmnet'] ` and were **ALL** successful. [01:11:41] 10serviceops, 10Release-Engineering-Team-TODO, 10SRE, 10Patch-For-Review, and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1267.eqiad.wmnet'] ` and were **ALL** successful. [01:12:14] 10serviceops, 10Release-Engineering-Team-TODO, 10SRE, 10Patch-For-Review, and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1277.eqiad.wmnet'] ` and were **ALL** successful. [01:41:43] 10serviceops, 10Release-Engineering-Team-TODO, 10SRE, 10Patch-For-Review, and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Dzahn) 4 more servers have been upgraded to buster: mw1266, mw1267 (appserver) mw1276, mw1277 (API server) are now... [02:23:36] 10serviceops: "envoy-tls-local-proxy" image on docker-registry.wikimedia.org has no tags - https://phabricator.wikimedia.org/T271381 (10Legoktm) Thanks for the explanation, should I dupe this to the existing task or leave it open with #upstream to track https://github.com/docker/distribution/issues/2747 ? [07:51:26] 10serviceops: "envoy-tls-local-proxy" image on docker-registry.wikimedia.org has no tags - https://phabricator.wikimedia.org/T271381 (10JMeybohm) [07:51:34] 10serviceops, 10Release Pipeline, 10Release-Engineering-Team, 10SRE, 10User-brennen: Remove obsoleted docker images - https://phabricator.wikimedia.org/T242604 (10JMeybohm) [07:52:05] 10serviceops: "envoy-tls-local-proxy" image on docker-registry.wikimedia.org has no tags - https://phabricator.wikimedia.org/T271381 (10JMeybohm) I prefer to dupe for more context. Hope that's fine with you. [08:31:55] 10serviceops, 10Release Pipeline, 10Release-Engineering-Team, 10SRE, and 2 others: Remove obsoleted docker images - https://phabricator.wikimedia.org/T242604 (10JMeybohm) [16:03:29] 10serviceops, 10SRE, 10Wikimedia-production-error: PHP7 corruption reports in 2020-2021 (Call on wrong object, etc.) - https://phabricator.wikimedia.org/T245183 (10hashar) @jijiki thanks for the investigation! We were kind of wondering whether the Apache reload might have triggered the opcache issue which i... [16:46:01] 10serviceops, 10SRE, 10Patch-For-Review, 10cloud-services-team (Kanban): Upgrade labweb servers to buster - https://phabricator.wikimedia.org/T269004 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by andrew on cumin1001.eqiad.wmnet for hosts: ` ['labweb1001.wikimedia.org'] ` The log can be fou... [18:06:52] 10serviceops, 10SRE, 10Patch-For-Review, 10cloud-services-team (Kanban): Upgrade labweb servers to buster - https://phabricator.wikimedia.org/T269004 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['labweb1001.wikimedia.org'] ` and were **ALL** successful. [18:13:40] we should upgrade conf2* from configcluster to configcluster_stretch... right? [18:14:43] just noticed on conf2001 when looking at something else because puppet talks about missing base_packages.jessie nowadays [18:28:38] 10serviceops, 10SRE: upgrade conf2* servers to stretch - https://phabricator.wikimedia.org/T271573 (10Dzahn) [18:37:53] 10serviceops, 10SRE: upgrade conf2* servers to stretch - https://phabricator.wikimedia.org/T271573 (10Dzahn) [18:40:25] 10serviceops, 10SRE: upgrade conf2* servers to stretch - https://phabricator.wikimedia.org/T271573 (10Dzahn) The differences between the jessie and the stretch role are that the latter uses `etcdv3` vs `etcd` and additionally includes `zookeeper` profiles. Additionally the old role has this code: ` 5... [18:55:52] 10serviceops, 10SRE: upgrade conf2* servers to stretch - https://phabricator.wikimedia.org/T271573 (10elukey) @Dzahn I can help with the zookeeper migration, it should be doable one host at the time without too many issues, but it needs to be done with care. The work for etcd might be more complicated, but it... [19:51:32] 10serviceops, 10Release-Engineering-Team-TODO, 10SRE, 10Patch-For-Review, and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Andrew) [19:52:24] 10serviceops, 10SRE, 10Patch-For-Review, 10cloud-services-team (Kanban): Upgrade labweb servers to buster - https://phabricator.wikimedia.org/T269004 (10Andrew) 05Open→03Resolved Getting Horizon on buster was a lot more trouble than I expected but this is done now. [21:14:37] 10serviceops, 10SRE: improve mw maintenance server switch over and discovery names - https://phabricator.wikimedia.org/T265936 (10Dzahn) The second part, having the inactive warning in MOTD is already done .. I see now that I am looking at it again: ` 115 # T199124 116 $motd_ensure = $ensure ? { 117... [21:17:39] 10serviceops, 10SRE: improve mw maintenance server switch over and discovery names - https://phabricator.wikimedia.org/T265936 (10Dzahn) [21:38:10] 10serviceops, 10SRE: improve mw maintenance server switch over and discovery names - https://phabricator.wikimedia.org/T265936 (10Dzahn) After revisting this today I think it can be splt into 3 separate parts: (cc: @rlazarus @Joe a) allow multiple maintenance servers per DC without enabling jobs on more than... [21:47:17] 10serviceops, 10SRE, 10conftool, 10Datacenter-Switchover: Disable maintenance scripts via conftool - https://phabricator.wikimedia.org/T266717 (10Dzahn) T265936 is partially a duplicate of this ticket but also adds the part that maintenance servers are web hosts for https://noc.wikimedia.org. Last switch-... [21:48:52] 10serviceops, 10SRE: make noc.wikimedia.org active/active (was: improve mw maintenance server switch over and discovery names) - https://phabricator.wikimedia.org/T265936 (10Dzahn)