[00:05:00] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by legoktm on cumin1001.eq... [00:37:33] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1404.eqiad.wmnet'] ` an... [00:49:30] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2294.codfw.wmnet', 'mw22... [01:05:09] 10serviceops, 10decommission-hardware: decommission rdb200[34].eqiad.wmnet - https://phabricator.wikimedia.org/T273140 (10wkandek) [01:05:38] 10serviceops, 10decommission-hardware: decommission rdb100[56].eqiad.wmnet - https://phabricator.wikimedia.org/T273139 (10wkandek) [01:05:57] 10serviceops, 10decommission-hardware: decommission scb200[1234].eqiad.wmnet - https://phabricator.wikimedia.org/T273138 (10wkandek) [01:06:15] 10serviceops, 10decommission-hardware: decommission thumbor100[34].eqiad.wmnet - https://phabricator.wikimedia.org/T273137 (10wkandek) [01:06:35] 10serviceops, 10decommission-hardware: decommission scb100[1234].eqiad.wmnet - https://phabricator.wikimedia.org/T273136 (10wkandek) [01:08:23] 10serviceops, 10decommission-hardware: decommission thumbor200[12].codfw.wmnet - https://phabricator.wikimedia.org/T273141 (10wkandek) [01:09:00] 10serviceops, 10decommission-hardware: decommission thumbor200[12].codfw.wmnet - https://phabricator.wikimedia.org/T273141 (10wkandek) [01:09:23] 10serviceops, 10decommission-hardware: decommission rdb200[34].codfw.wmnet - https://phabricator.wikimedia.org/T273140 (10wkandek) [01:09:58] 10serviceops, 10decommission-hardware: decommission scb200[1234].codfw.wmnet - https://phabricator.wikimedia.org/T273138 (10wkandek) [01:10:30] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1268.eqiad.wmnet'] ` an... [01:12:55] 10serviceops, 10decommission-hardware: decommission francium.eqiad.wmnet - https://phabricator.wikimedia.org/T273142 (10wkandek) [01:14:53] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by legoktm on cumin1001.eq... [01:33:15] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for host... [01:58:30] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2288.codfw.wmnet', 'mw2290.codfw.wmnet',... [02:44:17] 10serviceops, 10SRE, 10Patch-For-Review: Migrate onhost memcached to use a unix socket - https://phabricator.wikimedia.org/T273115 (10Legoktm) p:05Triage→03Medium [02:50:48] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1268.eqiad.wmnet'] ` an... [05:38:51] 10serviceops, 10Performance-Team, 10Platform Engineering, 10Wikimedia-Rdbms, and 3 others: Determine and implement multi-dc strategy for ChronologyProtector - https://phabricator.wikimedia.org/T254634 (10Krinkle) a:03Krinkle [12:46:45] 10serviceops, 10SRE: upgrade conf2* servers to stretch - https://phabricator.wikimedia.org/T271573 (10jcrespo) Sorry to bother you, but do you happen to know when this upgrade will happen? I am not blocked by it (so please don't change any plans because of my question), but depending on when it is probably goi... [14:23:03] 10serviceops, 10Prod-Kubernetes, 10SRE, 10Kubernetes, 10User-fsero: Upgrade Calico - https://phabricator.wikimedia.org/T207804 (10JMeybohm) [14:23:06] 10serviceops, 10Prod-Kubernetes, 10SRE, 10Kubernetes, 10Patch-For-Review: Refactor calico deploy strategy - https://phabricator.wikimedia.org/T267653 (10JMeybohm) 05Resolved→03Open Looks like we're missing an RBAC rule for calico-node: ` 2021-01-28 14:17:19.139 [INFO][31] tunnel-ip-allocator/ipam.go... [15:19:58] 10serviceops, 10Prod-Kubernetes, 10SRE, 10Kubernetes, 10User-fsero: Upgrade Calico - https://phabricator.wikimedia.org/T207804 (10JMeybohm) [15:20:01] 10serviceops, 10Prod-Kubernetes, 10SRE, 10Kubernetes, 10Patch-For-Review: Refactor calico deploy strategy - https://phabricator.wikimedia.org/T267653 (10JMeybohm) 05Open→03Resolved fixed with helm chart version: 0.1.9 [17:32:08] 10serviceops, 10SRE, 10Wikimedia-General-or-Unknown, 10Patch-For-Review, 10User-jijiki: Remove pear/mail packages from WMF MW app servers - https://phabricator.wikimedia.org/T195364 (10jijiki) 05Open→03Resolved a:03jijiki Packages do not exists on mw* app servers anymore. [17:43:11] 10serviceops, 10decommission-hardware: decommission francium.eqiad.wmnet - https://phabricator.wikimedia.org/T273142 (10Dzahn) a:03Dzahn [17:46:56] 10serviceops, 10decommission-hardware: decommission francium.eqiad.wmnet - https://phabricator.wikimedia.org/T273142 (10Dzahn) the replacement of this, called htmldumper1001 still has this comment in site.pp ` 1230 # new host that needs to be turned over to service owner 1231 node 'htmldumper1001.eqiad.wmnet... [17:47:55] 10serviceops, 10decommission-hardware: decommission francium.eqiad.wmnet - https://phabricator.wikimedia.org/T273142 (10Dzahn) --> T245567 [18:07:36] 10serviceops, 10decommission-hardware, 10Patch-For-Review: decommission francium.eqiad.wmnet - https://phabricator.wikimedia.org/T273142 (10Dzahn) 05Open→03Stalled This server is still in production. Before decom can start this kind of switch needs to happen: https://gerrit.wikimedia.org/r/c/operations... [18:33:48] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [18:37:44] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [18:38:59] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [19:50:57] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1264.eqiad.wmnet'] ` an... [19:55:37] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2247.codfw.wmnet'] ` an... [19:59:03] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2223.codfw.wmnet'] ` an... [21:06:43] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by legoktm on cumin1001.eq... [21:48:54] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2284.codfw.wmnet', 'mw22... [23:12:30] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by legoktm on cumin1001.eq... [23:14:09] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [23:14:29] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia...