[00:00:35] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [00:40:35] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1366.eqiad.wmnet'] ` an... [00:43:13] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2265.codfw.wmnet'] ` an... [01:16:25] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1319.eqiad.wmnet'] ` an... [01:19:16] 10serviceops: create mwdebug1004, mwdebug2003 and mwdebug2004 as mediawiki::canary_appserver - https://phabricator.wikimedia.org/T274023 (10Dzahn) [01:21:21] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1313.eqiad.wmnet'] ` an... [01:21:35] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Dzahn) [01:22:12] 10serviceops: create mwdebug1004, mwdebug2003 and mwdebug2004 as mediawiki::canary_appserver - https://phabricator.wikimedia.org/T274023 (10Dzahn) [01:22:17] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Dzahn) [01:22:37] 10serviceops: create mwdebug1004, mwdebug2003 and mwdebug2004 as mediawiki::canary_appserver - https://phabricator.wikimedia.org/T274023 (10Dzahn) [01:23:13] 10serviceops, 10Performance-Team, 10WikimediaDebug: create mwdebug1003 - ganeti VM with buster and appserver role - https://phabricator.wikimedia.org/T267248 (10Dzahn) T274023 is for mwdebug1004, mwdebug2003, mwdebug2004 [01:23:25] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Dzahn) p:05Medium→03High [01:23:31] 10serviceops: create mwdebug1004, mwdebug2003 and mwdebug2004 as mediawiki::canary_appserver - https://phabricator.wikimedia.org/T274023 (10Dzahn) p:05Triage→03High [01:24:31] 10serviceops: create mwdebug1004, mwdebug2003 and mwdebug2004 as mediawiki::canary_appserver - https://phabricator.wikimedia.org/T274023 (10Dzahn) [01:31:18] 10serviceops: create mwdebug1004, mwdebug2003 and mwdebug2004 as mediawiki::canary_appserver - https://phabricator.wikimedia.org/T274023 (10Dzahn) [02:00:37] 10serviceops, 10Performance-Team, 10WikimediaDebug: create mwdebug1004, mwdebug2003 and mwdebug2004 as mediawiki::canary_appserver - https://phabricator.wikimedia.org/T274023 (10Reedy) [06:50:24] 10serviceops, 10Code-Health-Objective, 10Performance-Team (Radar), 10Platform Team Initiatives (Session Management Service (CDP2)), and 2 others: Determine multi-dc strategy for CentralAuth - https://phabricator.wikimedia.org/T267270 (10Krinkle) >>! In the **task description**, @BPirkle wrote: > […] > > *... [15:09:19] legoktm, i see testreduce1001 has been buster-ed ... and mariadb is busted on it now .. so cannot run rt-testing right now. will file a phab task in a bit. [15:15:02] 10serviceops: mariadb failing on testreduce1001 - https://phabricator.wikimedia.org/T274034 (10ssastry) [15:15:32] 10serviceops, 10Parsoid (Tracking): mariadb failing on testreduce1001 - https://phabricator.wikimedia.org/T274034 (10ssastry) p:05Triage→03High [15:16:21] Given that the train is stalled right now, this can wait till Monday. [20:07:35] 10serviceops, 10Parsoid (Tracking): mariadb failing on testreduce1001 - https://phabricator.wikimedia.org/T274034 (10Legoktm) > Looks like testreduce1001 has been converted to buster but it seems to have broken mariadb install there. It's been buster since July 2020, it was just restarted yesterday, see https... [20:17:28] subbu: I think something is wrong with innodb, it's past my knowledge on how to fix it though