[08:47:49] 10serviceops, 10MW-on-K8s, 10SRE, 10Release Pipeline (Blubber), 10Release-Engineering-Team (Pipeline): Deployment infrastructure for PHP microservices - https://phabricator.wikimedia.org/T261369 (10akosiaris) >>! In T261369#6746098, @sbassett wrote: >>>! In T261369#6695002, @akosiaris wrote: >>> As I und... [09:50:12] 10serviceops, 10MW-on-K8s, 10SRE, 10Release Pipeline (Blubber), 10Release-Engineering-Team (Pipeline): Deployment infrastructure for PHP microservices - https://phabricator.wikimedia.org/T261369 (10Joe) >>! In T261369#6746098, @sbassett wrote: >>>! In T261369#6695002, @akosiaris wrote: >>> As I understan... [09:51:16] 10serviceops, 10MW-on-K8s, 10SRE, 10Release Pipeline (Blubber), 10Release-Engineering-Team (Pipeline): Deployment infrastructure for PHP microservices - https://phabricator.wikimedia.org/T261369 (10Joe) I also want to stress this is the wrong task to have the above discussions, please open a new one to k... [09:52:18] 10serviceops, 10MW-on-K8s, 10SRE, 10Release Pipeline (Blubber), 10Release-Engineering-Team (Pipeline): Deployment infrastructure for PHP microservices - https://phabricator.wikimedia.org/T261369 (10Joe) @jeena we already have a .pipeline directory for Shellbox, and it works as intended in creating multip... [13:18:44] 10serviceops, 10Graphoid, 10Platform Engineering, 10SRE: Final undeploy for graphoid - en.wiki - https://phabricator.wikimedia.org/T271495 (10Jseddon) p:05Triage→03Low [14:19:31] 10serviceops, 10MediaWiki-Authentication-and-authorization, 10Platform Team Workboards (Clinic Duty Team), 10Wikimedia-production-error: Error fetching URL "http://localhost:600...": (curl error: 28) Timeout was reached - https://phabricator.wikimedia.org/T265551 (10BPirkle) 05Open→03Resolved a:03BPir... [17:02:29] 10serviceops, 10Graphoid, 10Platform Engineering, 10SRE: Final undeploy for graphoid - en.wiki - https://phabricator.wikimedia.org/T271495 (10phuedx) 👋 FYI this deployment caused a noticeable increase in the HTML and asset size for all enwiki pages with at least one graph. {F34001243,size=full} That 179.... [18:08:42] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for host... [18:09:05] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for host... [18:10:06] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for host... [18:11:14] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for host... [18:48:48] when reimaging codfw appservers, I first skipped the special cases mcrouter proxy and scap proxy. Then I get back to an mcrouter case to move it around so it's not down during reimaging.. and I notice the IP and the hostname comment in mcrouter_wancache.yaml do not match. it was actually a different host that has that IP and was already reimaged.. without it being noticed... hrmm [18:49:20] fixing the yaml though so it reflects reality [19:25:55] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2241.codfw.wmnet'] ` and were **ALL** s... [19:26:17] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2242.codfw.wmnet'] ` and were **ALL** s... [19:29:40] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2255.codfw.wmnet'] ` and were **ALL** s... [19:31:27] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2258.codfw.wmnet'] ` and were **ALL** s... [19:58:05] 10serviceops: mc1024 broke - replace it or remove it from configs - https://phabricator.wikimedia.org/T272078 (10Dzahn) [19:58:55] 10serviceops: mc1024 broke - replace it or remove it from configs - https://phabricator.wikimedia.org/T272078 (10Dzahn) p:05Triage→03High [19:59:56] 10serviceops: mc1024 broke - replace it or remove it from configs - https://phabricator.wikimedia.org/T272078 (10Dzahn) [20:02:51] 10serviceops: mc1024 broke - replace it or remove it from configs - https://phabricator.wikimedia.org/T272078 (10Dzahn) [20:22:07] 10serviceops: mc1024 broke - replace it or remove it from configs - https://phabricator.wikimedia.org/T272078 (10Dzahn) mc2024 is the replication partner in codfw: ` <+icinga-wm> ACKNOWLEDGEMENT - Check health of redis instance on 6379 on mc2024 is CRITICAL: CRITICAL: replication_delay is 3066 600 - REDIS 2.8.... [20:45:52] 10serviceops: mc1024 broke - replace it or remove it from configs - https://phabricator.wikimedia.org/T272078 (10Dzahn) [20:46:29] 10serviceops: mc1024 broke - replace it or remove it from configs - https://phabricator.wikimedia.org/T272078 (10Dzahn) [21:26:00] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for host... [21:29:00] 10serviceops, 10MW-on-K8s, 10Release Pipeline, 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)): Request volume for Docker images and container filesystems on releases machines - https://phabricator.wikimedia.org/T272092 (10dduvall) [21:29:25] 10serviceops, 10MW-on-K8s, 10Release Pipeline, 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)): Request volume for Docker images and container filesystems on releases machines - https://phabricator.wikimedia.org/T272092 (10dduvall) p:05Triage→03Me... [21:30:26] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for host... [21:32:07] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for host... [21:33:16] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for host... [21:36:47] 10serviceops, 10MW-on-K8s, 10Release Pipeline, 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)): Request volume for Docker images and container filesystems on releases machines - https://phabricator.wikimedia.org/T272092 (10Dzahn) How many Gigabytes d... [22:09:17] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2268.codfw.wmnet'] ` and were **ALL** s... [22:11:07] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2269.codfw.wmnet'] ` and were **ALL** s... [22:11:58] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2270.codfw.wmnet'] ` and were **ALL** s... [22:44:06] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2236.codfw.wmnet'] ` and were **ALL** s... [22:53:11] 10serviceops, 10MW-on-K8s, 10Release Pipeline, 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)): Request volume for Docker images and container filesystems on releases machines - https://phabricator.wikimedia.org/T272092 (10dduvall) >>! In T272092#674...