[00:55:59] 10serviceops, 10MW-on-K8s, 10Operations, 10Patch-For-Review, and 2 others: RFC: PHP microservice for containerized shell execution - https://phabricator.wikimedia.org/T260330 (10tstarling) [07:15:16] 10serviceops, 10Excimer, 10Performance-Team, 10Sustainability (Incident Followup), 10User-jijiki: Verify Excimer's performance footprint - https://phabricator.wikimedia.org/T270152 (10jijiki) [08:42:55] 10serviceops, 10WMF-JobQueue, 10Patch-For-Review, 10Platform Team Workboards (Clinic Duty Team), and 2 others: Could not enqueue jobs: "Unable to deliver all events: 503: Service Unavailable" - https://phabricator.wikimedia.org/T249745 (10JMeybohm) >>! In T249745#6689455, @akosiaris wrote: > https://grafan... [10:12:39] 10serviceops, 10Operations: Memcached is listening to 127.0.0.1 after first puppet runs - https://phabricator.wikimedia.org/T261164 (10jijiki) 05Open→03Declined [10:23:18] _joe_: can I scap https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/649602/ ? [10:24:03] <_joe_> effie: +1, just keep an eye for errors afterwards [10:24:13] sure [10:24:18] tx [11:23:58] 10serviceops, 10Operations, 10Platform Engineering, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` mc1033.eqiad.wmnet ` The log can be... [11:24:05] 10serviceops, 10Operations, 10Platform Engineering, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` mc2033.codfw.wmnet ` The log can be... [11:29:06] 10serviceops, 10Operations, 10Platform Engineering, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10jijiki) [11:56:57] 10serviceops, 10Excimer, 10Performance-Team, 10Sustainability (Incident Followup), 10User-jijiki: Verify Excimer's performance footprint - https://phabricator.wikimedia.org/T270152 (10jijiki) @Krinkle is this for Q2 or we can schedule it for early Q3? [12:32:04] 10serviceops, 10Operations, 10Platform Engineering, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mc1033.eqiad.wmnet'] ` and were **ALL** successful. [12:34:40] 10serviceops, 10Prod-Kubernetes, 10Patch-For-Review, 10User-fsero: Set up PodSecurityPolicies in clusters - https://phabricator.wikimedia.org/T228967 (10JMeybohm) a:03JMeybohm [12:50:43] 10serviceops, 10Operations, 10Platform Engineering, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mc2033.codfw.wmnet'] ` and were **ALL** successful. [15:46:14] 10serviceops, 10Operations, 10MW-1.36-notes (1.36.0-wmf.18; 2020-11-17), 10Performance Issue, and 3 others: Strategy for storing parser output for "old revision" (Popular diffs and permalinks) - https://phabricator.wikimedia.org/T244058 (10Pchelolo) [15:47:51] 10serviceops, 10Operations, 10MW-1.36-notes (1.36.0-wmf.18; 2020-11-17), 10Performance Issue, and 3 others: Strategy for storing parser output for "old revision" (Popular diffs and permalinks) - https://phabricator.wikimedia.org/T244058 (10Pchelolo) 05Open→03Resolved a:03Pchelolo Now old revision vie... [15:53:28] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Add kubernetes 1.17+ typology annotations - https://phabricator.wikimedia.org/T270191 (10JMeybohm) [16:19:02] 10serviceops, 10Operations, 10Release-Engineering-Team-TODO, 10Patch-For-Review, and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by hnowlan on cumin1001.eqiad.wmnet for hosts: ` ['m... [16:21:30] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Add kubernetes 1.17+ typology annotations - https://phabricator.wikimedia.org/T270191 (10JMeybohm) Currently we're setting those per node in hiera. Would be nice to have that automated (T229397). [17:07:21] 10serviceops, 10Operations, 10MW-1.36-notes (1.36.0-wmf.18; 2020-11-17), 10Performance Issue, and 3 others: Strategy for storing parser output for "old revision" (Popular diffs and permalinks) - https://phabricator.wikimedia.org/T244058 (10jcrespo) Thanks for working on this. Let's monitor parsercache usag... [17:10:43] 10serviceops, 10Operations, 10MW-1.36-notes (1.36.0-wmf.18; 2020-11-17), 10Performance Issue, and 3 others: Strategy for storing parser output for "old revision" (Popular diffs and permalinks) - https://phabricator.wikimedia.org/T244058 (10Pchelolo) > maybe not necessary if it only hits memcache, haven't l... [17:11:14] 10serviceops, 10Operations, 10MW-1.36-notes (1.36.0-wmf.18; 2020-11-17), 10Performance Issue, and 3 others: Strategy for storing parser output for "old revision" (Popular diffs and permalinks) - https://phabricator.wikimedia.org/T244058 (10jcrespo) Ah, cool then. One less thing to worry about, but was goin... [17:30:06] 10serviceops, 10Operations, 10Release-Engineering-Team-TODO, 10Patch-For-Review, and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1265.eqiad.wmnet'] ` and were **ALL** successful. [18:31:28] 10serviceops, 10Excimer, 10Performance-Team, 10Sustainability (Incident Followup), 10User-jijiki: Verify Excimer's performance footprint - https://phabricator.wikimedia.org/T270152 (10Krinkle) p:05Triage→03Medium Q3 is fine. I think all we need is for it to be turned off on a few servers and then tur... [21:10:47] 10serviceops, 10Excimer, 10Performance-Team, 10Sustainability (Incident Followup), 10User-jijiki: Verify Excimer's performance footprint - https://phabricator.wikimedia.org/T270152 (10jijiki) @Krinkle I can do it after the holidays in January:) [22:10:55] 10serviceops, 10MW-on-K8s, 10Operations, 10Release Pipeline (Blubber), 10Release-Engineering-Team (Pipeline): Deployment infrastructure for PHP microservices - https://phabricator.wikimedia.org/T261369 (10Legoktm) Have we ever run composer in production/part of an automated build process like this? For M... [22:44:53] 10serviceops, 10MW-on-K8s, 10Operations, 10Release Pipeline (Blubber), 10Release-Engineering-Team (Pipeline): Deployment infrastructure for PHP microservices - https://phabricator.wikimedia.org/T261369 (10Jdforrester-WMF) npm services build the whole of their (non-dev) npm dependency graph into the image... [23:23:55] 10serviceops: ifup@eno1.service failed on some buster hosts - https://phabricator.wikimedia.org/T270220 (10RLazarus) [23:24:10] 10serviceops: ifup@eno1.service failed on some buster hosts - https://phabricator.wikimedia.org/T270220 (10RLazarus) p:05Triage→03High