[10:58:25] 10serviceops, 10Operations, 10SRE-tools, 10User-jijiki: Create a cookbook to perform a rolling reboot of a kubernetes cluster - https://phabricator.wikimedia.org/T260661 (10jijiki) [11:00:37] 10serviceops, 10Operations, 10SRE-tools, 10User-jijiki: Create a cookbook to perform a rolling reboot of a kubernetes cluster - https://phabricator.wikimedia.org/T260661 (10Volans) [12:11:20] 10serviceops, 10Operations, 10Performance-Team, 10User-jijiki: MediaWiki to route spefic keys to /*/mw-with-onhost-tier/ - https://phabricator.wikimedia.org/T264604 (10jijiki) [12:12:15] 10serviceops, 10Operations, 10Performance-Team, 10User-jijiki: MediaWiki to route spefic keys to /*/mw-with-onhost-tier/ - https://phabricator.wikimedia.org/T264604 (10jijiki) [12:12:32] 10serviceops, 10Operations, 10Performance-Team, 10User-jijiki: MediaWiki to route spefic keys to /*/mw-with-onhost-tier/ - https://phabricator.wikimedia.org/T264604 (10jijiki) [12:12:35] 10serviceops, 10Operations, 10Performance-Team, 10Patch-For-Review, and 2 others: Reduce read pressure on mc* servers by adding a machine-local Memcached instance (on-host memcached) - https://phabricator.wikimedia.org/T244340 (10jijiki) [14:39:00] 10serviceops, 10Operations, 10Product-Infrastructure-Team-Backlog, 10Wikifeeds: [Bug] The feed/featured endpoint is broken - https://phabricator.wikimedia.org/T263043 (10Pchelolo) [14:39:27] 10serviceops, 10Operations, 10Product-Infrastructure-Team-Backlog, 10Wikifeeds, 10Platform Team Workboards (Clinic Duty Team): Move feed assembly from RESTBase to Wikifeeds - https://phabricator.wikimedia.org/T263133 (10Pchelolo) 05Open→03Resolved [14:40:03] 10serviceops, 10Kubernetes, 10User-jijiki: Deploy kube-state-meterics - https://phabricator.wikimedia.org/T264625 (10JMeybohm) [14:40:23] 10serviceops, 10Kubernetes, 10User-jijiki: Deploy kube-state-metrics - https://phabricator.wikimedia.org/T264625 (10JMeybohm) [14:43:21] 10serviceops, 10Operations, 10SRE-tools: Create a cookbook for depooling one or all services from one kubernetes cluster - https://phabricator.wikimedia.org/T260663 (10JMeybohm) [15:50:54] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10User-jijiki: Update to kernel 4.19 on kubernetes nodes - https://phabricator.wikimedia.org/T262527 (10JMeybohm) a:05JMeybohm→03jijiki [16:06:42] 10serviceops, 10Operations, 10SRE-tools, 10User-jijiki: Create a cookbook to perform a rolling reboot of a kubernetes cluster - https://phabricator.wikimedia.org/T260661 (10JMeybohm) @MoritzMuehlenhoff wrote some generic code to do rolling reboots for groups of hosts that could probably be utilized here: h... [17:11:27] 10serviceops, 10Parsing-Team, 10Parsoid: parsoid apt repo rolled back breaks updates - https://phabricator.wikimedia.org/T264546 (10ssastry) @Dzahn @MoritzMuehlenhoff something you can fix? [17:43:33] 10serviceops, 10observability, 10User-jijiki: Handle unknown stats in rsyslog_exporter - https://phabricator.wikimedia.org/T210137 (10Volans) Any news on this by any chance? On this random host I was checking today it taking up 1/3rd of syslog. ` netbox-dev2001 0 ~$ grep -c rsyslog_exporter /var/log/syslo... [17:46:27] 10serviceops, 10Parsing-Team, 10Parsoid: parsoid apt repo rolled back breaks updates - https://phabricator.wikimedia.org/T264546 (10MoritzMuehlenhoff) Is this really worth fixing? jessie is totally end-of-life by Debian, not even Debian LTS covers it, so let's rather retire the repo for jessie for good. [17:55:03] 10serviceops, 10Operations, 10Patch-For-Review: docker-reporter-releng-images failed on deneb - https://phabricator.wikimedia.org/T251918 (10JMeybohm) 05Open→03Resolved This was some kind of connection issue to a apt repo while installing debmonitor-cli: ` Sep 28 14:51:47 deneb docker-report-releng[2358... [17:57:47] 10serviceops, 10Kubernetes, 10Patch-For-Review: Support TLS for service-to-service communication in k8s staging - https://phabricator.wikimedia.org/T260917 (10JMeybohm) I could use a pair of eyes on https://gerrit.wikimedia.org/r/q/bug:T260917 The PCC full diff (https://puppet-compiler.wmflabs.org/compiler10... [18:03:03] 10serviceops, 10Parsing-Team, 10Parsoid: parsoid apt repo rolled back breaks updates - https://phabricator.wikimedia.org/T264546 (10Firefishy) This jessie repo is used for all debian distros (including Ubuntu 20.04). Install notes are here: https://www.mediawiki.org/wiki/Parsoid/JS/Setup#Ubuntu_/_Debian [18:33:40] 10serviceops, 10Parsing-Team, 10Parsoid: parsoid apt repo rolled back breaks updates - https://phabricator.wikimedia.org/T264546 (10MoritzMuehlenhoff) >>! In T264546#6518541, @ssastry wrote: > @Dzahn @MoritzMuehlenhoff something you can fix? We don't have control over the apt caches of the people who've dow... [18:36:38] 10serviceops, 10Operations, 10Performance-Team, 10User-jijiki: MediaWiki to route spefic keys to /*/mw-with-onhost-tier/ - https://phabricator.wikimedia.org/T264604 (10Gilles) a:03aaron [18:36:57] 10serviceops, 10Operations, 10Performance-Team, 10User-jijiki: MediaWiki to route spefic keys to /*/mw-with-onhost-tier/ - https://phabricator.wikimedia.org/T264604 (10aaron) p:05Triage→03Medium [19:37:02] 10serviceops, 10Operations, 10Performance-Team, 10User-jijiki: MediaWiki to route specific keys to /*/mw-with-onhost-tier/ - https://phabricator.wikimedia.org/T264604 (10Aklapper) [23:04:55] 10serviceops, 10Operations, 10Parsoid, 10observability, 10User-jijiki: Create per cluster error rate alerts on Mediawiki servers - https://phabricator.wikimedia.org/T262078 (10colewhite) As of T256418, we have removed StatsD outputs from Logstash. Prometheus-ES-Exporter accepts an Elasticsearch query an...