[07:30:59] 10serviceops, 10MW-on-K8s, 10SRE, 10observability: Add support for scraping php applications to the kubernetes prometheus scraper - https://phabricator.wikimedia.org/T271822 (10Joe) >>! In T271822#6773823, @lmata wrote: > Hi Joe, > > Let us know if there is any support you'd like from our team on this tas... [07:35:03] 10serviceops, 10MW-on-K8s, 10SRE, 10observability: Logging options for apache httpd in k8s - https://phabricator.wikimedia.org/T265876 (10Joe) @lmata we really need to set up a meeting to tackle the questions here and in T271822 pretty soon; we're at the point where not figuring out this stuff will harm o... [08:49:22] 10serviceops, 10MW-on-K8s, 10SRE, 10observability: Add support for scraping php applications to the kubernetes prometheus scraper - https://phabricator.wikimedia.org/T271822 (10JMeybohm) >>! In T271822#6790319, @Joe wrote: > - Is there a way to tell prometheus to read from multiple ports from the same pod... [10:12:03] 10serviceops, 10docker-pkg: Add a verify step to docker-pkg - https://phabricator.wikimedia.org/T273427 (10JMeybohm) [10:33:04] 10serviceops, 10MW-on-K8s, 10SRE, 10observability: Add support for scraping php applications to the kubernetes prometheus scraper - https://phabricator.wikimedia.org/T271822 (10fgiunchedi) I dug a bit into upstream Prometheus issues and this is relevant: https://github.com/prometheus/prometheus/issues/3756... [10:57:42] 10serviceops, 10Dumps-Generation, 10Platform Engineering, 10SRE, 10Patch-For-Review: Upgrade snapshot hosts to Buster - https://phabricator.wikimedia.org/T269377 (10ArielGlenn) I have tested in deployment-prep all of the "other" dumps (not xml/sql) except for the wikidata and adds-changes dumps. Those ar... [15:00:43] 10serviceops, 10SRE: upgrade conf2* servers to stretch - https://phabricator.wikimedia.org/T271573 (10jcrespo) [15:01:52] 10serviceops, 10SRE: upgrade conf2* servers to stretch - https://phabricator.wikimedia.org/T271573 (10jcrespo) I just realized, after closer inspection, that the blocker is indeed real, and we need these in stretch or higher to revert T273182. Is there something I can do to help? [15:42:04] 10serviceops, 10SRE: upgrade conf2* servers to stretch - https://phabricator.wikimedia.org/T271573 (10jcrespo) See also related T224560 [17:34:52] 10serviceops, 10decommission-hardware, 10Patch-For-Review: decommission francium.eqiad.wmnet - https://phabricator.wikimedia.org/T273142 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: `francium.eqiad.wmnet` - francium.eqiad.wmnet (**PASS**) - Downtimed host o... [17:42:22] 10serviceops, 10Prod-Kubernetes: PodSecurityPolicies will be deprecated with Kubernetes 1.21 - https://phabricator.wikimedia.org/T273507 (10JMeybohm) p:05Triage→03Low [17:44:22] 10serviceops, 10MW-on-K8s, 10SRE, 10observability: Logging options for apache httpd in k8s - https://phabricator.wikimedia.org/T265876 (10lmata) noted @Joe! I'll reach out to you to coordinate a time to talk with the team. [18:05:29] 10serviceops, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar), and 3 others: Investigate possible performance degradation on mediawiki servers after Debian Buster upgrade - https://phabricator.wikimedia.org/T273312 (10Legoktm) Here's the list of most of the stuff that PHP links to that differs: curl,... [18:36:20] 10serviceops, 10Analytics-Radar, 10SRE, 10decommission-hardware, 10ops-codfw: decommission kraz.wikimedia.org - https://phabricator.wikimedia.org/T245279 (10wiki_willy) [18:40:48] 10serviceops, 10SRE: upgrade conf2* servers to stretch - https://phabricator.wikimedia.org/T271573 (10elukey) @jcrespo me and Giuseppe are discussing the problem, so your pings are not unseen, but the problem is complex since it requires a lot of clients to move to eqiad first (Pybals, etcd DNS configs, etc..)... [19:30:38] 10serviceops, 10decommission-hardware: decommission francium.eqiad.wmnet - https://phabricator.wikimedia.org/T273142 (10Dzahn) 05Stalled→03Open [19:31:04] 10serviceops, 10decommission-hardware: decommission francium.eqiad.wmnet - https://phabricator.wikimedia.org/T273142 (10Dzahn) [19:39:43] 10serviceops, 10decommission-hardware, 10ops-eqiad: decommission francium.eqiad.wmnet - https://phabricator.wikimedia.org/T273142 (10Dzahn) [19:42:01] 10serviceops, 10decommission-hardware, 10ops-eqiad: decommission francium.eqiad.wmnet - https://phabricator.wikimedia.org/T273142 (10Dzahn) a:05Dzahn→03None The serviceops part of this is done. dcops can now continue. [19:42:27] 10serviceops, 10decommission-hardware, 10ops-eqiad: decommission francium.eqiad.wmnet - https://phabricator.wikimedia.org/T273142 (10wiki_willy) a:03Cmjohnson [19:44:58] 10serviceops, 10Analytics-Radar, 10SRE, 10decommission-hardware, 10ops-codfw: decommission kraz.wikimedia.org - https://phabricator.wikimedia.org/T245279 (10wiki_willy) a:03Papaul [19:52:40] 10serviceops, 10Analytics-Radar, 10SRE, 10decommission-hardware, 10ops-codfw: decommission kraz.wikimedia.org - https://phabricator.wikimedia.org/T245279 (10Papaul) @wiki_willy this is a VM it doesn't go to me [19:55:25] 10serviceops, 10Analytics-Radar, 10SRE, 10decommission-hardware: decommission kraz.wikimedia.org - https://phabricator.wikimedia.org/T245279 (10wiki_willy) a:05Papaul→03None [19:59:18] 10serviceops, 10Analytics-Radar, 10SRE, 10decommission-hardware: decommission kraz.wikimedia.org - https://phabricator.wikimedia.org/T245279 (10Dzahn) a:03Dzahn [20:00:59] 10serviceops, 10Analytics-Radar, 10SRE, 10decommission-hardware: decommission kraz.wikimedia.org - https://phabricator.wikimedia.org/T245279 (10Dzahn) a:05Dzahn→03None Oh, this ticket is actually not ready at all (VM or not), per previous comments. this is supposed to be stalled. [20:02:14] 10serviceops, 10Analytics-Radar, 10SRE, 10decommission-hardware: decommission kraz.wikimedia.org - https://phabricator.wikimedia.org/T245279 (10Dzahn) 05Stalled→03Invalid I'll just close this as invalid because the template would not match a VM and the system is still in production. [20:02:18] 10serviceops, 10Analytics, 10SRE, 10vm-requests, 10User-Elukey: Create a replacement for kraz.wikimedia.org - https://phabricator.wikimedia.org/T244719 (10Dzahn) [20:02:55] 10serviceops, 10Analytics-Radar, 10SRE, 10decommission-hardware: decommission kraz.wikimedia.org - https://phabricator.wikimedia.org/T245279 (10wiki_willy) Thanks @Dzahn - I saw it listed under the "pending onsite steps (codfw)" column, so it threw me off for a sec. >! In T245279#6793672, @Dzahn wrote: >... [20:09:11] 10serviceops, 10Analytics-Radar, 10SRE, 10decommission-hardware: decommission kraz.wikimedia.org - https://phabricator.wikimedia.org/T245279 (10Dzahn) Yea, my bad, this is not fitting for a VM. But also when this was created we did not have the same decom cookbook and workflow yet. [20:10:44] 10serviceops, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar), and 3 others: Investigate possible performance degradation on mediawiki servers after Debian Buster upgrade - https://phabricator.wikimedia.org/T273312 (10Legoktm) >>! In T273312#6788663, @Legoktm wrote: > I started by profiling `api.php?...