[07:37:35] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Upgrade all TLS enabled charts to v0.2 tls_helper - https://phabricator.wikimedia.org/T253396 (10JMeybohm) So eventgate and eventstream use forked tls_helpers (currently even the forks slightly differ). As far as I can tell, this is due to... [07:39:42] 10serviceops, 10Packaging: Please provide our special component/php72 in buster-wikimedia - https://phabricator.wikimedia.org/T250515 (10MoritzMuehlenhoff) What @Legoktm said, to run CI tests on Buster and PHP 7.2 best to use the sury.org repository. Our component for stretch-wikimedia is a rebuild of sury.org... [07:48:37] 10serviceops, 10Operations, 10decommission, 10ops-codfw, 10Patch-For-Review: codfw: decom at least 15 appservers in codfw rack C3 to make room for new servers - https://phabricator.wikimedia.org/T247018 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: `mw[21... [07:53:05] 10serviceops, 10Operations, 10decommission, 10ops-codfw, 10Patch-For-Review: codfw: decom at least 15 appservers in codfw rack C3 to make room for new servers - https://phabricator.wikimedia.org/T247018 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: `mw[21... [08:14:14] 10serviceops, 10Operations, 10decommission, 10ops-codfw, 10Patch-For-Review: codfw: decom at least 15 appservers in codfw rack C3 to make room for new servers - https://phabricator.wikimedia.org/T247018 (10Dzahn) @Papaul All remaining old mw servers in rack C3 (mw2154 through mw2186) are also decom'ed... [08:14:29] 10serviceops, 10Operations, 10decommission, 10ops-codfw, 10Patch-For-Review: codfw: decom at least 15 appservers in codfw rack C3 to make room for new servers - https://phabricator.wikimedia.org/T247018 (10Dzahn) p:05High→03Medium a:05Dzahn→03Papaul [08:59:00] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Upgrade all TLS enabled charts to v0.2 tls_helper - https://phabricator.wikimedia.org/T253396 (10JMeybohm) Oh. I see that the current canary setup will not work with my suggestions and as I see it there is currently no way on how to do it w... [09:24:46] jbond42: still testing "cas enabled vhost" stuff on icinga2001 ? [09:24:53] eh, i meant the other channel :) [10:08:51] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Upgrade all TLS enabled charts to v0.2 tls_helper - https://phabricator.wikimedia.org/T253396 (10JMeybohm) And I now see T242861, so please ignore what I said (or at least what I was suggesting). I'll evaluate the route of merging the commo... [12:52:35] 10serviceops, 10observability, 10Patch-For-Review: "PHP opcache hit ratio" alert shouldn't bother on mwdebug*/scandium/etc - https://phabricator.wikimedia.org/T254025 (10Dzahn) https://puppet-compiler.wmflabs.org/compiler1002/22957/mwdebug1001.eqiad.wmnet/index.html vs. https://puppet-compiler.wmflabs.org/... [13:29:11] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Upgrade all TLS enabled charts to v0.2 tls_helper - https://phabricator.wikimedia.org/T253396 (10Ottomata) Thanks @JMeybohm yeah T242861 is complicated! [13:56:53] hello! [13:57:12] i just noticed that there are no kafka-jumbo entires in default-kubernetes-policy.yaml [13:57:21] but there are plenty of kafka-main ones [13:57:37] are the kafka-jumbo ones for e.g. eventgate-analytics defined elsewhere ? [13:59:21] akosiaris: ^ [13:59:50] OHHHH WAIT THEY ARE THERE [13:59:54] jus tnot commented so i can't grep them [13:59:54] ok [14:06:46] :-). Do comment them please :-) [14:54:23] 10serviceops, 10LDAP-Access-Requests, 10Operations, 10observability, 10Patch-For-Review: Grant Access to Logstash to Peter(peter.ovchyn@speedandfunction.com) - https://phabricator.wikimedia.org/T249037 (10Dzahn) Hi @AMooney any updates on this? [14:58:14] 10serviceops, 10LDAP-Access-Requests, 10Operations, 10observability, 10Patch-For-Review: Grant Access to Logstash to Peter(peter.ovchyn@speedandfunction.com) - https://phabricator.wikimedia.org/T249037 (10AMooney) Thanks for checking @Dzahn. It is currently in approval process. I will reach out when we a... [15:46:14] 10serviceops, 10Packaging: Please provide our special component/php72 in buster-wikimedia - https://phabricator.wikimedia.org/T250515 (10Jdforrester-WMF) OK, that works. [15:55:03] 10serviceops, 10Packaging: Please provide our special component/php72 in buster-wikimedia - https://phabricator.wikimedia.org/T250515 (10Jdforrester-WMF) [15:55:08] 10serviceops, 10Release-Engineering-Team-TODO, 10Release-Engineering-Team (Deployment services): upgrade MediaWiki appservers to Debian 10 (buster) - https://phabricator.wikimedia.org/T245757 (10Jdforrester-WMF) [19:18:01] 10serviceops, 10Mobile-Content-Service, 10Page Content Service, 10Patch-For-Review, 10Product-Infrastructure-Team-Backlog (Kanban): Migrate mobileapps to k8s and node 10 - https://phabricator.wikimedia.org/T218733 (10Mholloway) Awesome! Thanks, @akosiaris! I'll get that change going shortly. Maybe I'm... [19:49:30] _joe_: kafkacat -b kafka-main1001.eqiad.wmnet:9092 -p 0 -t 'eqiad.resource-purge' | grep mediawiki [19:49:41] papam! [19:49:44] <_joe_> Pchelolo: whoa [19:49:48] we have group0 purges [19:49:50] <_joe_> live during the presentation [19:50:51] <_joe_> \o/ [19:52:56] gutsy! [20:04:39] _joe_: another fyi: I've deployed kask for sessions everywhere [20:04:57] however, I've now realized that central auth was excluded from it and still talks to redis [20:05:10] so, we have cut the number of redis requests in half [20:05:18] but didn't eliminate them yet [20:05:21] <_joe_> I saw it was ~ 15k req/s [20:05:30] <_joe_> instead of the 30 we expected [20:05:36] I will find out what's up with central auth [20:06:54] but, it's one more step in this already way too long process [20:08:02] as for kafka purges - tell me whenever we should take next step. e.g. disable htcp or enable more groups [21:40:19] <_joe_> Pchelolo: let's talk tomorrow :)