[01:21:16] 10serviceops, 10Release-Engineering-Team-TODO, 10Release-Engineering-Team (Deployment services): Upgrade MediaWiki appservers to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Dzahn) [01:23:50] 10serviceops, 10Release-Engineering-Team-TODO, 10Release-Engineering-Team (Deployment services): Upgrade MediaWiki appservers to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Dzahn) [01:24:44] 10serviceops, 10Release-Engineering-Team-TODO, 10Release-Engineering-Team (Deployment services): Upgrade MediaWiki appservers to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Dzahn) >>! In T245757#6533938, @Dzahn wrote: > A new ticket for this has been created at T264991. Ignore m... [01:57:37] 10serviceops, 10DC-Ops, 10Operations, 10ops-eqiad: eqiad: Physical Moves for MediaWiki Servers - https://phabricator.wikimedia.org/T266164 (10Dzahn) [01:58:14] 10serviceops, 10DC-Ops, 10Operations, 10ops-eqiad: eqiad: Physical Moves for MediaWiki Servers - https://phabricator.wikimedia.org/T266164 (10Dzahn) One rack done, the other rack we will continue from week of Nov 16. [02:14:40] 10serviceops, 10Operations, 10Parsoid, 10Parsoid-Tests: Make testreduce web UI publicly accessible on the internet - https://phabricator.wikimedia.org/T266509 (10Dzahn) I will be afk for about 2 weeks. If this needs earlier attention (I assume not, based on low prio etc) please contact the subteam. [02:15:26] 10serviceops, 10Operations, 10Parsoid, 10Parsoid-Tests: Make testreduce web UI publicly accessible on the internet - https://phabricator.wikimedia.org/T266509 (10Dzahn) p:05Triage→03Medium [04:17:50] 10serviceops, 10DC-Ops, 10Operations, 10ops-eqiad: eqiad: Physical moves for MediaWiki servers - https://phabricator.wikimedia.org/T266164 (10Krinkle) [08:11:18] <_joe_> jayme: do we still need the .hfenv files for the admin part? [08:37:04] _joe_: yeah [08:38:11] unfortunately we do...the admin structure probably needs some refacoring as well at some point [08:49:46] <_joe_> ok, what I though :) [09:14:30] 10serviceops, 10Operations, 10Performance-Team, 10Traffic, and 2 others: Very long response time on frwiki main page - https://phabricator.wikimedia.org/T266865 (10Aklapper) +#Wikimedia-Incident if this was considered an incident [11:31:21] 10serviceops, 10Prod-Kubernetes, 10Release Pipeline, 10Patch-For-Review: Refactor our helmfile.d dir structure for services - https://phabricator.wikimedia.org/T258572 (10Joe) 05Open→03Resolved [13:06:54] 10serviceops, 10Push-Notification-Service, 10Product-Infrastructure-Team-Backlog (Kanban), 10User-jijiki: High latency on push notification service initialization - https://phabricator.wikimedia.org/T265258 (10MSantos) My assumption is that the memory leak issue in node-apn (T263058) is triggering high loa... [15:15:16] 10serviceops, 10Operations, 10Kubernetes, 10Patch-For-Review: Migrate to helm v3 - https://phabricator.wikimedia.org/T251305 (10JMeybohm) [15:15:18] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Upgrade kubernetes clusters to a security supported (LTS) version - https://phabricator.wikimedia.org/T244335 (10JMeybohm) [15:19:19] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Upgrade kubernetes clusters to a security supported (LTS) version - https://phabricator.wikimedia.org/T244335 (10JMeybohm) [15:23:43] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Build new kubernetes packages - https://phabricator.wikimedia.org/T266766 (10JMeybohm) [16:32:52] 10serviceops, 10Operations, 10Scap, 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)): Make a way to build Scap .deb in Docker - https://phabricator.wikimedia.org/T265501 (10LarsWirzenius) I can't actually find the `package_builder` host and can't check if I have login access or the ability t... [17:26:20] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Upgrade kubernetes clusters to a security supported (LTS) version - https://phabricator.wikimedia.org/T244335 (10akosiaris) > We are not able to go 1.19 because of calico only supporting 1.18 Looks like this isn't true. Judging from https://github.com/projectca... [17:44:00] 10serviceops, 10Operations, 10CommRel-Specialists-Support (Oct-Dec-2020): CommRel support for ICU 63 upgrade - https://phabricator.wikimedia.org/T267145 (10RLazarus) [17:51:50] 10serviceops, 10MW-on-K8s, 10Operations, 10observability: Logging options for apache httpd in k8s - https://phabricator.wikimedia.org/T265876 (10lmata) Just dropping a quick update here, we should schedule some time to review options. Had a brief exchange with @akosiaris and we'll get the team together for... [22:39:13] 10serviceops, 10Operations, 10Performance-Team, 10Patch-For-Review, 10User-jijiki: MediaWiki to route specific keys to /*/mw-with-onhost-tier/ - https://phabricator.wikimedia.org/T264604 (10jijiki) @aaron is there a timeline as to when those patches will be merged? [22:39:55] 10serviceops, 10Operations, 10Performance-Team, 10Patch-For-Review, and 2 others: Reduce read pressure on mc* servers by adding a machine-local Memcached instance (on-host memcached) - https://phabricator.wikimedia.org/T244340 (10jijiki)