[08:50:24] 10serviceops, 10MW-on-K8s, 10Operations: Logging options for apache httpd in k8s - https://phabricator.wikimedia.org/T265876 (10Joe) [08:51:28] 10serviceops, 10MW-on-K8s, 10Operations, 10observability: Logging options for apache httpd in k8s - https://phabricator.wikimedia.org/T265876 (10Joe) p:05Triage→03High [08:51:43] 10serviceops, 10MW-on-K8s, 10Operations: Create the base container images for running MediaWiki in a production environment - https://phabricator.wikimedia.org/T265324 (10Joe) a:03Joe [08:59:08] 10serviceops, 10Prod-Kubernetes, 10Release Pipeline, 10Patch-For-Review: Refactor our helmfile.d dir structure for services - https://phabricator.wikimedia.org/T258572 (10Joe) I think there are just a few dangling services that are managed by the analytics team, specifically: - eventgate-analytics-externa... [09:07:17] 10serviceops, 10MW-on-K8s, 10Operations, 10observability: Logging options for apache httpd in k8s - https://phabricator.wikimedia.org/T265876 (10Joe) Additional datapoint that was required: we should be sending ~ 10/15k messages per second to the central log server, depending on traffic. [09:38:39] 10serviceops, 10Operations, 10Kubernetes, 10Service-Architecture: Consider using a file-based xDS system for envoy in k8s - https://phabricator.wikimedia.org/T265879 (10Joe) [09:39:49] 10serviceops, 10Operations, 10Kubernetes, 10Service-Architecture: Upgrade envoy configuration to use the v3 API - https://phabricator.wikimedia.org/T265880 (10Joe) [09:46:30] 10serviceops, 10Growth-Structured-Tasks, 10Growth-Team, 10Release-Engineering-Team: Move mwaddlink-query from github to gerrit - https://phabricator.wikimedia.org/T261403 (10kostajh) The move is in progress, see https://www.mediawiki.org/wiki/Topic:Vw2qibn0ocvx95lp [09:48:08] 10serviceops, 10Operations, 10Kubernetes, 10Service-Architecture: Improve envoy configuration CI checks - https://phabricator.wikimedia.org/T265881 (10Joe) [09:51:16] 10serviceops, 10Operations, 10Kubernetes, 10Service-Architecture: Allow canarying new envoy configurations in kubernetes - https://phabricator.wikimedia.org/T265882 (10Joe) [10:35:00] 10serviceops, 10Operations, 10Kubernetes, 10Service-Architecture: Allow canarying new envoy configurations in kubernetes - https://phabricator.wikimedia.org/T265882 (10Marostegui) p:05Triage→03Medium [10:35:07] 10serviceops, 10Operations, 10Kubernetes, 10Service-Architecture: Improve envoy configuration CI checks - https://phabricator.wikimedia.org/T265881 (10Marostegui) p:05Triage→03Medium [10:35:13] 10serviceops, 10Operations, 10Kubernetes, 10Service-Architecture: Upgrade envoy configuration to use the v3 API - https://phabricator.wikimedia.org/T265880 (10Marostegui) p:05Triage→03Medium [10:35:19] 10serviceops, 10Operations, 10Kubernetes, 10Service-Architecture: Consider using a file-based xDS system for envoy in k8s - https://phabricator.wikimedia.org/T265879 (10Marostegui) p:05Triage→03Medium [11:19:28] 10serviceops, 10Operations, 10Growth-Team (Current Sprint), 10Patch-For-Review, 10User-Elukey: Reimage one memcached shard to Buster - https://phabricator.wikimedia.org/T252391 (10kostajh) @jijiki EditorJourney logging is now switched off. We may at some point want to re-enable but will wait for this wor... [11:23:23] 10serviceops, 10Prod-Kubernetes, 10observability, 10Kubernetes: Store Kubernetes events for more than one hour - https://phabricator.wikimedia.org/T262675 (10JMeybohm) a:03JMeybohm [11:42:55] 10serviceops, 10Operations, 10ops-eqsin: ganeti5002 was down / powered off, machine check entries in SEL - https://phabricator.wikimedia.org/T261130 (10akosiaris) Any news on this one? (just found out today about it while working on T265607) [13:16:26] 10serviceops, 10Prod-Kubernetes, 10Release Pipeline, 10Patch-For-Review: Refactor our helmfile.d dir structure for services - https://phabricator.wikimedia.org/T258572 (10Ottomata) AH! Sorry was on vaca for 2 week and have really been trying to focus on a lagging goal from last quarter. Ok, I'll prioriti... [13:47:02] Hello! [13:47:04] https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/634976 [13:47:21] jayme: yt? how easy is this to deploy/apply ? [13:55:01] ottomata: I can take a look in a few. It's usually possible to make those changes a no-op, so "very easy" :) [13:55:24] k, just wasn't sure if k8s would see that as a new deployment or something [13:58:08] jayme: q [13:58:40] in eventgate-main, resources.replicas are in values.yaml, but in eventgate-analytics, they are in the e.g. values-eqiad.yaml and values-codfw.yaml [13:58:43] is there a reason? [13:58:46] just want to be consistent [13:59:04] i think i'd rather put the default for prod in values.yaml, and then override just in staging and canary values [13:59:14] like eventgate-main has [14:02:05] Don't know if there is a special reason. But your suggestions sounds good to me [14:03:14] k [14:36:14] akosiaris: got some labs/private commits to puppet-merge [14:36:16] ok if i merge them? [14:36:47] ah wait i can skip them [14:36:50] i'm trying to merge ops/puppet [14:36:53] i left them unmerged [14:37:43] ottomata: thanks I 'll merge them [14:51:10] I will be less than 5' for the meeting [14:51:15] I need to switch venues [14:52:18] <_joe_> less than 5 minutes late you mean? [15:50:10] 10serviceops, 10Operations, 10ops-eqsin: ganeti5002 was down / powered off, machine check entries in SEL - https://phabricator.wikimedia.org/T261130 (10RobH) So we got some movement on this Friday/replies today. Dell Singapore is being very difficult and require a local contact number. I've gone ahead and... [16:18:34] 10serviceops, 10Add-Link, 10GrowthExperiments-NewcomerTasks, 10Operations, and 2 others: Service operations setup for Add a Link project - https://phabricator.wikimedia.org/T258978 (10MGerlach) >>! In T258978#6532612, @Joe wrote: > - Logging: log in `json format` to stdout Added json-logging to the scrip... [16:31:35] 10serviceops, 10RESTBase, 10Platform Engineering (Icebox): Make internal services use RESTRouter instead of RESTBase - https://phabricator.wikimedia.org/T234816 (10Aklapper) 05Stalled→03Open The previous comments don't explain who or what (task?) exactly this task is stalled on (["If a report is waiting... [16:31:38] 10serviceops, 10RESTBase, 10Epic, 10Platform Team Initiatives (RESTBase Split (CDP2)), and 2 others: Split RESTBase in two services: storage service and API router/proxy - https://phabricator.wikimedia.org/T220449 (10Aklapper) [16:32:44] 10serviceops, 10RESTBase, 10Platform Engineering (Icebox): Make internal services use RESTRouter instead of RESTBase - https://phabricator.wikimedia.org/T234816 (10Pchelolo) 05Open→03Declined We have decided to eliminate RESTBase altogether, this task is no longer valid. [16:32:47] 10serviceops, 10RESTBase, 10Epic, 10Platform Team Initiatives (RESTBase Split (CDP2)), and 2 others: Split RESTBase in two services: storage service and API router/proxy - https://phabricator.wikimedia.org/T220449 (10Pchelolo) [16:55:56] 10serviceops, 10Growth-Structured-Tasks, 10Growth-Team, 10Release-Engineering-Team: Move mwaddlink-query from github to gerrit - https://phabricator.wikimedia.org/T261403 (10kostajh) https://github.com/dedcode/mwaddlink is now imported at https://gerrit.wikimedia.org/r/plugins/gitiles/research/mwaddlink, s... [16:56:12] 10serviceops, 10Growth-Structured-Tasks, 10Growth-Team, 10Release-Engineering-Team: Move dedcode/mwaddlink from github to gerrit - https://phabricator.wikimedia.org/T261403 (10kostajh) [18:40:36] 10serviceops, 10Operations: improve mw maintenance server switch over and discovery names - https://phabricator.wikimedia.org/T265936 (10Dzahn) [20:19:02] 10serviceops, 10Operations, 10Scap, 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)): Make a way to build Scap .deb in Docker - https://phabricator.wikimedia.org/T265501 (10jijiki) [20:22:58] 10serviceops, 10Prod-Kubernetes, 10Release Pipeline, 10Patch-For-Review: Refactor our helmfile.d dir structure for services - https://phabricator.wikimedia.org/T258572 (10Ottomata) Ok! Done. [20:23:48] 10serviceops, 10Operations, 10Growth-Team (Current Sprint), 10Patch-For-Review, 10User-Elukey: Reimage one memcached shard to Buster - https://phabricator.wikimedia.org/T252391 (10jijiki) @kostajh Thank you! 💃🏼 [20:29:10] 10serviceops, 10Operations, 10Growth-Team (Current Sprint), 10Patch-For-Review, 10User-Elukey: Reimage one memcached shard to Buster - https://phabricator.wikimedia.org/T252391 (10jijiki) [21:23:21] 10serviceops, 10DBA, 10MediaWiki-Parser, 10Parsoid, 10Platform Team Workboards (Green): CAPEX for ParserCache for Parsoid - https://phabricator.wikimedia.org/T263587 (10Pchelolo) I guess we have to begin here. TLDR of the problem is that we will not have enough space in MySQL for ParserCache for transi... [23:51:27] 10serviceops, 10Release-Engineering-Team: replace production deployment servers - https://phabricator.wikimedia.org/T265963 (10Dzahn) [23:51:50] 10serviceops, 10Release-Engineering-Team: replace production deployment servers - https://phabricator.wikimedia.org/T265963 (10Dzahn) [23:52:16] 10serviceops, 10Release-Engineering-Team: replace production deployment servers - https://phabricator.wikimedia.org/T265963 (10Dzahn) [23:52:34] 10serviceops, 10Release-Engineering-Team: replace production deployment servers - https://phabricator.wikimedia.org/T265963 (10Dzahn)