[01:55:23] 10serviceops, 10Operations, 10ops-codfw: (OoW) restbase2009 lockup - https://phabricator.wikimedia.org/T227408 (10Eevans) >>! In T227408#5364314, @jijiki wrote: > @Eevans I was under the impression we have more work to be done on the server. Shall we mark this task as resolved? I was under that impression t... [05:15:17] 10serviceops, 10DBA, 10Patch-For-Review, 10Performance-Team (Radar): phased rollout of dbctl, etcd-backed database configuration in Mediawiki - https://phabricator.wikimedia.org/T229070 (10Marostegui) \o/ [09:17:36] 10serviceops, 10Operations, 10Patch-For-Review, 10Release-Engineering-Team-TODO (201907), 10Wikimedia-Incident: docker-registry: some layers has been corrupted due to deleting other swift containers - https://phabricator.wikimedia.org/T228196 (10fsero) @greg thanks for following this, i definitely would... [09:19:08] 10serviceops, 10Release Pipeline: Staging k8s ci namespace limitranges - https://phabricator.wikimedia.org/T229073 (10fsero) @thcipriani is granular per namespace, you can submit a CR with changed values anytime, i will bump those values and refer to this phab task so you can see how is done [10:28:08] 10serviceops, 10Operations: SRE FY19-20 Q1 goal: complete the transition to PHP7 - https://phabricator.wikimedia.org/T219127 (10jijiki) [11:09:59] 10serviceops, 10Release Pipeline, 10Patch-For-Review: Staging k8s ci namespace limitranges - https://phabricator.wikimedia.org/T229073 (10fsero) @thcipriani you can launch the pipeline again and it should work, however a better fix is to change limits in blubber default values in the chart, 1m is not realist... [12:34:32] 10serviceops, 10Operations, 10ops-codfw: (OoW) restbase2009 lockup - https://phabricator.wikimedia.org/T227408 (10jijiki) @Papaul Can you lets us know what are our options (if any?) [14:34:34] 10serviceops, 10Operations, 10Release-Engineering-Team-TODO (201907), 10Wikimedia-Incident: create swift container-to-container synchronization metrics - https://phabricator.wikimedia.org/T229117 (10fsero) [14:36:37] 10serviceops, 10Operations, 10Release-Engineering-Team-TODO (201907), 10Wikimedia-Incident: create a docker_registry_codfw swift container backup - https://phabricator.wikimedia.org/T229118 (10fsero) [15:41:23] 10serviceops, 10Release Pipeline: Staging k8s ci namespace limitranges - https://phabricator.wikimedia.org/T229073 (10thcipriani) 05Open→03Resolved a:03fsero Working now! https://integration.wikimedia.org/ci/blue/organizations/jenkins/blubber-pipeline-publish/detail/service-pipeline-test-and-publish/8/p... [16:25:35] 10serviceops, 10Operations, 10Release-Engineering-Team-TODO (201907), 10Wikimedia-Incident: create a docker_registry_codfw swift container backup - https://phabricator.wikimedia.org/T229118 (10herron) p:05Triage→03Normal [16:25:47] 10serviceops, 10Operations, 10Release-Engineering-Team-TODO (201907), 10Wikimedia-Incident: create swift container-to-container synchronization metrics - https://phabricator.wikimedia.org/T229117 (10herron) p:05Triage→03Normal [16:31:27] can anyone help with https://phabricator.wikimedia.org/T229051 [16:31:28] ? [16:31:30] fsero: ? [16:31:31] :) [16:33:18] I am afraid I can't help you :/ [16:33:32] is that a router firewall change? [16:34:39] No is a Calico policy change [16:34:46] 10serviceops, 10Operations, 10Core Platform Team Workboards (Clinic Duty Team), 10Patch-For-Review, and 3 others: Use PHP7 to run all async jobs - https://phabricator.wikimedia.org/T219148 (10CCicalese_WMF) [16:34:54] ottomata: could you please make a cr in puppet and in deployment charts? [16:35:13] The Calico policy lives in both repos but will end up in deployment charts [16:35:27] When all clusters config are managed via helmfile [16:35:39] I cannot do it merge right now sorry [16:35:55] But I can do it next Monday and if you prepare the CRS will be even easier [16:36:05] ok found it i think [16:36:06] will do. [16:36:19] default-kubernetes-policy.yamo [16:36:21] yaml [16:36:22] ya? [16:36:32] Yep [16:36:41] great, thank you, will submit change [17:35:33] 10serviceops, 10Operations, 10Release Pipeline, 10Release-Engineering-Team-TODO, and 4 others: Kask functional testing with Cassandra via the Deployment Pipeline - https://phabricator.wikimedia.org/T224041 (10CCicalese_WMF) [18:25:13] 10serviceops, 10Core Platform Team, 10CPT Initiatives (Session Management Service (CDP2)), 10Core Platform Team Workboards (Clinic Duty Team), and 2 others: Package table_properties utility for Debian - https://phabricator.wikimedia.org/T226551 (10WDoranWMF) [18:25:18] 10serviceops, 10Core Platform Team, 10Operations, 10Release Pipeline, and 5 others: Kask functional testing with Cassandra via the Deployment Pipeline - https://phabricator.wikimedia.org/T224041 (10WDoranWMF) [18:30:12] howdy. I see wikimedia is looking at using envoy. was wondering what you'll be using it for, and if anyone wanted to chat with us about it :) [18:30:36] 10serviceops, 10Operations, 10Release Pipeline, 10Release-Engineering-Team-TODO, and 4 others: Kask functional testing with Cassandra via the Deployment Pipeline - https://phabricator.wikimedia.org/T224041 (10WDoranWMF) [18:30:39] 10serviceops, 10CPT Initiatives (Session Management Service (CDP2)), 10Core Platform Team Workboards (Clinic Duty Team), 10User-Clarakosi, 10User-Eevans: Package table_properties utility for Debian - https://phabricator.wikimedia.org/T226551 (10WDoranWMF) [18:33:49] 10serviceops, 10MediaWiki-General, 10Operations, 10CPT Initiatives (PHP7 (TEC4)), and 3 others: Some pages will become completely unreachable after PHP7 update due to Unicode changes - https://phabricator.wikimedia.org/T219279 (10WDoranWMF) [18:46:17] 10serviceops, 10Operations, 10Release Pipeline, 10CPT Initiatives (RESTBase Split (CDP2)), and 3 others: Deploy the RESTBase front-end service (RESTRouter) to Kubernetes - https://phabricator.wikimedia.org/T223953 (10WDoranWMF) [18:46:20] 10serviceops, 10RESTBase, 10CPT Initiatives (RESTBase Split (CDP2)), 10Epic, 10User-mobrovac: Split RESTBase in two services: storage service and API router/proxy - https://phabricator.wikimedia.org/T220449 (10WDoranWMF) [18:47:29] 10serviceops, 10CPT Initiatives (Multi-DC (TEC1)): k8s liveness check(?) generating session storage log noise - https://phabricator.wikimedia.org/T227514 (10WDoranWMF) [18:53:53] um. so the serviceops people are almost all eu tz except for mutante :-D [18:59:33] Ryan_Lane: I can't speak on this topic well -- much better people are _joe_ and fsero, but I suspect they won't be around until Monday EU time -- but AIUI the first use case was both TLS termination on traffic between services, as well as using it as a connection pooling layer for PHP code talking to backends. Presently nginx is used for both of those but with some issues [19:03:35] 10serviceops, 10Parsoid-PHP, 10CPT Initiatives (Parsoid REST API in PHP (CDP2)), 10Patch-For-Review: Allow to avoid installing HHVM from the mediawiki puppet module and profile - https://phabricator.wikimedia.org/T228976 (10WDoranWMF) [19:03:39] 10serviceops, 10Parsoid-PHP, 10CPT Initiatives (Parsoid REST API in PHP (CDP2)): Deploy Parsoid-PHP with Mediawiki to scandium for RT and performance testing - https://phabricator.wikimedia.org/T228069 (10WDoranWMF) [19:07:05] <_joe_> Ryan_Lane: I'd be very happy to [19:07:46] <_joe_> Ryan_Lane: basically for now it's going to be static routes (using dns-based discovery) + local tls termination both within k8s and outside of it [19:08:25] <_joe_> but in a not-so distant future I want to implement a xDS layer and get rid of centralized load-balancing at the applayer as much as possible [19:09:15] <_joe_> Ryan_Lane: I'm writing some puppet code / other things to install envoy today, so it might happen quite soon [19:09:40] <_joe_> well at first I guess I'll do some benchmarking, just cross my ts and dot my is [19:37:21] ah, awesome [19:37:36] _joe_: did you see the announcement recently about packages being made available? [19:44:19] lemme PM you the maintainer's email address. he'd love to chat [20:52:01] <_joe_> Ryan_Lane: no, in fact we're building our own packages, but given we're using the "official" build method, we might as well use the official ones