[00:02:53] 10serviceops, 10MW-on-K8s, 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)): Missing docker iptables nat rules for releases hosts - https://phabricator.wikimedia.org/T276869 (10dduvall) [00:03:33] 10serviceops, 10MW-on-K8s, 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)): Missing docker iptables nat rules for releases hosts - https://phabricator.wikimedia.org/T276869 (10dduvall) [00:22:38] 10serviceops, 10MW-on-K8s, 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)): Missing docker iptables nat rules for releases hosts - https://phabricator.wikimedia.org/T276869 (10Legoktm) Including `profile::docker::builder` would be wrong since that also pulls in a bunch of other stuff to build... [00:29:23] 10serviceops: Phase out legacy "uploader" docker-registry.wikimedia.org user - https://phabricator.wikimedia.org/T275581 (10Legoktm) 05Open→03Resolved a:03Legoktm https://gerrit.wikimedia.org/r/669964 [06:52:15] 10serviceops, 10Code-Health-Objective, 10Performance-Team (Radar), 10Platform Team Initiatives (Session Management Service (CDP2)), and 2 others: Determine multi-dc strategy for CentralAuth - https://phabricator.wikimedia.org/T267270 (10tstarling) >>! In T267270#6808265, @Krinkle wrote: > * How do our loca... [08:49:25] 10serviceops, 10Code-Health-Objective, 10Performance-Team (Radar), 10Platform Team Initiatives (Session Management Service (CDP2)), and 2 others: Determine multi-dc strategy for CentralAuth - https://phabricator.wikimedia.org/T267270 (10Gilles) @tstarling thank you for your in-depth response. Based on what... [10:13:32] 10serviceops, 10Analytics-Radar, 10Cassandra, 10ContentTranslation, and 9 others: Rebuild all blubber build docker images running on kubernetes - https://phabricator.wikimedia.org/T274262 (10JMeybohm) [10:17:13] 10serviceops, 10MW-on-K8s, 10SRE: Figure out appropriate readiness and liveness probes - https://phabricator.wikimedia.org/T276908 (10Joe) [10:22:33] 10serviceops, 10MW-on-K8s, 10SRE: Figure out appropriate readiness and liveness probes - https://phabricator.wikimedia.org/T276908 (10Joe) Ideally, the liveness probe needs to check if the container is running (more or less), while the readiness probe should check that the service is still responding. What... [11:39:52] 10serviceops, 10Product-Infrastructure-Team-Backlog: Allow `push-notifications` service to accept production environment flag for APNS requests - https://phabricator.wikimedia.org/T274456 (10jijiki) [13:23:21] 10serviceops, 10MediaWiki-JobQueue: jobqueue-eventbus grafana alerts UNKNOWN - https://phabricator.wikimedia.org/T276926 (10fgiunchedi) [13:40:29] hi! just fyi we're going to deploy another new termbox image in a few minutes [14:43:41] 10serviceops, 10DNS, 10SRE, 10Traffic, and 3 others: DNS for GitLab - https://phabricator.wikimedia.org/T276170 (10wkandek) gerrit.wikimedia.org lives on a second IP address on gerrit1001. Should we follow that model here as well? [15:00:04] 10serviceops, 10Prod-Kubernetes, 10observability, 10Kubernetes: Kubernetes 1.16 dropped deprecated cadvisor metric labels pod_name and container_name - https://phabricator.wikimedia.org/T275618 (10akosiaris) Using the python grafcli package I fetched all the dashboards under the Service/ folder, then sent... [15:25:42] 10serviceops, 10Prod-Kubernetes, 10observability, 10Kubernetes: Kubernetes 1.16 dropped deprecated cadvisor metric labels pod_name and container_name - https://phabricator.wikimedia.org/T275618 (10akosiaris) Run the same thing on the Kubernetes/ folder. I only had to update 3 dashboards * Kubernetes DNS *... [15:26:49] 10serviceops, 10Prod-Kubernetes, 10observability, 10Kubernetes: Kubernetes 1.16 dropped deprecated cadvisor metric labels pod_name and container_name - https://phabricator.wikimedia.org/T275618 (10akosiaris) 05Open→03Resolved a:03akosiaris I am gonna resolve this, I think we 've updated all we cared... [15:26:51] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Upgrade kubernetes clusters to a security supported (LTS) version - https://phabricator.wikimedia.org/T244335 (10akosiaris) [15:45:41] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Run helm test after deploy - https://phabricator.wikimedia.org/T276949 (10JMeybohm) p:05Triage→03Medium [16:11:21] 10serviceops, 10Product-Infrastructure-Team-Backlog: Allow `push-notifications` service to accept production environment flag for APNS requests - https://phabricator.wikimedia.org/T274456 (10jijiki) @Dmantena in order to help me understand what the requirements are here, can you provide us with the request flo... [16:15:43] 10serviceops, 10Wikidata, 10Wikidata-Termbox, 10wdwb-tech-focus: Missing alerts for Termbox staging and test services - https://phabricator.wikimedia.org/T276550 (10JMeybohm) It was more a matter of a day than month (as we just upgraded the kubernetes version in staging). Also we don't enable monitoring fo... [18:26:20] 10serviceops, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)): Replace production deployment servers and update them to Buster - https://phabricator.wikimedia.org/T265963 (10Papaul) [20:30:22] 10serviceops, 10DNS, 10SRE, 10Traffic, and 3 others: DNS for GitLab - https://phabricator.wikimedia.org/T276170 (10Dzahn) >>! In T276170#6896563, @wkandek wrote: > gerrit.wikimedia.org lives on a second IP address on gerrit1001. Should we follow that model here as well? It would be appropriate in the "exa... [21:00:33] 10serviceops: mc1024 broke - replace it or remove it from configs - https://phabricator.wikimedia.org/T272078 (10jijiki) 05Open→03Resolved a:03jijiki The server is resting in piece, new servers have been bought, we can close this task [23:33:52] 10serviceops, 10DNS, 10SRE, 10Traffic, and 3 others: DNS for GitLab - https://phabricator.wikimedia.org/T276170 (10wkandek) Let's go with the simpler solution and use the CNAME. [23:49:03] 10serviceops, 10MW-on-K8s, 10Release-Engineering-Team: Investigate how we can provide an mwdebug functionality on kubernetes - https://phabricator.wikimedia.org/T276994 (10jijiki)