[07:14:25] 10serviceops, 10Operations, 10Traffic, 10Patch-For-Review: Certificate *.wikipedia.org valid until 2020-06-20 - https://phabricator.wikimedia.org/T251726 (10ayounsi) ACKing the alerts again with that task as comment. [08:04:03] 10serviceops, 10Operations, 10Prod-Kubernetes, 10Kubernetes: Add TLS termination to services running on kubernetes - https://phabricator.wikimedia.org/T235411 (10JMeybohm) [08:04:06] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Upgrade all TLS enabled charts to v0.2 tls_helper - https://phabricator.wikimedia.org/T253396 (10JMeybohm) [08:09:13] 10serviceops, 10Operations, 10Security-Team, 10vm-requests, 10PM: Eqiad: 1VM request for Peek (PM service in use by Security Team) - https://phabricator.wikimedia.org/T252210 (10Dzahn) @chasemp Yes, i can create the VM. But I would ask you to please add your new cluster name, description and contact on h... [08:26:37] 10serviceops, 10Operations, 10Security-Team, 10vm-requests, and 2 others: Eqiad: 1VM request for Peek (PM service in use by Security Team) - https://phabricator.wikimedia.org/T252210 (10Dzahn) Ah, wait, so i was about to create it and already added peek2001.codfw.wmnet to DNS but then noticed it asks for e... [12:30:58] 10serviceops, 10Packaging, 10Patch-For-Review: prepare mcrouter package for Debian Buster - https://phabricator.wikimedia.org/T251574 (10jbond) mcrouter 0.4.1 is now available in buster-wikimedia ` $ sudo -E reprepro ls mcrouter mcrouter | 0.37.0-1~jessie0 | jessi... [12:31:04] 10serviceops, 10Packaging, 10Patch-For-Review: prepare mcrouter package for Debian Buster - https://phabricator.wikimedia.org/T251574 (10jbond) 05Open→03Resolved [12:31:12] 10serviceops, 10Operations, 10Patch-For-Review, 10Performance-Team (Radar), and 2 others: Upgrade memcached for Debian Stretch/Buster - https://phabricator.wikimedia.org/T213089 (10jbond) [12:58:49] 10serviceops, 10Operations, 10Security-Team, 10vm-requests, 10PM: Eqiad: 1VM request for Peek (PM service in use by Security Team) - https://phabricator.wikimedia.org/T252210 (10chasemp) >>! In T252210#6168163, @Dzahn wrote: > Ah, wait, so i was about to create it and already added peek2001.codfw.wmnet t... [12:59:13] 10serviceops, 10Operations, 10Security-Team, 10vm-requests, 10PM: Eqiad: 1VM request for Peek (PM service in use by Security Team) - https://phabricator.wikimedia.org/T252210 (10chasemp) [13:07:17] 10serviceops, 10Operations, 10Security-Team, 10vm-requests, 10PM: Eqiad: 1VM request for Peek (PM service in use by Security Team) - https://phabricator.wikimedia.org/T252210 (10Dzahn) Creating VM peek2001.codfw.wmnet in cluster ganeti01.svc.codfw.wmnet with row=B vcpus=1 memory=2GB disk=20GB link=privat... [13:07:27] 10serviceops, 10Operations, 10Security-Team, 10vm-requests, 10PM: Eqiad: 1VM request for Peek (PM service in use by Security Team) - https://phabricator.wikimedia.org/T252210 (10Dzahn) 05Stalled→03Open [13:12:33] 10serviceops, 10Operations, 10Security-Team, 10vm-requests, 10PM: Eqiad: 1VM request for Peek (PM service in use by Security Team) - https://phabricator.wikimedia.org/T252210 (10chasemp) >>! In T252210#6168983, @Dzahn wrote: > Creating VM peek2001.codfw.wmnet in cluster ganeti01.svc.codfw.wmnet with row=... [13:21:39] 10serviceops, 10Operations, 10Security-Team, 10vm-requests, and 2 others: Eqiad: 1VM request for Peek (PM service in use by Security Team) - https://phabricator.wikimedia.org/T252210 (10Dzahn) a:03Dzahn [13:45:51] 10serviceops, 10Operations, 10Security-Team, 10vm-requests, and 2 others: Eqiad: 1VM request for Peek (PM service in use by Security Team) - https://phabricator.wikimedia.org/T252210 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: `peek2001.codfw.wmnet` - pee... [13:53:30] 10serviceops: Integrate kube-metrics-server into our infrastructure - https://phabricator.wikimedia.org/T249929 (10apakhomov) I investigated possibility to run metric server outside the cluster. And what I did to run in this way: 1. Ran k8s cluster 2. Deployed required manifests APIService from https://github.co... [14:09:23] 10serviceops, 10Operations, 10decommission, 10ops-codfw, 10Patch-For-Review: codfw: decom at least 15 appservers in codfw rack C3 to make room for new servers - https://phabricator.wikimedia.org/T247018 (10Papaul) @Dzahn Please to not resolve yet. I still have mgmt DNS and switch port to remove. Thanks [14:15:18] 10serviceops, 10Operations, 10Security-Team, 10vm-requests, 10PM: Eqiad: 1VM request for Peek (PM service in use by Security Team) - https://phabricator.wikimedia.org/T252210 (10Dzahn) @chasemp The VM has been created and I installed the OS and signed the puppet cert request. It is in site.pp with the "i... [14:16:37] 10serviceops, 10Operations, 10Security-Team, 10vm-requests, 10PM: Eqiad: 1VM request for Peek (PM service in use by Security Team) - https://phabricator.wikimedia.org/T252210 (10chasemp) 05Open→03Resolved >>! In T252210#6169212, @Dzahn wrote: > @chasemp The VM has been created and I installed the OS... [14:21:35] 10serviceops, 10Operations, 10Security-Team, 10vm-requests, 10PM: Eqiad: 1VM request for Peek (PM service in use by Security Team) - https://phabricator.wikimedia.org/T252210 (10Dzahn) Alright, done. Your SSH user exists now. [15:28:43] 10serviceops, 10MediaWiki-General, 10Operations, 10Core Platform Team Workboards (Clinic Duty Team), and 3 others: Revisit timeouts, concurrency limits in remote HTTP calls from MediaWiki - https://phabricator.wikimedia.org/T245170 (10AMooney) [17:57:10] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Make helm upgrades atomic - https://phabricator.wikimedia.org/T252428 (10JMeybohm) [17:57:51] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Make helm upgrades atomic - https://phabricator.wikimedia.org/T252428 (10JMeybohm) 05Open→03Resolved tiller has been updated in all clusters and namespaces so this is resolved now [22:26:56] 10serviceops, 10Performance-Team (Radar): Avoid php-opcache corruption in WMF production - https://phabricator.wikimedia.org/T253673 (10ori) A contractor is going to ask for the same thing that upstream is asking for: a reproduction case. There's not much to go on without one. Have you tried depooling an apps... [22:49:01] 10serviceops, 10Packaging, 10Patch-For-Review: prepare mcrouter package for Debian Buster - https://phabricator.wikimedia.org/T251574 (10Andrew) I did a simple hand test of this (setting on one host using the local mcrouter port, getting on another host using that host's local mcrouter) and it looks good. I...