[05:08:52] 10serviceops, 10SyntaxHighlight: Package latest python3-pygments for apt.wikimedia.org - https://phabricator.wikimedia.org/T276298 (10Legoktm) [08:06:54] 10serviceops, 10MW-on-K8s, 10SRE, 10observability: Keep calculating latencies for MediaWiki requests that happen k8s - https://phabricator.wikimedia.org/T276095 (10JMeybohm) p:05Triage→03Medium [08:08:20] 10serviceops, 10MW-on-K8s, 10SRE, 10observability: Logging options for apache httpd in k8s - https://phabricator.wikimedia.org/T265876 (10JMeybohm) p:05High→03Medium Lowering prioiry to medium as of discussion with @Joe [08:28:52] 10serviceops, 10Add-Link, 10Growth-Team, 10Prod-Kubernetes: linkrecommendation: Can't connect to MySQL server on 'm2-master.eqiad.wmnet' - https://phabricator.wikimedia.org/T276268 (10akosiaris) a:03akosiaris I think I got an idea about the issue for this. m2-master seems to have changed we need the enti... [08:45:25] 10serviceops, 10Add-Link, 10Growth-Team (Current Sprint), 10Patch-For-Review: Use Envoy for making GET requests to lang.wikipedia.org/api.php - https://phabricator.wikimedia.org/T276217 (10kostajh) So AIUI what needs to happen: 1. add the mwapi-async listener to the helmfile values.yaml ([done in the WIP... [08:49:56] <_joe_> jayme: can you assist? ^^ [08:50:19] <_joe_> we also need a wikitech page explaining how it should be done [08:50:38] _joe_: sure, on it already [08:51:28] _joe_: I was guessing a bit about the Host: header part yesterday...can you confirm that this is how it's done? [08:51:43] <_joe_> lemme see the task [08:52:32] also I wanted to ask if there is a reason we have a mwapi-async but no mwapi listener configured [08:53:55] 10serviceops, 10Add-Link, 10Growth-Team (Current Sprint), 10Patch-For-Review: Use Envoy for making GET requests to lang.wikipedia.org/api.php - https://phabricator.wikimedia.org/T276217 (10JMeybohm) >>! In T276217#6877578, @kostajh wrote: > So in the helm chart, we should be setting `MEDIAWIKI_API_URL` to... [08:55:57] 10serviceops, 10Add-Link, 10Growth-Team (Current Sprint), 10Patch-For-Review: Use Envoy for making GET requests to lang.wikipedia.org/api.php - https://phabricator.wikimedia.org/T276217 (10Joe) >>! In T276217#6877578, @kostajh wrote: > So AIUI what needs to happen: > > 1. add the mwapi-async listener to t... [08:57:51] 10serviceops, 10Add-Link, 10Growth-Team (Current Sprint), 10Patch-For-Review: Use Envoy for making GET requests to lang.wikipedia.org/api.php - https://phabricator.wikimedia.org/T276217 (10kostajh) Thanks @Joe and @JMeybohm, I'll give it a try. [08:58:07] <_joe_> kostajh: does addlink need authenticating? [08:58:52] <_joe_> I am thinking we should write a library to connect to the mediawiki api and publish it [08:58:54] _joe_: no it does not [08:59:06] <_joe_> ok then all you need is the Host header [08:59:50] <_joe_> volans: do you think adding a method to get a request to the mw api would be out of scope for pywmflib ? [09:02:01] <_joe_> kostajh: oh so your method only works with wikipedias, I see, I was about to comment on how you go wiki_id => url won't work in some cases [09:03:14] _joe_: yeah eventually we'd like to make that more generic so it works with other wikis but this is good enough for now [09:03:48] _joe_: yet another mw api client? :-P if it doesn't require special auth sure I don't see a problem. Consider that the package it's installed on all buster hosts [09:04:30] <_joe_> volans: well this one should "do the right thing" in production [09:05:20] then yeah go for it :) [09:13:01] <_joe_> volans: the idea would be to then let python services in production use it [09:15:58] makes sense to me, I will just need to keep somehow a track of who's using it, in order to know what to restart when releasing new versions [09:16:15] <_joe_> like debmonitor, you mean? [09:16:26] <_joe_> it's docker images [09:16:38] ah then simpler :) [09:16:46] <_joe_> so they will pick up the new version on next release, unless we force a rebuild [09:17:06] <_joe_> but first I want to identify what this library should do [09:17:15] <_joe_> and I should really work on other stuff now [09:18:43] eh :) feel free to reuse wmflib.requests.http_session if that helps [09:27:38] 10serviceops, 10SRE, 10User-jijiki: Enable TLS on memcached - https://phabricator.wikimedia.org/T271967 (10Joe) Can I ask how do we intend to perform the transition from non-tls to tls in detail? I see a series of pitfalls with our current setup and the code I see in puppet, but please be explicit about the... [09:47:19] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Update Kubernetes cluster staging-eqiad to kubernetes 1.16 - https://phabricator.wikimedia.org/T276305 (10JMeybohm) [09:47:56] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Update Kubernetes cluster staging-eqiad to kubernetes 1.16 - https://phabricator.wikimedia.org/T276305 (10JMeybohm) p:05Triage→03Medium [09:49:31] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Update Kubernetes cluster staging-eqiad to kubernetes 1.16 - https://phabricator.wikimedia.org/T276305 (10JMeybohm) [10:30:01] https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/667934 I did this last night akosiaris feel free to toss it, no idea why it didn't get autolinked on the task [10:30:05] see folks later [10:33:08] apergos: Yeah it was correct, I 've uploaded a similar somewhat more finegrained change. But fwiw this is all going away. That file will cease to exist by EOQ. Thanks btw! [10:38:37] 10serviceops, 10Add-Link, 10Growth-Team, 10Prod-Kubernetes, 10Patch-For-Review: linkrecommendation: Can't connect to MySQL server on 'm2-master.eqiad.wmnet' - https://phabricator.wikimedia.org/T276268 (10akosiaris) Fixed in the commit above, service-checker now on deploy1001 says: ` $ service-checker-sw... [11:30:50] 10serviceops, 10Add-Link, 10Growth-Team, 10Prod-Kubernetes: linkrecommendation: Can't connect to MySQL server on 'm2-master.eqiad.wmnet' - https://phabricator.wikimedia.org/T276268 (10kostajh) >>! In T276268#6877907, @akosiaris wrote: > Resolving per comment above, feel free to reopen It works! 🎉 Thank... [11:34:32] 10serviceops, 10Add-Link, 10Growth-Team (Current Sprint), 10Patch-For-Review: Use Envoy for making GET requests to lang.wikipedia.org/api.php - https://phabricator.wikimedia.org/T276217 (10kostajh) >>! In T276217#6877602, @kostajh wrote: > Thanks @Joe and @JMeybohm, I'll give it a try. One more question,... [11:56:31] sure, just tryina learn a wee bit! [13:09:02] 10serviceops, 10Add-Link, 10Growth-Team (Current Sprint), 10Patch-For-Review: Use Envoy for making GET requests to lang.wikipedia.org/api.php - https://phabricator.wikimedia.org/T276217 (10JMeybohm) >>! In T276217#6878056, @kostajh wrote: > One more question, as we just need read-only access, should be usi... [17:09:49] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Update Kubernetes cluster staging-eqiad to kubernetes 1.16 - https://phabricator.wikimedia.org/T276305 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jayme on cumin1001.eqiad.wmnet for hosts: ` ['kubestage1001.eqiad.wmnet',... [17:33:46] akosiaris: I hear you wrote a tutorial on how to determine CPU/memory limits for k8s deployments? :) [17:34:29] legoktm: https://wikitech.wikimedia.org/wiki/User:Alexandros_Kosiaris/Benchmarking_kubernetes_apps [17:34:36] needs some love, but the basic premise is there [17:34:53] ty [18:11:43] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Update Kubernetes cluster staging-eqiad to kubernetes 1.16 - https://phabricator.wikimedia.org/T276305 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['kubestage1002.eqiad.wmnet', 'kubestage1001.eqiad.wmnet'] ` and were **ALL**... [19:09:01] 10serviceops, 10SRE: Upgrade docker-registry servers to Debian Buster - https://phabricator.wikimedia.org/T272550 (10Legoktm) [19:34:24] 10serviceops, 10Release-Engineering-Team: switch prod Phabricator from phab1003 to phab1001 - https://phabricator.wikimedia.org/T238956 (10Cmjohnson) [19:53:23] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Update Kubernetes cluster staging-eqiad to kubernetes 1.16 - https://phabricator.wikimedia.org/T276305 (10JMeybohm) The actual steps where: * disable puppet on master and nodes * stop apiserver, controller manager, scheduler on master * Cl... [20:37:38] where do I run puppet for changes to conftool-data to take effect? https://gerrit.wikimedia.org/r/c/operations/puppet/+/668190 [20:54:23] legoktm: maybe on the puppetmaster itself? https://wikitech.wikimedia.org/wiki/Conftool#In_puppet [20:54:55] I've replied in -sre, same question :) [21:35:50] 10serviceops, 10SRE: Upgrade docker-registry servers to Debian Buster - https://phabricator.wikimedia.org/T272550 (10Legoktm) registry1003 is now pooled, I did a test pull to it specifically and it worked fine.