[08:36:02] 10serviceops, 10MediaWiki-General, 10Operations, 10Patch-For-Review, 10Service-Architecture: Create a service-to-service proxy for handling HTTP calls from services to other entities - https://phabricator.wikimedia.org/T244843 (10JMeybohm) [08:36:13] 10serviceops, 10Operations, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Move wikifeeds to use TLS only - https://phabricator.wikimedia.org/T255878 (10JMeybohm) [09:10:43] 10serviceops, 10Machine Learning Platform, 10ORES, 10Okapi, and 3 others: ORES redis: max number of clients reached... - https://phabricator.wikimedia.org/T263910 (10Joe) Checkin in to report that calls from OKAPI have stopped tonight. Thanks @RBrounley_WMF (and the team)! So if we still see the starvatio... [09:22:13] 10serviceops, 10Operations, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Move zotero to use TLS only - https://phabricator.wikimedia.org/T255869 (10JMeybohm) [09:22:18] 10serviceops, 10Operations, 10Prod-Kubernetes, 10Kubernetes: Move mathoid to use TLS only - https://phabricator.wikimedia.org/T255875 (10JMeybohm) [09:22:37] 10serviceops, 10MediaWiki-General, 10Operations, 10Patch-For-Review, 10Service-Architecture: Create a service-to-service proxy for handling HTTP calls from services to other entities - https://phabricator.wikimedia.org/T244843 (10JMeybohm) [09:23:05] finally \o/ - "only" restbase and mw left [09:28:35] 10serviceops, 10MediaWiki-General, 10Operations, 10Patch-For-Review, 10Service-Architecture: Create a service-to-service proxy for handling HTTP calls from services to other entities - https://phabricator.wikimedia.org/T244843 (10JMeybohm) [09:56:47] <_joe_> jayme: I think rb will be doable once we merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/630562 [09:57:30] <_joe_> mediawiki /should/ be doable as well but I'm sure there is something that calls via http once every quarter and if we stop that everything breaks horribly [10:00:10] hm. If thats the only way to figure it out... [10:08:02] <_joe_> i have an idea how to track outliers [10:19:43] <_joe_> basically all requests that come from envoy, thus TLS, include a request id in their logs [10:20:01] <_joe_> in the second-to last field [10:21:19] <_joe_> tail -f /var/log/apache2/other_vhosts_access.log | awk '{if ($NF-2 == "-") { print $_ }}' basically :P [10:22:02] <_joe_> err -1 [10:47:49] cool [10:50:09] I could use a pair of eyes on https://gerrit.wikimedia.org/r/q/bug:T260917 where I think I overlooked something. PPC is at https://puppet-compiler.wmflabs.org/compiler1002/25621/ and I'm missing the "defaultsecret: notdefault" for staging zotero there [13:13:07] 10serviceops, 10Operations, 10Performance-Team, 10Patch-For-Review, and 2 others: Reduce read pressure on mc* servers by adding a machine-local Memcached instance (on-host memcached) - https://phabricator.wikimedia.org/T244340 (10jijiki) **What happens when onhost memcached in unavailable? ** https://phabr... [13:14:45] 10serviceops, 10Operations, 10Patch-For-Review, 10User-jijiki: Test onhost memcached performance and functionality - https://phabricator.wikimedia.org/T263958 (10jijiki) **What happens when onhost memcached in unavailable? ** https://phabricator.wikimedia.org/T244340#6211682 @elukey @aaron With the con... [13:51:02] serviceops friends, any objections to me rolling out https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/631530 and its ancestor patch shortly? [13:54:15] ah, it looks like this would be a full scap sync [13:54:46] oh, no, sync-file can take a directory [13:54:49] okay [13:56:34] 10serviceops, 10Operations, 10Performance-Team, 10Patch-For-Review, and 2 others: Reduce read pressure on mc* servers by adding a machine-local Memcached instance (on-host memcached) - https://phabricator.wikimedia.org/T244340 (10jijiki) >>! In T244340#6197415, @Krinkle wrote: > If the local-memcached's bl... [14:09:46] cdanis: while having a limited idea about the impact of those changes I'm not aware of any general issues that should prevent rollouts :) [14:49:33] <_joe_> cdanis: you already did that? [14:49:42] <_joe_> I'm perplexed by that change [14:49:54] <_joe_> sorry I wasn't looking at this chan :/ [14:50:59] <_joe_> why is wikimediafoundation.org a separate docroot? last I checked we had ported it to use the standard docroot [14:51:15] 10serviceops, 10Operations, 10Wikimedia-production-error: PHP7 corruption reports in 2020 (Call on wrong object, etc.) - https://phabricator.wikimedia.org/T245183 (10CDanis) [14:51:30] 10serviceops, 10Operations, 10Wikimedia-production-error: PHP7 corruption reports in 2020 (Call on wrong object, etc.) - https://phabricator.wikimedia.org/T245183 (10CDanis) [14:51:54] _joe_: it was added in https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/631529 [14:53:11] <_joe_> I would've appreciated if that was ran by the people who spent a lot of time deduplicating all of our docroots [14:53:14] <_joe_> like Reedy [14:53:24] <_joe_> anyways, it's the end of my work week, 🤷 [14:54:07] <_joe_> we can have a better solution at a later time [15:13:02] 10serviceops, 10Machine Learning Platform, 10ORES, 10Okapi, and 3 others: ORES redis: max number of clients reached... - https://phabricator.wikimedia.org/T263910 (10calbon) I just had to do another service restart. [15:26:43] 10serviceops, 10CX-cxserver, 10Language-Team (Language-2020-October-December), 10Release-Engineering-Team (Pipeline): Migrate apertium to the deployment pipeline - https://phabricator.wikimedia.org/T255672 (10Pginer-WMF) [15:57:33] 10serviceops, 10Machine Learning Platform, 10ORES, 10Okapi, and 3 others: ORES redis: max number of clients reached... - https://phabricator.wikimedia.org/T263910 (10Joe) Small status update: in order to grant everyone a quieter weekend (hopefully!), we've repooled eqiad and raised manually the max client... [17:39:15] 10serviceops, 10Push-Notification-Service, 10Patch-For-Review, 10Product-Infrastructure-Team-Backlog (Kanban): Push notification service should make deletion requests to MediaWiki for invalid or expired subscriptions - https://phabricator.wikimedia.org/T260247 (10Mholloway) >>! In T260247#6508622, @Joe wro... [18:16:18] cdanis: all good on the f.wm.o change? [18:16:27] Krinkle: deployed, works :) [18:16:32] EMS is happy [18:16:37] great [18:18:11] j.oe: 1) yes, it was, but for f.wm.o it was symlinked to std-docroot instead of actually point there directly, which actually helped: 2) yes, it was (patch by me), which replces the symlink with a n actual directory and adds the file in question, and 3) not asked, but yes, it was already done via DNS, but this other matrix feature doesn't support DNS SRV yet, so oh well. now we have both. [18:19:06] we can probably remove the dns entry if EMS are confident that won't cause anything unexpected on their end. [18:19:13] your (plural) call [19:10:17] 10serviceops, 10Operations, 10Performance-Team, 10Patch-For-Review, and 2 others: Reduce read pressure on mc* servers by adding a machine-local Memcached instance (on-host memcached) - https://phabricator.wikimedia.org/T244340 (10Krinkle) Short summary of IRC convo: Per [doc](https://docs.google.com/docume...