[05:10:16] 10serviceops, 10Operations, 10Patch-For-Review, 10Performance-Team (Radar): Reduce read pressure on memcached servers by adding a machine-local Memcache instance - https://phabricator.wikimedia.org/T244340 (10Joe) >>! In T244340#6135546, @Krinkle wrote: >>>! In T244340#5853430, @jijiki wrote: >> The idea i... [07:53:47] 10serviceops, 10Patch-For-Review: restrouter.svc.{eqiad,codfw}.wmnet in a failed state - https://phabricator.wikimedia.org/T242461 (10JMeybohm) a:03JMeybohm [07:59:25] 10serviceops, 10Patch-For-Review: restrouter.svc.{eqiad,codfw}.wmnet in a failed state - https://phabricator.wikimedia.org/T242461 (10JMeybohm) 05Open→03Resolved [12:11:22] 10serviceops, 10Core Platform Team, 10MediaWiki-General, 10Operations, and 3 others: Revisit timeouts, concurrency limits in remote HTTP calls from MediaWiki - https://phabricator.wikimedia.org/T245170 (10tstarling) To do: * Deploy the SwiftFileBackend config change, maintaining it at 900 * Reduce MultiHtt... [13:07:33] 10serviceops, 10MediaWiki-General, 10Operations, 10Patch-For-Review, 10Service-Architecture: Use envoy for TLS termination on the appservers - https://phabricator.wikimedia.org/T247389 (10Joe) As of today, all appservers use envoy too. [13:12:14] <_joe_> hnowlan: so you're migrating cpjobqueue as well? [13:13:05] _joe_: yep [13:13:14] Hoping this one is a bit less traumatic [13:13:31] <_joe_> have yo just migrated a few jobs or you're moving everything? [13:14:01] Nothing migrated so far, this is the first deployment. Everything but updateBetaFeaturesUserCounts is disabled [13:14:12] and that's still enabled in scb for the time being [13:14:21] <_joe_> updateBetaFeaturesUserCounts ahahah our beloved job [13:15:54] heh, is there history there? [13:28:14] hnowlan: there's always $history ;) [13:32:36] <_joe_> hnowlan: that's the first job we migrated from the old redisfromhell to use changeprop :) [13:32:57] <_joe_> so I guess Petr picked it for historical significance [13:33:11] <_joe_> it's basically the lowest-impact job we have [13:33:23] <_joe_> if it doesn't run for a few hours, some counters are inaccurate. [13:33:31] <_joe_> counters only *us* care about [13:47:26] heh, cool. yeah I have a vague idea of how to test it [13:47:38] in terms of the visibility [15:00:49] _joe_: do you know how often those counts update when things are functioning normally?