[06:29:43] <_joe_> I think people should only use "sql" from the maintenance servers, yes [13:19:58] 10serviceops, 10Scap, 10Release-Engineering-Team (Watching / External), 10User-jijiki: Allow scap sync to deploy gradually - https://phabricator.wikimedia.org/T212147 (10jijiki) Some things to consider while implementing this: * We have a predefined sets of canary hosts in `/etc/dsh/group/mediawiki-appser... [13:49:15] 10serviceops, 10Operations, 10Thumbor, 10Patch-For-Review, and 2 others: Assess Thumbor upgrade options - https://phabricator.wikimedia.org/T209886 (10jijiki) [14:09:48] \o Hi _joe_, just thinking about some more of the wikidata ssr things, is it possible to put varnish infront of the ssr service thing (termbox) inside the cluster without exposing it externally? [14:10:27] <_joe_> addshore: *why* [14:12:01] good question :) [14:12:27] <_joe_> ok, once you have an answer, I can suggest what's the best solution in my opinion :) [14:12:43] _joe_: let me write a few sentences then send them you way to cover the situation [14:13:31] <_joe_> sure :) [14:19:31] Currently the parser cache is split per UI language for wikidata, snippets of html are then stored in "extension_data" in the parser cache and re ordered after retrieval from the pcache based on the users preferences etc. [14:19:31] The new term box will also include some more user specific information, so it can't really land in the parser cache, such as why they are being shown vertain languages and why they are in the given order. [14:19:31] One of the simplest things would be to not store the termbox in the pcache in any form, and instead after retrieving the pcache for the wikidata page hit the ssr to get the termbox to add to the output (kind of ESI) [14:19:31] Rather than always hitting he service it would proabbly make sense to have a layer of caching infront of the service (hence I thought varnish?). [14:19:31] With the longer term plan of having the wikibase UI infront of mediawiki the issue with the parser cache goes away as then we are just talking about general caching stratergy of the UI. [14:19:36] _joe_: ^^ :) [14:20:15] There are of course more evil things we could think about too, and it is possible that all of this will also be covered in the rfc, I just wanted to gather some more of your thoghts [14:20:27] <_joe_> addshore: sorry, in a meeting, but i will read in a few [14:20:32] ack! [14:37:07] <_joe_> you know what's great about parsercache? That it's a cache managed by the application, not a shotgun cache put in front of it. [14:37:47] <_joe_> the idea of having layers of cache between applications, instead than inside them, is an ill result of the poor architectural choice that restbase has been [14:37:54] regarding paser cache, another option could be sharding the parser cache for wikidata further! not sure if that is better or worse [14:38:15] <_joe_> so, how I see it [14:38:51] <_joe_> you should cache the immutable parts of the termbox in some caching system managed by the termbox [14:39:19] <_joe_> see why I would prefer a different approach? [14:40:16] <_joe_> and you still have the problem of invalidating this cache on wikidata edits [14:40:44] <_joe_> which is, again, introducing a change propagation problem between mediawiki and your service [14:41:21] <_joe_> this problem is the same wether you use varnish in front of the termbox or if you just use its own cache. [14:41:59] Splitting the parser cache further for wikidata (per user for logged in user), and keeping it the same as currently is for anon users is probably the easy route then [14:42:19] <_joe_> oh sorry [14:42:34] <_joe_> you're saying anons will all have the same parsercache output? [14:43:29] <_joe_> this is the usual, semi-unsolvable problem [14:43:31] yes, so (I'll have to triple check), anons would still only have a parser cache output split based on UI language [14:44:05] <_joe_> but you still can't have the pc split per-user [14:44:19] but logged in users could essentially end up having their own pcache entries for pages [14:44:26] <_joe_> yeah, no. [14:44:33] <_joe_> that's not really viable, imho? [14:44:54] <_joe_> so either you get clever and save the immutable bits in $cache [14:45:09] <_joe_> or, well, I dunno [14:45:23] <_joe_> is the wikidata termbox expensive to parse/render? [14:46:03] <_joe_> this also implies that parsing happens outside of mediawiki? [14:46:56] <_joe_> to explain better, what do you get back from the MW API? wikitext or HTML? [14:47:59] not wikitext, we do nothing with wikitext, so just html, and only for page views, not via the action api at all [14:48:42] <_joe_> yes what I mean is that [14:48:50] <_joe_> user requires page to mediawiki [14:48:59] <_joe_> mediawiki calls SRR for the termbox [14:49:07] <_joe_> SSR calls the mw api for HTML [14:49:28] <_joe_> and the api uses the parsercache [14:49:49] <_joe_> so the expensive part of the whole ordeal - converting wikitext to html - is done before you hit the api [14:50:02] <_joe_> and cached properly by MediaWiki [14:50:12] <_joe_> or is there something that I misunderstood? [14:50:39] <_joe_> I'm by no means an expert on how parsercache works [14:51:16] so currently we just have: user page req -> mediawiki -> pcache -> transform some of the pcache data in PHP and shove it in the html-> present to user [14:51:43] and the pcache is split per UI language [14:52:47] <_joe_> ok my question is [14:53:05] <_joe_> SSR will call the action api to fetch the html generated from wikitext, right? [14:53:51] no, it will just be fetching some json, no html or parser stuff [14:53:55] *parsed [14:54:08] <_joe_> ok so, where does wikitext get processed? [14:54:16] there is no wikitext :) [14:54:28] <_joe_> ok, so, why do you need aggressive caching? [14:54:41] <_joe_> is reading those json and rendering them so expensive? [14:54:51] <_joe_> how good is your uncached latency? [14:54:55] so, that also is a good question, and something someone raised, do we even need this caching in the first place [14:55:07] the reason we have it is, well, it is in mediawiki [14:55:07] <_joe_> you want front-end caching [14:55:23] <_joe_> which we already do in varnish at the edge [14:55:36] <_joe_> but I'd only cache internal calls if they're expensive [14:55:42] the rendering / parsing is actually just mostly a json -> html transformation [14:56:02] <_joe_> so you avoid that pesky problem of having to invalidate said cache [14:56:18] <_joe_> it's all a cost/benefit calculation [14:56:24] <_joe_> and for that, you need [14:56:30] <_joe_> *evil drumroll* [14:56:32] <_joe_> DATA [14:56:37] yup, right, thanks for the chat again, i think I have some more things to go away and think about :) [14:57:09] I want to have a separate pcache hit rate for wd.org, or at least for wikidata content vs wikitext etc. [14:57:13] <_joe_> we can try to gather info on the page views in wikidata that would require a termbox generation [14:57:30] <_joe_> and yes, that too helps :) [14:58:22] it might even make sense for us to start using the pcache in a totally different way [14:59:03] we could avoid the split if we just put JSON in there that is used to render the page, rather than html [14:59:18] interesting [14:59:21] thanks as always _joe_ :) [15:00:36] <_joe_> yw, I did nothing of substance though :P [16:37:18] 10serviceops, 10Operations, 10Proton, 10Reading-Infrastructure-Team-Backlog, and 3 others: Document and possibly fine-tune how Proton interacts with Varnish - https://phabricator.wikimedia.org/T213371 (10Tgr) >>! In T213371#4902627, @pmiazga wrote: > There is a queue mechanism. Right, sorry, I still need... [16:40:16] 10serviceops, 10Operations, 10Proton, 10Reading-Infrastructure-Team-Backlog, and 3 others: Document and possibly fine-tune how Proton interacts with Varnish - https://phabricator.wikimedia.org/T213371 (10Tgr) >>! In T213371#4906133, @Tgr wrote: > (I also have the impression that it's the default behavior f... [16:53:08] 10serviceops, 10TechCom, 10TechCom-RFC: RfC: Standards for external services in the Wikimedia infrastructure. - https://phabricator.wikimedia.org/T208524 (10jijiki) [17:30:47] _joe_: we can also talk about re-allocating imagescalers at allhands ? [17:30:54] <_joe_> sure [17:31:40] ok [18:26:28] 10serviceops, 10Operations, 10User-Joe: SRE FY2019 Q3 goal: Ramp-up serving traffic to PHP 7 - https://phabricator.wikimedia.org/T212828 (10Jdforrester-WMF) [18:26:33] 10serviceops, 10Operations, 10Core Platform Team (PHP7 (TEC4)), 10Core Platform Team Backlog (Watching / External), and 4 others: Set up a beta feature offering the use of PHP7 - https://phabricator.wikimedia.org/T213934 (10Jdforrester-WMF) 05Open→03Resolved a:03Joe [18:27:47] 10serviceops, 10Operations, 10User-Joe: SRE FY2019 Q3 goal: Ramp-up serving traffic to PHP 7 - https://phabricator.wikimedia.org/T212828 (10Jdforrester-WMF) [18:32:55] 10serviceops, 10Thumbor, 10User-jijiki: First page of a specific PDF files on Commons does not render a preview - https://phabricator.wikimedia.org/T213771 (10Quiddity) >>! In T213771#4879327, @Catrope wrote: > What's especially strange about this is that https://upload.wikimedia.org/wikipedia/commons/thumb/... [18:44:35] _joe_: hey, I need to sign your GPG key so I can re-encrypt all of releng's pwstore entries. Give me your details at All Hands (I'll try to remember to remind in person as well) [20:49:39] 10serviceops, 10Operations, 10Phabricator, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External): Reimage both phab1001 and phab2001 to stretch - https://phabricator.wikimedia.org/T190568 (10Paladox) "* 429 Too Many Requests" someone will want to whitelist deploy1001 ip so it's not hit by... [20:49:44] 10serviceops, 10Operations, 10Phabricator, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External): Reimage both phab1001 and phab2001 to stretch - https://phabricator.wikimedia.org/T190568 (10Dzahn) [20:58:40] 10serviceops, 10Operations, 10Phabricator, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External): Reimage both phab1001 and phab2001 to stretch - https://phabricator.wikimedia.org/T190568 (10Dzahn)