[17:53:03] Emanny: hello [17:53:57] cjim hi there [17:56:08] Emanny hello [17:56:26] cjim, Emanny - hi, y'all setting up IRC for the first time? [17:56:58] yup! [17:57:16] Cool, welcome :) [17:58:17] Thank you. [21:44:46] discussion of https://www.mediawiki.org/wiki/Requests_for_comment/CentralNotice_Caching_Overhaul_-_Frontend_Proxy here in 15 min [21:57:00] hey mwalker [21:57:33] greetings [21:57:53] mwalker: feel free to correct me when I summarise what we want today :) [21:59:54] ok! about to start. [22:00:04] Tim won't be with us for this one sadly. [22:00:06] #startmeeting CentralNotice Caching Overhaul - Frontend Proxy | Channel is logged and publicly posted (DO NOT REMOVE THIS NOTE). https://meta.wikimedia.org/wiki/IRC_office_hours [22:00:06] Meeting started Wed Jul 30 22:00:06 2014 UTC and is due to finish in 60 minutes. The chair is sumanah. Information about MeetBot at http://wiki.debian.org/MeetBot. [22:00:06] Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. [22:00:06] The meeting name has been set to 'centralnotice_caching_overhaul___frontend_proxy___channel_is_logged_and_publicly_posted__do_not_remove_this_note___https___meta_wikimedia_org_wiki_irc_office_hours' [22:00:12] #chair sumanah brion mark [22:00:12] Current chairs: brion mark sumanah [22:00:19] #link https://www.mediawiki.org/wiki/Architecture_meetings/RFC_review_2014-07-30 [22:00:21] hey all [22:00:24] hello [22:00:27] Hi! [22:00:34] First off - about next week & next month [22:00:38] #topic Future RFC stuff [22:00:47] #info Please check out https://www.mediawiki.org/wiki/Requests_for_comment/API_roadmap - Brad Jorsch is updating it 'as "notify people of upcoming changes", "chance for people to object to upcoming changes", and "todo list" all at once.' [22:00:57] (Brad isn't here right now) [22:01:11] Aw, anomie... [22:01:12] (API changes! ever looming!) [22:01:15] #info A reminder that I'm going on vacation in mid-August, and will take that opportunity to stop running these meetings and stop coordinating the process overall. I'm working with Rachel Farrand, Quim Gil, and the architecture committee to make sure stuff still happens. [22:01:30] #info Next week there is no RfC meeting on IRC because of Wikimania. Are there specific topics that you want to discuss in an architecture meeting at Wikimania? [22:01:48] sumanah: "stop running these meetings" forever? Or just for a while? [22:02:25] #info marktraceur: I personally will be running the meeting on Aug 16 and then I will stop forever. I am working with others to ensure that the process keeps going. It may change. [22:02:32] KK [22:03:01] Re Wikimania: who here is going? do you want to talk about any specific architecture/RfC stuff there? I know Daniel Kinzler suggested talking about "Using factory functions for component instantiation" there [22:03:21] o/ [22:03:59] well, we don't have to discuss it here - "[Wikitech-l] Anyone going to WikiMania?" will do [22:04:12] hey atgo :) [22:04:23] hey sumanah ! [22:04:27] * awjr waves [22:04:36] atgo: you are just in time! because now that those announcements are over: [22:04:37] #topic CentralNotice Caching Overhaul - Frontend Proxy RfC [22:04:43] #info Today we're talking about Matt Walker's proposal to "have a static javascript blob, injected via skin hook, in every content namespace page that will call up to a special CentralNotice banner server". This has frontend caching implications. [22:05:04] #link https://www.mediawiki.org/wiki/Requests_for_comment/CentralNotice_Caching_Overhaul_-_Frontend_Proxy [22:05:26] I'm now gonna summarize for mwalker - say if I'm wrong or omitting something important here Matt :) [22:05:31] #info The RFC has evolved in the last few weeks. As Matt explained just before he updated it: "I'm going to unify the URL to bits.wikimedia.org; varnish will initially serve the request before forwarding that to a backend; I'll add a bit to talk about CSP and how I dont currently care about it w/ justification that the first load is REALLY important; I'm going to add a note on where the 200GB comes from" [22:05:58] #info this is a followup to the meeting in October 2013 https://www.mediawiki.org/wiki/Talk:Requests_for_comment/CentralNotice_Caching_Overhaul_-_Frontend_Proxy#IRC_meeting_2013-10-02_34156 [22:06:20] #info Matt is about to leave the Foundation so in today's meeting we want to clear up any Ops concerns so that Anne Gomez (Fundraising tech product manager atgo ), ejegg, Jeff_Green, AndyRussG, and others can carry on implementing it after Matt's gone [22:06:33] one implementation point up front -- right now the JS is injected via skin hook because we're proposing to host this on bits. if instead we agree to host it on some wiki url, e.g. en.wm.o/banners than it can be a simple fetching JSONP” and would indeed work around CSP issues [22:17:24] ori, ESI can work, but I haven't spent a huge amount of time figuring out the best way of doing it [22:17:30] ESI just scares me :D [22:17:34] brion: but if it's async would you not still get the page bump? [22:17:48] ori: ESI has only recently been possible now that text is on Varnish, and nothing and no one uses it yet [22:17:50] ESI is still kinda borked with our current varnish deployment anyways [22:17:53] So it's a bit scary for those reasons [22:17:56] Oh well and there you go [22:18:00] ori: yes, but the jsonp fetch is already async so that’s what you’re already getting with this proposal afaict [22:18:06] otherwise we'd be using it for the Zero banners [22:18:32] bblack: how/why is it it borked? (or would answering that carry us too far off-course?) [22:18:42] if you want to make the jsonp fetch sync, you need to use document.write(‘ , and that returned some JS that set some variables and also wrote a CSS rule into the head that does .bannerContainer { height: 247px; } .... [22:26:29] or something like that [22:26:56] ori, if we host this on the same domain and use ESI, we'd still have to have a backend server to create the banner -- not a problem it just means the backend has to do slightly more work -- it will delay the entire page request though [22:27:12] that fixes the page bump but introduces a delay yeah [22:27:18] RoanKattouw, ah; but I dont want the script to request another something... the downloaded script must have the entire banner to be injected [22:27:27] and we dont know the hight of the banner because that's a browser dependent flow thing [22:27:34] and not all banners are the same height [22:27:45] do we know what exactly is borked with the ESI solution? [22:27:53] bblack, ^ [22:28:20] mwalker: Right, so banners are neither pure images nor pure HTML but a mix of those two? [22:28:52] brion: I need to head off in a few minutes - you'll be finishing up the meeting? [22:28:54] it's been a while since we last looked, I couldn't recall details right now. I'm pretty sure most of the problem has to do with us running the upstream-unsupported -plus variant of varnish3 for streaming, but I'm not 100% sure that ESI works correctly even for mainline varnish3 [22:28:58] RoanKattouw, *nods* [22:29:02] some even have their own js [22:29:04] Right [22:29:05] #info discussion of ESI solution [22:29:09] sumanah: confirmed [22:29:18] Yeah then I'm not sure that the bump can be fixed without using ESI [22:29:45] i tend to agree with RoanKattouw, fixing the bump *and* not introducing a round-trip between client and server requires ESI or something very like it [22:29:49] bblack: :( [22:29:52] thanks [22:29:54] Hmmm banners with js and js with banners [22:29:54] but the bottom line is ESI is deeply broken in our currently-deployed varnish, and we've had impetus to fix it already from Zero and not found any easy way to do so, yet. Looking at it was postponed until the varnish v4 upgrade a while back [22:30:09] Unless FR practices significant self-restraint in terms of standardizing banner heights or only using existing blank space or something like that [22:30:18] RoanKattouw, hah! [22:30:32] you can look at this RFC as a stepping stone to ESI... the backend nodejs server would only need small modifications to support it once it works [22:30:37] bblack: Yeah I made that statement not as a "this is why we need ESI now" but more as a "this is why this isn't going to happen soon" [22:30:48] in the meantime, this would reduce page bump by the time of 1 RTT [22:30:49] #info ESI is very broken with our currently deployed varnish; expected to be working for 4 upgrade but thta’s a ways off [22:30:57] * ori nods [22:31:20] Thanks all, catch you later. [22:31:31] yeah i don’t hate this solution, but it depends on how much work people think it will be and how much of that is going to be reusable for the ESI better solution later [22:31:43] i have the feeling that it would be easy to reuse, but i could be wrong [22:32:25] mwalker: So if I understand this all correctly, the JSONP request initiated from the head will return a JSON blob that contains the banner HTML and all the needed data? [22:32:27] #info rfc proposal doesn’t fix “page bump” when banners are loaded, but reduces its delay by 1 RTT between client and server [22:32:33] RoanKattouw, that's correct [22:32:43] Like, it won't just give us a banner name that we then have to go fetch separately [22:32:45] OK awesome [22:32:46] That's good [22:32:48] and then it'll wait for the RL downloaded centralnotice controller to inject itself [22:32:52] Yeah [22:33:51] ori: what do you think? would the backend for the jsonp blob be adaptable to a source for ESI banners once ESI is available for doing it “right"? [22:33:57] mwalker: Re CSP if you really want to you can make the head JS an RL top queue module and I think that'll still work. It'll just run a bit later I think, because RL needs to init first [22:34:13] or would we need to do things more differently twice [22:34:30] Actually I think I would recommend doing that, because what you're currently talking about makes it very difficult to ever change the head JS code [22:34:36] brion: it sounds like it could be [22:35:38] bblack: what do you say --- if we went with this approach, is there anything that could be done to make it easier to transition to ESI in the future? [22:35:46] RoanKattouw: that’s a good point, would hate to have to clear all the caches [22:35:54] RoanKattouw, that's basically what we do right now... and it's slow [22:36:03] hmmm [22:36:21] *slow because it serializes requests that *could* happen in parallel [22:37:03] clearing varnish caches once in a blue moon on a major transition (e.g. to ESI here) isn't *that* huge of a deal, so long as we're ok with the cache clear happening slowly [22:37:19] it's different if you need the cache cleared all at once or things are horribly broken for the user [22:37:29] (where slowly is like, over a period of a day or three) [22:37:38] mwalker: What does it serialize exactly? Your code vs the other top modules you mean? [22:37:38] hosting it on the same domain solves the *cant change the head js* problem -- you just then cannot change the URL location [22:37:51] Yes that would also wokr [22:38:11] #info using JS blob now means a cache clear required for ESI transition later, but this bblack thinks it may be managable (especially if can be done gradually) [22:38:14] RoanKattouw, right now it's RL->CentralNotice->BannerRequest->Injection; where it could be RL->Join(CentralNotice, BannerRequest)->Injection [22:38:43] what this RfC proposes is actually Join(RL->CentralNotice, BannerRequest)->Injectiong [22:38:45] (And note that that serializes things just as much, just in the reverse order so the impact is less: RL init waits for CN rather than the other way around) [22:39:21] No, I don't think that's accurate [22:39:33] with that last point, no, if I assume that browsers will execute requests in parallel as they see them [22:39:54] For