[01:40:08] gwicke: Hi. [01:40:23] hi Katie [01:40:44] what's up? [01:41:07] I was looking at and it seemed kind of strange to me. [01:41:30] Like we want to set up API rate-limiting to protect ourselves from ourselves? [01:42:39] I have a draft reply to Brandon's e-mail as well, but I was trying to do a bit more research first. [01:45:27] Katie: we have internal services that hit other services quite hard as well [01:45:45] I'm not sure what you mean. [01:46:11] let me find a link to an old amazon post for you [01:46:24] Is the one about making everything API-able? [01:46:34] Is it * [01:46:56] yes, and designing internal services as if they were external [01:47:54] Okay. [01:48:27] https://twitter.com/gonzohacker/status/585439006588035072 :-) [01:48:58] gwicke: If RESTBase is hitting the MediaWiki Web API, the response of adding rate limits seems kind of weird. [01:49:08] Since we presumably control RESTBase. [01:50:22] in a perfect world, we control everything & there are no bugs or unexpected request surges [01:50:58] Right. [01:51:10] in the real world, we use several techniques to make our system robust [01:51:19] one of them is code review & careful design [01:51:25] Sure. [01:51:35] another is rate limits and other infrastructure measures [01:51:57] operating systems with process isolation, containers etc are all examples of the same principle [01:52:15] In the non-perfect world, there are finite resources. [01:52:26] sure, we could write it all in assembly & run it in ring 0, but in reality, that's a bit risky [01:52:29] And technical debt from new features and functionality has a real and recurring/ongoing cost. [01:52:49] we'd also move very slowly if we did that [01:52:52] which is a huge cost [01:52:58] So when someone says "we need API rate limits" and the answer to "why?" is "so we don't shoot ourselves in the foot," I'm left boggling a bit. [01:53:21] Anyway, I guess I'll reply on-list. [01:53:43] But my question to Brandon was roughly: what problem are we trying to solve other than protect us from ourselves. [01:53:48] basically, lets say the PDF render service mis-behaves in some way and starts to produce more requests than anticipated [01:53:55] Okay. [01:54:01] I'd say to fix the PDF renderer. [01:54:02] we'd prefer that to not impact more important services [01:54:11] and yes, we'll fix it too [01:54:19] but in the meantime, better to avoid it impacting others [01:54:30] belt & suspenders, basically [01:54:50] So the rate limits are for internal tools, then? [01:55:04] no, they are primarily for external users [01:55:13] External users being the PDF renderer and RESTBase? [01:55:27] but they can help to make things more robust internally too [01:55:52] no, external users would be third-party clients hitting our apis [01:56:19] It sounds like inconveniencing everyone because the Wikimedia Foundation occasionally deploys buggy code that hurts Wikimedia wikis. [01:56:36] I don't think anyone expects perfection. [01:56:38] no [01:56:53] I have a security bug open that allowed me to bring down the api cluster with a mobile phone [01:57:00] But before we add rate limits, it'd be nice if there were a defined problem outside of "so-and-so enabled a gadget globally" or whatever. [01:57:03] by opening a single HTML page in mobile firefox [01:57:28] A single HTML page making how many requests? [01:57:28] I think that particular end point is now locked down a bit further & would now require more requests [01:57:32] And what type of requests? [01:57:44] but I'm sure there are others [01:57:48] There will always be expensive API requests. [01:58:08] we have had api outages caused by careless users [01:58:25] What kind of users do you mean? [01:58:26] And when? [01:58:27] many weren't even malicious [01:58:44] Almost all of the times I can remember API troubles related to the Wikimedia Foundation. [01:59:07] Or related to Wikimedia, I guess. [01:59:31] That is, we're more like (D)DoS ourselves than anyone else. [01:59:34] likely * [01:59:40] I'm going to try that sentence again. [01:59:46] That is, we're more likely to (D)DoS ourselves than anyone else. [01:59:59] we had both [02:00:38] I'd prefer to see rate limits put into RESTBase or the PDF renderer or whatever else before being put into the Web API. [02:00:54] I haven't done the research to be able to know which is more common [02:01:20] ironically, most requests in RB are very cheap [02:01:35] they only hit RB storage, and take a dozen ms or so [02:01:55] Katie: For a real-world example, a user had the 'look up on Wikipedia' search button on their e-reader being held down (probably had it lying oddly on the floor). [02:01:57] RB storage is a custom cache layer? [02:02:31] Katie: This triggered millions of search queries (hundreds a second) for the days before they presumably found that it was lying oddly, or it lost power. [02:03:03] You can make hundreds of requests a second from an e-reader? [02:03:05] Katie: This didn't take the search cluster down, but it did add massive load without any user benefit. [02:03:13] Katie: Blame Barnes & Noble. :-) [02:03:28] Was the person working in a data center or something? [02:03:45] Nope, home connection. Speedy but not that bad. [02:03:56] Hundreds a second? [02:04:02] Have you ever tried to hammer an API? [02:04:03] the old security bug is https://phabricator.wikimedia.org/T64615 [02:04:03] It's not surprising. Requests are cheap to fire. [02:04:08] Not that cheap. [02:04:21] Katie: I have not tried to violate criminal law, now. [02:04:23] these days, any phone can send hundreds of requests per second [02:04:26] Yeah. [02:04:35] Really? [02:04:49] Sadly. [02:04:52] Well, no. [02:04:53] Not sadly. [02:04:58] It's the march of technology. [02:05:10] But it's something we need to deal with. [02:05:15] Katie: install wrk, for example [02:05:21] https://github.com/wg/wrk [02:05:46] or apachebench [02:05:51] I've used ab. [02:05:59] or, to a lesser degree, a web browser ;) [02:06:10] I'm incredulous about an e-reader on a residential connection making that many requests for a sustained period. [02:06:26] Going to https://en.wikipedia.org/wiki/Barack_Obama makes a few hundred requests alone. [02:06:40] gtg, ttyl [02:06:57] Fair enough, I guess. [02:07:08] Then that raises the question why it matters. [02:07:20] Since our search infrastructure presumably handles millions of requests anyway. [02:07:47] Because it risks slowing down or in some cases disrupting the service for everyone just to serve a normally out-of-control script, bot or (in this case) button. [02:09:32] Given the already high level of traffic, I'm not sure that argument really holds well. [02:10:16] It seems to point to a similar conclusion that came from research into preventing hotlinking for example. [02:10:46] The fact that you've grown to tolerate the slow service caused by a few bad apples doesn't mean we should continue to inflict that on everyone else. :-) [02:10:50] Anyway, I'm off. [02:10:56] Byeeeeee. [02:11:23] A few bad apples can be blocked without imposing limits on everyone else. [02:11:30] And the services are usually fast for me. [02:13:33] You'd think if an e-reader sends so many requests, it probably happens pretty frequently. [02:13:47] I mean, people hold down buttons accidentally often enough. [02:37:54] Katie, here is a copy of the Amazon quote I promised: https://plus.google.com/+RipRowan/posts/eVeouesvaVX [02:38:25] specifically, 5) All service interfaces, without exception, must be designed from the ground up to be externalizable. That is to say, the team must plan and design to be able to expose the interface to developers in the outside world. No exceptions. [02:39:29] really gtg now, ttyl! [10:04:09] [[Tech]]; Angely roy; /* How do you make a wiki? */Someone help me please; https://meta.wikimedia.org/w/index.php?diff=13431563&oldid=13416264&rcid=6751140 [14:58:07] wait wut ? nik is no longer working for WMF ? [15:20:49] thedj: About a month now? [15:21:18] thedj: sadly :( [15:22:19] thedj: He's still around-ish on IRC tho [22:48:33] spagewmf: we don't generally post the transcript for these meetings anymore, do we? (where "we" is "you guys that have been doing this") [22:49:02] ^ that is a conversation moving over from #wikimedia-office [22:49:29] ^ and "these meetings" is the RFC review that just happened [22:50:41] robla: someone links to the Meetbot HTML from the phab ticket. (Which I just learned is not ideal because not backed up: https://meta.wikimedia.org/wiki/Meetbot#Logs ) [22:51:29] sadly, I knew that already because we lost many transcripts in 2014-ish [22:52:38] Sumana would copy them over to MediaWiki because of that. [22:53:14] robla: right, :-( at https://meta.wikimedia.org/wiki/IRC_office_hours#Past_office_hours_.28no_log_published_yet.29 [22:54:06] sadly, there is a *really* awful amount of administrivia that is both overly complicated and sadly necessary because of our crappy tools [22:54:57] * robla puts on headphones in attempt not to get distracted by loud office conversations [22:57:37] robla: agreed. The VE team ran a lot of office hours, maybe they (James_F, Elitre) had a better process. I would love to see meetbot backed up so a link to its HTML summary is good enough [22:58:27] It'd be more lovely if the meetings were in #wikimedia-meetings. [23:02:57] James_F: what's better about #wikimedia-meetings ? https://meta.wikimedia.org/wiki/IRC/Channels doesn't mention #wikimedia-meetings, the only references I find to it are from 2009 [23:03:14] spagewmf: -office is a stupid channel name. [23:03:22] they have popcorn i hear.. [23:03:36] James_F: how very Wikipedian of you to get fussy with the name ;-) [23:03:41] robla: Thank you. :-) [23:03:56] James_F: oh yeah, I agree it sounds like "We ran out of milk on third floor" :) [23:04:55] * robla doesn't want to embark on a rename, given the already ridculous morass of random wiki pages and phab tickets that point to #wikimedia-office [23:07:52] spagewmf: I think we should replace docs/design.txt with a reasonably complete summary of mw:Architectural_guidelines [23:08:08] * robla heads off to file Phab ticket for that [23:13:05] robla: as .wiki that someone copies to mw.org, or as .md that doxygen builds to https://doc.wikimedia.org/mediawiki-core/master/php/ ? FWIW hashar and Krinkle like the latter, DanielK_WMDE_ and others like the former [23:14:06] robla: current situation is embarrassing, https://doc.wikimedia.org/mediawiki-core/master/php/design_8txt.html agggghh [23:14:25] https://phabricator.wikimedia.org/T111283 [23:15:46] spagewmf: your link is pretty much exactly why I linked to Github from T111283 [23:15:50] spagewmf: That's not a good example as that file was never written with doygen in mind [23:15:53] Here's a better one https://doc.wikimedia.org/mediawiki-core/master/php/group__FileBackend.html#file_backend_design [23:16:03] From https://github.com/wikimedia/mediawiki/blob/HEAD/includes/filebackend/README#L3 [23:17:56] The main reason for maintaining in-git alongside code documentation: reviewed by developers, maintainable in an atomic way (e.g. can expect a developer to update docs at the same time, incl. new contirbutors instead of after merging remembering to update; if it was merged), and versioned alongside MediaWiki (e.g. keep docs for 1.25 table and without distractions). There's more reasons I could think of [23:17:56] . [23:18:14] stable* [23:18:39] but I agree without doubt that the current rendering for "pages" in Doxygen in shit, even the "better" examples like FileBackend. [23:19:13] Krinkle: yup, filebackend/README is great and yes I agree on the docs in git benefits. Doxygen now supports .md and I think devs would find it easier than all that doxygen stuff, [23:19:58] Krinkle: +1 [23:20:07] there's also success stories from git-core, nodejs, karma and others that maintain docs in a similar way. [23:20:22] the reason why I started thinking about docs/design.txt is that I think there are many elements of AaronSchulz's RFC we discussed today that should be in there [23:20:28] but a) I couldn't get .md to work locally, and b) there's a problem where the README shows up in MediaWiki > Introduction > Introduction (phab filed) [23:21:27] I think we should consider seeking alternatives to Doxygen. It's not made for what we use it for. It's screaming constantly (if you care to look at the warning output), it wants to be put out of its misery. [23:21:53] For one, it doesn't even support PHP. We pull the raw php file source through a regex filter (maintenance/mwdoc-filter.php) to make it kind of work. [23:21:57] spagewmf: i liked your suggestion to just transclude files from the git repo directly into wiki pages. but purging is tricky... [23:21:59] Breaking lots of comments in the process. [23:22:23] Krinkle: sure, let's talk about it. [23:22:28] * robla will be very sad if important architecture conversations take a backseat to Doxygen discussions [23:22:40] * Krinkle was responding to pings. [23:22:52] By all means carry on :) [23:23:08] robla: well, how to organize and maintain documentation is a pretty important issue. but we shouldn't get stuck in bikeshedding about the ideal tool. [23:23:36] Krinkle: no problem...I'm just worried about our general ADD we have about architectural conversations. Not specifically the Doxygen tangent [23:23:37] Krinkle the other approach that DanielK_WMDE_ mentions is https://phabricator.wikimedia.org/diffusion/EWBA/browse/master/docs/lua.wiki in git, that "a person/process" transcludes to https://www.mediawiki.org/wiki/Extension:Wikibase_Client/Lua [23:23:50] I think for architecture, it may make sense to iterate on mediawiki.org. It's not that tightly coupled with current software day-to-day changes and use by third-parties. [23:24:31] Krinkle: I'm just trying to get https://www.mediawiki.org/wiki/Architecture_guidelines the attention it *needs* [23:24:50] we don't link to it from docs/design.txt [23:25:10] I've never read docs/design.txt [23:25:21] https://phabricator.wikimedia.org/T111283 [23:25:47] Krinkle: i'm especially keen to improve medium-level documentation, and i think that should be on git. thinks like information flow or control flow, documentation of how the parser cache works, or how to control web cache, etc. [23:25:49] Krinkle: exactly. what is the "must read" material we have from your perspective? [23:26:15] robla: 10 years of commit history and wikitech-l posts to warm up :D [23:26:27] But more seriously, let's see. [23:26:54] "Security for developers" is it from my perspective [23:27:06] DanielK_WMDE_: Yeah, I've been trying to get a few pieces into place with Aaron with regards to caching [23:27:20] robla: i have been working on a draft, but got stuck. https://www.mediawiki.org/wiki/User:Duesentrieb/Architecture_Guidelines [23:27:26] (very much Work In Progress) [23:27:35] I don't mind much where it lives, but I noticed that time and time again I was unable to review Aaron's patches because the changes seemed arbitrary – I mean, it wasn't obvious why it would be done this way, the code allowed a dozen other ways. [23:27:35] * robla needs to head out of the office in a couple of minutes [23:27:50] see https://github.com/wikimedia/mediawiki/blob/89a167b500dc857f9def03bd83d951161d312d4d/includes/objectcache/ObjectCache.php#L43-L70 [23:27:58] Things like that help a lot. [23:28:12] but needs better discoverability. It's tucked away in Doxygen right now. [23:28:45] exposing it on pages on mediawiki.org would help, i think. [23:28:45] robla tweak or rewrite docs/design.txt and any delivery approach you take (leave as .txt, doxygen-ify, or rename to .wiki) is fine for now. [23:28:55] Having a better hierchy in the code and a way for the documentation platform to interpret that as highlevel modules/categories would help a lot by reducing the number of links you can click from doc.wm.o/mw [23:30:15] E.g. going to https://doc.wikimedia.org/mediawiki-core/master/php/, I want to see a list of components from includes/ and each going to a page with introductory info, and list of contained classes and configs. [23:30:57] Introducing the high level concepts of what makes MediaWiki what it is. Up to the reader to explore the ones they're interested in. [23:33:12] Krinkle: as you know, those ObjectCache comments became https://doc.wikimedia.org/mediawiki-core/master/php/classObjectCache.html#details , but I don't know if doxygen will have good material until someone points it out to me. [23:33:15] Compare to https://doc.wikimedia.org/mediawiki-core/master/js/#!/api/mw.Api for example – I've removed almost all on-wiki documetnation related to JS modules since this satisfies that need quite well imho. [23:33:52] spagewmf: Yeah, there is no good user experience between Doxygen home -> /classObjectCache.html#details [23:33:56] you'd have to know where to look [23:34:40] Anyway, we're talking about documentation again :) [23:35:37] good conversation y'all! I'm going to go afk here, but I may eventually respond to pings :-) [23:39:11] Krinkle: I love it :) a) I'll see about adding to mw.org's {{MW file}} a parameter "showLinkToDoxygenItsActuallyGood" . b) developers could have a sprint to improve doxygen, but hard to prioritize it if "we should consider seeking alternatives to Doxygen." [23:39:45] I mean improve doc.wikimedia.org/mediawiki-core/master/php/