[00:55:41] You're here? [01:01:50] no [06:11:59] So the citoid extension seems to be not working on en wiki only: https://phabricator.wikimedia.org/T104944 [06:12:21] The only thing I can think of that would cause that is for the mediawiki namespace message to be broken, but there are no edits to it [06:14:30] hmm, when was the deploy? [06:15:06] possibly an i18n thing? That's the only thing that would have gone through this week [06:23:07] no, it hasn't been cut yet, much less deployed.... herm [06:36:50] Well, it's definitely a namespace message issue. It's not coming through on en wiki only for some reason [09:24:05] omg phabricator font URLs changed so I need to update blocked URLs [10:10:44] a [10:10:46] Hi there [10:11:05] I am trying to understand the wikimedia and especially the wikipedia internal architecture [10:11:24] I am designing a world wide distributed system [10:11:50] with similar requirements (read heavy, document based ...) to the wikipedia [10:12:09] since I know that different people around the world can edit wikipedias documents [10:12:24] I wonder if the documents hit different data centers distributed world wide [10:12:41] no [10:13:15] so you mean the world is working with a single database center? And only read requests are forwarded to local installments? [10:13:29] (beside a backup datacenter?) [10:15:16] nemo? [10:46:31] BeaverP2: see https://meta.wikimedia.org/wiki/Wikimedia_servers#Hosting [10:48:40] BeaverP2: Hi [10:48:45] https://wikitech.wikimedia.org/wiki/Clusters [10:49:01] BeaverP2: Wikimedia uses cache PoPs to distribute caching and read traffic. [10:49:18] write traffic (e.g. POST requests) go the primary data center. [10:49:56] There are currently two application data centres (e.g. one that can be made primaries) and several caching clusters (read-only) [10:52:27] BeaverP2: Having multiple places accept changes is a cool idea but highly complex in practice. First of all, primary key conflicts when creating new objects. And secondly, merge conflicts for objects with revisions (e.g. who's save wins when there are more than one in a small timeframe). [10:53:17] One way to resolve that is with distributed consensus and live replication. However doing that for application level objects is quite hard. Using it for determining which server is the master (and change this over time) is a lot more feasible. [10:53:18] :) [10:54:41] thanks for the links [10:54:43] i read those [10:54:55] I understand the problems I am currently facing it myself [10:55:24] The system that i design right now is somewhat similar to the wikipedia but aims to the market instead of knowlege [10:55:32] While it requires a large infrastructure to handle global write traffic in a single data centre, you can take at least Wikipedia as example where this scales just fine. [10:55:50] " aims to the market instead of knowlege" what a false dichotomy [10:56:33] the single master datacenter when it comes to write really reduces the inherent complexity of the global system [10:56:52] BeaverP2: Yeah. [10:57:09] We currently only do manual switch overs, but thats mostly because we have legacy issues to support. [10:57:12] i wonder if people editing the main document storage accept the latency of roughly one second [10:57:17] When designing a new system I'd strongly recommend not to make that mistake. [10:58:06] i use a principle that uses a main storage [10:58:16] and the actual services are compiled using the main storage [10:58:24] e.g. make sure your system can designate A or B as primary. Even have it change from time to time just because you can. I haven't tried it at scale but I found Raft Consensus quite neat. [10:58:41] so the main storage does not fiddle with indexes and how to represent things in order retrieve them fast within a certain query type [10:58:54] BeaverP2: Yeah, the latency is non-trivial. However we do what we can to mitigate it. [10:59:05] Raft is the stuff everyone like cassandra uses right? [10:59:49] for one, such request will go through the caching centre, so you don't rely on the oceanic backbone of your user's ISP. But rather from the nearest caching centre phone home through your own high speed network [11:00:01] i thought about faking the response but with the current approach you have something like do a change, write to main document store, distributed services listens to it update shards done [11:00:10] but the latency would be awesome [11:00:46] depending on how the app exposes information, you can also handle requests in a non-master centre and have it only do the database write to the master. [11:01:04] krinkle thats the idea i explore [11:01:10] but i also need local data [11:01:26] we're currently experimenting with that. e.g. the secondary data centre handles the request, and makes a db connection only for one or two queries at most to the main data centre (over WAN) [11:01:27] that is written locally and stored in bulk in the main [11:01:44] this data is set to can go lost in terms of data center failure [11:02:27] I don't know what cassandra uses, but I'd expect it to use one of the consensus algorithms yeah [11:02:49] the interesting architectural choice is realy compiling services entirely from a main helps with recovery due if a shard can not be recovered it can be recompiled from daata [11:02:51] It only works if everything plays together though. The more sytsems you have the harder it gets [11:02:57] they use raft [11:03:04] mysql, memcached, nginx, varnish, load balancers. [11:03:11] everything well-contained. [11:03:12] i factored all those out [11:03:15] cool [11:03:26] if you check infinity cache its roughly only 200kb of work [11:04:12] i currently plan with cassandra since it is embeddable but since I have shards using its own storage engine (just columnwise tables) i asked myself why i need cassandra [11:04:35] so it comes down that each shard uses the tech available in java but the storage is done by us [11:04:49] embeddable in what way? [11:05:07] having java code (project is entirely java) that can be started within the same process [11:05:20] Aye, I see. [11:05:28] in order to be used it must be also automatically configurable [11:06:04] but for the distributed services we have to implement that much that every distributed store is an unnecessary dependency [11:06:27] if you look at the internals all boils down to a memory mapped file and a way to store things along with indexes [11:06:38] sure the query language but thats not much to do also [11:06:58] the optimization is beat to death with hardware and inmemory calculations [11:07:24] along with caches that do not replicate and are bound with a single shard and therefore shared nothing [11:09:18] The only thing that gets reused throughout the project is jGroups (membership) and zeroMQ for inter process communication [11:09:38] an embedded jetty is used to reuse web tech [11:11:00] But since now the storage engine (cassandra) is also gone it looks simple but freightening (lot of wheels to be reinvented) but it is easier since we control protocols and implementation [11:11:15] does anyone have some figures for the data center and traffic costs? [11:11:49] Wikis down :/ [11:15:06] Anyhow Krinkle thanks for the help [11:15:53] BeaverP2: You're welcome [11:15:57] BeaverP2: Yeah, caching is important. [11:16:27] BeaverP2: Cache propagation is also a good one to get right. Try to allow indefinite caching and purge proactively. [11:16:30] for the sake of simplicity I will go for the single site and use main logical datacenter being composed of multiple datacenters closely related (latency < 50) scenario [11:17:30] The cache is build in by doing shard duplication (even if the replication criteria is met) [11:17:47] The only thing to get right is the consistency [11:18:05] BeaverP2: I don't have the numbers here. [11:18:08] usually users will not care if a shard is used that is lets say 30 seconds behind master shards [11:18:21] but even then, it'll be hard to put them in perspective. [11:18:37] what numbers are you referring to? [11:18:40] costs? [11:19:37] the goal is to build a system where a special cache strategy is not used but build in feature of each service [11:19:43] but i am still unsure [11:20:04] the simulations i do on my local machine looks good but i do not know yet [11:20:24] What kind of data are you caching? How is it accessed? How does it end up in context of end-users? [11:20:29] it is implemented by having asynchronious slave shards that listen to a master shard collection in order to update themself [11:20:55] there are many datas some are similar to wikipedia knowledge pages [11:21:09] others are statistics others are reviews and alike [11:21:38] for example having statistics those are done locally and are written to the master only once every x hours [11:21:46] after being aggregated [11:22:17] having shards with special location preference is another concept that is hard to get right and that i design for but leave out in the first implementation [11:23:04] the enduser chain is like -> request -> web / android application -> frontend ->* services -> service ->* shards [11:23:57] consistency is achived on writes by modifying the document store and get a certain timestamp back and requesting a service view that regards to the timestamp [11:24:20] So you can ensure to see a version of the global data that are at least including the given timestamp [11:24:33] but it will be a logical timestamp [11:24:55] but making the document store as the only guaranteed persistence store the main bottle neck [11:25:03] Hm.. I'd recommend against timestamps [11:25:06] so I learn from the cassandra implementation in that regard [11:25:11] logical timestamp [11:25:17] more like a timestamp on a snapshot [11:25:17] meaning what [11:25:28] is it deterministic? [11:25:39] in what regard? [11:25:55] it is the document stores internal way to group modifications [11:26:10] hashes may safe trouble down the line [11:26:24] ? in what regard? [11:26:48] think about the following scenario. You have lets say 100k writes per second to the document store (to be extreme) [11:26:50] are all modifications explicit and direct with a specific time bound? [11:26:58] Or can changes propagate implicitly? [11:27:04] e.g. data that is included from an external source [11:27:13] or from configuration data [11:27:20] ah now i understand [11:27:34] I've found that in many cases timestamps will bite you. [11:27:36] external data is handled differently and can not be timestamped [11:27:49] timestamps (or sequence numbers of modification blocks) [11:27:51] as soon as you have any data involved that doesn't have a reliable timestamp, you are forced to make up your own timestamp [11:28:00] are only guard against the document store [11:28:01] e.g. track when you first saw it or something like that [11:28:12] but that ends up a mess [11:28:16] and isn't deterministic [11:28:38] Now adays I'd probably start with hashes, even if only the hash of a timestamp. [11:28:48] I only plan for versionized documents and logical timestamps to track those document store changes [11:29:29] in order to get a timestamp after you have seen something you have to ask the document store for a save to use timestamp reference (i marker that is guaranteed to come after your modification is propagated) [11:30:15] do you actually need the time value of the timestamp, or merely a string to identify a version? [11:30:30] e.g. will you compute it into offsets, or compare with < or > [11:30:53] for example a resource is a document store entry but the binary is stored otherwise in order to keep sync the binary is stored in a special service and a document is created after the resource is stored and the timestamp of that document can be savely used [11:31:08] for whatever resource usage or timestamping [11:31:25] i have a logical clock [11:31:47] and only thing something outside can say the event came before the logical timestamp i optained from master right now [11:32:18] the versioning of a particular document (and the data items it reflects) is build in the document store and is independent [11:33:02] look at it like you can ask the document store about all changes happening from timestamp x to y which might include changes you already know (since having timestamp x does not mean timestamp x was over) [11:33:16] so a certain point in time is always a range since it is a logical timestamp [11:33:38] and the range is at least twice the time it takes for a document node to talk to its master document node [11:34:15] so the internal logical clock divergates at most one timestamp (which is the reason why the internal timestamp resolution is different than the external the services use) [11:34:58] thats why i have to plan with closely related datacenters when it comes to the main document storage [11:35:05] just to get the clock right [11:35:26] or i rely on the gps time which is guranteed to be closely to a certain bound [11:35:43] that is not determined now [11:35:56] cassandra for instances uses the clock and argues with the gps signal [11:36:10] i am not sure which route to go [11:36:22] its left out for testing [11:36:45] depends also on the gurantees a installment will provide [11:37:14] does this makes sense to you? [11:56:26] BeaverP2: I didn't know about logical clocks. interesting [11:57:00] Still, it seems like a lot of work to maintain that throughout different layers of the infrastructure if you have more than a single monolithic code base. [11:57:23] it was popular 15 years ago but today a global clock called gps exists [11:57:29] a hash of the meta data, document or event data or whatever you have access to would be easier to orchestrate [11:57:49] and is deterministic in case things need to be replayed from logs in case of out of sync [11:57:56] krinkle that is why the system is designed as a monolitic code base :) [11:58:07] But runs on more than one server. [11:58:22] yeah one could use a hash of a system state but again you need a coordinateor [11:58:26] I just try to avoid attaching meaning to sequence or time stamps as a principle. [11:58:30] saves a lot of trouble in my experience. [11:58:40] But you can use it as a distributed increasing integer. [11:59:04] for example each shard has a modification operation count and each shard can only be modified sequencially makes this also a logical clock [11:59:20] you are totally right [11:59:34] but i need a clock (logical or physically) in order to handle failure [12:00:02] so i have this shard modification count + a sequence of checksums (your hashs) [12:00:10] in order to detect divergent shards [12:00:23] along with a modification log for each shard making local recovery [12:00:58] global recovery does not makes sense since the system can just replicate a master shard or even compile it again based on the main document store [12:02:01] so the only thing to get really right is some mean of consistency in terms of read requests (to ensure that the result is only xxx seconds behind the core state of the system) [12:02:22] and for write i hit the main core since wikipedia shows that it is good enough [12:02:24] :) [12:03:04] so basically the duplication of shards is the idea to control read throughput [12:03:35] and write throughput is limited by the possibility of the master core and its processing capabilities [12:03:56] since documents are single islands (similar to mongodb without being interpreted and therefore searchable) [12:04:08] it scales linear and is no bottle neck at all [12:04:37] the write latency is simply the internal latency of the core and the latency of the network to reach that core [12:04:49] i hope that makes sense to you [12:06:08] for example our wiki part would be something like a sharded store where the database is shared based on the language first and then is shared based on the document id [12:06:44] it also maintains an index for the document titles and sections in order to allow linkage [12:07:17] this index is currently planed to be sharded independently of the documents but being closely related [12:08:09] so if you want to display a wikipage you will hit the shard storing the page document and one copy of all shards indexing the linkable structure [12:08:31] since the system will detect the high utilization of index shards those are replicated more often [12:08:37] done [12:08:57] this way every service is realized [12:09:42] a shard size is different and is planned to be about 10GB max in size and once it hits the size it is split [12:10:02] easy enough system i hope [12:10:28] so thanks alot krinkle to verify that wikipedia is using a (dual)core center [12:10:40] helps alot with what the system looks like [12:11:01] currently i read the new version of the zeromq documentation [12:11:05] Yeah, feel free to join #wikimedia-operations and ask for details [12:11:17] maybe we can drop the dependency of jgroups [12:11:40] what does wikipedia use for its internal communication? [12:11:48] i only read php [12:11:58] this channel is mostly for on-wiki technology such as gadgets and templates, not the engineering and operations of the site as a whole. [12:11:58] so using the database layer as communication layer? [12:12:07] ah ok [12:12:15] and operations is? [12:12:27] The DevOps and data centre ops folks [12:12:41] ok thats nice [12:12:47] will join there [13:57:49] A detailed description of the error is shown below: [13:57:49] $.ucFirst is not a function TypeError @line51 [13:57:55] any iddea what is causing such errors? [14:11:43] hi Steinsplitter [14:12:03] Steinsplitter: missing dependency on 'jquery.mwExtension' [14:14:30] MatmaRex: maybe you can take a look? would be higly appricated :) https://commons.wikimedia.org/wiki/MediaWiki:Gadget-AjaxQuickDelete.js [14:14:39] thanks :) [14:16:00] Steinsplitter: looks like Rillke just fixed it. 'jquery.mwExtension' [14:16:02] agh [14:16:05] https://commons.wikimedia.org/w/index.php?title=MediaWiki:Gadget-AjaxQuickDelete.js&diff=165220958&oldid=164696809 [14:16:35] he was fast O_O, thanks aniwyay :) [14:33:22] guillom: around ? [14:33:42] matanya: in a few minutes, unless it's urgent? [14:34:29] guillom: no, just when you have got a second, have a look: https://he.wikipedia.org/wiki/%D7%A9%D7%99%D7%97%D7%AA_%D7%9E%D7%A9%D7%AA%D7%9E%D7%A9:Matanya [14:35:22] matanya: Yes, matt told me yesterday. I commented at https://phabricator.wikimedia.org/T104914 [14:36:15] guillom: i was referring to the fact i got the message 3 times [14:36:22] Oh; hmm. [14:36:47] Well, I certainly only sent it once. [14:36:55] So I don't know what happened. [14:37:25] you think flow related [14:41:21] Probably. [14:47:31] Does someone know who and how can see the statsd statistics for Wikimedia wikis? https://gerrit.wikimedia.org/r/#/c/217752/ [14:47:57] edit.failures.session_loss [14:49:22] I also wonder whether https://wikitech.wikimedia.org/wiki/Graphite#Statsd is up to date [14:59:56] Nemo_bis: iirc it is in graphite, so if you have access to that, you may see it [15:31:11] I don't, what questions [16:34:55] PL-NL perpetual domination ;) https://www.mediawiki.org/w/index.php?title=Developers/Maintainers&diff=next&oldid=1640820 [16:37:49] this cannot end up well [16:52:13] hah [18:06:37] Kindly Requests for new languages/Wikipedia Saraiki [18:07:15] be verfied and approved [18:07:18] saraiki, https://meta.wikimedia.org/wiki/Language_committee ? [18:07:54] I don't see Saraiki listed under https://meta.wikimedia.org/wiki/Requests_for_new_languages [18:08:16] ah, it's just not linked from the list: https://meta.wikimedia.org/wiki/Requests_for_new_languages/Wikipedia_Saraiki [18:08:40] Saraiki is listed dear [18:09:55] in https://meta.wikimedia.org/wiki/Category:Open_requests_for_new_languages [18:12:02] Dear andre__ look ino mattar [18:12:51] Frakir [18:14:24] ? [18:14:37] Requests for new languages/Wikipedia Saraiki [18:14:58] saraiki: you have the wrong channel [18:15:21] what i do [18:15:44] I don't know where the right channel for this is, but it's not here [18:16:02] saraiki, have you contacted the Language committee? [18:16:54] ai what link? [18:17:03] see the link I pasted above [18:18:15] ok [20:31:33] a question [20:31:40] are there graphs and etc of blacklogs? [20:32:25] blacklogs? [20:32:47] backlogs [20:32:54] sorry for the typo [20:36:04] ToAruShiroiNeko, what backlogs? [20:36:23] wikipedia has boat load of backlogs [20:36:25] deletion backlog [20:36:34] ask wikipedia? [20:36:44] I dont know, anything is a backlog [20:36:55] it is a technical question [20:37:01] do you know statistics for these? [20:37:10] no [21:51:45] is there a simple way to get number of deleted files and pages on wikipedia? [21:52:51] ToAruShiroiNeko: http://deletionpedia.org/en/Special:Statistics ?:) [21:54:10] doesnt look accurate [21:56:50] ToAruShiroiNeko: https://en.wikipedia.org/wiki/Special:Log/delete [21:57:12] yea, i dont know about the sum of all pages [21:59:26] umm I am not cointing them one by one :p [22:00:45] API and then open result in ooffice ? [22:16:08] mutante a straight answer would be most useful. can I quary it at least? [22:21:29] ToAruShiroiNeko: yes [22:21:39] ToAruShiroiNeko: https://www.mediawiki.org/wiki/API:Deletedrevs [22:22:09] "You need the deletedhistory right to use this module." i don't know who gets that right [22:22:21] on each wiki [22:24:34] Administrators, basically