[07:37:50] Wow, haven't been in here in over a year. [09:39:54] Hi all! [09:40:32] does anyone knows what are the ways to do incremental updates? [09:40:43] so far I got rss, api call and socket.io [09:40:48] is there any other? [10:11:17] luizk: incremental dumps http://dumps.wikimedia.org/other/incr/wikidatawiki/ [10:12:05] mmm [10:12:08] cool! [10:12:40] Lydia_WMDE: https://phabricator.wikimedia.org/T124757 [10:12:49] Adrian_WMDE: merci! [10:17:39] benestar: Are you online? [10:45:53] jzerebecki / aude by any chance around? :P [10:51:24] addshore hello [11:02:43] hi HakanIST [11:04:54] addshore is it possible for you to update https://www.wikidata.org/wiki/Wikidata:Creating_a_bot#Wikibase_api_for_PHP according to client available at the github link? [11:05:52] HakanIST: perhaps! [11:05:59] your better off reading the README actually [11:06:05] it might make more sense to get rid of those examples! [11:07:13] with the readme I came so far as to add statements but still cannot add references [11:11:15] HakanIST: okay! [11:11:18] give me a second! [11:12:40] HakanIST: there is a ReferenceSetter service you can use ;) [11:14:54] addshore https://paste.ee/p/IKHCh , part 1 is working but part 2 is not [11:15:06] addshore don't know if you've seen https://www.wikidata.org/wiki/Wikidata:Project_chat#Wikilinks_and_redirects :) [11:17:59] HakanIST: so statementCreator::create returns the GUID of the statement you have created [11:18:41] if you want to create a reference on the statement you have you created you need to pass that GUID into the $statement param in ReferenceSetter::set [11:20:10] Alphos: oooooh :) [11:20:35] addshore ok will try that [11:20:39] :) [11:21:06] i came up with a cute sql query ; sadly, i can't say it's super-duper fast when it gets to bigger wikis, and jynus wasn't really cool with that :D [11:21:56] read your PM to see what i mean ^^' [11:25:52] :) [11:41:19] Lydia_WMDE, hi, you there? :-) [11:41:39] was wondering if you could assign https://phabricator.wikimedia.org/T115792 to someone.... it's been months now, and this bug is very annoying [11:42:45] +1 [12:00:40] Jhs: looking [12:01:13] thx :) [12:03:46] Jhs: sjoerddebruin: adrian is currently working on solving the language list issues. so i'll see if this will be solved as part of that. [12:03:58] :) [12:04:01] Lydia_WMDE, cool :) [13:42:27] hey Thiemo_WMDE, now I am :) [13:55:18] Thiemo_WMDE: in entitytermsview, I really don't get what I should test.. [13:55:39] what do you expect to happen when the userLanguages config is empty? [13:55:41] benestar: Sorry, we are in story time right now. [13:55:52] k [13:56:00] benestar: I already started working on your patches and will continue that. [15:35:55] Jonas_WMDE: your +2 won't do anything since it needs a rebase... [15:36:20] manual rebase? [15:36:44] Just want to tell @Thiemo_WMDE that I am fine with this change [15:56:29] Jonas_WMDE: yes, manual rebase, but I'm not sure if Thiemo is still working on this locally so I'm waiting for his response [15:58:20] benestar: I will continue working on this tomorrow. I will do more changes and rebase your patches on top of these changes. This will happen tomorrow, hopefully. [15:59:06] benestar: I created two subtasks for this: https://phabricator.wikimedia.org/T92759 If you have time, you can start working on this. I have to go now. I will continue working on this tomorrow morning. [16:00:33] benestar: The reason for all this is this change (the "value" method): https://gerrit.wikimedia.org/r/#/c/256195/13/view/resources/jquery/wikibase/jquery.wikibase.entitytermsforlanguagelistview.js [16:01:01] This slows the UI down. We can get rid of this, mostly. [16:03:41] ok, thanks for your work Thiemo_WMDE [16:03:48] I will have a look into this [16:40:43] DanielK_WMDE: OMG I moved something to use only TitleValue :D https://gerrit.wikimedia.org/r/#/c/266524/ [16:40:46] benestar: got my ping here? https://phabricator.wikimedia.org/T123828 [16:41:53] yes, I know the gadget is broken [16:42:07] I have no other fix than just showing the heading for users of that gadget though [16:42:12] perhaps I should just implement that? [16:43:14] addshore: wanna see!! [16:43:24] try again [16:43:49] benestar: everything better than nothing. [16:44:00] I was in the misunderstanding that you were busy or something. [16:44:10] i *am* busy but nvm :P [16:44:14] university stuff... [16:53:33] addshore: \o/ [16:53:33] any review of that thing would also be great ofc ;) [16:55:55] sjoerddebruin: does the gadget work for you now? [16:56:07] Jep [16:56:08] still needs some style adjustments but the headings should be displayed again [16:56:20] do you have a nice "copy" icon? [16:56:32] no icon yet [16:56:50] * an idea for :P [16:57:07] Ah, so. Isn't there a OOjs UI-one? [16:57:47] For "copy some text"? No, sorry. [16:59:25] Hm [17:00:16] WikiFont, Wikicons, stuff is hard to find anyway. [17:01:28] But, I think copy is very hard to describe with a icon. [17:01:41] https://commons.wikimedia.org/wiki/File:Copy_font_awesome.svg [17:02:04] Yeah, but then with this icon. https://commons.wikimedia.org/wiki/File:CiteArticle.svg [17:02:06] not sure if we are allowed to embed that in our software though [17:03:20] I also like the addition of a arrow. http://static1.squarespace.com/static/512f7f28e4b0e0699d176018/t/53eb2ad3e4b0ba68f27e9ad1/1394012260426/iconmonstr-copy-7-icon-256.png [17:35:14] sjoerddebruin: when invalidating your cache you should get some nice results :) [17:35:48] Only downside is the white background. ;) [17:36:25] Nice icons. [17:36:42] Some margin/padding between add and insert btw [17:37:27] it doesn't have a white background =o [17:37:31] at least it shouldn't [17:37:48] it's only a thick white line [17:37:51] Ah, yes. [17:37:57] Not needed imo. [17:44:08] erm... https://www.wikidata.org/wiki/Special:Contributions/Ndeofm [17:44:50] Wikitext, what's that? ;) [17:45:15] well https://phabricator.wikimedia.org/T124356#1966209 :P [17:58:50] DanielK_WMDE: It would be nice, if you could have a (preliminary) look at https://phabricator.wikimedia.org/T124737 [18:05:01] hoo: i nmeed to remember why I was convinced we need eu_touched [18:07:17] hoo: the tricky bit is that we need to track usages for all target languages. These get created on *view*, not on save. They don't trigger a LinksUpdate at all. [18:07:40] still need to think about it more. [18:08:36] DanielK_WMDE: Ok... it took me quite some time to get my head around it [18:08:47] and I might still have missed something [18:09:02] me too ;) [18:17:15] hoo: commented. [18:42:04] DanielK_WMDE: Replied… I still think we can eventually go without any additional fields [18:47:43] hoo: you are basically right, but nasty race conditions make this very hard to do. Especially ApiStashEdit. See my reply. [18:51:55] hoo: actually - how does ApiStashEdit prevent the stashed ParserOutput to be purged when the edit is saved? Or does it actually use a different key? [18:53:29] DanielK_WMDE: Stashed edit only saves to object cache [18:54:18] Not sure how exactly they are applied to the database on save [18:54:18] ah, that would help. I think we trigger only on the ParserCache update, not on re-parse, right? [18:54:27] Still a potential race condition, but not as problematic [18:54:56] * DanielK_WMDE goes to refactor ObjectCache to use DI [18:55:43] I think we save additional entries on re-parse… but it should be trivial to do that for stashed edits [18:55:57] also in case we do, that would only mean more entries, not less [18:56:11] and the superfluous ones would get purged of post edit [18:56:29] not nice, I agree… but almost everything is nicer than needing to touch tons of rows on purge [18:57:05] * not do that for stashed edits [18:57:55] oh, I'm wrong [18:57:58] we only listen to doParserCacheSaveComplete [18:58:36] so that would not be a problem at all (maybe only the way stashed edits are applied later on, but that shouldn't be a problem) [18:58:48] because we already really handle that right npw [19:07:58] DanielK_WMDE: I think we need to be clear that the table is only for purging. It might work for other things, but we shouldn't try to model it for that. [19:11:44] hoo: how is touchign tons of rows on purge worse than deleting tons of rows on purge? [19:15:30] DanielK_WMDE: Well, I assume that most entity usage will stay in place [19:16:11] hoo: but we will not know that when the purge happens. we only know that once the page is re-parsed. [19:16:15] so we can't update before that [19:16:26] Sure [19:16:33] how? [19:16:38] but I don't think that's a real problem [19:17:02] I mean we don't handle that right now as well [19:17:03] Took me a couple of weeks to find the right mechanism for purging [19:17:35] hm [19:17:38] We do, kind of. We mark the entries as stale via eu_touched. The we purge the stale ones eventually. [19:17:40] We would only purge old rows once we have the new usages [19:17:57] Yeah, but we purge the stale ones on links update runs [19:18:03] and how do we know that these other rows are not used be other renderings? [19:18:05] and we would just keep doing that [19:18:28] We don't... and we don't care… if we purge of rows, all caches are lost [19:18:34] so only the new ones matter for caching [19:19:15] i'm confused. so, when something gets added to the parser cache, we add tracking entries (and don't remove any, because we don't know which ones are still needed) [19:19:20] When exactly do we ever remove any? [19:19:48] DanielK_WMDE: On links update, when core updates its secondary tables [19:20:18] hoo: so only on edit, not on purge [19:20:24] which is essentially what we already do right now... in that hook we insert the (then) new usages and delete ALL old ones [19:20:38] only on edit and purge with linksupdate [19:20:46] might be good enough, though i'm afraid portal pages will accumulate a LOT of stale usages this way [19:20:48] but not the regular purge [19:20:51] they are hardly ever edited [19:21:09] like the Main page [19:21:39] Links update also runs when a template included gets changed [19:21:39] because that might also change core's tracking tables [19:21:42] hm, true [19:22:09] We might get some stale entries here and there, but I don't think we should try to avoid that [19:22:16] we need to make sure we have everything relevant [19:22:25] hoo: ok. so, purge all tracking entries on LinksUpdate, and add tracking entries when a PO is put in the PC. [19:22:27] having more is not nice, but a necessary evil here [19:22:33] yes [19:22:36] But how do we tackle the race condition? [19:22:49] Which one exactly? [19:23:01] LinksUpdate is a job. Putting things into the parser cache happens synchonously. [19:23:30] so the purge will happen *after* we add the tracking entries for the primary rendering [19:23:58] we'd need to put that into a job, and make sure it happens after the LinksUpdate job. I don't think there is a mechanism for that at the moment [19:24:40] hoo: i came across all this shit wile implementing usage tracking. which is why it took so long... [19:25:06] iirc, i added eu_touched after we had problems with this race condition [19:26:06] The gap we have is: Someone edits "foo" (with uselang=en), then someone visits it with uselang=es, then links update runs and throws away the entries for the Spanish user [19:26:15] Right? [19:26:34] the entries that are unique to the Spanish user [19:26:36] hoo: no need even for the spanish version [19:26:54] you edit, a rendering for "en" gets saved and tracked. Then linksUpdate runs. [19:27:38] but in that case LinksUpdate would have that very same ParserOutput [19:28:51] hoo: ah, true - if you don't just purge everything in LinksUpdate, but just prune the ones not in the new PO. Which makes sense of course, and it'S what LinksUpdate usually does for other links tables. [19:29:00] yes [19:29:02] There is currently no code for that, but it's easy enough to write [19:29:22] Actually we already do that in DataUpdateHookHandlers::doLinksUpdateComplete [19:29:42] we take the "current" entries from that ParserOutput, save them to cache [19:29:42] So, ok - that leaves the gap you mentioned: if there are additional things to track for a rendering in another language that gets cached before the LinksUpdate runs. [19:29:49] and then delete everything older than wfTimestampNow() [19:29:58] so if taht's a problem it's also there in the current implementation [19:30:35] hm, i think the intention was to use page_touched there, not wfTimestampNow() [19:30:43] I recall some fiddeling there [19:30:58] I think we tried that but it didn't work out for some reason [19:31:02] I remember reviewing a patch [19:31:20] So it's essentially current behaviour [19:31:34] yea... can't remember why. And wfTimestampNow() worked well enough. I probably forgot about the edge case you mentioned. Or decided that it's Good Enough For Now (tm) [19:35:26] I guess in the end we will have to stick with a solution that works well enough [19:36:03] Especially if we decided to make usage tracking more fine grained (which I guess we have to), we need to make sure it works performance wise [19:36:39] hoo: there is also Gabriel's generic dependency tracking stuff coming up [19:36:49] I hope we can migrate to that at some point [19:36:55] My guess is that it's about 6 months out [19:37:15] Can you give me a pointer? [19:38:28] hoo: https://phabricator.wikimedia.org/T102476 and https://phabricator.wikimedia.org/T105766 [19:38:54] uh, there was another one somewhere [19:39:52] hoo: ah, right, this may also be relevant, but it's really more a replacement for our ChangeDispatcher stuff: https://phabricator.wikimedia.org/T88459 [19:41:52] I'll read through them, once I find the time [19:43:52] If you really think that these mechanisms are just a few months out, this discussion (and the other ones) are moot [19:44:04] We shouldn't waste time on that [19:44:28] But it might be time to really learn about nodejs for me [19:44:29] :P [22:07:56] addshore: pong [22:09:36] pong ;) [22:46:01] https://phabricator.wikimedia.org/T124805 i'm requesting permission to periodically run a query for each wiki to find sets of { item -> wikilink -> redirect -> page -> other item }, hope jynus won't object ^^' [23:07:54] Alphos: You just want a list of redirect that have a wikidata item, right? [23:08:37] not quite ; the page they redirect to must have a *different* wikidata item [23:08:57] although... [23:09:16] well, that's fairly obvious actually ; one item can't have two wikilinks to the same wiki [23:09:36] Just use https://tools.wmflabs.org/multichill/queries/wikidata/redirect_items.sql to get a list like https://tools.wmflabs.org/multichill/queries/wikidata/redirect_items_nlwp.txt [23:09:40] And have a bot go over it [23:10:40] wow, addshore wasn't aware of it when he suggested i did it :x [23:11:15] Aware of what Alphos? [23:11:24] although mine gives both the originating and terminal item and links [23:11:34] that that sql existed ;) [23:11:49] This runs in several seconds [23:12:12] so does mine [23:12:43] ok, so why create a ticket? [23:13:33] because i talked to jynus and he wasn't too happy about running my queries multiple times when it took about an hour for arwiki, so i'd rather find a better way to get the same info ^^' [23:13:40] (i just didn't know your way existed) [23:19:52] hm, we don't seem to find the same number of items [23:20:27] yours give 3322 items, mine 1238... [23:23:15] Alphos: Usually when a redirect sticks around something went wrong so databases might be out of sync? [23:23:28] Time for bed. Good luck [23:23:33] nighty [23:23:38] i'll compare the two, see what i find