[09:44:15] Lydia_WMDE ddddaily? [09:45:00] Jonas_WMDE: not in the office [09:45:06] Thiemo_WMDE: can you call jonas in? [09:45:12] or tell glorian to do it? [09:45:57] IRC daily instead? [09:46:05] Jonas_WMDE: Are you there? [09:46:38] Thiemo_WMDE here [09:47:50] IRC daily sounds good to me [09:47:51] Jonas_WMDE: I have no microphone right now. :-((( [09:48:18] ok no problem [09:48:59] ok, who'll start? [09:49:24] lets chat after the daily how push the sprint forward [09:51:31] Glorian can [09:51:38] My status: I need an other coffee. At the moment I'm working on a minor wikidata.git cleanup I had in my Git stashes. Then I want to clean my inbox (luckily no mess there today). I will talk with Daniel about "create MediaInfo entities in statement editing API". [09:51:45] Glorian can't join us because this channel is still +r [09:52:00] why is that still on? that's extremely restrictive :( [09:53:34] anyway. my status: Stas movwed structured search forward, we should be able to start using it soon. Also, there weas in interesting paper published on WDQS performance. we should get in touch with the researchers. [09:54:36] Today I'll work on auto-creation of MediaInfo (and talk to Thiemo about it). I'll also follow up on the Multi-Content_Revisions stuff (talk to Tim Starling about the BlobStore interface, and to Jaime about the representation of revision content in the DB) [09:54:50] If I feel brave, I'll install Neon on my laptop. [09:55:18] Jonas_WMDE: ? [09:55:42] sjoerddebruin: thank you [09:56:02] No problem. :) [09:56:07] hey Glorian_WMDE! [09:56:12] hey [10:00:22] I rebased some stuff and addressed gerrit comments and then I like to push the current sprint. I will talk to you about the individual tickets then. [10:01:13] DanielK_WMDE will you do a task breakdown with Thiemo_WMDE ? [10:01:46] Jonas_WMDE: probably. i have to look at the code again to see where we are at [10:01:51] Jonas_WMDE: I think this is what we will do. Do a bit of code experiment and then break the task down. [10:02:06] we also need to decide how much refactoring we'll do, and with how much cruft we are willing to live [10:02:45] What is the status of T115269 [10:03:32] is this blocked on announcement of Lydia_WMDE [10:04:40] Jonas_WMDE: yes. and then on waiting for people to fix their software. [10:05:20] we also want to prepare a bot run. Maybe we can talk to addshore about it when he's at the office [10:05:41] Lydia_WMDE: what do you think should the timeline be for announcing and implementing the change to Quantity serialization? [10:05:54] *waves* [10:06:36] DanielK_WMDE I already saw a swagger change here and there ;) [10:07:29] DanielK_WMDE: a bot run for what? :D [10:08:53] addshore: change bounds on quantities, for specific proeprties. Something like "for P1234, remove all +/-0 bounds" or "for P5678, change +/-1 to +/-0.5". [10:09:20] ooooh, okay, sounds easy enough :) [10:09:24] Jonas_WMDE: hm? [10:12:14] addshore: yea, but needs a bit of discussion :P [10:14:52] DanielK_WMDE: Do you want to have a look at https://gerrit.wikimedia.org/r/197873 ? Otherwise I will just merge it. [10:16:26] Jonas_WMDE: Do you want to have a look at https://gerrit.wikimedia.org/r/302408 ? Otherwise I will merge it. [10:18:27] Thiemo_WMDE ok with me [10:18:44] Ok, thanks. [10:19:20] Thiemo_WMDE: uh... i'll have to look and figure out what this does, and why [10:20:10] DanielK_WMDE: You did this before. :-) You could just check the reason for your -1 and if I fixed it and +2 the patch then. ;-) [10:37:49] Thiemo_WMDE: my vote was +1, no -1. i still don't see the point, but i'm fine with it [10:42:31] Thiemo_WMDE, everyone @WMDE: lunch at little saigon? [10:53:35] for when you're back from the lunch: Please review https://gerrit.wikimedia.org/r/#/c/299284/ [13:06:16] aude: Do you know who updated the sites table after tcy wiki was created? [13:06:21] There's nothing in SAL :( [14:16:19] Thiemo_WMDE: https://gerrit.wikimedia.org/r/#/c/295218/ https://gerrit.wikimedia.org/r/#/c/294521/ https://gerrit.wikimedia.org/r/#/c/294040/ [14:19:53] I almost forgot how much fun it is to dick around the parser's internals [14:20:02] * digg [14:20:04] * dig [14:21:08] well... you can also do that... [14:21:40] :P [14:22:06] and I also noticed that the parser profiling is totally screwed as of recently [14:22:21] but not sure how related that is to my thing [14:23:37] Oh wow https://de.wikipedia.org/wiki/Modul:Wikidata2Lemma [14:23:41] this is... just wow [14:24:36] efficient [14:33:15] hoo: sorry, forgot to log it [14:33:16] ( [14:33:18] :( [14:39:42] Ah, that's fine :) [14:39:49] Just wanted to make sure it has been done correctly [14:41:05] no problems, except that addWiki.php doesn't tak ecare of this properly [14:41:12] maybe it's best manually though [14:41:26] and we need to add tcy to the interwiki sorting order [15:20:40] https://gerrit.wikimedia.org/r/303556 wow [15:20:52] Didn't realize we were a) still doing that and b) it was so slow [15:21:06] \o/ [15:27:00] what are we doing? [15:27:07] o_O [15:27:43] I love how I could just live hack it on mw1017 and then benchmark api calls :) [15:28:57] * aude can do code reviews later, once i'm not tethering [15:43:19] The number of interface and classes we have around retrieving terms (in some way) is gigantic [15:43:52] I thought about splitting the TermIndex interface [15:44:03] but before doing that, I should probably find out what we have so far [15:50:05] aude: Do you have any ongoing refactors in the above area? [15:50:15] don't want to interfere with tha [15:50:15] t [15:50:32] or maybe it even eases my pain (quite likely) [16:01:32] DanielK_WMDE: here? [16:07:55] Is it just me or does the "in other languages" doesn't show the so far process anymore during saving? [16:10:15] hoo: yea [16:10:19] sjoerddebruin: sorry, what? [16:11:07] DanielK_WMDE: https://gerrit.wikimedia.org/r/303556 [16:11:27] hoo: i lloked at it. i'd merge, but jenkins sais no. [16:11:42] In order to mock WikibaseClient::getTermLookup, would you rather have a setter WikibaseClient::setTermLookup [16:11:59] or should I mock it via our MockClientStore (the underlying TermIndex) [16:12:18] Earlier, you could see what edits were processed already (the underline would go away then). I don't see any indication anymore during the process. [16:12:20] getting mocktermindex that is in sync with the other mock data is not as easy as it sounds [16:13:09] sjoerddebruin: ah. no idea, sorry.. [16:13:32] hoo: when exactly does it need to be mocked, and to do what? [16:14:07] Will make a ticket then [16:14:11] DanielK_WMDE: The user is in Scribunto_LuaWikibaseLibrary [16:14:17] but we don't have control over that at all [16:14:23] this is an integration test [16:14:30] Scribunto_LuaWikibaseLibrary is instantiated via Scribunto [16:14:34] outside of our reach [16:14:47] hoo: so how did the mock data get in there in the past? [16:14:50] so we need to mock WikibaseClient or its store (which is already mocked) [16:15:13] we put a mock store into the global WikibaseCLient instance?... ugh... [16:15:16] We used an EntityRetrievingTermLookup with the MockRepository EntityLookup [16:15:21] Yes, that's needed [16:15:29] and well enclosed to only act for that test [16:15:29] ); [16:15:58] hoo: for which test, exactly? [16:16:13] anyway... [16:16:14] DanielK_WMDE: The lua integration tests [16:16:28] tests derived from Scribunto_LuaWikibaseLibraryTestCase [16:16:32] (various classes) [16:16:46] these are the tests that are actually written in Lua [16:18:14] hoo: I see... no chance to get a hold of the WikibaseLuaBindings and call setters on it? [16:18:55] No [16:19:06] We could make itself register to a global variable [16:19:14] but that sounds even worse [16:21:08] hoo: we could make it WikibaseClient manage the WikibaseLuaBindings instance. Would be cleaner anyway. [16:23:40] hoo: dunno, make a patch, so i can complain ;) [16:23:52] it's all kind of ugly :( [16:24:11] DanielK_WMDE: But then WikibaseLuaBindings would need to "register" with WikibaseClient [16:24:20] because we have no control over what Scribunto does [16:25:34] why can't we have a singleton in WikibaseClient? oh, the UsageAccummulator, I gues... [16:27:12] hoo: so if we were using MediaWikiServices, you could just call MediaWikiTestCass::setService and be done [16:27:24] *sigh* [16:28:15] hm… but how is that different from setting it in WikibaseClient? [16:28:16] just make a setter in WikibaseClient, and complain if it's called outside a test. [16:28:53] hoo: it's not different, it's the same thing, but with a mechanism to automatically clean up afterwards, and to ensure consistency between services (to different digrees) [16:29:13] yeah, that sounds good… I hacked that together myself more or less [16:29:32] Would be nice if we could change our top level service factories to MWServices [16:29:33] hoo: changing global state in test fixtures is ugly, but *leaking* global state is a *problem*... [16:30:07] hoo: yes. put them into MWServices directly, or at least make them derive from ServiceContainer [16:30:44] Do we have a ticket for that? [16:30:55] don't think so, no [16:32:52] I also think we should make a list of Term related services and then try to consolidate/ split them as needed [16:32:57] that's a bit of a mess now [16:34:20] yea, so much to do, so little time. [16:36:19] True :/ [16:48:45] SMalyshev: Hello! I need to restart WDQS for JVM upgrade. I'll do that after pulling the latest code. Any issue with that? [17:37:58] it looks that nobody answered me or I've missed it in the logs [17:38:04] any Lua gurus here now? [17:38:23] Base: What do you need? [17:39:12] hoo I need to know if I can getEntity by knowing wiki and sitelink title in it [17:39:40] I do not see how to get entity ID with this info [17:39:48] it would be overtrivial via API [17:40:12] but https://www.mediawiki.org/wiki/Extension:Wikibase_Client/Lua looks like missing needed functionality or I am missing it, thus asking [17:42:17] That's not implemented: https://phabricator.wikimedia.org/T74815 which is unresolved because of https://phabricator.wikimedia.org/T142093 [17:42:26] The access functionality is easy to do [17:42:30] but there's a long tail to it [17:46:45] so basicly as of now if you implement it and I use it in module it wouldn't update cache for entities once they are changed? considering that I need it to track whether there is no new sitelink created that is too bad :( [17:47:08] any idea how long it will take to figure out that second task? [17:47:26] i see that the first one is there for 2 years already [17:48:06] Base: Exactly… the first one lay around for ages [17:49:16] I only opened the RfC ticket recently… not sure how fast we can pick this up [17:49:41] I would like to get it resolved soon-ish, as it's been a feature that people have asked for since forever [17:51:49] it looks that I would need to add a parameter where people will manually put entity id, though too bad this wouldn't make all 11k of transclusions of the template I am going to enhance have new feature, usage wouldn't be straightforward too. (Just for the record, https://uk.wikipedia.org/wiki/Шаблон:Не_перекладено this template, though it is irrelevant as the problem is rather general) [17:53:21] hoo, yeah would be really cool if it would be implemented soon. It is really hard to believe that it really isn't there yet [17:54:09] I can't promise anything, but I'll try to push it mor [17:54:09] e [17:57:48] I think it would be useful for that structured commons thing which I believe is what is now work goes on too, so hopefully it gets good priority. Ok thanks you for info, I'll go figuring out what best temporary means I can use [18:01:10] aude: Do you know whether we still need property-suggester.wikidata-dev.eqiad.wmflabs [18:01:25] it's one of those instances that are not accessible (need a reboot) [18:01:30] but I'm not sure we need it at all [18:01:48] wikidata-suggester.wikidata-dev.eqiad.wmflabs for building the suggester data [18:01:51] no idea about that box [18:04:57] hoo: talking about the suggester, has the ratio for identifiers changed too? [18:05:20] Or did we adjust it completely some time ago [18:05:33] What do you mean by ratio? [18:05:57] Ehm, that one number. :P [18:06:04] I delete all property relations where the first property is an ext-id [18:06:21] So something can relate to an ext-id [18:06:29] but ext-id don't relate to anything [18:07:10] but that's just a very ugly workaround [18:07:20] For https://phabricator.wikimedia.org/T132839 [18:07:21] Well earlier when I mirrored the VIAF for a item, more suggestions showed up. [18:07:28] but I guess we don't have the resources to pick that one up [18:09:18] Well, can we at least update the data again after today's dump? [18:09:24] huh… that's weird VIAF is an external id [18:09:35] sjoerddebruin: Yes… the process is already running ;) [19:20:40] hoo|away: script ready ret? [20:54:31] sjoerddebruin: Nope… but it's in the last phase [20:54:50] I guess another 10-20 minutes before I can apply it [20:55:34] Okay :) [21:11:37] Alphos: if nobody responds here, just post to the talk page [21:12:09] Alphos: or ping Jonas_WMDE, best during a time europeans are at work.... [21:12:44] DanielK_WMDE duly noted [21:12:46] oh wait, i just replied to something you posted last night ;) [21:15:48] yes you did :D [21:16:11] DanielK_WMDE I am here on duty ;) [21:28:41] Jonas_WMDE: oh hey! [21:29:00] did you see what Alphos posted last night? I didn't look into it, but I thought you might be interested [21:29:11] no when? [21:29:14] by the way, in quite a few examples, there are BIND clauses that are plain unnecessary. instead of http://tinyurl.com/hnb226m (12170 ms), why not replace the bound values where needed ? http://tinyurl.com/zgvywop (same results, 1656 ms) [21:29:18] that [21:31:29] I guess BIND disables the use of indexes even in trivial cases. [21:31:37] maybe the blazegraph people would be interested [21:31:55] SMalyshev: what do you think --^^ [21:33:10] hmm interesting, let me look [21:34:01] that should act as a simple asias in this context... [21:34:41] it should... but it may be it interprets it differently... let me see the explains [21:34:49] Alphos: it's always goo to know who to ping :) [21:35:07] SMalyshev: i didn't confirm, maybe it was a fluke [21:37:31] naw, looks legit. i got faster results, but also something like an 8x speedup without the bind on the first try. may be caching though. now both versions are super fast. [21:37:42] are queries run via the web interface cached? [21:37:45] that's probably caching [21:37:50] yes, all is cached [21:38:07] SMalyshev: is there a way to bapass it? with ?foo=amsjdf or something? [21:38:24] SMalyshev: DanielK_WMDE: https://jira.blazegraph.com/browse/BLZG-1141 [21:38:29] DanielK_WMDE: yes, it caches by url, so change the query (even ws) or add dummy param [21:38:34] DanielK_WMDE: Just put garbage into a comment [21:39:48] The computation of definiteVars for bind expressions could be improved: currently, variables bound in BIND expressions are never considered definite (as BIND in general may fail), but trivial cases such as BIND (?x as ) might be considered "safe". [21:39:54] hoo: that looks hairy [21:39:58] that's probably the reason it's slow [21:40:13] Indeed… scary bug [21:40:14] it doesn't think bind is constant [21:40:30] I'll ping them about it anyway [21:41:07] then again, why bother binding a var instead of using its value directly ? ;) [21:41:20] hoo, Alphos: good find! to we need a phab ticket for this? [21:41:23] *de we [21:41:26] **do we [21:41:33] * DanielK_WMDE is already on his way to bed [21:41:43] * Alphos is already IN bed [21:41:49] laptop <3 [21:41:50] hahaha [21:42:18] DanielK_WMDE: wouldn't hurt to have phab ticket [21:42:32] Alphos: in this query, i agree, there is no pint. other than perhaps to meake clear what each ID means. [21:42:39] makes easier for me to track when such issues are fixed and re-check them when new BG release happens [21:42:42] Alphos: Sometimes it makes a query more expressive or shows of the flexibility better IMO [21:43:05] hoo of course, but in this case, i don't think that's true [21:43:27] it just makes the query more complex, which is prolly why it's taking 8x more [21:44:03] Yeah, in that case I don't see much value in it either [21:44:27] also, i apologize in advance, i might go really dumb in a short span but for quite some time, in the next few minutes :x [21:44:54] (cluster headache incoming. if you hear me screaming all the way from berlin, that's it) [21:48:37] Hi, can someone tell me what's wrong with my group concat in this query? [21:48:40] SELECT ?item ?itemLabel (GROUP_CONCAT(?memberOf) As ?memberOf) WHERE { ?item wdt:P31 wd:Q185441 . ?item wdt:P463 ?memberOf } [21:49:10] Alphos, SMalyshev, hoo: https://phabricator.wikimedia.org/T142437 [21:49:42] Alphos: oh :( [21:49:45] patricksevat: You need to group by ?item [21:49:54] thanks, DanielK_WMDE [21:50:10] also you will need to do that in a subquery and only select the labels after [21:51:30] patricksevat: SELECT ?item ?itemLabel ?memberOf WHERE { SERVICE wikibase:label { bd:serviceParam wikibase:language "en" .} { SELECT ?item (GROUP_CONCAT(?memberOf) As ?memberOf) WHERE { ?item wdt:P31 wd:Q185441 . ?item wdt:P463 ?memberOf } GROUP BY ?item } } [21:51:37] Sorry, formatting sucks [21:51:53] @hoo, thx! [21:52:18] I've been breaking my head on this for an hour :x [21:53:40] Yeah, especially grouping with labels is a little counter intuitive [21:57:14] I would really love for us to at some point tackle the problems described in https://phabricator.wikimedia.org/T132839#2270026 [21:57:26] The suggester often is annoyingly dumb [21:57:38] and our workarounds are not really helping either [21:57:56] yeah [21:58:09] and I ranted about the lack of suggestions for references earlier [21:59:57] Think that might be caused partly by my workaround [22:00:02] but not sure about that [22:00:06] I only touch ext-id related stuff [22:00:14] we just need more references ;) [22:00:44] btw, since last weeks deploy I have double suggestions at some moments: https://www.dropbox.com/s/d1u6i7ala59u6qz/Schermafdruk%202016-08-09%2000.00.42.png?dl=0 [22:04:09] This is also still a huge problem. https://phabricator.wikimedia.org/T102324 [22:06:36] Would this be hard to add btw? We do the same when there are no statements. https://phabricator.wikimedia.org/T77972 [22:07:10] Showing it at least when no suggestions are available is already a huge improvement. [22:07:27] sjoerddebruin: Well, it's easy probably [22:07:45] dec 2014, damn [22:07:46] time flies [22:07:54] but might end up being hard if we want to do it right (keeping application logic and settings like these separate) [22:09:40] not really hard, but quite some work [22:10:14] But a huge improvement for adding sources. :)