[00:32:15] wmde/WikibaseInternalSerialization/rel/6558a27 : jeroendedauw The build has errored. https://travis-ci.org/wmde/WikibaseInternalSerialization/builds/109721925 [10:29:55] Can somebody help with WDQS? I still don't understand the syntax and I just want to fetch items without statements. [10:32:45] hey sjoerddebruin [10:32:50] hi hoo [10:32:52] Without any statement? [10:33:19] Yeah, for nlwiki. [10:33:40] I'm now working with this, but I think it could be easier. http://tools.wmflabs.org/autolist/autolist1.html?lang=nl&props=31,279&q=link%5Bnlwiki%5D%20and%20noclaim%5B31%5D%20and%20noclaim%5B279%5D%20and%20noclaim%5B910%5D%20and%20noclaim%5B646%5D%20and%20noclaim%5B373%5D%20and%20noclaim%5B18%5D%20and%20noclaim%5B17%5D%20and%20noclaim%5B214%5D%20and%20noclaim%5B227%5D%20and%20noclaim%5B641%5D%20and%20noclaim%5B508%5D%20and%20noclaim%5B3 [10:33:40] 61%5D%20and%20noclaim%5B136%5D%20and%20noclaim%5B1482%5D%20and%20noclaim%5B156%5D%20and%20noclaim%5B155%5D%20and%20noclaim%5B625%5D%20and%20noclaim%5B1269%5D%20and%20noclaim%5B1080%5D%20and%20noclaim%5B856%5D%20and%20noclaim%5B571%5D%20and%20noclaim%5B527%5D# [10:33:51] :P [10:34:11] sjoerddebruin: So items with links to nlwiki, but not statements (at all) [10:34:22] Yep! [11:11:52] :) [11:15:01] reified [11:27:23] Thiemo_WMDE: do you want to add anythign to https://github.com/wmde/DataTypes/pull/56 ? [12:02:34] Adrian_WMDE: aude: https://phabricator.wikimedia.org/T127044 [12:26:01] Is wd back from read-only? [12:27:30] Wesalius_: it should be [12:27:53] sjoerddebruin: ERROR (set_label) : The wiki is currently in read-only mode [12:28:00] sjoerddebruin: when using quick statements [12:28:04] sjoerddebruin: teste just now [12:28:09] *tested [12:29:06] There are some server issues, but recovery would be ahead. [12:31:01] sjoerddebruin: ok [12:34:29] addshore: https://grafana.wikimedia.org/dashboard/db/wikidata-edits :O [12:44:47] sjoerddebruin: its running fine again [12:45:32] Nice [12:59:46] JeroenDeDauw: Thiemo_WMDE benestar|cloud is it just me or all the tags gone on https://packagist.org/packages/wikibase/data-model ? [12:59:58] e.g. compared to https://packagist.org/packages/wikibase/data-model-services [13:02:06] addshore: do you know? [13:02:13] aude: ? [13:02:14] *looks* [13:02:21] all the tags are gone https://packagist.org/packages/wikibase/data-model [13:02:29] so composer install for wikibase complains about not finding stuff [13:02:44] hmm, odd [13:03:00] tags are still on github https://github.com/wmde/WikibaseDataModel/tags [13:03:01] e.g. https://integration.wikimedia.org/ci/job/mwext-testextension-php55-composer/308/console [13:03:07] yeah [13:03:20] i'm just adding "repositories" to composer.json for the branch [13:03:22] that works [13:03:33] maybe re running the hook may work too [13:05:09] mayb [13:05:10] e [13:05:45] Thiemo_WMDE: why a +2 for https://gerrit.wikimedia.org/r/#/c/271234/ ? [13:15:47] addshore: i clicked "update" on packagist and think it's ok now [13:15:51] :) [13:16:06] it probably just got some foobar api result and for some reason accepted it and removed the tags :p [13:16:09] wtf happend [13:16:15] :/ [13:16:25] *fubar [14:42:00] hoo: is there something we can do to improve the scribunto tests? [14:42:01] https://integration.wikimedia.org/ci/job/mwext-testextension-php55/1474/testReport/(root)/ [14:42:07] they are especially slow :( [14:42:55] aude: You mean the tests by Scribunto itself? [14:43:00] https://phabricator.wikimedia.org/T126274 [14:43:03] yeah [14:43:15] ours also that involve scribunto [14:44:52] aude: hm... all client tests (including Scribunto) run in less than 16s for me [14:45:07] :( [14:45:19] still some are to slow and we have one test that waits for the next second (so up to one second) for entity usage [14:45:49] but I don't think we can speed up the Lua tests... we could try to reduce the number of test cases, though [14:45:54] :/ [14:45:58] not sure that's desired, though [15:51:53] hey Lydia_WMDE :) [15:52:44] benestar: hey [15:53:17] Lydia_WMDE: how will the media info stuff be displayed? will it get its own page or be integrated in file pages? [15:53:50] benestar: in the prototype it gets its own page - later integration with the file page [15:53:59] :O [15:54:03] 'cause I wonder if I should start refactoring our current view infrastructure a bit to register additional entity views or if we are going to do a different thing anyways [15:54:26] but perhaps we'd need that refactoring anyways... [15:54:50] i believe so [15:55:29] created a ticket and added it to the sprint [15:55:56] :) [16:26:16] sjoerddebruin: did you ever find out how to select items without statements? (I tried, but everything I try just times out) [16:36:51] I wish merging would create redirects more reliably... it seems like every time I check things like items with no statements or sitelinks, half of them were merged but not redirected [17:14:49] nikki: Can you file a bug for that, please [17:14:54] that sounds troublesome [17:21:56] hoo: https://www.wikidata.org/wiki/User:Addshore/Identifiers/0#good_to_convert [17:22:06] so, not viaf yet [17:33:34] commons.wikimedia.org/wiki/Special:FilePath/Wikidata.png?width=150 [17:47:18] hoo: done https://phabricator.wikimedia.org/T127213 [18:44:04] 16:38 mw.config.values.wbSiteDetails: JMJisVJc -> zIuK1nTc (changed after 8 days, 20 hours, 7 minutes and 33 seconds) [18:44:04] 16:43 mw.config.values.wbSiteDetails: zIuK1nTc -> JMJisVJc (changed after 5 minutes and 3 seconds) [18:44:05] 16:48 mw.config.values.wbSiteDetails: JMJisVJc -> zIuK1nTc (changed after 5 minutes and 2 seconds) [18:44:05] 16:53 mw.config.values.wbSiteDetails: zIuK1nTc -> JMJisVJc (changed after 5 minutes and 3 seconds) [18:44:05] 16:58 mw.config.values.wbSiteDetails: JMJisVJc -> zIuK1nTc (changed after 5 minutes and 3 seconds) [18:44:14] Looks like this module may be malfuctioning [18:56:47] Krinkle: We updated the data today [18:56:49] so, could be [18:56:53] benestar: ping [18:57:04] hey hoo :) [18:57:13] how are you doing? [18:57:16] Im also slightly alarmed that such a module even exists. [18:57:19] benestar: How does property info store updating/ cache invalidation work? [18:57:52] Krinkle: Well, it would be an api module these days [18:57:54] I guess [18:57:54] they are updated when Content::getSecondaryDataUpdates gets called [18:58:15] hm [18:58:18] afaik, see PropertyHandler::getEntityModificationUpdates [18:58:27] hoo: I mean, it seems a bit overkill to have an entire module for one value. As opposed to a generic 'data module' or even a file module + data [18:58:48] even more so to have it be named in the mw.* namespace [18:58:55] thugh wbSiteDetails will be unique of course [18:59:57] Looks like it cashes the version has per-server [19:00:08] so during change, it will flip flop for a while [19:00:18] RL caches that just fine [19:00:29] shouldn't need caching of any sort there [19:00:55] not unless it's in memcached, or it's a verified key (e.g. a key that can be verified to be latest in some way) [19:01:19] also getModifiedHash is deprecated. [19:01:21] Krinkle: I think Ori was complaining [19:01:45] We're not using get modified hash in there for a while now [19:02:59] Ah, different branch. I see [19:03:00] it uses getDefinitionSummary now [19:04:00] Is any of the data included in there cached in some way? [19:04:06] maybe settings->getSetting( [19:04:12] or siteStore->getSites [19:04:21] I think we have all relevant data in there [19:05:20] May wanna simplify and remove getDefinitionSummary() and instead add enableModuleContentVersion() { return true;} [19:05:34] which is effectively the same as implementing getDefinitionSummary as sha1( getScript ) [19:05:37] and then have the version in the configuration? [19:05:41] ag [19:05:44] * ah ok [19:05:51] if that's fast enough [19:06:03] if it's run ofte, Ori is going to complain again [19:06:24] I don't understand [19:06:33] What exactly woud run too often? [19:07:00] He was complaining about us doing the hashing to often, thus we implemented the cached hash [19:07:10] the hasing [19:07:13] the hashing [19:07:26] which hashing [19:08:07] Whatever caching is applied indirectly by methods called in getDefinitionSummary, you could apply to the methods called in getScript. [19:09:08] Although I imagine there is an advantage to having getScript run uncached [19:09:13] and instead debounce thrhough the version hash [19:09:22] But in that case it needs to be fixed to not vary by server [19:09:36] unless it can verify that a key is stale [19:11:37] hoo: What is the purpose of this module in the end? How is the module consumed? [19:11:54] it looks a lot like sitematrix [19:12:04] which doesn't seem expensive to compute [19:12:11] Krinkle: We use it in the UI in various places [19:12:21] it's based on sites, not interwiki [19:12:24] so unlike sitematrix [19:12:35] what is it doing that is expensive? [19:12:38] (which is, unlike the name may suggest, not based on sites) [19:12:48] Krinkle: Getting a hash of all the content [19:13:21] hoo: I'm assuming the cache is not to debounce 1 call to sha1() [19:13:28] so the content is expensive to fetch, not the hashing [19:13:38] Is it making mulitple sql queries? [19:14:46] Krinkle: No, it's mostly to not do the json and sha1 [19:14:54] the data itself is in CACHE_ACCEL [19:15:00] thus not expensive to get [19:15:33] put in to CACHE_ACCEL by something outside SiteModule(Worker)?, right? [19:15:41] yeah [19:16:11] It's cached in startup module for 5min, the expiry of 10min just means it only gets called every other time. [19:16:16] still per-wiki [19:16:21] though no longer per-lang and per-skin I suppose [19:16:33] hm [19:16:52] I think that should really be fetched from the API when needed [19:17:17] I'm glad I didn't write that one :P [19:17:51] I'm just trying to figure out why it kept varying for ~ 25 minutes [19:18:08] too many layers [19:18:17] Maybe take a good look at it and see what can be simplfied [20:13:26] nooo I'm getting 502 errors all of a sudden [20:14:09] nikki: WHERE? [20:14:18] oh. on wikidata [20:14:22] trying to preview a page [20:15:24] :/ link? [20:15:33] it seemed to work again after refreshing a few times (although at first it didn't load any of the styling), but I still wasn't expecting it [20:16:30] the page I was previewing is https://www.wikidata.org/wiki/User:Addshore/Identifiers/1 [20:19:44] interesting [20:20:09] That addshore keeps messing in his user space... [20:20:13] I have to say though that I'm very very happy that 502s (and similar errors) are so rare here, I spent too long in the past on sites where those were a daily part of life :P [20:20:19] :> [20:22:12] nothing relevant in the logs at first sight [20:22:53] hoo: see #tech [20:23:22] ah ok, fine then [20:32:52] aude: can you have a look at EntityParserOutputGenerator? https://github.com/wikimedia/mediawiki-extensions-Wikibase/blob/master/repo/includes/ParserOutput/EntityParserOutputGenerator.php#L163 [20:33:46] those lines seem to be duplicates of what ReferencedEntitiesDataUpdater already does [20:37:25] hoo: still around? [20:37:32] yeah [20:37:41] looking at random identifier stuff [20:37:49] hoo: what should happen when ParserOutputGenerator is told to generate html but no entity view is defined? [20:38:10] that can happen when an extension defines an entity type but doesn't register a view for it [20:38:12] slap the caller with an exception [20:38:33] it would be trivial for them to give us a null view, if that's desired [20:38:43] Stryn: tech says this happens anywhere? (I'm not in there, atm) [20:38:51] because I just got such an error on dewiki [20:38:52] otherwise it's just not a task we can reasonably solve [20:39:42] Vogone: yes it happened (or still happends) everywhere, not just on WD. Related task: https://phabricator.wikimedia.org/T127227 [20:40:22] ty :) [20:51:26] nikki: thank you! [20:51:45] what did I do? [20:52:04] sort identifiers :) [20:52:06] ah :) [20:52:33] yeah, I figured I should do some of the more obvious ones already, because the discussion about it isn't getting anywhere [20:52:41] *nod* [20:52:44] that is helpful [20:52:57] Definitely :) [20:53:08] and i think if we have a bunch more most people will be able to get the system and then it'll be easier once they see how it affects items [20:54:08] You also struck through the ones I already converted way before I even thought about doing that :) [20:54:19] \o/ [20:55:08] Unless something goes wrong, I think we can do all that are considered "good to convert" tomorrow [20:55:20] And it looks good so far [20:56:18] If too many items get touched we might not be able to proactively purge all pages, though… but that's not more than an annoyance [20:57:07] hoo: how are we going to handle conflicts with the authority control gadget? [20:57:21] benestar: We just don't bother [20:57:36] it keeps overwriting us, but as it uses the same formatter URLs that doesn't matter [20:58:06] But there's a performance side to that… I posted that on https://www.wikidata.org/wiki/MediaWiki_talk:Gadget-AuthorityControl.js#Performance [20:58:15] it should be changed however to not care about identifier datatype properties [20:58:25] Exactly [21:12:27] nikki: no, waiting for hoo. ;) [21:13:06] huh? [21:13:31] oh the no statements thing? [21:13:47] Yeah, I thought you were looking in to it? [21:14:12] I was, but I couldn't figure it, then I got distracted and forgot [21:14:22] Would post in project chat then [21:14:29] I wasn't able to find something to filter for [21:14:46] pretty sure it's trivial, but would need to wrap my head around it [21:14:52] There are a lot of examples, but nothing like this. :) [21:16:46] We basically need all known predicates and the filter not exists any value [21:16:51] * then [21:17:09] not all known predicates, that would be easy [21:17:17] all predicates that represent statements [21:17:25] or statement triples [21:18:36] jzerebecki: why do we still run php53lint? [21:18:42] hehehe [21:18:45] so someone noticed [21:19:23] benestar: I made a mistake in a regex, so it wasn't skipped for master anymore. already fixed. [21:19:23] sjoerddebruin: what do you want about statements? [21:20:30] benestar: create a list of items on nlwiki without statements [21:21:18] I'm using a link[nlwiki] and noclaim-construction in WDQ for that now, but it keeps showing merged items. [21:35:26] sjoerddebruin: http://quarry.wmflabs.org/query/7447 [21:35:34] still running but in theory that would give you the result [21:36:56] benestar: Yeah, but is that easy to use? [21:37:12] how do you define "easy to use"? [21:37:40] Well, now it's sorted on creation date and shows the current statements for P31 and P279 [21:37:50] (the thing I use now) [21:38:10] I'm sure there's a way to do that in SPARQL... but not today :P [21:39:09] I'm just annoying. :P [21:40:36] i tried it in sparql but it always times out [21:40:56] and it makes sense: there is a huge data set of items with a link to nlwiki but only very few that have no statements [21:41:44] Is a page onwiki, with direct links to the item, limited to the newest 100 items possible? [21:42:17] benestar: well, not sure [21:42:23] can you publish your SPARQL query? [21:44:02] well, I didn't get very far [21:46:45] hoo: interestingly, I can't even create a short url because the buttons only get visible when there are results [21:46:50] however, my query only times out :P [21:47:45] if it's useful, you might want to pastebin it [21:47:53] otherwise I'll think about that some more tomorrow [21:48:03] or just post it somewhere on Wikidata and someone will do amgic [21:48:06] * magic [21:48:40] Nice. Btw, do you guys see broken captions here? https://nl.wikipedia.org/wiki/Gebruiker:Sjoerddebruin#Prijzenkast [21:48:45] that's what I got so far http://pastebin.com/RfDAqQZu [21:49:14] going via rank is an interesting approach [21:49:22] not sure it's good performance wise, though [21:50:40] hoo: well, using a wikibase:statement doesn't work because those triples are filtered out [21:50:46] however, all statements have a rank so... [21:52:19] hoo: ∃ is always shitty in query languages [21:53:17] depends on the selectivity [21:53:26] but in the general case, yeah probably