[06:55:15] Good morning everyone! [08:53:26] Tobi_WMDE_SW: https://gerrit.wikimedia.org/r/#/c/312001/ [08:53:41] thx Jonas_WMDE [08:56:31] got it Jonas_WMDE [08:56:42] it's because the changes are not deployed yet on test.wikidata.org! [08:56:55] möp [08:57:07] the issue is, we're always running the latest version of the tests there.. [08:57:34] probably worth looking into only running the tests from the branch that is actually deployed there [09:09:50] Jonas_WMDE: ok, actually jenkins IS always trying to check out the deployed version! [09:10:00] but Wikidata is special because we do not deploy regularly.. [09:10:32] the script uses https://test.wikidata.org/w/api.php?action=query&meta=siteinfo&siprop=general to get the current deployed version of mediawiki and then tries to checkout that version of the extension [09:10:59] but wikidata does not have a wmf/1.28.0-wmf.20 branch yet! is is still on .19! [09:11:01] https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/extensions/Wikibase,branches?skip=50 [09:11:05] Jonas_WMDE: ^^ [09:11:11] that's the cause of the problem [09:25:09] com.bigdata.rdf.internal.NotMaterializedException: Vocab(2):XSDUnsignedInt(1029222) /// darnit :-( [09:26:38] hm, it stopped doing it, this is normal :-/ [09:26:50] * Alphos won't even comment on the matter :D [10:52:13] Hallo. I have updated the CI jobs for Wikidata/Wikibase to disable Zend garbage collection. That apparently is a source of some segfault https://phabricator.wikimedia.org/T142158 [10:52:20] all patches courtesy of Paladox :D [10:56:25] hashar: thank you! [11:59:30] Good afternoon, WikidataFacts [11:59:47] good morning ;) [12:00:21] one could think that you live in two very different time zones :D [12:00:29] What timezone are you? :O [12:01:12] CET, I just got up late :D [12:01:29] Better start working then. :P [12:05:53] well, I could try to get somewhere with the uppercased Esperanto person labels bot… [12:06:10] I could just run it from PAWS, and ignore that it takes a long time (just let it run) [12:06:16] would that be frowned upon? [12:06:37] No objections from my side. [12:51:39] Hello friendly people in this round. [12:51:40] Does anybody have experiences with deploying a wikidata-query-service together with Blazegraph? We'd want to attach this stack to a custom Wikibase instance. [12:51:50] It lives at https://base.transformap.co [12:57:33] Lydia_WMDE do you know or who would know? [12:57:39] ^ [12:58:18] jon[m]: https://www.mediawiki.org/wiki/Wikidata_query_service/User_Manual has infos [12:58:28] Oh, say hi to Lydia_WMDE (IRC) , we're friends from Maxime. I think we once walked to Diefenbach together? [12:58:42] hehe [12:58:44] hi! [12:58:49] Yes, we're going to put this into a Docker deployment. [12:59:13] Let's see how it will work out. Currently investigating triple stores. Any other recommendations, despite Blazegraph? [12:59:22] hello Lydia_WMDE ! [12:59:38] jon[m]: for use with wikibase? blazegraph :D [12:59:42] harmonia: hey! :) [12:59:52] oké [13:00:27] Lydia_WMDE: I passed the million edits monday :p [13:00:50] harmonia: wohooooooooooooo! :D [13:01:06] <3 [13:01:07] ^^ [13:01:17] jon[m]: if you are happy with your wikibase install and if things work out i'd be happy to have you as one of the users of wikibase on wikiba.se [13:01:17] ^/me is happy [13:01:24] you should be! :D [13:01:30] cupcakes? [13:01:38] kittens! [13:01:41] hehe [13:01:42] cupcakes and tea! [13:01:51] * Lydia_WMDE serves some for the channel [13:02:04] * harmonia give pu-ehr tea to all the chan [13:02:07] gives* [13:02:29] Kitten-shaped cupcakes! [13:02:33] lol [13:03:04] with matcha icecream? [13:03:09] :p [13:03:56] Anybody who ever put this beast (Wikidata Query Service + Blazegraph) into Docker containers? Any preliminary work would right now be very helpful. [13:03:56] We're starting to read the documentation from above at and may use https://github.com/lyrasis/docker-blazegraph . https://www.mediawiki.org/wiki/Wikidata_query_service/User_Manual#Standalone_service [13:13:02] jon[m]: gehel and SMalyshev might be able to tell [13:15:25] Thanks. [13:15:42] Let's wait if they are able to respond to the mention. [13:43:21] * jon[m] are working in https://github.com/TransforMap/docker-wikibase-query-service [14:06:20] We managed to push some commits which get Blazegraph running in a Docker container. [14:06:21] Now we are figuring out how to create an RDF export of our Wikibase instance. Any pointers, again? [14:08:34] Found https://www.mediawiki.org/wiki/Wikidata_Toolkit which had been used to generate http://korrekt.org/page/Introducing_Wikidata_to_the_Linked_Data_Web [15:03:14] sjoerddebruin I think I fixed T146318 could you please verify? [15:03:15] T146318: DuplicateReferences doesn't support adding a second copied reference to the same statement - https://phabricator.wikimedia.org/T146318 [15:05:10] Maybe we shouldn't test on the same item. :P [15:07:03] It seems to save, but it doesn't update the current view. If you know what I mean, Jonas_WMDE. [15:07:22] not sure [15:07:48] It just stays on saving here. [15:08:36] argh [15:10:18] do you get any js errors? [15:11:21] TypeError: null is not an object (evaluating 'this._referencesListview.option') [15:23:23] sjoerddebruin so when I use the gadget I can reproduce the error [15:23:31] when I use the my script it works https://www.wikidata.org/wiki/User:Jonas_Kress_(WMDE)/duply.js [15:23:48] Note that someone did some edits to the gadget one. [15:26:57] yes I saw that, but I can not read the diff unfortunately :( [15:33:58] Sorry sjoerddebruin :( [15:39:20] :( [15:57:14] We would still be stuck with generating a Wikibase dump for the Wikidata Toolkit Client to convert to RDF. Any suggestions appreciated. [16:06:55] does anyone here know about the OSM/wikidata work? [16:07:16] I wonder why the wikidata tags on OSM are not being autopopulated from the wikipedia tags [16:07:21] aude: ? [16:08:15] dennyvrandecic (IRC): A friend who's sitting next to me is also jumping on IRC now. They have had this exact question at the State of the Map in Brussels last weekend. [16:08:39] ha, very timely :) [16:08:54] Nobody knows how to create the JSON dumps of Wikibase? https://www.wikidata.org/wiki/Wikidata:Database_download [16:09:32] dennyvrandecic: We had the OSM/wikidata discussion on the recent State of the Map conference... [16:10:16] *nod* [16:10:35] Automatic 'imports' into OpenStreetMap are done only in very few cases. [16:12:00] The reason the wikidata tag is not set automatically is that often the wikipedia tag does not correspond to the 'real' object... e.g. on a monument the wikipedia tag is set to the subject, e.g. an old king. Setting the wikidata tag therefore automatically would be wrong. [16:12:29] often? [16:13:00] in a lot of cases. The 'policy' for OSM on imports is that you have to check each object manually. [16:13:04] ah, so the wikipedia tags have been used rather indiscriminately [16:13:57] we have a lot of cases where a wikipedia tag is used not to encode equality, but rather relatedness [16:14:00] is that fair to say? [16:14:03] Yeah :-/ [16:14:03] That's partly the fault of OSM users, and we do not want to 'legitimate' this wrong data by adding the wrong WD keys [16:14:18] understood, makes sense [16:14:40] is there an effort ongoing to solve that? [16:14:46] e.g. for a monument the correct wikidata tag would then be "subject:wikidata"=Q* . [16:15:08] There is not a coordinated effort ongoing yet. [16:15:22] i.e. if someone took the data, analyzed it, and created a dataset that has a higher correctness [16:15:31] but it would make a nice "weekly task". [16:15:47] is there a chance of uploading it, or would it still require manual individual assessment [16:16:11] Even if you analyze it, it would only simplify the manual correction. Yes, the current policy is that you still have to check each object manually. [16:16:31] We on OSM have a big history of 'failed' imports, so we are rather strict on new imports. [16:16:54] understood. it is unfortunate though [16:17:08] I met one user at SotM, who tried to do that automatically - he was banned rather quickly^^. [16:17:14] if we have a dataset with 0.99 correctness, it is likely that it will be better than the human evaluation [16:17:42] The OSM data working group would still have objections, even if its 99%. [16:18:02] is there any indication human evaluation is better than 99%? :D [16:18:07] or even reaches it? [16:19:10] we're talking about 700k items here :) I am not sure many humans will maintain a high correctness on such a dataset [16:19:48] No, there is not. But at OSM, we do not thrust external datasets in general. [16:19:49] Then it may takes time until every OSM wp key has its corresponding WD key. We have time :-) [16:20:32] sure :) I won't fight policies of other collaborative projects. I know how these are made on the wikimedia projects and how useful it is to critizise them as an outsider [16:20:34] I do not say it is impossible to do an automated import here - it is just very improbable that the OSM community would agree to do it automaticly [16:21:04] would you have a link to the process? [16:21:32] Let's believe the WikiData case could be rather different of any other import, given the collaborative alignment and importance of both datasets to the Public Domain. [16:21:42] http://wiki.openstreetmap.org/wiki/Automated_Edits_code_of_conduct [16:23:45] thank you for the link, i will take a look [16:24:24] I assume a tool where an OSM contributor can oauth in and can then click yes or no or skip would be OK? [16:24:55] Oh that sounds lovely. I have heard of those Wikidata games only lately. [16:24:58] Yeah, I guess it would do [16:25:04] great thanks [16:25:08] and speciesATosm, since I have you here [16:25:15] I understand relation IDs are not persistent [16:25:26] which is why it is discouraged to link from Wikidata to OSM [16:25:27] Also an option to "fixme" flat an entry would be nice. [16:25:36] speciesATosm: good idea re fixme [16:26:01] oh, relation IDs are pretty stable. [16:26:01] only nodes and ways are not, because they are often migrated to relations. [16:26:10] ah [16:26:11] OK [16:26:13] good to know [16:26:22] so relations are fine, nodes and ways rather not [16:26:24] (noting) [16:26:38] thats why there is a OSM relation id property on WD, but not a node/rel property. [16:26:47] one more question - even if they are not stable, would you ever reuse a previous id for a different object? [16:27:00] speciesATosm: makes sense! [16:27:26] not by chance - you have to manually talk to the API to do that. [16:27:37] ah, great [16:27:52] but there are no "redirects" kept from old ids to the new one? [16:28:13] unfortunately not. [16:28:13] there are some workarounds, but very few users actually use that. [16:28:27] ok, thanks, just trying to understand and learn [16:28:44] i am not criticising :) just learning what the restrictions are [16:28:52] But there are plany to use the "wikidata" tag on an object to have some sort of "stable" identifier for an object. [16:29:52] that sounds nice [16:30:30] is there a datadump extract with such metadata, keys, labels, etc. but without the actual geo-data? [16:30:53] because it acts as some sort of UUID for an object. And an OSM "thing" can be a 'collection' of more than one items, e.g. the different segments of a street. [16:31:14] You can easily create such a datadump with tools like osmconvert. [16:31:24] ok, great [16:31:34] ah, different segments makes sense, indeed [16:32:22] osmconvert can generate csv-files and statistics from a planet dump. [16:32:35] great, I'll fetch that and take a look [16:33:01] try it with a smaller extract first, e.g. liechtenstein. [16:33:23] good idea [16:33:51] osm has no plans to include historic boundaries and data like the spread of certain species, i assume? [16:34:18] no, only current data. [16:34:18] For historical data, there is the OpenHistoricalMap. [16:34:25] thanks! [16:34:34] For the spread of certain species, I would suggest to use Umap. [16:35:11] Or create a fork of OSM, as OHM did would make even more sense. [16:35:22] openhistoricalmap is down. ah well, there's the wiki page on it on osm [16:36:13] oh, sad :-/ [16:36:58] another question [16:37:12] is the relation between administrative regions somehow encoded in osm? [16:37:18] i have seen that the level is encoded [16:37:36] but was wondering if also what county belongs to which state is there, or similar [16:37:44] and whether this is used for quality checks [16:37:55] 'level' is used for the 'building level' an item is on. [16:38:43] sorry, admin_level [16:39:00] To derive the 'political' boundaries an item is in, we usually use nominatim, the geocoder. It has a very nice API. [16:39:27] But please don't DDOS it with 700k items^^ [16:39:37] :D [16:39:42] jon[m]: sorry for the late answer. I'm busy at Ops offsite this week, but WDQS in docker seems like a kool idea! I'd be happy to help, but saddly not before next week [16:39:43] wasn't the plan :) [16:40:06] for 700k items I'd probably just download the data completely and try to setup a local install of nominatim [16:40:29] gehel: Thanks, it seems to be progressing. [16:41:06] Now solved the often confusing documentation by using our own pad. MediaWiki, Wikidata and Wikibase want to be carefully distinguished domains. [16:41:09] Yeah, setting up nominatim would be the way to go. [16:41:42] gehel (IRC): Make sure to track our repository, which we are going to enrich with our findings. [16:41:50] https://github.com/TransforMap/docker-wikibase-query-service [16:42:16] but there's no one checking, e.g. "oh, this county is supposed to be in this state, but the state boundaries do not cover the county"? [16:42:42] jon[m]: I'm now watching that repo! [16:43:30] It might make sense at some point to provide a data container with a recent set of blazegraph data... [16:43:39] But that would be a pretty big one... [16:44:01] There are some automatic relation checkers which at least check for unclosed relations, but I don't know of any who check the topology of relations - seems like a good idea to implement something that .... fetches the relation of states etc from Wikidata and check if they are correct in OSM too :-) [16:45:23] yeah, just thinking loud :) [16:46:09] ok, thank you so much, speciesATosm for answering all my questions [16:46:14] and your patience [16:46:17] this was superhelpful [16:46:30] jon[m]: thanks for connecting [16:47:33] speciesATosm: oh, one more thing: regarding something like taxa map, we could also host, say a kml file on commons, and link to that? [16:47:46] that would also allow to use that as an overlay over an osm generated map easily? [16:48:33] You could use Umap to display a map with an external file easily. I don't know if kml is supported, I usually use geojson. [16:48:47] or geojson [16:49:03] i don't care so much about the specific format, as long as it is open and easily usable by the tools :) [16:49:16] yes, i guess i should have said geojson, i am merely old :) [16:49:17] DanielK_WMDE: DanielK_WMDE_: here? [16:51:18] This is really easy to do with Umap. Create a new Umap, and use an external like as datasource. [16:51:38] s/like/link/ [16:51:49] sweet :) [16:52:28] my dream come true would be to for the wikidata sparql service to transparently be able to draw relations based on a query result [16:52:34] i think this is already possible [16:52:49] and then mix that with a umap/custom geojson [16:53:01] well, sounds like we're pretty close to that :) [16:54:05] dennyvrandecic: do you mean osm relations or any WD relations between entities? [16:54:54] in this case osm relations. I.e. make a query like { ?county inState wd:California } and display the result on a map [16:55:01] so to quickly see which counties are missing, etc. [16:55:32] i have to check the work the foundation did recently, I think that covers this already mostly [16:55:42] wikimedia foundation [16:55:51] sorry, i should be more specific while talking to you :D [16:57:05] For displaying OSM relations, just directly link them: https://www.openstreetmap.org/relation/1155955 - your browser should dynamically load the boundaries and display them. [16:57:31] yes, that's one - but I want *all* the counties of california [16:57:57] Ah - maybe an overpass query would do it... [16:58:11] (noting down) [16:58:24] osm has plenty of cool tools in its infrastructure [17:01:43] maybe something like that: http://overpass-turbo.eu/s/iDy - if you dig through https://wiki.openstreetmap.org/wiki/Overpass_API/Overpass_QL, there are also functions to query objects inside some boundaries... [17:02:39] cool [17:02:44] I'll play around with that [17:02:57] yay, a proprietary query language! :D [17:03:57] Yeah :-/ [17:03:57] But there was nothing like that available ... maybe sometimes when OSM is available as RDF, some thing like GeoSPARQL would do it. [17:04:24] I actually have no idea how appropriate something like sparql is for osm [17:04:25] :) [17:04:41] osm is a unique resource in its availability [17:04:57] it might well be that a proprietary language will remain the best way [17:05:14] but let's see. i like standardization :) [17:05:25] me neither ;-) [17:05:25] But at least there were some attempts by http://linkedgeodata.org/ [17:05:53] It was down for some years, and I don't know how active it is maintained. [17:06:41] yeah, I would prefer not to discuss their approach to these things publicly :D [17:28:52] ow, speciesATosm, that query on http://overpass-turbo.eu/s/iDy you sent me know merely gives me error messages [17:29:05] speciesATosm: it seems I am not a good influence on OSM infra :D [17:29:07] An error occured during the execution of the overpass query! This is what overpass API returned: [17:29:07] Error: runtime error: open64: 2 No such file or directory /osm3s_v0.7.52_osm_base Dispatcher_Client::1 [17:55:24] dennyvrandecic: Maybe try it with a smaller BBOX? I never had this error - Just tested, it worked now. [17:55:34] trying [17:55:44] ah yeah indeed [17:55:57] yeah, maybe the bounding box was bad [17:56:07] oh wow, i am surprised how much work is still left in OSM [17:56:19] i would have assumed that municipalities and counties etc. are pretty complete [17:56:22] fun! [17:58:42] * DanielK_WMDE_ is always happy to see dennyvrandecic play with data :) [17:58:50] :D [17:58:53] data is awesome [17:59:38] * Harmonia_Amanda has no idea what they are talking about but she is always good for a little datalove [18:06:47] speciesATosm: one last question, if you don't mind - when I search for my adress, I can find it in OSM (yay) - but when I am in edit view in the same area, there is nothing, no house, no object to put the adress on [18:06:55] is there some magic going on with adresses? [18:10:25] sjoerddebruin: bot is running now :) [18:10:51] dennyvrandecic: i'm making slides for a keynote, and would like to give some historical background. do you have some slides handy about How It All Started in 2011? [18:10:54] WikidataFacts: :) [18:11:10] dennyvrandecic: i remember you presenting something like this at wikimania a few years ago... [18:11:23] err, how it all started in 2005, i mean :P [18:14:51] haha, holy shit! https://commons.wikimedia.org/wiki/Category:Wikidata_presentations_in_English [18:15:37] just 56? That's not too many [18:16:37] Hi room, speciesATosm (IRC) and me are still evaluating how to create a valid Wikibase RDF dump for importing into Blazegraph for the Query Service. [18:16:56] Unfotunately the dumpRdf.php from Wikibase doesn't create an output that the munge.sh script likes. [18:16:57] DanielK_WMDE_: I might have lost my slidesets from back then :( [18:17:06] or have them only on some disks somwhere [18:17:19] never mind then. [18:17:22] oh wait [18:17:25] I should have one thing [18:17:28] Now we're trying dumpJson.php again, but the wdtk-client does not like our inputfile. What can we do? [18:17:34] we had a 10 year SMW presentation last year [18:17:38] i should have a slideset on that [18:17:44] yay! [18:17:48] dennyvrandecic: You are based in the US? Nominatim is also using TIGER data when looking up addresses. Could you send me the link to the query you are using? [18:17:53] I'll share with you DanielK_WMDE_ [18:18:11] dennyvrandecic: oh, i wasn't thinking of the original SMW presentation; rather the slides you had about 2005 when you presented wikidata at wikimania in... 2012, i think. [18:18:21] thanks! [18:22:29] speciesATosm: https://www.openstreetmap.org/search?query=1430%20Grove%20Street%2C%20Alameda%2C%20CA#map=19/37.76077/-122.23498 [18:27:37] shared two presentations I had here, DanielK_WMDE_ [18:29:15] dennyvrandecic: yes, the address is only found in the TIGER dataset, not yet in OSM. You may add it :-) [18:33:24] dennyvrandecic: thank you! [18:33:32] i find myself writing an actual text for this o_O [18:33:41] usually i just make slides, and then wing it... [18:45:12] jon[m]: which specific problems do you have with dumpRdf? [18:51:21] DanielK_WMDE: do you know what is the procedure for updating wikidata build that is deployed? I've made 1.4.0 for purtle, but I have no idea if it was deployed and where [18:51:29] and Special:Version just says 1.0 [20:10:26] DanielK_WMDE: DanielK_WMDE_: still there? [20:10:47] speciesATosm: thanks! that's good to understand! [20:14:42] DanielK_WMDE: I did find the slides from wikimania 2011&12 here, we go: http://simia.net/download/wikidata-wikimania-2011.pptx and http://simia.net/download/wikidata-wikimania-2012.pptx [21:10:26] gehel (IRC) + Lydia_WMDE (IRC) : Our progress of setting up a dockerised wikibase query service is stuck at the second installation step: [loading data](https://github.com/TransforMap/docker-wikibase-query-service/issues/1). [21:29:31] jon[m]: still around? [22:07:45] dennyvrandecic: thank you! i'll look at them tomorrow. [22:15:43] SMalyshev: can you give these folks a hint? i suspect they are missing the right params to dumpRdf.php https://tree.taiga.io/project/transformap/task/353 [22:16:35] jon[m]: ah, you are still here. talk to SMalyshev. [22:22:34] DanielK_WMDE: I have hard time understanding what's going on... it just says "not accepted" but I have no idea what that means [22:22:46] jon[m]: have any details on that? [22:23:58] ah ok I found the log [22:24:05] SMalyshev: does the munger eat ttl? or does it want triples? [22:24:06] looks like the base URL is not set [22:24:32] * DanielK_WMDE__ didn't find the log [22:24:41] if your entities are not wikidata, you need to set base url [22:26:40] SMalyshev: i see: Error munging null org.wikidata.query.rdf.tool.exception.ContainedException: Didn't get a revision id for [(_:genid-d83776a395a04eafa83b5e9cccbeb6de-genid1, http://www.w3.org/1999/02/22-rdf-syntax-ns#type, http://www.w3.org/2002/07/owl#Restriction [22:26:51] jon[m]: try munge.sh .... -- -w base.transformap.co [22:26:54] is that caused by a missing base URL? [22:27:26] DanielK_WMDE: hmm that's something else [22:27:41] SMalyshev: you can log in using your github id to leave a comment, it seems [22:28:06] .oO(what's tiga.io anyway?) [22:29:40] DanielK_WMDE: not allowed to comment [22:29:47] hmpf [22:29:55] .oO(what's a ContainedException?) [22:30:11] it's a kind of exception in WDQS [22:30:22] SMalyshev: well, let's hope jon[m] reads his backlog. [22:30:52] SMalyshev: yea, i gathered that much :D [22:31:55] DanielK_WMDE: well, it's an internal class to classify exceptions by deadliness :) Contained one is not very deadly as we can recover from it [22:32:12] but that really doesn't matter too much if you see one something's wrong :) [22:32:59] I added comment to github one, hopefully he'll see it. [22:34:12] "We hope you enjoy using Taiga.io, your Open Source, Agile Project Management Tool." [22:34:28] so apparently it's an agile project management tool :) [22:36:42] heh [22:37:45] SMalyshev: btw, thank you for the image meta-data search. it's nice to see something that people have been wanting for a decade just happen :) [22:38:39] it's magic when all the pieces finally drop into place [22:38:47] DanielK_WMDE: heh, np :) I think once we've refactored to make search filters sane and making indexing sane making this work became not hard at all [22:39:10] yea, good tools make good work! [22:39:40] yeah I think without structure in indexing and searching it'd be much harder... I wouldn't dare to touch query parser before :) [22:40:09] heh, i have never ventured there...