[09:04:23] New patchset: Raimond Spekking; "Consistency tweaks in preparation for re-adding to Translatewiki.net" [mediawiki/extensions/RDFIO] (master) - https://gerrit.wikimedia.org/r/42223 [09:21:45] New patchset: Raimond Spekking; "Consistency tweaks in preparation for re-adding to Translatewiki.net" [mediawiki/extensions/RDFIO] (master) - https://gerrit.wikimedia.org/r/42223 [10:47:40] Change merged: Siebrand; [mediawiki/extensions/RDFIO] (master) - https://gerrit.wikimedia.org/r/42223 [15:52:52] hey #semantic-mediawiki, yaron! i have written a utility to transform mysql tables into page schemas, is there anyway to programmatically submit and create them without using screen scraping ? [15:53:10] Hello! [15:53:12] Wow!!! [15:53:16] Awesome. [15:53:30] it seems pretty useful [15:53:31] I don't understand the question, though. [15:54:02] How is screen scraping involved? [15:54:25] well, it generates the xml and saves it on the wiki, but then the front end has to be used to select "generate pages" and then confirm that [15:55:27] Ah, okay, right. That's not quite "screen scraping", though I don't know what it's called. [15:55:52] Page Schemas unfortunately doesn't have an API, but that's something that it would make sense to add. [15:55:56] yeah 'programmatically access the form front end' was kinda long [15:56:08] Right. :) [15:56:14] It sounds like you're trying to submit the data to the wiki without having to use the web interface, right? [15:56:48] Frank__ i have no problem submitting the data, it's just triggering the template, form, properties generation [15:56:48] Frank__: not quite submit the data, but really just press a button. (I think.) [15:56:54] Yeah. :) [15:57:02] Ah, OK. [15:57:22] nostriluu: how many categories are in your overall schema, by the way? [15:58:12] 53 [15:58:44] Wow! Well, that explains it. :) [15:59:11] yes, we're using this to do mockups so it will need constant refreshes [15:59:20] It really shouldn't be too hard to make an API call for it... [15:59:39] well that would be brilliant, there is lots of neato stuff that could, nay should be done with schemas [15:59:48] Would you want to be able to select which pages get generated, or is it enough to generate/overwrite everything every time? [16:00:18] for now just updating everything would be fine [16:01:21] i should mention, cough cough, my code is in nodejs, so the api call would have to be at the api.php level [16:01:21] Right, okay. [16:01:40] Yeah, that's what I was planning to do. [16:01:46] perfect ! [16:01:51] Ooh, Node.js, very nice. [16:01:58] i like it very much. [16:02:04] i wish smw were written in it :) [16:02:15] Why not all of MediaWiki? :) [16:02:20] even better. [16:02:42] and use it for the template language, and we are in the glistening harmonious universe. [16:03:05] Wow, that's mind-blowing. [16:03:32] i don't use the word glistening casually [16:04:12] Maybe MediaWiki could just interpret itself, or something. [16:05:43] So - is there any chance of you publishing this code? It might be useful for other people. [16:05:49] for sure [16:06:07] i could put it on github with a link from the page schemas page ? [16:06:09] Alright, great - I could link to it from the PS page; or it could go on there directly. [16:06:18] Right! [16:07:31] So how does the process work, by the way? (Ideally, at least.) You update the wiki's data structure directly from the DB schema every night or so, using Page Schemas, and then... do an import of the actual data? [16:08:29] this is just intended for an up front generation based on schemas. i've written a pile of java code that does that sort of stuff, but i'm moving to node for new work [16:09:40] unfortunately due to the need to deal with templates and a fair amount of arbitrariness the code is pretty gross in places, which has kept me from releasing it. not sure how to deal with that aside from coding in php :\ [16:10:51] Would PHP be any better about it? [16:11:01] i heard a while ago there's some effort to build out smw's api.php, would definitely be interested in participating in building reusable libs [16:11:11] well i mean php could reuse chunks of actual mw/smw for things like parsing [16:12:00] Oh - are you still just talking about turning SQL data structure calls into PS XML, or other stuff as well? [16:12:30] all kinds of other stuff. the java code is really very useful but it has shortcomings due to the forementioned [16:13:04] Right. [16:14:03] Yeah, sooner or later, you might have to start writing PHP. :) [16:15:02] What features do you want to see added to SMW's API, by the way? You can already do complete queries with it, I think.. [16:16:39] yes, i've used that quite a bit based on scraping and the new api.php stuff. [16:17:00] i had a list, but would have to think about it when my brain is out of this current task [16:17:43] i should also properly package this up for github but happy to share it in its raw form if you want to look at it [16:20:01] Yeah, I'm definitely curious! [16:20:36] Assuming this code works for you, it should be fairly easy to transport the logic to some other language, like PHP, if necessary. [16:21:42] ok, it's here: http://zooid.org/~vid/MysqlToSMW.js [16:22:39] it's a bit twisty since this is my first real node prog and i haven't picked up a framework for that sort of thing yet [16:24:46] Oh, neat! It doesn't look that bad. [16:25:22] I figured the utility would parse SQL - but this connects to a database and gets table description data directly, which seems like the easier approach. [16:25:48] i just reordered it to make it a bit easier to follow if you reresh [16:26:15] yes much easier i think [16:26:24] Alright, cool. [16:27:09] it does AllowedValues but it would like it to support values from category where it recognizes a relationship using _id [16:27:24] I'm confused about one thing - you have special handling if the type of the field contains "_id". [16:27:30] Right. [16:27:31] yes, that's by convention in our case [16:27:43] By "type" there, do you mean "name"? [16:28:01] I've never heard of a field type (like "varchar") ending in "_id". [16:28:17] ah ! thanks :) [16:30:24] Okay - that makes sense, then. [16:30:48] Yes, that's specific to your DB - though, on the other hand, that's a fairly common convention, from what I know. [16:30:55] Ruby on Rails uses the same thing. [16:33:52] i did try to keep the code general but that's a big win for this app [16:34:31] and i can also generate a cheap relationship diagram [16:35:45] which is something smw could really use [16:37:39] Oh - interesting. Is that a separate utility? [16:48:00] no, i just now added properties if it notices an _id relationship. works great. [17:06:38] nostriluu: by "relationship diagram" did you mean a visual diagram? [17:08:06] yes, it's a private wiki but i've temporarily opened it: http://mockwiki.rightscase.equalit.ie/w/Sandbox [17:08:36] unfortunately graph format linking doesn't work anymore:\ [17:12:27] Oh... I get it - you have some utility that adds a property to the wiki every time it sees an "_id" column in a SQL table description. [17:12:57] yup. very helpful for this project [17:14:35] i moved the image here: http://floaty.zooid.org/~vid/rcase_schemas.png [17:14:58] That might make sense as an addition to Page Schemas, by the way - something that indicates, "this property points to pages in this category". [17:15:15] Oh, nice. [17:15:18] that's the main reason i was originally interested in page schemas [17:15:41] That's interesting. [17:16:28] i would have preferred to use smw+ for all this with it's formal linked data approach, but it seems a bit stuck [17:16:33] its [17:16:58] No comment. [17:17:18] i wish i knew what was going on there, it seems like a big loss for everyone [17:18:14] One thing that it definitely makes sense to add to Page Schemas is "this property points to pages that use this form" - to enable smart red links; which is similar, though not the same, as "points to pages in this category". [17:19:05] yeah page schemas could be the cornerstone of a lot of good stuff and especially make large wikis more managable [17:19:17] (large == lots of classes) [17:19:56] Yeah... maybe this year all the Page Schemas stuff can really become a focus. [17:20:18] along with api.php interaction i hope [17:20:22] Your SQL-to-PS utility is a good start to that, definitely. [17:20:46] Right... what other kinds of API actions are you envisioning, besides generating pages, if any? [17:23:27] i can't really think offhand since there's been a break since my last intense period, but anything to help deal with ambiguities around templates. for example, figuring out what properites are on a rendered page without trying to reproduce the templating sytsem [17:25:29] i'll get back to you on that one though since my brain is in another universe now [17:31:17] Alright. [17:31:51] Yeah, I understand - I'm still recovering from vacation. :) [17:37:27] yay, more improvements, handles autocomplete on category now (using _id) [17:41:38] Cool. [17:43:00] Ideally Page Schemas would be able to directly store "property points to category" information, to enable both autocompletion and red links. [17:45:58] yes. it'd also be great if the generation triggered by api call would not be queued [17:53:22] adding in 'has default form' to those fields would pretty much make it an app generator from an existing DB, aside from link tables [17:58:12] Hi. I am trying to set up properties that have type quantity and was wondering in order for the search to work do the values on the pages being searched need to be # "space" unit or is #unit ok? For example, an item gives a 5% damage boost....the property has [[Has Type:Quantity]][[Corresponds To:1 %]]. Will a semantic search find "5%" or does it need to be "5 %"? Thanks [18:34:57] nostriluu: you mean, you don't want "generate pages" to use the job queue? [19:08:07] Change merged: Jeroen De Dauw; [mediawiki/extensions/SemanticResultFormats] (master) - https://gerrit.wikimedia.org/r/40356 [19:09:01] Change merged: Jeroen De Dauw; [mediawiki/extensions/SemanticMediaWiki] (master) - https://gerrit.wikimedia.org/r/41220 [20:00:38] yaron, that's right, cause with many entities it takes quite a while [20:01:22] That's true, although... trying to do everything at once might not work. [22:30:32] New patchset: Jeroen De Dauw; "Change param definitions to work with Validator 1.0" [mediawiki/extensions/SemanticMediaWiki] (master) - https://gerrit.wikimedia.org/r/39858