[09:12:37] <Auregann_WMDE>	 hey hjb_ to get in touch with the Wikibase community, there's also a mailing-list https://lists.wikimedia.org/mailman/listinfo/wikibaseug and a Telegram channel :)
[09:28:24] <Cyberpower678>	 Okay.  I'm at my wits end here.
[09:29:04] <Cyberpower678>	 I've been trying to get IABot to interface with Wikidata, but in literally every experiment I tried when it comes to updating the data, I get an error from the API.
[09:29:36] <Cyberpower678>	 I'm trying to follow, what I am now considering the very poor API documentation, but I'm not getting anywhere.  I have been struggling with this for the last week now.
[09:30:03] <Cyberpower678>	 Is there an API expert here familiar with the Wikibase functions?
[09:31:36] <Cyberpower678>	 Anyone?
[09:31:47] <Cyberpower678>	 Because I'm literally about to cry here.
[09:31:51] <Cyberpower678>	 :(
[09:33:06] <Cyberpower678>	 Matthew: addshore: perhaps?
[09:33:30] <Cyberpower678>	 or FastLizard4
[09:33:50] <addshore>	 hi
[09:34:06] <addshore>	 what are you trying to do?
[09:34:14] <addshore>	 =]
[09:34:47] <Cyberpower678>	 addshore: so the gist of it, is I'm using wbgetentities to fetch the QID and look for URL/Archive URL properties.
[09:34:58] <addshore>	 yup
[09:35:43] <Cyberpower678>	 Noting down the sequences, and hashes, I take them, process them as I would on Wikipedia, and attempt to update the identity with either wbsetidentity, wbsetclaim, or wbsetreference. 
[09:35:48] <Cyberpower678>	 I get errors for all three
[09:36:05] <addshore>	 wbsetidentity? O_o
[09:36:28] <addshore>	 can you give me an example request your giving?
[09:36:32] <addshore>	 and or the error you are getting
[09:36:51] <Cyberpower678>	 *wbeditidentity
[09:36:58] <addshore>	 and also, what action on a claim are you trying to do? update the value? add a ref? change a ref? etc?
[09:37:17] <Cyberpower678>	 addshore: serialization exceptions, wikibase-invalid errors, etc..
[09:37:39] <Cyberpower678>	 addshore: add a ref if it doesn't exist, or update it if it does.
[09:37:55] <Cyberpower678>	 I understand I needs a staks sequence and a reference hash.
[09:38:07] <addshore>	 staks sequence?
[09:38:29] <Cyberpower678>	 addshore: statement.  The GUID
[09:38:36] <addshore>	 gotcha
[09:39:13] <Cyberpower678>	 But now I'm getting serialization errors.  I'm so depressed about this, and after I promised to release IABot at the Wikiconference last November.
[09:40:16] <Cyberpower678>	 Ideally, I would like to edit the identity in one swoop, rather then one claim at a time.
[09:40:19] <addshore>	 what are you sanding to the api?
[09:40:31] <addshore>	 you can edit it all at once using wbeditentity
[09:40:40] <Cyberpower678>	 Well in the latest experiment, I tried to edit a single reference.
[09:40:45] <addshore>	 but then you don't get "nice summaries"
[09:40:59] <Cyberpower678>	 addshore: I get errors with no error messages.  So I don't even know what I'm doing wrong.
[09:41:10] <addshore>	 no message at all? O_o
[09:41:18] <addshore>	 boy oh boy you must be doing something very wrong
[09:41:23] <Cyberpower678>	 Mmmmhhmmm
[09:41:42] <Cyberpower678>	 I figured.  But the documentation is not very helpful in this case.
[09:42:17] <addshore>	 so, let me make an example on test
[09:42:24] <Cyberpower678>	 SO for example, I'm using wbsetreference just to see if I can even fashion a functioning statement, and am sending the following
[09:42:28] <Cyberpower678>	 {"P1065":{"snaktype":"value","property":"P1065","datavalue":{"value":"http:\/\/centraalmuseum.nl\/ontdekken\/object\/#o:830","type":"string"},"datatype":"url"},"P2960":{"snaktype":"value","property":"P2960","datavalue":{"value":{"time":"+2019-01-01T23:59:59Z","calendarmodel":"http:\/\/www.wikidata.org\/entity\/Q1985727"},"type":"time"},"datatype":"time"},"P813":{"snaktype":"value","property":"P813","datavalue":{"value":{"t
[09:42:28] <Cyberpower678>	 ime":"+2019-01-01T23:59:59Z","calendarmodel":"http:\/\/www.wikidata.org\/entity\/Q1985727"},"type":"time"},"datatype":"time"}}
[09:42:47] <Cyberpower678>	 with the statement Q24066189$8959A0C5-1366-409A-807B-3B5FABFD3593
[09:43:36] <Cyberpower678>	 addshore: ^ What I expect is that the https://www.wikidata.org/wiki/Q24066189 described at URL property gets and archive URL reference.
[09:44:20] <Cyberpower678>	 What I get is the error "Failed to get reference from reference Serialization The snak serialization should be an array"
[09:44:32] <Cyberpower678>	 It's the only error I got so far with an actual message.
[09:46:14] <addshore>	 so
[09:46:18] <addshore>	 Get the entitiy
[09:46:18] <addshore>	 https://test.wikidata.org/wiki/Special:ApiSandbox#action=wbgetentities&format=json&ids=Q164486
[09:46:27] <addshore>	 https://www.irccloud.com/pastebin/N6mRIceH/
[09:46:36] <addshore>	 https://www.irccloud.com/pastebin/wC3BZ3Y1/
[09:46:51] <addshore>	 (of course, add your own token)
[09:48:08] <Cyberpower678>	 But that's what I'm doing I'm building the JSON data, and the serializing it with json_encode
[09:48:32] <Cyberpower678>	 And then I get a serialization error.
[09:51:10] <Cyberpower678>	 addshore: Can you check to see where I went wrong with my test?
[09:51:14] <addshore>	 so, Cyberpower678 it looks like your error "The snak serialization should be an array" is pretty specific
[09:51:32] <addshore>	 your list of snaks is not an array, it is a single object with {}
[09:51:40] <addshore>	 {"P1065":{"sna
[09:51:41] <addshore>	 should be
[09:51:44] <addshore>	 {"P1065":[{"sna
[09:51:48] <Cyberpower678>	 So how do I get PHP to generate it as an array?
[09:52:11] <addshore>	 well, its not even that, your missing 1 level of depth there
[09:52:34] <Cyberpower678>	 I am.  I thought it imitated what the docs are using.
[09:53:27] <addshore>	 not the docs on api.php at least ( i copied the first one there and it worked)
[09:53:38] <Cyberpower678>	 The docs have it showing that the properties are the highest level array indexes.
[09:54:46] <addshore>	 Then the next issue you have is "timezone field required"
[09:55:19] <addshore>	 Cyberpower678: if you are doing this in php you should have a look at my libraries
[09:55:30] <addshore>	 https://github.com/addwiki/wikibase-api
[09:56:01] <addshore>	 they allow you to use all of the internal data objects and serialization logic used in wikibase itself, so you wont need to worry about any of this trying to do it yourself stuff
[09:56:43] <Cyberpower678>	 does it communicate with the API or does it simply prepare the queries?
[09:57:03] <Cyberpower678>	 Because IABot has it's own internal set of functions to talk to the API with.
[09:58:50] <addshore>	 it talks to the api
[09:59:19] <addshore>	 where is the IABot code? I could probably write something to allow you to use an adapter
[10:01:02] <addshore>	 infact, with a little bit of cleanup, all you would need is something that implements https://github.com/addwiki/mediawiki-api-base/blob/master/src/MediawikiApiInterface.php
[10:01:15] <Cyberpower678>	 addshore: https://github.com/cyberpower678/Cyberbot_II/tree/test/IABot
[10:01:36] <Cyberpower678>	 The wikidatawiki extension is not present as I haven't committed it yet.
[10:02:05] <Cyberpower678>	 But I can't pastebin it if you'd like
[10:03:09] <addshore>	 so a fixed copy of your query / json you posted above iss:
[10:03:13] <addshore>	 https://www.irccloud.com/pastebin/ODuiD31Y/
[10:03:31] <addshore>	 for the json specifically that is
[10:03:31] <addshore>	 {"P93":[{"snaktype":"value","property":"P93","datavalue":{"value":"http:\/\/centraalmuseum.nl\/ontdekken\/object\/#o:830","type":"string"},"datatype":"url"}],"P151":[{"snaktype":"value","property":"P151","datavalue":{"value":{"time":"+2019-01-01T00:00:00Z","timezone": 0,"before": 0,"after": 0,"precision":
[10:03:31] <addshore>	 11,"calendarmodel":"http:\/\/www.wikidata.org\/entity\/Q1985727"},"type":"time"},"datatype":"time"}],"P25":[{"snaktype":"value","property":"P25","datavalue":{"value":{"time":"+2019-01-01T00:00:00Z","timezone": 0,"before": 0,"after": 0,"precision": 11, "calendarmodel":"http:\/\/www.wikidata.org\/entity\/Q1985727"},"type":"time"},"datatype":"time"}]}
[10:04:37] <addshore>	 you can actually use the parsevalue api module to get JSON objects for simple string values, so you give it "1st June 2019", and it will give you the json back, but you probably dont want that overhead
[10:06:07] <addshore>	 Cyberpower678: I was planing on squashing all of the libraries under https://github.com/addwiki into a single repo to make working on them easier, and then do subtree splits for releases of individual things (probably)
[10:06:33] <addshore>	 I can probably work on that this evening a little bit and give you a poke with a fixed up easy 1 composer package to use that you can pass your own thing into that calls the api etc
[10:07:36] <Cyberpower678>	 addshore: I just need the JSON.
[10:07:54] <addshore>	 well, for just the JSON, you have all of the existing
[10:08:00] <addshore>	 *existing php libraries used by wikibase
[10:08:12] <addshore>	 you can just look at that wikibase-api library as an example of how to pull them all together
[10:08:32] <addshore>	 Cyberpower678: https://github.com/addwiki/wikibase-api/blob/master/composer.json#L16-L19
[10:10:21] <Cyberpower678>	 Include those four lines?
[10:10:23] <addshore>	 Cyberpower678: getting a revision object from the api result using the libs https://github.com/addwiki/wikibase-api/blob/master/src/Api/Service/RevisionGetter.php#L94-L101
[10:12:43] <Cyberpower678>	 addshore: I included the four lines in my composer file.  What now.
[10:13:10] <Cyberpower678>	 addshore: What I would like to do is edit the indentity instead of each individual reference.
[10:13:29] <Cyberpower678>	 But that gets me an invalid wikibase error.
[10:14:18] <addshore>	 what wikibase error? again, it is probably due to the had crafted serialization
[10:14:38] <addshore>	 *hand
[10:14:41] <Cyberpower678>	 Well let me try.
[10:14:54] <Cyberpower678>	 It's not hand crafted.  It was done by json_decode
[10:15:00] <Cyberpower678>	 *json_encode
[10:17:19] <Cyberpower678>	 Or just update the clai
[10:19:07] <Cyberpower678>	 addshore: I've gotten myself all confused now.  I'm totally flustered.
[10:19:10] <Cyberpower678>	 :-(
[10:19:17] <addshore>	 :<
[10:19:33] <addshore>	 well, we got the last api call you poked me toward working :)
[10:20:11] <Cyberpower678>	 addshore: Alright.  What's the best way to edit all of the references in all of the claims, in one shot?
[10:20:32] <Cyberpower678>	 That would be wbeditentity right?
[10:20:36] <addshore>	 wbeditentity, it is the only way to touch multiple parts of the entity at once
[10:25:19] <Cyberpower678>	 So I'm trying to do something like {"claims":{"P973":{"id":"Q24066189$8959A0C5-1366-409A-807B-3B5FABFD3593","references":{"snaks":{"P1065":{"snaktype":"value","property":"P1065","datavalue":{"value":"http:\/\/centraalmuseum.nl\/ontdekken\/object\/#o:830","type":"string"},"datatype":"url"},"P2960":{"snaktype":"value","property":"P2960","datavalue":{"value":{"time":"+2019-01-01T23:59:59Z","calendarmodel":"http:\/\/www.wikida
[10:25:20] <Cyberpower678>	 ta.org\/entity\/Q1985727"},"type":"time"},"datatype":"time"},"P813":{"snaktype":"value","property":"P813","datavalue":{"value":{"time":"+2019-01-01T23:59:59Z","calendarmodel":"http:\/\/www.wikidata.org\/entity\/Q1985727"},"type":"time"},"datatype":"time"}}}}}} to it
[10:26:07] <Cyberpower678>	 addshore: ^ it comes back with "wikibase-api-invalid-claim"
[10:26:19] <addshore>	 so, it looks like you have some of the same issues here, the snaks inside the property snak list are not in an array, the timevalue serialization is wrong and missing stuff expected by the api
[10:26:46] <addshore>	 Another thing I fixed in the last one is removing the hours, mins and seconds, wikidata only supports precision to the day in date time value currently
[10:27:01] <addshore>	 So 1) {"claims":{"P973":{"i >>> {"claims":{"P973":[{"i
[10:27:14] <addshore>	 2) T23:59:59Z >> T00:00:00Z
[10:27:15] <stashbot>	 T23: Identify features Gerrit users would miss in Phabricator - https://phabricator.wikimedia.org/T23
[10:27:35] <addshore>	 3) the timevalue serializations needs this in them too ,"timezone": 0,"before": 0,"after": 0,"precision": 11,
[10:27:51] <addshore>	 then give that another go and see what happens
[10:31:59] <Cyberpower678>	 addshore: still an error.
[10:32:06] <addshore>	 whats the message now?
[10:32:21] <addshore>	 and paste me what you are sending? :)
[10:32:47] <Cyberpower678>	 addshore: https://pastebin.com/Bh1r0y26
[10:33:00] <Cyberpower678>	 Same thing.  Invalid claim
[10:33:47] <addshore>	 "P1065":{" also needs to be "P1065":[{"
[10:34:03] <addshore>	 so every time there is a snak list indexed by property id, that element needs to be an array of objects
[10:38:00] <Cyberpower678>	 addshore: still no good.
[10:38:46] <Cyberpower678>	 https://pastebin.com/fSAKX7kV
[10:39:39] <addshore>	 references should also be an array
[10:39:48] <addshore>	 probably also need the rank and type in the statement object
[10:40:03] <addshore>	 if you look at what wbgetentities gives you, if you send back the same thing, it will work, that is the expected format
[10:41:06] <addshore>	 statement is missing the mainsnak too
[10:41:08] <Cyberpower678>	 addshore: I thought anything I don't send, won't get touched.
[10:41:26] <Cyberpower678>	 But I don't want to touch the mainsnak.
[10:41:29] <addshore>	 yes, but if you are touching the statement, you have to give it a valid statement
[10:42:19] <Cyberpower678>	 Oh god.  Wikidata.
[10:43:02] <Cyberpower678>	 addshore: going back to your API, can I do this with your API, or at least have the codes necessary to create a proper wbeditidentity statement?
[10:43:32] <Cyberpower678>	 Well I mean have it create the JSON needed and I can just send it off with my bot's code?
[10:43:40] <addshore>	 Right, so this is example valid json modified from yours
[10:43:44] <addshore>	 https://www.irccloud.com/pastebin/euUo7MeY/
[10:45:12] <addshore>	 So, regarding the library, why do you not want it to do the actual api call?
[10:45:30] <addshore>	 I mean, there is totally something I could hack together to capture the url params or something
[10:45:31] <Cyberpower678>	 Because IABot has the functions built in and uses OAuth.
[10:45:40] <Cyberpower678>	 It's deeply integrated.
[10:46:45] <addshore>	 Could do something like this
[10:46:48] <addshore>	 https://www.irccloud.com/pastebin/s5iI1QEn/
[10:46:50] <addshore>	 maybe
[10:47:09] * addshore scroll back up to the code you sent
[10:47:46] <addshore>	 oh dear god https://github.com/cyberpower678/Cyberbot_II/blob/test/IABot/APII.php is large, eww
[10:47:46] <addshore>	 :P
[10:48:12] <Cyberpower678>	 addshore: it integrates with multiple APIs.  MW, Wayback, etc...
[10:48:59] <addshore>	 is there a function in there to just, post?
[10:49:21] <Cyberpower678>	 Well I have a specific Edit function.
[10:49:35] <Cyberpower678>	 Pass the data to it, and it does the rest.
[10:49:50] <addshore>	 well, if you make a specific generic post with auth function
[10:50:17] <addshore>	 then I could write a class that implements the interface from my library, and then you could just use the whole library, but backed by your crazy api class and methods
[10:51:32] <Cyberpower678>	 addshore: I wouldn't call it crazy.  It just kind of grew to that size.  I am rewriting it.
[10:51:37] <Cyberpower678>	 https://pastebin.com/CZTaEUEe
[10:52:00] <Cyberpower678>	 This is the extension I'm working on.  You can edit the edit function to however you want it.
[10:52:35] <Cyberpower678>	 Bear in mind that this is still a work in progress here, as I've been struggling with this.
[10:54:21] <addshore>	 https://www.irccloud.com/pastebin/tGIU06Vm/
[10:54:52] <addshore>	 if you literally have something that does that, then i can write something to join the 2 things together, and you can have a nice easy interface to wikidata
[10:55:48] <Cyberpower678>	 Go ahead and add it to the extension.
[10:55:53] <Cyberpower678>	 Oh wait.
[10:56:59] <addshore>	 well, if you make something that can do authed api calls when given the params generated by my lib
[10:57:08] <addshore>	 then i can make something that will work for you
[10:57:09] <addshore>	 :)
[10:59:25] <addshore>	 Cyberpower678: step #1, add the addwiki/wikibase-api library to your composer.json with version ~0.7 and see if it installs :)
[10:59:51] <Cyberpower678>	 https://pastebin.com/KAnesNyq
[11:00:02] <addshore>	 cool
[11:00:05] <addshore>	 what class is that in?
[11:00:27] <addshore>	 got a meeting will be back in 15 mins or less
[11:00:32] <Cyberpower678>	 The wikidatawikiAPI that extends API
[11:04:38] <Cyberpower678>	 composer installed the packages
[11:15:29] <Cyberpower678>	 addshore: ping
[11:15:37] <addshore>	 pong
[11:15:42] <addshore>	 2 more mins
[11:18:51] <addshore>	 something like this could work
[11:18:54] <addshore>	 https://www.irccloud.com/pastebin/CSIletW5/
[11:19:53] <addshore>	 then you can use this CustomAddwikiMediawikiApi object to make the WikibaseApi object from the wikibase-api library for use in services etc,
[11:20:12] <Cyberpower678>	 addshore: what's in the request?
[11:20:25] <addshore>	 all of the url params needed
[11:20:40] <addshore>	 then you can follow the docs on https://github.com/addwiki/wikibase-api#load--general-setup
[11:20:53] <addshore>	 but of course, you can ignore the MediaWikiApi bit there, as you will use your custom one
[11:22:43] <addshore>	 so for example:
[11:22:48] <addshore>	 https://www.irccloud.com/pastebin/qHzEQ0d1/
[11:24:06] <addshore>	 the wikibase factory has a whole load of services for doing a whole load of things, basically each api module has its own service
[11:24:12] <addshore>	 ping me if you have questions etc
[11:27:55] <Cyberpower678>	 addshore: I'm still a little confused by your code .
[11:30:08] <Cyberpower678>	 addshore: the RevisionSaver is that the thing that saves the entity and modifies it?
[11:30:57] <addshore>	 well, revision saves a revision, if you have one
[11:31:18] <addshore>	 there are more docs, for example, creating a new statement https://github.com/addwiki/wikibase-api#create-a-new-statement
[11:31:32] <addshore>	 adding a ref to a statement https://github.com/addwiki/wikibase-api#add-a-reference-to-a-statement
[11:34:40] <Cyberpower678>	 addshore: do each of these calls change the entity, or is there a final call for that to submit the entity?
[12:19:04] <Addax>	 I'm trying to figure out when a given entity was updated (to see if changes were made). I'd LIKE to do this when gathering my initial data (i.e., "get all businesses, and show when they were updated as part of the query" - but P5017, the field that seems to control this, is always empty in the query results.
[12:19:37] <Lucas_WMDE>	 Addax: P5017 is only for references, apparently
[12:19:42] <Lucas_WMDE>	 I think you want `schema:dateModified`
[12:19:46] <Addax>	 If I query for specific entities (i.e., grabbing an individual QID) I can see a last update timestamp, so it has to be there *somewhere* - I just don't know how to get it without querying each QID
[12:19:52] <Addax>	 let me try that
[12:20:34] <Lucas_WMDE>	 e. g. http://tinyurl.com/y5f6srfh
[12:21:06] <Addax>	 oh man, you are a freaking godsend
[17:31:03] <addshore>	 SMalyshev: do multiple updaters run for for the wdqs servers for prod?
[17:31:10] * addshore was just looking at https://phabricator.wikimedia.org/T217897
[17:31:27] <SMalyshev>	 addshore: well, yes, each server has its own updater
[17:31:33] <addshore>	 ack
[17:49:04] <CP678>	 addshore: thanks for helping me to understand entities better.  I just made my first successful test edit. https://www.wikidata.org/w/index.php?diff=prev&oldid=881169091
[17:49:16] <addshore>	 CP678: woooo!
[17:49:23] <addshore>	 using the library or your stuff? =]
[17:49:34] <addshore>	 or both? :P
[17:50:17] <CP678>	 When I was looking at your library, it helped me to understand how to correctly do it.  Rather than build the entity, I should just download it, modify directly, and then upload back into WikiData,
[17:50:31] <CP678>	 So I'm using my own stuff.
[17:51:18] * Cyberpower678 tries to avoid throwing on additional libraries when possible..
[17:52:13] <Cyberpower678>	 addshore: remind me where BRFA is on Wikidata?
[17:52:25] <Cyberpower678>	 Because I'm finally ready to open on.
[17:52:58] <addshore>	 :)
[17:53:30] <addshore>	 Cyberpower678: https://www.wikidata.org/wiki/Wikidata:Requests_for_permissions/Bot i think
[17:54:09] <Cyberpower678>	 addshore: Did I mention I was learning this while not sleeping. :P. I've been awake for 36 hours now.
[17:54:18] <Cyberpower678>	 :p
[17:55:57] <addshore>	 i did think that you should be in a different timezone
[18:32:05] <sjoerddebruin>	 Cyberpower678: woot!
[18:32:20] <Cyberpower678>	 :-)
[18:33:57] <addshore>	 SMalyshev: does each server only run 1 copy of wdqs currently, or do some run multiple copies?
[18:36:00] <SMalyshev>	 addshore: each server runs one instance of wdqs, but that includes 2 instances of blazegraph (for main and category namespaces)
[18:42:35] <abian>	 Cyberpower678: Great news I see :D
[18:42:55] <Cyberpower678>	 :-0
[18:42:57] <Cyberpower678>	 :-)
[18:43:02] <abian>	 Cyberpower678: Do you think it would be nice to move that repository to https://gerrit.wikimedia.org ?
[18:43:35] <abian>	 And keep Github as a mirror (that's the default behavior of gerrit projects)
[18:48:44] <addshore>	 SMalyshev: thanks! I'm gonna write a rather substantial comment on that ticket this evening
[18:49:35] <SMalyshev>	 addshore: i suspect caching has little to do with what's going on that ticket btw...
[18:50:52] <addshore>	 I still think requests with a specific revid is the right way to go, and im gonna basically comment with some data to back that up
[18:51:09] <SMalyshev>	 addshore: specific revid would absolutely kill performance
[18:51:17] <SMalyshev>	 I tried it
[18:51:48] <addshore>	 which bit kills the perf?
[18:51:58] <SMalyshev>	 using specific revid
[18:52:09] <SMalyshev>	 it converts one fetch to 20 fetches
[18:52:10] <addshore>	 on a single day im looking at there are 1.1 million edits, and the prod wdqs updaters end up making between 826,878 and 926,848 requests to special entity data
[18:52:31] <addshore>	 so, it is clear the batching is removing some duplication, but not a whole lot
[18:52:57] <SMalyshev>	 addshore: see the ticket, batching doesn't remove a lot because there's no global revid map (yet)
[18:53:20] <addshore>	 the thing with using revid, is other updaters could also use it and then share the varnish cache
[18:53:29] <SMalyshev>	 but fetching by specific revid would prevent forward-hopping, which would mean global revid map would be useless
[18:53:50] <addshore>	 on this given day 58% of the WDQS updater calls to special entity data came from us, 42% came from 3rd parties
[18:53:55] <SMalyshev>	 addshore: yeah, the problem is they shouldn't share that cache - they shouldn't load that revision at all
[18:54:40] <SMalyshev>	 addshore: the problem is like this: if there's 20 edits to the same item, last thing we want is 20 fetches
[18:55:23] <SMalyshev>	 fetching by revid mandates that (with slight relief from in-batch aggregation but only small one)
[18:59:21] <addshore>	 well, looking at the current batching, it isn't reducing the load count by all that much though? thought it looks like the plan would be to alter that?
[19:00:07] <addshore>	 From the load on wikibase side of things, what I see is 1.1 million revisions, thus there should be 1.1 php requests rendering data for special entity data, and then those being cached in varnish
[19:00:40] <SMalyshev>	 addshore: you don't want 1.1 million requests, it's too slow
[19:00:56] <SMalyshev>	 because each load should then be converted to sparql query
[19:01:02] <SMalyshev>	 the whole point is that it should instead load the latest available rev
[19:01:16] <addshore>	 instead though, we get 21 million PHP requests, because nothing hits the cache
[19:01:36] <SMalyshev>	 this is especially important when catching up because you only load each item once per edit
[19:02:34] <SMalyshev>	 I've tried to switch to revids, and it produced huge drop in update performance, it wasn't able to keep up with lags because it had to run sparql update on every revision
[19:02:56] <SMalyshev>	 and sparql updates are very expensive, much more than rdf loads
[19:03:55] <SMalyshev>	 so yes 21 mln is a lot, and we need to solve this, but just using revids would kill performance for the updater, especially when lagged, so that won't work
[19:04:28] <addshore>	 does the rdf include the revid?
[19:04:32] <SMalyshev>	 we could use something smarter like "give me data for this revid or later" but this could get in trouble with caching I'm afraid
[19:04:44] <SMalyshev>	 addshore: yes revid is part of RDf
[19:06:31] <SMalyshev>	 the problem with "revid or later" would be that different servers would start with different revids... so I'm not sure that would give much
[19:06:56] <SMalyshev>	 addshore: btw 1.1M edits looks low... for how long is that?
[19:07:07] <addshore>	 thats for the whole of march 1st
[19:07:21] <SMalyshev>	 oh just one day? that looks ok then
[19:07:51] <SMalyshev>	 but can varnish keep all of that around?
[19:08:41] <addshore>	 well, my comment in this ticket is just generally going to be with some info about what I see is happening, rather than any specific ideas to tackle the issue
[19:08:50] <addshore>	 I guess the can varnish hold X is for brandon :)
[19:11:37] <SMalyshev>	 addshore: ok, some data is always useful... I think using "revid or later" could allow some caching to happen, though I am not sure it'd be beneficial, since if we have revs X, X+1,...., X+20 on same item, and X+1 gets cached, then every other client, even those that could get directly to X+20, would still get X+1 and would have to re-load X+2 etc. potentially if we're really unlucky we still get 20 updates instead of one
[19:11:39] <addshore>	 SMalyshev: heh, looking at wikidata, there were 23.8 million cache misses on the 1st march, and 21 million were from instances of the wdqs updater
[19:12:01] <addshore>	 If I did my query write, but im currently also taking a laptop appart
[19:12:24] <addshore>	 also 54 million cache passes, but thats another kettle of fish
[19:12:26] <SMalyshev>	 we have 14 updaters... 
[19:12:43] <SMalyshev>	 so about 15M of those are ours I guess
[19:13:20] <SMalyshev>	 the question is how big of a problem are they? I mean if we miss one cache hit but earn one less sparql update that'd be a win... 
[19:13:38] <SMalyshev>	 I guess I need to add collecting metrics of how many revisions we skip when updating
[19:14:15] <SMalyshev>	 well I can't really know how many I can just know when we jumped a revision - i.e. when rdf loads revision that is higher than we expected from change event
[19:14:30] <SMalyshev>	 but I can't know how many it skipped
[19:15:01] <addshore>	 well, I can probably infer that from comparing the edit count on the day vs the # of requests from each host
[19:15:08] <addshore>	 with some degree of accuray
[19:17:04] <addshore>	 time for some food over here :)
[19:17:11] <sjoerddebruin>	 Send me some as well...
[19:19:49] <addshore>	 SMalyshev: how long does the sparql query take to execute to update given a revision?
[19:20:12] <addshore>	 is that known or vaugly predictable ?
[19:20:47] <SMalyshev>	 addshore: well, it's hard to say per revision, as we only do a batch but it's rarely we can go over 15 items/second in updater
[19:21:28] <SMalyshev>	 usually it's more around 10... though it's only main update, we also do other queries before/after
[19:21:39] <addshore>	 I think really the updater should use revid for requests, but just internally be clever about when it commits an entity into blazegraph
[19:21:39] <SMalyshev>	 ah wait no it's actually counting those too
[19:22:09] <SMalyshev>	 addshore: it can't be clever really - if it has data for revision X its the only thing it can send to Blazegrapg
[19:22:20] <SMalyshev>	 it can't send data for the revision it doesn't have
[19:22:45] <addshore>	 but the batching is already partially clever, just perhaps in the wrong way
[19:23:34] <SMalyshev>	 do you mean aggregating edits and not writing them out? this is possible, but will introduce delays and we don't really know if it's useful because we might be on the end of the edit sequence already so waiting is useless
[19:23:52] <SMalyshev>	 and of course if we're catching up waiting would be totally useless
[19:24:41] <SMalyshev>	 we could probably use larger batches when loading events and then send them out in smaller batches to sparql... that _might_ be faster, but not 100% sure about it
[19:26:56] <SMalyshev>	 addshore: batching is clever in the right way, it's just very nearshighted since it loads like 500-700 events, sometimes even less, and when active editing is going on repeats may be distanced more in event stream (even if very close in time stream)
[19:28:13] <addshore>	 I mean, essentially this is the same problem as wikidata dispatching to clients (wikipedia), just in a slightly different flavour
[19:29:07] <SMalyshev>	 not sure what wikidata dispatching means
[19:30:03] <SMalyshev>	 I think also if we already seen rev Y, and get change for rev X, X < Y, we might still load the RDF since we don't keep the fact we already have Y in memory, only in the database...
[19:30:07] <SMalyshev>	 let me check
[19:32:16] <SMalyshev>	 ah no we do a query agains db but not data load
[19:46:51] <Cyberpower678>	 abian: I'm not good with Gerrit
[19:47:35] <Cyberpower678>	 Should IABot pre-emptivly add archive URLs, or should it wait until the originals are down?
[19:49:18] <abian>	 Cyberpower678: Preemptively would be great
[19:50:10] <abian>	 Ah, I thought you meant saving the pages, hmm...
[19:50:56] <abian>	 I would wait because adding an archive URL for each URL might be even impossible
[19:53:35] <abian>	 If you really see it's possible, adding the archive URLs would let people access to the page version that is actually referenced
[19:56:12] <abian>	 "Maybe it would be beneficial to have IABot just add the archive URLs to all links since they are merely properties bound to the original?" → So this means it's more efficient to add all the links?
[20:27:12] <addshore>	 well
[20:27:23] <addshore>	 there might even be value in keeping the origional url?
[20:27:29] <addshore>	 abian: Cyberpower678 ?^^
[20:27:41] <addshore>	 what do the archive urls even look like?
[20:28:34] <abian>	 Something like https://web.archive.org/web/<timestamp>/<url>
[20:28:42] <abian>	 https://web.archive.org/web/20190312202759/https://www.wikidata.org/wiki/Wikidata:Main_Page
[20:29:01] <addshore>	 I wonder if that could be done automagically for URLs that already have a retrieval date
[20:29:07] <addshore>	 I wonder what the value is in changing the data?
[20:29:35] <abian>	 It's indeed possible to do it automagically if you have a retrieval date
[20:30:12] <Cyberpower678>	 addshore: abian: IABot can easily add archive URLs to the originals.  Very easy for the bot
[20:30:13] <abian>	 If you look for a timestamp for which there's no snapshot, the previous, existent one is loaded
[20:31:22] <abian>	 But, if it's really efficient and possible, adding all the archive URLs looks great to me too
[20:33:10] <Cyberpower678>	 addshore: https://www.wikidata.org/w/index.php?title=Q1003178&diff=prev&oldid=881282130
[20:33:22] <Cyberpower678>	 This is an example.
[20:34:06] <addshore>	 aah yes, that looks sane
[20:34:46] <Cyberpower678>	 addshore: If not for you, I'd still be banging my head on the desk on how stupid hard the API is for entities.
[20:37:08] <addshore>	 :P
[20:37:12] <addshore>	 its quite simple really ;)
[20:37:18] <addshore>	 just dont go messing aorund with the json :D
[20:37:23] <addshore>	 right, im off