[00:01:43] oh, the other core dump I got from random traffic on mw1088 is also from wikidata serialization [00:02:29] https://bugzilla.wikimedia.org/show_bug.cgi?id=71479#c1 ? [00:03:16] that stacktrace isn't totally helpful as it seems to be re-throwing the exception [00:03:30] or such and the error is deeper in the code some place [00:06:11] (03Merged) 10jenkins-bot: Call $reference->getSnaks() separately in ReferenceSerializer [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/165133 (owner: 10Aude) [00:06:39] it's not the same backtrace [00:06:44] (03PS1) 10Aude: log SerializationException in EntityContentDataCodec [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/165140 [00:06:46] ah [00:07:03] i think ^ might help [00:08:23] (03CR) 10Aude: [C: 032] Don't alter the object passed to ByPropertyListSerializer::getSerialized [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/165131 (owner: 10Hoo man) [00:08:28] (03CR) 10Aude: [C: 032] Don't alter the object passed to ByPropertyListSerializer::getSerialized [extensions/Wikibase] (wmf/1.25wmf1) - 10https://gerrit.wikimedia.org/r/165132 (owner: 10Hoo man) [00:08:38] (03CR) 10Aude: [C: 032] Call $reference->getSnaks() separately in ReferenceSerializer [extensions/Wikibase] (wmf/1.25wmf1) - 10https://gerrit.wikimedia.org/r/165134 (owner: 10Hoo man) [00:12:45] (03PS1) 10Aude: log SerializationException in EntityContentDataCodec [extensions/Wikibase] (wmf/1.25wmf1) - 10https://gerrit.wikimedia.org/r/165141 [00:13:15] (03CR) 10Aude: [C: 04-2] "branch only for now" [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/165140 (owner: 10Aude) [00:24:49] (03CR) 10Aude: [C: 032] log SerializationException in EntityContentDataCodec [extensions/Wikibase] (wmf/1.25wmf1) - 10https://gerrit.wikimedia.org/r/165141 (owner: 10Aude) [00:25:10] (03Merged) 10jenkins-bot: Don't alter the object passed to ByPropertyListSerializer::getSerialized [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/165131 (owner: 10Hoo man) [00:27:12] aude: would you like me to deploy that? i can ask greg-g [00:27:39] (03Merged) 10jenkins-bot: Don't alter the object passed to ByPropertyListSerializer::getSerialized [extensions/Wikibase] (wmf/1.25wmf1) - 10https://gerrit.wikimedia.org/r/165132 (owner: 10Hoo man) [00:28:40] (03Merged) 10jenkins-bot: Call $reference->getSnaks() separately in ReferenceSerializer [extensions/Wikibase] (wmf/1.25wmf1) - 10https://gerrit.wikimedia.org/r/165134 (owner: 10Hoo man) [00:29:13] ori: if you like [00:29:35] i'll need maybe 10 minutes to update our build etc [00:29:40] sure [00:33:50] (03Merged) 10jenkins-bot: log SerializationException in EntityContentDataCodec [extensions/Wikibase] (wmf/1.25wmf1) - 10https://gerrit.wikimedia.org/r/165141 (owner: 10Aude) [00:36:45] (03PS1) 10Aude: Update Wikidata, fixes for serialization issues [extensions/Wikidata] (wmf/1.25wmf1) - 10https://gerrit.wikimedia.org/r/165148 [00:40:16] should i +2? [00:40:24] wait for jenkins [00:45:27] aude: filed https://bugzilla.wikimedia.org/show_bug.cgi?id=71724 for the other core dump I collected [00:45:56] thanks [00:49:46] (03CR) 10WikidataJenkins: "Build Successful" [extensions/Wikidata] (wmf/1.25wmf1) - 10https://gerrit.wikimedia.org/r/165148 (owner: 10Aude) [00:49:55] https://www.wikidata.org/wiki/Q14203619 [00:49:59] is the item [00:50:17] shouldn't be anything strange there [00:51:17] (03CR) 10Aude: [C: 032] Update Wikidata, fixes for serialization issues [extensions/Wikidata] (wmf/1.25wmf1) - 10https://gerrit.wikimedia.org/r/165148 (owner: 10Aude) [00:54:30] how does HashBagOStuff::get end up calling serialize? Is it that using $key as an array index causes it to be cast to string? [00:56:55] ori: i think it uses multiple layers of caching .... in process with HashBagOfStuff [00:57:00] if not found, then the memcached [00:57:05] if still not found, then db lookup [00:57:17] not sure about serialize [00:59:32] (03Merged) 10jenkins-bot: Update Wikidata, fixes for serialization issues [extensions/Wikidata] (wmf/1.25wmf1) - 10https://gerrit.wikimedia.org/r/165148 (owner: 10Aude) [01:52:35] how does HashBagOStuff::get end up calling serialize? [01:52:39] $casToken = serialize( $this->bag[$key][0] ); [01:52:52] return $this->bag[$key][0]; [01:55:13] APCBagOStuff and a few other classes use the value for $casToken, without serialization [06:42:01] So long no Amir1 [06:56:37] (03CR) 10Nikerabbit: UserLanguageLookup: Only return languages we support (033 comments) [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/164725 (owner: 10Hoo man) [07:37:37] amir has a real life .. [07:58:45] (03CR) 10Thiemo Mättig (WMDE): UserLanguageLookup: Only return languages we support (033 comments) [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/164725 (owner: 10Hoo man) [08:19:54] (03PS1) 10Adrian Lang: Remove EntityIdParser test [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/165174 [08:27:17] (03PS1) 10Adrian Lang: Fix EntityStore usage in tests [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/165176 [08:33:08] (03CR) 10Tobias Gritschacher: [C: 031] Cleanup some EntityStore JavaScript code [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/164978 (owner: 10Thiemo Mättig (WMDE)) [08:37:20] (03CR) 10Adrian Lang: "Could be merged, but I have a minor comment." (031 comment) [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/164978 (owner: 10Thiemo Mättig (WMDE)) [08:44:35] (03PS2) 10Thiemo Mättig (WMDE): Cleanup some EntityStore JavaScript code [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/164978 [08:46:23] (03PS1) 10Thiemo Mättig (WMDE): Get rid of ORMTable::getFieldPrefix [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/165178 [08:47:27] (03CR) 10Thiemo Mättig (WMDE): "You are right. I reverted that." (031 comment) [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/164978 (owner: 10Thiemo Mättig (WMDE)) [08:55:22] (03CR) 10Thiemo Mättig (WMDE): [C: 032] Fix EntityStore usage in tests [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/165176 (owner: 10Adrian Lang) [08:55:33] (03CR) 10Nikerabbit: UserLanguageLookup: Only return languages we support (033 comments) [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/164725 (owner: 10Hoo man) [08:55:41] (03CR) 10Thiemo Mättig (WMDE): [C: 032] Remove EntityIdParser test [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/165174 (owner: 10Adrian Lang) [09:03:24] (03CR) 10Adrian Lang: [C: 032] Cleanup some EntityStore JavaScript code [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/164978 (owner: 10Thiemo Mättig (WMDE)) [09:08:39] (03Merged) 10jenkins-bot: Fix EntityStore usage in tests [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/165176 (owner: 10Adrian Lang) [09:11:13] (03Merged) 10jenkins-bot: Remove EntityIdParser test [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/165174 (owner: 10Adrian Lang) [09:13:28] (03Merged) 10jenkins-bot: Cleanup some EntityStore JavaScript code [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/164978 (owner: 10Thiemo Mättig (WMDE)) [10:03:33] [13WikibaseDataModel] 15filbertkm created 06getallsnaks (+1 new commit): 02http://git.io/RNToUQ [10:03:33] 13WikibaseDataModel/06getallsnaks 14584bc04 15aude: Call getQualifiers and getReferences outside of iterator_to_array for getAllSnaks... [10:04:09] [13WikibaseDataModel] 15filbertkm 04force-pushed 06getallsnaks from 14584bc04 to 1467832f1: 02http://git.io/m9RPcA [10:04:09] 13WikibaseDataModel/06getallsnaks 1467832f1 15aude: Call getQualifiers and getReferences outside of iterator_to_array for getAllSnaks... [10:05:28] [travis-ci] wmde/WikibaseDataModel/getallsnaks/584bc04 : aude The build has errored. http://travis-ci.org/wmde/WikibaseDataModel/builds/37271543 [10:05:31] aaaaaa [10:05:34] [13WikibaseDataModel] 15filbertkm opened pull request #216: Call getQualifiers and getReferences outside of iterator_to_array (06master...06getallsnaks) 02http://git.io/oZKI8w [10:11:12] (03PS1) 10Adrian Lang: Remove unused destroy event from listview, improve destroy method [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/165181 [10:13:18] (03PS1) 10Adrian Lang: Replace widgetBaseClass in wbtooltip [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/165182 [10:14:06] (03PS1) 10Adrian Lang: Remove listview.liValue and unused listview.prefixedClass [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/165183 [10:14:46] (03PS1) 10Adrian Lang: Do not use second parameter to jQuery constructor [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/165184 [10:17:59] [13WikibaseDataModel] 15filbertkm 04force-pushed 06getallsnaks from 1467832f1 to 1463d006f: 02http://git.io/m9RPcA [10:17:59] 13WikibaseDataModel/06getallsnaks 1463d006f 15aude: Call getQualifiers and getReferences outside of iterator_to_array... [10:56:13] (03CR) 10Nemo bis: "This commit seems to be trying to fix bug 37459. Please clarify on the bug report what is left to fix, and change the code to use Language" [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/164725 (owner: 10Hoo man) [11:24:25] DanielK_WMDE__: https://bugzilla.wikimedia.org/show_bug.cgi?id=71519 [11:24:48] daaaaa [11:24:51] DanielK_WMDE__: https://bugzilla.wikimedia.org/show_bug.cgi?id=71519 [11:41:15] DanielK_WMDE__: https://bugzilla.wikimedia.org/show_bug.cgi?id=71519#c22 [11:41:18] blob [11:44:28] q [12:01:22] !nyan [12:01:22] ~=[,,_,,]:3 [12:19:01] [13WikibaseDataModel] 15mariushoch comment on pull request #216 1463d006f: Is clone needed here maybe? 02http://git.io/MKCwaQ [13:02:07] hi #wikidata - is there anything I can do to help with the Q183 issue? [13:07:24] (03PS1) 10Adrian Lang: Simplify and generalize array handling in SqlUsageTracker [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/165193 [13:07:26] (03PS1) 10Adrian Lang: Remove reindexEntityIds from SqlUsageTracker [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/165194 [13:07:58] (03CR) 10Adrian Lang: "Some comments; I also added two change sets on top of this." (037 comments) [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/161477 (https://bugzilla.wikimedia.org/64586) (owner: 10Daniel Kinzler) [13:08:08] (03CR) 10Adrian Lang: [C: 04-1] Introduce usage tracking service. [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/161477 (https://bugzilla.wikimedia.org/64586) (owner: 10Daniel Kinzler) [13:09:19] manybubbles: it's very strange issue [13:09:40] i think i somewhat narrowed down the issue but am confused how it happens [13:09:44] yeah and I don't really know the code but I can be useful to hear ideas. I know it isn't a good time either [13:12:46] we are using SplObjectStorage, with hash => traversable object [13:13:09] http://pastie.org/9627735 is what it fails on but only certain situations it is triggered [13:13:39] i see ["snaks":"Wikibase\DataModel\Reference":private]=> [13:13:40] int(7518994243564923343) [13:14:00] where as it should be like ["snaks":"Wikibase\DataModel\Reference":private]=> [13:14:01] object(Wikibase\DataModel\Snak\SnakList) [13:14:34] actually seems the hash is added a second time and it picks the first one (wrong, strange) [13:15:24] if i omit execution of certain code, then it works or chnge the order or some [13:20:33] aude: bleh. Is this something you can reproduce locally with the right data or do you need it in eval.php? [13:20:36] (03CR) 10jenkins-bot: [V: 04-1] Simplify and generalize array handling in SqlUsageTracker [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/165193 (owner: 10Adrian Lang) [13:20:38] (03CR) 10jenkins-bot: [V: 04-1] Remove reindexEntityIds from SqlUsageTracker [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/165194 (owner: 10Adrian Lang) [13:20:42] i am using eval.php [13:21:32] hmmm - if you could get it locally you could add ugly debugging code everywhere [13:21:34] problem is that the code i'm looking at is not new at all, but other code changed [13:22:52] figures :) [13:23:38] i think we are close though [13:24:07] (03PS2) 10Adrian Lang: Simplify and generalize array handling in SqlUsageTracker [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/165193 [13:24:09] (03PS2) 10Adrian Lang: Remove reindexEntityIds from SqlUsageTracker [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/165194 [13:24:38] just trying to figure out what did change that causes the issue now [13:24:48] whereas no problem before [13:28:32] looks like that object has some serialization/unserialization wonkiness. any changes in that area? [13:28:52] yes, there have been [13:29:53] i just wonder how it gets int(7518994243564923343) added [13:30:11] if i call getHash() [13:30:59] hmmmm [13:34:09] (03CR) 10Adrian Lang: [C: 04-1] "Just one thing. Reviewing this is basically impossible, though, because gerrit seems to not recognize the renames." (031 comment) [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/161453 (owner: 10Henning Snater) [13:48:20] [13WikidataBrowserTests] 15tobijat pushed 1 new commit to 06master: 02http://git.io/M2G5rg [13:48:20] 13WikidataBrowserTests/06master 1436ef3f8 15Tobias Gritschacher: Add @firefox tags as required by WMF CI... [13:54:14] [13WikidataBrowserTests] 15tobijat pushed 1 new commit to 06master: 02http://git.io/akhb1Q [13:54:14] 13WikidataBrowserTests/06master 14159f1d5 15Tobias Gritschacher: Add WB_REPO_USERNAME to CI settings [14:32:37] (03PS1) 10Thiemo Mättig (WMDE): Fix bad assumptions in EntityParserOutputGenerator [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/165203 [14:35:32] (03PS1) 10Aude: Get snaks in ReferenceSerializer from cloned $reference [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/165205 (https://bugzilla.wikimedia.org/71519) [14:36:22] (03PS2) 10Thiemo Mättig (WMDE): Fix bad assumptions in EntityParserOutputGenerator [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/165203 [14:37:33] (03CR) 10Aude: "looks like bug 71520 ?" [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/165203 (owner: 10Thiemo Mättig (WMDE)) [14:48:02] back [14:54:49] manybubbles: now that i have narrowed down the specific issue, [14:54:59] 1) think i have a workaround to avoid the problem https://gerrit.wikimedia.org/r/165205 [14:55:10] 2) maybe tim wants to look again, as i think there is a php bug [14:56:19] it'd be hard to pin down a php bug unless you could reproduce it locally [14:56:22] http://pastie.org/9627921 (a bit evil) is what i'm doing to be able to hit the code [14:56:42] and see what might help [14:57:04] maybe i could import all the properties and item to vagrant and reproduce locally [14:57:19] and try with different versions of php and hhvm [14:58:14] probably worth trying. I don't see why we can't try that patch during SWAT though [14:58:16] if it works it works [14:58:24] then we can figure out why [14:58:55] if you think [14:59:24] would be better if hoo or someone looks [15:00:47] (03CR) 10Manybubbles: "Does cloning clear the odd reference or never set it? Would it make more sense to filter it out then try not to make it in the first plac" [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/165205 (https://bugzilla.wikimedia.org/71519) (owner: 10Aude) [15:02:25] aude: yeah - I'm not a real code reviewer for wikidata - i don't know it well enough [15:02:55] manybubbles: it shouldn't be set [15:03:08] on other referencelists in the item, it does not break [15:03:55] aude: i'm just suggesting filtering it out rather than trying not to set it - either way is fine [15:04:21] maybe [15:04:39] we could be smarter about finding the correct reference [15:04:43] e.g. instanceof [15:05:00] but sounds evil [15:09:02] aude: evil might be required temporarily [15:10:03] we can probably put up for swat this evening, 1am [15:10:17] so folks can look at it [15:11:06] k [15:11:07] https://bugzilla.wikimedia.org/show_bug.cgi?id=71519#c24 [15:11:11] details [15:13:17] aude: the docs say getHash just delegates to some other function - maybe call that directly? [15:14:37] http://us1.php.net/manual/en/splobjectstorage.gethash.php [15:14:53] hash isn't the right word for that I think [15:15:50] not sure [15:40:11] DanielK_WMDE__: https://bugzilla.wikimedia.org/show_bug.cgi?id=71479#c1 remains an issue [15:40:23] other items, not Q183 [15:43:23] DanielK_WMDE__: https://bugzilla.wikimedia.org/show_bug.cgi?id=71479#c21 [15:59:23] DanielK_WMDE__: examples https://bugzilla.wikimedia.org/show_bug.cgi?id=71479#c9 [15:59:42] ignore Q72 [15:59:46] different issue, fixed [15:59:51] look at setclaim [16:12:47] (03CR) 10Daniel Kinzler: Get snaks in ReferenceSerializer from cloned $reference (031 comment) [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/165205 (https://bugzilla.wikimedia.org/71519) (owner: 10Aude) [16:16:05] (03PS2) 10Aude: Get snaks in ReferenceSerializer from cloned $reference [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/165205 (https://bugzilla.wikimedia.org/71519) [16:17:02] (03CR) 10Aude: Get snaks in ReferenceSerializer from cloned $reference (031 comment) [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/165205 (https://bugzilla.wikimedia.org/71519) (owner: 10Aude) [16:41:15] Extension:Data Transfer [16:41:26] * https://www.mediawiki.org/wiki/Manual_talk:Pywikibot/Wikidata#CSV [16:49:39] aude: Saw my github comment? [16:49:49] Maybe we need clone [16:55:11] hoo|train: hey. aude just left. should be back online in a bit [16:55:43] Ok, I see [16:56:09] Most of the Q183 timeouts happened in DM code paths [16:56:25] AFAIR [16:57:43] hoo|train: yea... but when profiling, we found that it's rendering that is dog slow... that should not matter when a client page uses the label... [16:58:16] and the hash issue katie found? strange.... [16:58:36] Indeed [16:59:10] But why did client explode then? [17:00:28] (03CR) 10Jeroen De Dauw: "Again, having a failing test case that verifies this actually fixes the issue would be great." (031 comment) [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/165205 (https://bugzilla.wikimedia.org/71519) (owner: 10Aude) [17:03:05] (03CR) 10Jeroen De Dauw: [C: 032] Get rid of ORMTable::getFieldPrefix [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/165178 (owner: 10Thiemo Mättig (WMDE)) [17:03:48] Wow... I made it through the mobile network "blackhole" here w/o loosing connection [17:04:11] But I wonder why they don't fix it :( [17:06:15] (03CR) 10Jeroen De Dauw: [C: 032] Fix bad assumptions in EntityParserOutputGenerator [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/165203 (owner: 10Thiemo Mättig (WMDE)) [17:13:26] (03PS16) 10Daniel Kinzler: Introduce usage tracking service. [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/161477 (https://bugzilla.wikimedia.org/64586) [17:14:24] (03CR) 10Daniel Kinzler: Introduce usage tracking service. (037 comments) [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/161477 (https://bugzilla.wikimedia.org/64586) (owner: 10Daniel Kinzler) [17:24:05] (03CR) 10Daniel Kinzler: Simplify and generalize array handling in SqlUsageTracker (036 comments) [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/165193 (owner: 10Adrian Lang) [18:38:55] (03PS1) 10Daniel Kinzler: Factor connection management into separate class. [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/165260 [18:48:42] (03CR) 10jenkins-bot: [V: 04-1] Factor connection management into separate class. [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/165260 (owner: 10Daniel Kinzler) [18:53:44] (03CR) 10Daniel Kinzler: "The mock is mocking me by not mocking the mocks. What the mock?!" [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/165260 (owner: 10Daniel Kinzler) [19:01:47] hoo: back [19:02:04] i don't know that my changes in the data model are needed at all [19:02:12] :S [19:02:21] What about using clone for such things? [19:02:24] Needed or not? [19:02:28] the one to reference serializer is needed [19:02:36] probably not for now [19:05:57] Ok [19:06:15] How did you debug what is slow for such large items? [19:06:46] aude: ^ [19:06:54] s/debug/find out/ [19:06:58] only for Q183, on terbium with eval.php [19:07:04] same as Q72 [19:07:29] and you turned on the profiler? [19:07:34] no [19:07:45] and not necessarily 'slow' [19:07:49] mh [19:07:53] just what causes the exception [19:08:03] thiemo is doing profiling [19:08:03] Yeah... but what about timeouts while editing [19:08:11] and OOM when viewing? [19:08:13] not yet [19:08:29] probably more tomorrow [19:08:33] Ok [19:09:18] From Thursday on I'll have some time again so that I'll be able to do some stuff then [19:09:44] But if I want to do something I have to be updated with what others are doing [19:09:54] https://gist.github.com/filbertkm/63ac7756cb530014014a is the sort of stuff i was trying [19:09:58] it's evil [19:10:23] On terbium? [19:10:26] but allowed me to essentially bisect the code path and find out what was leading to the problems [19:10:29] yes [19:10:32] :/ [19:10:48] it's a last resort and would prefer to locally reproduce the bug [19:11:37] the entity builder code, and specificaly RefererencedEntitiesFinder was interfering [19:11:50] if that was skipped, then the exception did not occur [19:12:03] mh [19:12:08] but then had to dig more [19:12:11] how long would it take to locally import a jsonDump [19:12:18] solving it there would not really solve this [19:12:21] (in a more or less beefy machine) [19:12:26] no idea [19:12:38] We don't have a script for that anyway :S [19:12:40] i have xml dumps from august and sept [19:12:43] loaded into postgres [19:12:51] Might be good enough [19:12:55] but why postgres? [19:12:57] so i can try stuff with that [19:13:04] serializeing and such [19:13:13] Yeah... getting that working would be nice [19:13:14] it has postgis :) [19:13:20] so i can geocode the coordinates [19:13:34] geocode? [19:13:45] geo queries [19:13:51] Is that a datatype they have? [19:13:51] show them on a map, etc. [19:14:08] yes, special support for geo [19:14:11] it's what osm uses :) [19:14:14] Nice :) [19:14:32] it's more for my bot, but helps to debug also [19:16:13] foo [19:16:15] (03PS3) 10Aude: Get snaks in ReferenceSerializer from cloned $reference [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/165205 (https://bugzilla.wikimedia.org/71519) [19:17:20] i will see about some tests for this [19:17:36] aude: Did you see the bit about strings in json in the mail? [19:18:59] I wonder if that's relevant to wbEntity we send to browsers also [19:19:15] might also be something that slows down the browsers a bit on *huge* items [19:21:04] which mail? [19:21:28] Ok, that's not being send to browsers that way [19:21:58] Subject: Problems with overly large items [19:21:58] Date: Tue, 07 Oct 2014 00:28:39 +0200 [19:22:00] oh, from tim? [19:22:03] to wikidat-tech [19:22:04] utf16 [19:22:05] yep [19:22:05] yes [19:22:50] (03CR) 10Manybubbles: "If we had a failing test case then we probably would have a real solution instead of a workaround. We'll need both eventually but for now" [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/165205 (https://bugzilla.wikimedia.org/71519) (owner: 10Aude) [19:23:56] (03CR) 10Aude: "I am always in favor of more tests, but not sure how best to add a test case for a reference becoming corrupt?" [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/165205 (https://bugzilla.wikimedia.org/71519) (owner: 10Aude) [19:25:08] (03CR) 10Aude: Get snaks in ReferenceSerializer from cloned $reference (031 comment) [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/165205 (https://bugzilla.wikimedia.org/71519) (owner: 10Aude) [19:34:39] hey, I'm trying to set up WD locally, currently stuck at connecting a Q with an article, the API keeps returning {"error":{"code":"no-external-page","info":"The external client site did not provide page information (The external client site did not provide page information)","messages":[{"name":"wikibase-api-no-external-page","parameters":[],"html":{"*":"The external client site did not provide page information"}}]}} [19:34:45] what did I miss? [19:35:05] MaxSem: The page has to exist on the client wiki [19:35:13] well, it does:) [19:35:20] in the typical dev setup it will need to exist on the real wikimedia wiki [19:35:40] eh, I'd rather kep it local [19:35:41] Eg. you can connect enwiki:Berlin but not enwiki:fsdlkfhadskfjha [19:35:44] Oh [19:35:55] because I'm integrating with WikibaseClient [19:36:04] MaxSem: You can still do that [19:36:13] I'm doing all my client testing in such a setup [19:36:46] I once had to hack it for some stuff, but it shouldn't affect anything but repo [19:36:57] so client doesn't care what's in the sites table [19:37:00] or have to add an entry to the sites table for your wiki [19:37:05] (concerning API urls) [19:37:07] * aude always calls my wiki enwiki [19:37:10] so... I did sites setup, it recognises the enwiki prefix as pointing [19:37:22] ..to somwhere... [19:37:30] MaxSem: Wikipedia [19:37:31] apparently, to a wrong place:P [19:37:37] that setup script is Wikimedia-specifc [19:37:51] but people oppose moving it out of Wikibase into WikimediaMaintainence [19:38:01] 1 | enwiki | mediawiki | wikipedia | local | en | http | gro.aidepikiw.ne. | | 0 [19:40:52] aha [MediaWikiSite] call to external site failed: ?action=query&prop=info&redirects=1&converttitles=1&format=json&titles=Test+island [19:41:22] m( [19:41:31] How can that happen? [19:42:02] where does it get the server url from? [19:42:41] MaxSem: Which script did you use? The one in Wikibase? [19:42:45] I think it uses sitematrix [19:42:50] possible that it's broken [19:42:53] nah, filled sites manually [19:43:02] it has tests, but no one regularly uses it, I think [19:43:19] we do [19:43:22] not regularly [19:43:28] but used it for fa.wikivoyage [19:43:39] I tried to go with sitematrix, but its just too unsuitable for a setup with weird wiki names [19:44:12] Something different, but can someone estimate a time of arrival for a new dump with language links? http://dumps.wikimedia.org/wikidatawiki/ [19:44:29] FakirNL: Can't you use the json dumps? [19:45:09] well, I don't know, it's Ivan Krestinin who uses the dumps (for his merge project), but I usually work based on Krestinins bot run [19:50:45] aha, site_data [19:51:06] eww, serialize() [21:32:45] Maarten Dammers [21:49:26] JeroenDeDauw: Thanks for the update :) [21:49:38] This is more in line with what we saw in production [21:49:45] and makes this less weird [23:05:08] aude: just reading your updates on bug 71519 [23:07:37] TimStarling: still not sure the underlying reasons, why the references are getting 'corrupt' [23:08:04] but you have narrowed it down to SplObjectStorage::getHash()? [23:08:20] is $reference an SplObjectStorage? [23:08:49] yes or what we do to extend it [23:09:03] although our code there has not changed in quite some time [23:10:23] https://github.com/wmde/WikibaseDataModel/blob/master/src/Reference.php#L77-L86 [23:10:26] https://github.com/wmde/WikibaseDataModel/blame/master/src/Snak/SnakList.php#L112-L114 [23:12:32] and https://github.com/wmde/WikibaseDataModel/blame/master/src/Internal/MapValueHasher.php#L35-L54 [23:14:53] https://github.com/wmde/WikibaseDataModel/blame/master/src/Internal/MapValueHasher.php#L35-L54 uses splobjectstorage [23:14:56] aaah [23:15:01] https://github.com/wmde/WikibaseDataModel/blob/master/src/ReferenceList.php [23:28:48] (03PS4) 10Aude: Get snaks in ReferenceSerializer from cloned $reference [extensions/Wikibase] - 10https://gerrit.wikimedia.org/r/165205 (https://bugzilla.wikimedia.org/71519) [23:39:28] awesome [23:40:30] so you've confirmed that that fixes the exception, but not necessarily the segfault? [23:40:54] it avoids the problem [23:41:38] it might appear somewhere else [23:43:57] do you think it can be deployed today? [23:44:35] sure [23:45:29] at minimum, we can put it in the deployment branch [23:45:46] and then decide what is best to do for master [23:45:58] sure [23:46:15] (03PS1) 10Aude: Get snaks in ReferenceSerializer from cloned $reference [extensions/Wikibase] (wmf/1.25wmf1) - 10https://gerrit.wikimedia.org/r/165397 (https://bugzilla.wikimedia.org/71519) [23:47:02] it will probably take 15 min to update our build, depending on how fast jenkins is today [23:59:08] (03CR) 10Aude: [C: 032] Get snaks in ReferenceSerializer from cloned $reference [extensions/Wikibase] (wmf/1.25wmf1) - 10https://gerrit.wikimedia.org/r/165397 (https://bugzilla.wikimedia.org/71519) (owner: 10Aude)