[10:30:12] 17m items soon <3 [10:37:22] Lydia_WMDE: What's the status of https://phabricator.wikimedia.org/T128954? [11:36:02] Is it just me or does the query service no longer take query from the URL? [11:36:47] Ctrl + R fixed it [12:34:48] how can i use the api (e.g. sparql) to get the german label for an entity? [12:38:33] f.e. a list of german labels of professions [12:39:18] i know i can use .json f.e. https://www.wikidata.org/wiki/Special:EntityData/Q901.json but maybe there is a way to get a complete list in one request ? [12:40:33] are you aware of https://query.wikidata.org/? [12:41:07] babadoo : SERVICE wikibase:label {bd:serviceParam wikibase:language "de" . } [12:42:52] Alphos: thank you [12:43:31] https://query.wikidata.org/#select%20%3Fitem%20%3Fdelabel%20where%20%7B%0A%3Fitem%20wdt%3AP31%20wd%3AQ28640%20.%0A%3Fitem%20rdfs%3Alabel%20%3Fdelabel%20filter%20(lang(%3Fdelabel)%20%3D%20%22de%22)%20.%0A%7D is a pretty simple example [12:44:10] (e.g. it wouldn't find anything that's an instance of a subclass of profession) [12:45:52] nikki: great. this is very useful. [13:13:08] does anyone remember any tickets or discussions about how entering the subdomain/language code/whatever you want to call it when adding a sitelink doesn't always select the one you mean? [13:13:45] like typing "es" will select esperanto (eo) because that's earlier in the list, not spanish (es) [13:14:07] i remember [13:14:09] I could have sworn there were on-wiki complaints at least, but I can't find anything :/ [13:14:13] I get the same with "nl" [13:17:00] what does it select for nl for you? [13:20:55] "cr" if you type too fast [13:23:02] oh, because it matches on just the n [13:23:20] that's not quite the same thing I was thinking of [13:23:28] (although presumably still annoying) [16:02:23] DanielK_WMDE_: around? [16:08:02] hoo: hey [16:08:27] trying to prep the RFC session for tonight. DI for core and all that fun [16:08:50] So I'm looking at making sure WikiPageEntityMetaDataLookup doesn't do master queries unless we really need to [16:08:59] I look at all (indirect) callers [16:09:01] at least I hope [16:09:15] and ran into a few nasty corner cases [16:09:40] yay, nasty corner cases [16:09:43] Like things we do very fast after an edit (change dispatching/ handling) [16:10:03] change dispatchign is usually delyed [16:10:18] Is it really? Can we trust that? [16:10:30] I remember the problems we ahd with UpdateRepo before we delayed jobs there [16:10:32] No. For some edits, it happens really quickly after [16:10:39] for most edits, it happens a while later [16:10:52] yeah, I also presumed that [16:11:45] do the master lookups get triggered on the client wikis? [16:11:47] or in the repo? [16:12:00] on the client, it should always be delayed by quite a bit [16:12:10] DanielK_WMDE_: both [16:12:25] do we have stats on how often this happens? [16:12:34] Kind of, yes [16:12:39] and do we have stats on how often the master lookup actually succeeds? [16:12:40] https://phabricator.wikimedia.org/P2202 [16:12:44] no [16:12:52] https://gerrit.wikimedia.org/r/#/c/279135/ [16:12:58] I marked all possible corner cases there [16:13:08] * aude waves :) [16:13:12] that would be good to know. if they don't succeed, the item probably got deleted (or something went very wrong) [16:13:20] and tried to hack it in the coalescer, but ignore that [16:13:21] maybe we can optimize for that case, if it is common [16:13:26] aude: hey :) [16:13:26] dispatching and change handling can happen very fast after an edit on repo [16:13:28] aude: hey :) [16:13:44] aude: Adding you to the draft as well, one second [16:13:47] ok [16:13:51] aude: it *can* happen very fast, but I'd be surprised if that was the case often [16:14:30] DanielK_WMDE_: Fallbacking to master in these cases wont hurt [16:14:35] i found this the case when investigating pages that mysteriosuly had no language liknks [16:14:50] hoo: it won't? why? [16:15:09] hoo: in the log, i only see one line related to WikiPageEntityMetaDataLookup [16:15:12] DanielK_WMDE_: Because it's not affecting the performance of the web request [16:15:21] also we only do it in very few cases [16:15:24] hm? how is that? [16:15:24] yeah [16:15:39] we still have to wait for a response from the other DC, no? [16:15:40] I'm talking about ChangeRunCoalescer only right now [16:15:45] even if it's empty [16:15:59] If the job runs in a DB without master, yes [16:16:03] * DC [16:16:49] ///XXX: we could avoid loading the entity data by merging the diffs programatically [16:16:50] /// instead of re-calculating. [16:16:55] hoo: --^ [16:17:10] that would be the Real Solution for ChangeRunCoalescer [16:17:17] true [16:17:19] I could do that [16:17:26] shouldn't be very hard [16:17:37] and would save us a lot of pain [16:17:51] I want to get taht over with fast, becuase Aaron really wants this now [16:18:31] If there are no other really problematic cases, that would work [16:18:31] look at the places I commented on in the draf [16:18:31] t [16:21:25] I doubt we can find a way to avoid having fallbacks in there for the places where we need to load badges [16:21:46] not sure about OutputPageBeforeHTMLHookHandler… but I guess it's triggered immediately post edit [16:23:00] hoo: don't we aggressively cache badges? We should... [16:23:08] i think it was lua where this was an issue, afaik [16:23:29] DanielK_WMDE_: As far as I know we don't at all [16:23:30] if an item was deleted or didn't exist, then we check master [16:23:36] aude: Yes, Lua was the primary issue [16:23:37] and that's notnice [16:24:02] but the underlying problem is that WikiPageEntityMetaDataLookup has these fallbacks in place [16:24:14] which make it unpredictable [16:24:57] I guess Lua is just the place where this surfaces the most as we have more failing getEntity calls there [16:26:38] DanielK_WMDE_: We don't cache badges at all (except with other content in the ParserCache) [16:26:45] but there's explicit badge cache [16:26:49] or badge lookup fwiw [16:26:52] hoo: weeeelll.... we should, then :D [16:26:59] they hardly ever change [16:27:05] * no explicit badge cache [16:27:17] that's try [16:27:17] * true [16:27:37] we could have a badge lookup... or a badgedsitelinklookup [16:27:48] hoo: dropping the automatic fallback to master makes me very very uncomfortable. It basically removes a consistency guarantee from the service. Not good. [16:28:13] I don't think it really does [16:28:14] We could however be more agressive about logging such fallback, and we should try to avoid loading entities as much as we can. [16:28:35] if you want the latest id, we couldn't guarantee consistency before [16:28:45] because if there's something on the slave, you'll get it [16:29:04] if you explicitly want a certain revision it's different [16:29:17] true [16:29:22] I would be ok with even letting the default fallback in tehre [16:29:44] not sure thaht's nice with the APIs/ SpecialEntityData, but I doubt it's a huge issue [16:35:12] DanielK_WMDE_: To get to something actionable: 1) Make ChangeRunCoalescer no longer load entities 2) Create a BadgeLookup or BadgedSiteLinkLookup 3) (Solution for OutputPageBeforeHTMLHookHandler) 4) disable automatic fallbacks [16:35:13] aude: ^ [16:35:21] does that sound like a reasonable plan? [16:35:35] SiteLinkWithBadgesAndFriesAndCokeLookup :P [16:36:01] hoo: sounds good, but i'm unsure about the implications. will have to check when my head feels less fuzzy. [16:36:23] I think I checked all callers of the methods that would change semantics [16:36:33] but I might have missed some or even one [16:37:10] * missed one or even more [16:51:59] I wonder how much caching badges per entity actually gains us [16:52:11] full lookups will still require to read it from the json [16:52:42] full => non-cached [16:59:37] Maybe we should actually create the tracking table for these [16:59:53] I already see this "tiny" change in semantics is getting out of hands [17:16:24] hoo: i was thinking af caching info about badges, not which entity has which badges. [17:17:02] Though we could dump that into pageprops: 'item-badges' => [ "wikiid" => [ "Q123", "Q456" ] ]; [17:17:03] What do you mean by infos about badges? [17:17:09] Badges are just a list of entity ids [17:17:12] local labels, mostly [17:17:15] no further metadata involved... is there? [17:17:31] i don't think so, but i'd have to check [17:27:45] DanielK_WMDE_: Adopting the change coalescer with https://phabricator.wikimedia.org/T113468 would be ok, I guess [17:27:45] but without it, I'm not sure [17:27:45] also wouldn't bother doing that with the old format [17:28:08] also we would need to iterate over all changes and do diff step by step [17:28:08] * a diff [17:28:08] but that sounds ok-ish [17:29:05] hoo: sounds like a plan [17:29:48] would still not be totally equivalent to what we do now [17:29:54] but maybe good enough [17:30:08] eg. except for in sitelinks, you can catch reverts [17:30:16] * can't [17:34:22] hoo: oh, right - merging requires full diffs. the compact representation would need to include enough info to allow merging - which basically means it would be the same as the full diff... [17:34:26] needs more thought [17:34:42] Indeed :/ [17:38:37] Shall we move on with explicitly allowing fallbacks in some cases? [17:44:45] hoo: there should be a way to force a lookup on master. not sure we need a mode for "fall back to master if the page is missing entirely on the clinet". [17:45:12] DanielK_WMDE_: We already have a mode to force master and use it (almost) everywhere we do writes [17:45:22] hoo: actually, i think we added that mode because without it, change handling sometimes failed [17:45:27] except for the one place I corrected in repo in my patch [17:45:38] Not sure [17:45:51] we might also just added it because the same mechanism existed in core at that time [17:45:52] but was removed later on [17:46:00] having three modes is definitly an option. but how do we get around using the master fallback during change handling? [17:46:10] We don't want to just skip the update... [17:46:23] Change handling is a nice case [17:46:31] because we even know the revision ids [17:46:38] and they at least briefly existed [17:46:46] actually, we might skip coalescing in such a case [17:46:51] so very unlikely to get a miss on both the slave and the master [18:33:43] Easy chocolate https://gerrit.wikimedia.org/r/279159 [20:06:26] Hm, add statement is gone? [20:06:44] ugh [20:06:47] all js is broken [20:07:14] * hoo claps [20:09:04] oh wait [20:09:21] labs is down so the labels project script is breaking everything again [20:09:22] What JS error do you get? [20:09:26] Can't reproduce [20:09:39] oh, ok [20:09:46] that sounds like a bug in it [20:10:04] they ask people to include mw.loader.load( '//labels.wmflabs.org/gadget/loader.js' ); in their global js [20:11:06] :/