[10:57:38] Lydia_WMDE: todays issue with language links is another reason we really need to work on the Sites stuff. I have created a few tickets to track this mess. Have a look at https://phabricator.wikimedia.org/T137537 and its connections. [12:20:43] what database does wikidata run on? [13:23:22] hashar, jzerebecki: can we make the mwext-mw-selenium-composer job non-votig? It seems to be failing timeout often, if not always. [13:24:20] docnvk: we store json blobs in mysql. quite dumb, really. sparql queries are handled by a blazegraph server [13:25:02] some searched are handled by elastic, but we don't make very good use of it yet. aude and stas are working on that. [13:32:49] okay, thanks DanielK_WMDE__ [13:55:11] DanielK_WMDE__: we can, but I'd suggest we fix it instead... [14:38:58] DanielK_WMDE__: jzerebecki I made them non voting [14:39:20] via https://gerrit.wikimedia.org/r/#/c/293732/1 [14:39:27] and the tracking task is https://phabricator.wikimedia.org/T137561 [14:39:33] all I can do for now. At home with kids ...; [14:39:37] be back later [14:42:48] hasharAway: thx for the analysis [14:45:14] jzerebecki: we should fix them, but as long as they are broken, they shouldn't block ci, i think. [14:45:49] DanielK_WMDE__: sure they should block [14:47:15] DanielK_WMDE__: a change broke them, so the change needs to be fixed or reverted [14:48:08] jzerebecki: is that so? my impression was that it's related to limited resources on the test system. [14:48:21] DanielK_WMDE__: no that is not the case [14:48:36] * hoo cries… damn mysql [14:48:56] "Error: 1137 Can't reopen table" grrr [14:48:58] jzerebecki: do you know what change broke them, then? a change to the repo under test? or to the ci? [14:49:31] hoo: hey hoo! what do you think of https://phabricator.wikimedia.org/T137539 ? [14:49:41] DanielK_WMDE__: Implemented it already [14:49:51] hoo: \o/ [14:49:54] but the integration tests on Maria wont work [14:49:58] I'm using a union now [14:50:03] DanielK_WMDE__: h ashar said firefox upgrade, but i'm trying to find out if that is true because e.g. MobileFrontend selenium runs are passing [14:50:52] hoo: oh, fancy. that's a lot of SQL core, though. how big can the id batches get? [14:50:59] 100 should be ok, but 10000 wouldn't be... [14:51:08] *SQL code [14:51:30] DanielK_WMDE__: Up to 500 on a page deletion on Wikidata [14:51:43] Likely less, but certainly not more than that [14:51:47] will the union work with 500 elements? [14:51:51] oh wait, no… different languages fun [14:51:55] It should [14:52:02] "should" :P [14:52:04] but not testable [14:52:14] I can try it in production [14:52:30] did you try the union query against the production db? [14:52:32] how fast is it? [14:52:39] also, can you link me to the patch, please? [14:52:45] ah yes MF is using chrome [14:53:18] DanielK_WMDE__: Not yet in gerrit, I can upload it in a sec [14:53:23] the queries are blazingly fast [14:53:38] haven't been able to measure the time in production [14:53:44] (to low) [14:53:49] nice! [14:55:13] DanielK_WMDE__: https://gerrit.wikimedia.org/r/293736 [14:55:23] tests will fail though (as said) [14:56:03] hoo: does sqlite support unions? (does our wrapper for sqlite?) [14:56:28] DanielK_WMDE__: Yes [14:57:05] not sure about postgres, though… but we don't support that anyway [14:57:37] I'll do a test case with a large batch in production now [15:02:37] DanielK_WMDE__: Getting 106ms query time for Q1 to Q1000 on commonswiki (cold) [15:03:24] very nice! [15:03:25] https://phabricator.wikimedia.org/T137539#2371909 (with 1k entity ids) [15:04:08] good point about the __METHOD__ [15:04:40] grr https://integration.wikimedia.org/ci/job/mwext-Wikibase-repo-tests-sqlite-hhvm/10285/console [15:04:51] Looks like the SQLite wrapper gets it wrong :/ [15:05:25] hoo: looks like sqlite doesn't support LIMIT in sub-queries... [15:05:33] It does [15:05:39] not quite straight forward maybe [15:05:51] You just have to use sub-subqueries [15:05:53] Like [15:06:04] SELECT * FROM ( … LIMIT 1 ) UNION SELECT * FROM ( … LIMIT 1 ) [15:06:28] But our wrapper doesn't do that [15:06:30] *sigh* [15:06:49] let me see how much slower it is with 1000 round trips [15:06:52] it's silly that sqlite requires that [15:07:10] yea, we could fall back to round trips [15:07:16] actually, the db wrapper could also do that [15:07:55] hm [15:08:36] hoo: i think, for now, doing it the "stupid" way is easiest, and jcrespo seems tho think it's good enough [15:08:46] so let's just do that for now [15:09:01] the union thing is more elegant, but needs more work, it seems [15:09:11] so even if we do that, let's do the stupid solution irst [15:10:55] DanielK_WMDE__: Non-Union it takes 1.10s [15:11:01] :/ [15:12:04] maybe we don't need the limit on sqlite even [15:12:17] if we use UNION without all, it should get us distinct values anyway [15:12:27] so taht's probably good enoughf or anything htat's not MySQL [15:12:58] DanielK_WMDE__: ^ [15:16:08] hoo: uh, how would you do that? check what db engine we are running against, and construct specialized sql? [15:16:10] ugh [15:16:22] Yes [15:16:22] i'd rather do the same thing for all dbs [15:16:42] and live with the round trips for now. how much longer does that take anyway? [15:17:03] moin :) [15:17:04] takes 11 times longer [15:17:21] hey aude \o/ [15:17:44] Short benchmarks: hey aude \o/ [15:17:46] doh [15:17:50] * https://phabricator.wikimedia.org/T137539#2371909 [15:21:46] hoo: i suggest two patches: one with only the stupid code, and a follow-up with the "use a union if supported" logic. [15:21:58] DanielK_WMDE__: Ok [15:22:00] Can do thta [15:22:10] you can share code by building the subquery array always, and just runnign them directly if unions are not supported [15:22:23] hoo: i think it's useful to be able to discuss them separately [15:52:24] DanielK_WMDE__: Both changes uploaded [15:52:35] the first one looks a bit clunky, though :/ [16:10:26] aude: See your review queue… I'm calling it a week [16:10:46] Would be nice to get the database stuff merged before we branch on Tuesday [16:10:53] Will probably do some work tomorrow, but not sure