[09:11:04] 10Wikibugs: Separate wikibugs username, into grrrrit and wikibugs - https://phabricator.wikimedia.org/T153732#3070161 (10Nemo_bis) [22:12:09] been poking around but not finding the right bits...i use irssi through which i use alt [0-9qwertyuiop], and it seems xmonad has grabbed plenty of these for itself, how can i fix that without having to change my mod key to something other than alt? [22:12:38] doh wrong room .. [22:12:49] ebernhardson hi, it seems that searching with phabricator with some querys is slow, see https://phabricator.wikimedia.org/T159575 [22:12:50] please [22:13:44] paladox: how do i dump out the queries phabricator is issuing? [22:14:02] Not sure what you mean by dump out the queries? [22:14:08] i want to see what it sends to elasticsearch [22:14:09] We could implement logging if you want? [22:14:21] twentyafterfour ^^ [22:14:38] logging is a bit over the top, because then i have to go dig in some logs. for cirrus we use https://en.wikipedia.org/wiki/Special:Search?search=kennedy&fulltext=1&cirrusDumpQuery and it works great [22:15:06] as in, i just attach some query string param and it gives me the query instead of doing whatever it usually does [22:15:21] oh [22:15:37] Hmm [22:17:01] additionally, you should probably set some timeouts to not allow 1m45s queries to run. i imagine phabricator also doesn't have any kind of limits on concurrent queries it's issuing, but basically using that query you could probably DOS the prod search cluster by spamming it [22:17:37] Oh [22:17:57] twentyafterfour is currently rewriting the elasticsearch engine to allow it to use a cluster. [22:18:44] cluster is a start, the other big thing i would like to see there is some sort of semaphore that limits the number of concurrent queries that can be issued at a time. In testing i've been able to put an entire cluster out of comission by just spamming more and more queries without regards to concurrency limits [22:19:57] Ah [22:25:10] but anyways, for me to do anything about that long query i have to see what query its actually issuing [22:26:06] Yep, i doint have access to iridium. We have a test instance phabricator.phabricator.eqiad.wmflabs [22:30:15] ebernhardson: We could sniff some traffic ;-) [22:34:11] RainbowSprinkles: possible, but sounds annoying :P [22:34:43] Just dont get ran over [22:35:55] httpry doesn't show anything useful on :9200 :\ [22:35:57] That's no fun [22:37:33] Oh, cuz we use 9243 now [22:38:56] yea its all encrypted [22:39:39] Blah, so doubt we can sniff much :\ [22:39:56] RainbowSprinkles we could use phloh in phabricator code [22:40:16] well, its possible to tcp dump and then load the appropriate keys into wireshark with the dump, but its not pain free [22:40:19] I remeber twentyafterfour used it, and it was used in phab-01 now phabricator [22:42:27] I don't even know what that is [22:43:01] ebernhardson: Something similar to cirrusDumpQuery is probably pretty easy [22:43:03] Lemme have a look [22:45:19] yea i put on the ticket that a query string argument to just dump it would be nice [22:45:37] ah [22:45:38] phlog($result); [22:45:45] under function [22:45:45] protected function resolveTags($tags) { [22:45:49] https://phabricator.wikimedia.org/rPHAB37e755fe9180c6c680e5e730c69e62966d40d065 [22:47:39] debugging in PHP is so easy [22:47:42] if ( isset( $_REQUEST['phabDumpQuery'] ) ) { [22:47:42] echo var_export($data, true); [22:47:43] die( 1 ); [22:47:43] } [22:47:49] oh [22:47:52] :) [22:48:46] RainbowSprinkles: i prefer json_encode($data, JSON_PRETTY_PRINT) so i can copy/paste into a shell to run and tweak :P [22:48:58] It's already json encoded [22:49:01] Lemme move it a tad [22:51:27] Well that didn't work :( [22:51:29] Damn [22:51:35] Well, it's not giving me my datas [22:56:29] :( [23:03:22] RainbowSprinkles you can test internal php changes on phabricator (formaly phab-01) [23:03:37] as it also uses elasticsearch 5.x [23:05:16] I was testing on prod ;-) [23:05:37] Oh :) [23:07:53] http://s2.quickmeme.com/img/d2/d22f09bcf5a4d922838ebc6a1cd512490a6292d73ac694f22abc6add22826082.jpg [23:08:12] Oh yeah, php opcode caching [23:08:19] We have to restart apache for shit to pick up [23:08:23] * RainbowSprinkles sighs [23:08:32] RainbowSprinkles: can't you just trigger a script that clears the opcode cache? [23:08:32] lol [23:09:15] http://php.net/manual/en/function.opcache-invalidate.php although i suppose you have to get it into a script thats not already cached :P [23:09:35] I can write a script that sends SIGUP to apache ;-) [23:13:10] Bam, now it works [23:13:19] Append ?phabDumpQuery=1 to a search [23:13:34] Here's the one we're trying to debug: [23:13:36] { "_source": false, "query": { "bool": { "must": [ { "simple_query_string": { "query": "beta feature mediawiki.org", "fields": [ "title^4", "body^3", "cmnt^2", "tags", "_all" ], "default_operator": "and" } } ], "should": [ { "simple_query_string": { "query": "beta feature mediawiki.org", "fields": [ "titl", "body", "cmnt" ], "analyzer": "english_exact", [23:13:36] "default_operator": "and" } } ] } }, "from": 0, "size": 10000 } [23:13:51] { "_source": false, "query": { "bool": { "must": [ { "simple_query_string": { "query": "test", "fields": [ "title^4", "body^3", "cmnt^2", "tags", "_all" ], "default_operator": "and" } } ], "should": [ { "simple_query_string": { "query": "test", "fields": [ "titl", "body", "cmnt" ], "analyzer": "english_exact", "default_operator": "and" } } ] } }, "from": 0, "size": 101 } [23:14:21] { "_source": false, "query": { "bool": { "must": [ { "simple_query_string": { "query": "beta feature mediawiki.org", "fields": [ "title^4", "body^3", "cmnt^2", "tags", "_all" ], "default_operator": "and" } } ], "should": [ { "simple_query_string": { "query": "beta feature mediawiki.org", "fields": [ "titl", "body", "cmnt" ], "analyzer": "english_exact", "default_operator": "and" } } ] } }, "from": 0, "size": 10000 } [23:14:46] ebernhardson ^^ [23:14:54] that query is the one that takes along time [23:14:58] https://phabricator.wikimedia.org/maniphest/query/0AxVqqrEaGe_/?phabDumpQuery=1 [23:15:04] that's asking for 10k results ? [23:15:37] Really. lol no wonder it's slow [23:15:50] Pasted to phab [23:15:54] (pretty printed) [23:16:01] My chat logs i think reached 1 gb after that query paladox [23:16:03] https://phabricator.wikimedia.org/T159575#3072374 [23:16:19] thanks [23:16:21] Zppix lol [23:16:29] ebernhardson: If I had to guess....pull 10k results and do pagination on the application end? [23:16:37] That seems....easy to make Elastic sad [23:16:54] RainbowSprinkles: then your a crazy person who needs an asylum [23:17:18] Jury's still out on my sanity :p [23:18:22] lol, anyways, we should probaly do pagination on elasticsearch side? Or get the application side to send less out to elasticsearch, ie do it in batches. [23:18:49] Phab should only request one page of results at a time. [23:18:55] And use offsets [23:19:14] so, asking for 20 results takes ~60ms (including the time for curl to spin up a shell process and such) [23:19:40] So clearly it's the # of results thats a problem. Ok, so this is actionable now [23:20:02] hmm, even asking for 10k still shouldn't take that long curling that gives me results in 100ms, and says there are 125 total results [23:20:31] I'm wondering if PHP is crapping out with a 10k result set json payload [23:20:52] Can json even cope with that much well json [23:21:22] ebernhardson: Either way, the problem is clear. We *shouldn't* be requesting 10k results to show a few dozen [23:21:24] :) [23:21:33] hmm, yea i would suspect something in phabricator itself, because running the query as-is even with the 10k result request runs in elasticsearch reasonably quick [23:21:36] offsets exist for a reason :) [23:22:19] if you are really paranoid about how expensive the query is, there are even scrolls :P [23:22:33] Well clearly the query itself is fine [23:22:38] yup [23:22:39] It's just the result set size that's dumb [23:22:49] Ok, so we know what's up now. I'll remove my live hack and let's repurpose the bug a tad into fixing this [23:25:30] I say query limit set to ~100 [23:25:56] it should set a limit per page. [23:26:05] ^ [23:28:07] Summarized all this in a comment on the task [23:28:19] thanks [23:29:05] RainbowSprinkles your link breaks for me [23:29:06] https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-from-size.html%20from/size%20together%20to%20do%20this%20pagination [23:29:12] shows a 404 [23:29:31] Yeah I already fixed that [23:29:36] Missed a | [23:29:45] Oh [23:30:19] Whoops, fixed again [23:30:22] Typing is hard [23:30:25] lol [23:30:44] RainbowSprinkles maybe having something automated do it like siri? [23:30:45] lol [23:31:11] lolno [23:31:29] ebernhardson: Thanks for your help, I think you can safely ignore this now since it's clearly not Elastic's fault :) [23:31:58] :) [23:32:07] Well not entirely RainbowSprinkles [23:32:13] i also added a note the query building seems incomplete, the request contains a filter on items tagged Flow but there is no filter in the search [23:33:42] Responded. [23:33:48] Ugly answer :( [23:34:19] tldr: All the other filter-by fields don't actually filter in Elastic. All elastic does is do the fulltext search, which is later filtered down by various DB stuff in Phab [23:34:26] eww, doesn't seem that hard to include those relationships :P [23:34:30] Which....probably explains the "fetch 10k results cuz we'll toss most of them"