[04:44:45] inflatador: decoms complete, last remaining thing is patch to remove the site.pp stanza and the references to the hosts in the `[elastic,open]search` config files [07:36:40] good morning, the upgrade of cumin2002 to bookwork is about to start and we noticed that there is an sre.wdqs.data-reload --task-id T386098 for wdqs1022 in progress that was started shortly after the announcment of the upgrade window. brouberol, btullis it would be killed by the upgrade process, if there is a better way to stop it in a given position to make it easier to restart it now it would [07:36:41] T386098: Run a full data-reload on wdqs-main, wdqs-scholarly and wdqs to capture new blank node labels - https://phabricator.wikimedia.org/T386098 [07:36:46] be a good time to do so :) [07:37:59] volans: I don't know much/anything about this cookbook. inflatador and ryankemper were managing the process [07:38:32] I'm still around. Looking... [07:38:55] oh wow, why are you still around? (you shouldn't) but thanks a lot! [07:40:20] weird schedule today, basically ended up working two half days morning and night [07:40:27] I actually think the new hdfs-based reload process is supposed to be able to resume [07:40:38] but in any case we only lose like 2 days of progress at worst anyway [07:40:46] pausing the cookbook, one sec [07:40:51] great ,thx [07:42:33] ryankemper: thanks [07:42:34] volans: done. should i nuke my whole tmux or is it okay if i've still got sessions open (not performing any work) just to keep the scrollback buffer [07:42:59] ryankemper: no problem but they will be gone [07:43:06] most likely [07:43:09] not 100% sure [07:43:44] okay I'll give it a viking funeral [07:43:57] does it survive a reboot? [07:44:02] :D [07:44:39] I just kill-server'd it so the question will have to remain unanswered :P [08:07:16] Is it possible to search for pages / files in a category in commons using an advanced search field? [08:07:31] and if so, does that happen throughout the whole cat tree, or just at a top level? [12:30:40] addshore: Deepcat might be your friend: https://www.mediawiki.org/wiki/Help:CirrusSearch#Deepcategory [12:31:40] It searches up to some depth in the category tree. The default depth seems to be 5, [12:43:22] absolutly perfect! [12:43:52] "The depth of the tree is limited by 5 levels currently (configurable) and the number of categories is limited by 256 (configurable)." [12:44:02] configurable by user? or is that a backend thing? [13:09:01] \o [13:35:01] addshore: sadly, configurable by the administrator, its a wiki level setting [13:35:06] .o/ [13:35:12] ebernhardson: ty! [13:35:25] we've pondered what it should look like, but need a coherant answer to what parameterized keywords look like. maybe named parameters? unknown yet [14:10:57] Good morning team! I was having some discussion with TJ in email about one CirrusSearch issue I am currently having with my site: vimwiki.org, the issue basically is that the exact phase search seems stops working randomly, say I have a phrase in Chinese "无生法忍"，when I try to do an exact phrase match search for "无生法忍" (with quotes) [14:10:57] for [the first time of the day], it works fine, returned 126 pages; then if I remove the quotes and search for 无生法忍 (without quotes), it returns 1053 results, which also makes sense because Cirrus search tokenize it into multiple words; however, now if I search for "无生法忍" again (with quotes) for exact phrase match, it still return [14:10:58] 1053 results [14:10:58] instead of 126... and on the [following day], this seems happens again, worked once, then stopped working since after.. [14:11:46] thats curious, cirrus does have a query cache but it's only enabled for certain queries and phrase matching wont trigger it [14:11:56] is this a public wiki? [14:12:49] it is half public, I have some namespaces locked down only to logged in users, but I have Cirrus configured to only index some of the public namespaces. [14:13:14] do you have a url i can poke at? [14:14:45] Sure, try this one, it is the exact phrase search for “无生法忍”(with quote): https://vimwiki.org/w/index.php?search=%E2%80%9C%E6%97%A0%E7%94%9F%E6%B3%95%E5%BF%8D%E2%80%9D&title=Special%3A%E6%90%9C%E7%B4%A2&profile=advanced&fulltext=1&ns0=1&ns1=1&ns14=1&ns4100=1&ns4200=1 [14:15:01] it currently shows 1,053 results, but actual exact matched pages should be 120 or so. [14:20:20] Paulxu20: i noticed you have something odd going on with namespaces there, but i can't quite pinpoint what [14:21:42] for example your link has namespace 4100 and 4200 enabled in search, but then pressing enter in the loaded page performs a second search without those namespaces [14:21:53] which changes the result count [14:24:24] my suspicion would be these new namespaces were perhaps not fully added? Sadly I'm not entirely sure on the procedure for adding namespaces [14:26:00] but at least my take on whats happening is that the result count changes because the namespace selection changes. As for why the namespace selection changes...my guess would be to go back through the docs on adding namespaces. Certainly we have custom namespaces and they work in other circumstances [14:26:45] Can you elaborate more about this "pressing enter in the loaded page performs a second search without those namespaces"? did you mean press enter in the address bar in browser? [14:27:11] Paulxu20: in the search box, what i mean is that re-submitting the search on the page does not search the same set of namespaces [14:27:39] so if i follow the link you gave, push enter in the search box, I end up at https://vimwiki.org/w/index.php?search=%E2%80%9C%E6%97%A0%E7%94%9F%E6%B3%95%E5%BF%8D%E2%80%9D&title=Special%3A%E6%90%9C%E7%B4%A2&profile=advanced&fulltext=1&ns0=1&ns1=1&ns14=1 [14:27:53] which critically no longer has &ns4100=1&ns4200=1 at the end which selects the two namespaces [14:28:29] 4100 is probably 印顺导师, and 4200 妙境长老 [14:29:54] Ok I removed those additional namespaces in the URL, let's try only search in main namespace: https://vimwiki.org/w/index.php?search=%E2%80%9C%E6%97%A0%E7%94%9F%E6%B3%95%E5%BF%8D%E2%80%9D&title=Special%3A%E6%90%9C%E7%B4%A2&profile=advanced&fulltext=1&ns0=1 [14:30:52] if you try above, it returns 500 results, irrespective of whether "无生法忍" is quoted or not [14:33:17] Paulxu20: hmm, thats going to be a language analysis problem. When i run "无生法忍" (with quotes) through the analysis chain for zh it returns 4 separate ideographic characters [14:33:23] Trey314159: expected? ^ [14:34:25] ebernhardson: in a meeting at the moment [14:35:10] you can similarly see (although less specifically) how the terms are broken up here: https://vimwiki.org/w/index.php?search=%E2%80%9C%E6%97%A0%E7%94%9F%E6%B3%95%E5%BF%8D%E2%80%9D&title=Special%3A%E6%90%9C%E7%B4%A2&profile=advanced&fulltext=1&ns0=1&cirrusDumpResult&cirrusExplain=pretty [14:36:23] everything there seems to have a score for all.plain:"无 生 法 忍"~1 which would imply the phrase matching is finding the tokens close enough to each other [14:38:27] Sorry I am not familiar with how search engine works behind the scene :( My limited understanding is that the language analyzer should NOT tokenize it 无生法忍 if it is quoted? [14:40:48] well, in theory a tokenizer will break the text into semantic units, typically words but we try to call them tokens instead of words, because word is a bit too specific. Unfortunately I don't know enough about chinese to know what's supposed to happen here, but i suspect that is not 4 distinct semantic units. [14:41:08] But chinese tokenization is hard, it's all done with a bunch of statistics and something called a "hidden markov model", [14:43:02] there is also a translation layer that, iirc, converts traditional into simplified, but i think that happens pre-tokenization [14:45:03] that is beyond my knowledge :) but the issue here seems not about analyzer? because it works sometime, I have been testing it while we talk, I think it works 80% of the time, but fails 20% of the time.. I tried the same search on https://zh.wikipedia.org/, it works pretty well, did not fail. [14:46:14] Paulxu20: hmm, you can reproduce the changing result count with the second query? For the one using only main namespace i get the same 1-20 of 500 each time [14:47:09] yes with this search I get 71 results now: https://vimwiki.org/w/index.php?search=%22%E6%97%A0%E7%94%9F%E6%B3%95%E5%BF%8D%22&title=Special%3A%E6%90%9C%E7%B4%A2&profile=advanced&fulltext=1&ns0=1 [14:47:24] which should be the correct number [14:48:08] if I remove the quote in the search box and click on search button again, I get 500 results [14:49:38] it feels like it does have something to do with cache? [14:49:51] there is not cache [14:49:55] s/not/no/ [14:50:14] the only cache that exists is enabled for the Special:Statistics query, and the morelikethis: keyword, otherwise cirrus has no cache [14:50:24] ok [14:51:51] one difficulty that i'm not sure what to do with...queries that look the same submit a different thing [14:52:21] So like, https://vimwiki.org/w/index.php?search=%E2%80%9C%E6%97%A0%E7%94%9F%E6%B3%95%E5%BF%8D%E2%80%9D&title=Special%3A%E6%90%9C%E7%B4%A2&profile=advanced&fulltext=1&ns0=1&uselang=en and https://vimwiki.org/w/index.php?search=%22%E6%97%A0%E7%94%9F%E6%B3%95%E5%BF%8D%22&title=Special%3A%E6%90%9C%E7%B4%A2&profile=advanced&fulltext=1&ns0=1&uselang=en look the same to me, but the urlencoded [14:52:24] query is different [14:53:36] Paulxu20: oh, are you sending the wrong quotation characters? Your example query, https://vimwiki.org/w/index.php?search=%22%E6%97%A0%E7%94%9F%E6%B3%95%E5%BF%8D%22&title=Special%3A%E6%90%9C%E7%B4%A2&profile=advanced&fulltext=1&ns0=1, has curly quotations and not ascii [14:54:04] hmm, maybe? [14:54:37] the example query https://vimwiki.org/w/index.php?search=%E2%80%9C%E6%97%A0%E7%94%9F%E6%B3%95%E5%BF%8D%E2%80%9D&title=Special%3A%E6%90%9C%E7%B4%A2&profile=advanced&fulltext=1&ns0=1 clearly has the curly quotes [14:54:52] “ vs " [14:56:50] ebernhardson I think you are right!! shame on me, lol! I recently changed the font a little bit, the “ vs " looks very similar on my browser! [14:57:05] That explains everything, lol! [14:57:13] Paulxu20: glad we could figure it out :) [14:57:22] Thank you so much for the help! [14:57:30] they are indeed very hard to tell the difference, i didn't notice until i started urlencod'ing th equery [14:59:06] Indeed! [15:38:51] Picking up my son, back in ~30 [16:11:29] back [19:00:08] Is this still the best place to find Search Platform OKRs? https://w.wiki/EWg7 I'm working on my ITC [19:00:19] or I will, once I get back from lunch...see ya in ~40 [19:00:42] inflatador: I've not been good at keeping things up to date :( [19:00:53] search goals are a mess [19:01:16] SRE are a bit better: https://docs.google.com/spreadsheets/d/14RI5QScIY2fbZ01XZGXHIsSCbB2n-VjWkPvfKG41krQ/edit?gid=0#gid=0 [19:01:16] gehel it has "opensearch migration" on it, that's good enough for me ;) [21:04:49] traffic is moving back into eqiad now, flowing through the new dns-disc endpoints [21:06:54] hmm, one curious bit is now our metrics says "dnsdisc", because that's the cluster we are shipping requests through...might have to rework parts of the dashboard if we want per-cluster data [21:15:32] interesting [21:15:49] I feel like I missed the boat on that one ;) [21:16:21] it's the last step of all the stuff we did to setup search.discovery.wmnet [21:16:34] so in theory, you can move traffic with etcd/conftool now? [21:17:27] yeah, I guess so! [21:19:35] I'll get a task started for cleaning up the dashboards ;( [21:19:57] most of the dashboard is fine, it's just the numbers at the top. Maybe we can get those from envoy [21:20:37] I was just looking at https://grafana.wikimedia.org/goto/URArVnPHg?orgId=1 [21:21:12] Looks like it's showing 'dnsdisc' getting all the traffic now? [21:21:28] yea, the qps and percentiles at the top of that dashboard are wrong now, because they are looking at metrics that cirrus is recording [21:26:11] ah, thanks for that! And it looks like you addressed the mwconfig stuff as well [21:30:31] OK, I popped T397377 for this [21:30:32] T397377: Update Elasticsearch/OpenSearch dashboards and alerts post-Envoy enablement - https://phabricator.wikimedia.org/T397377 [21:33:14] been poking at metrics, but the envoy side might also not know :S [21:34:23] Is it something we could fix by updating our exporters? [21:36:13] there are maybe request rates we could get from elasticsearch, but they wouldn't have the p50/p95 latencies [21:37:22] We're still getting the latencies, right? Just not separated per DC? [21:38:01] right. The problem is that cirrus records the metrics, and cirrus doesn't know which cluster it's talking to anymore. It talks to wherever discovery does [21:38:37] envoy also records metrics, but it records them against the envoy cluster, and the discovery endpoint is an envoy cluster. Maybe they are there, but i haven't yet found metrics by destination [21:40:38] i wonder if thats going to trigger any of our alerts...looking [21:42:33] no they look like they should be fine, but it's now alerting on the aggregated latency for both clusters, since it all reports as `dnsdisc` now [21:48:00] This might be a dumb question, but can we do some kind of join between cirrus metrics and other mediawiki metrics to get what we want? [21:48:45] doubt it, there are no joins in prometheus :P The data we want is probably there somewhere, we can't be the only people that want to know latencies. Maybe it's supposed to come from envoy running on the server (but we terminate with nginx)? [21:50:40] how do we measure latency on WDQS? [21:52:43] hmm, i don't see latency percentiles on the main wdqs grafana dashboard. On some of the other dashboards i see latency as reported by varnish / ats [22:07:22] I'm looking at the mysql dashboard https://grafana.wikimedia.org/goto/mtE0D7PNg?orgId=1 , but I haven't found aggregated QPS yet [22:08:44] just spitballing