[10:54:40] <pfischer>	 dcausse: Is T414066 a blocker or could we try to request it from DPE (see https://phabricator.wikimedia.org/T403298#11499023)
[10:54:40] <stashbot>	 T414066: Download enterprise structured content snapshots in hdfs - https://phabricator.wikimedia.org/T414066
[10:56:15] <dcausse>	 pfischer: T414066 is not a blocker, we'll be using our credentials for this, some code has already be written https://gitlab.wikimedia.org/repos/search-platform/discolytics/-/blob/main/discolytics/cli/import_enterprise_dumps.py
[10:56:24] <dcausse>	 we need to schedule it from airflow
[10:56:45] <dcausse>	 broader question is perhaps where to land these in hdfs in case others are willing to use them
[10:57:11] <dcausse>	 but my understanding is that folks are mainly interested in html dumps not the structured ones
[11:00:47] <pfischer>	 Okay, but we have a legit use case here too, so once DPE has a pipeline in place, it could/should download both. Since we do not have a timeline for our own parsing/tokenizing, we’ll rely on those Enterprise dumps for as long as necessary.
[11:02:02] <pfischer>	 We could hand over the import_enterprise_dumps so it no longer is our responsibility to maintain that pipeline.
[11:04:05] <dcausse>	 pfischer: definetly
[11:04:33] <dcausse>	 *definitely
[11:40:52] <dcausse>	 lunch
[15:00:45] <ebernhardson>	 \o
[15:02:40] <inflatador>	 .o/
[15:08:21] <dcausse>	 o/
[15:20:51] <ebernhardson>	 sigh..mjolnir feature collection has been failing for awhile
[15:22:39] <ebernhardson>	 Exception: Did not collect equal number of rows per feature
[15:22:53] <ebernhardson>	 not sure how important it really is to get that going right now thoug
[15:23:50] <dcausse>	 yes... I've been ignore glent failures as well... :(
[15:23:59] <dcausse>	 *ignoring
[15:27:32] <dcausse>	 we can file a task and ponder that for later... I filed T412673 for glent
[15:27:33] <stashbot>	 T412673: Glent generate_query_similarity_candidates fails with NPE - https://phabricator.wikimedia.org/T412673
[15:27:44] <ebernhardson>	 yea i suppose thats the right course
[15:28:52] <dcausse>	 moving T414066 & T414070 to the workboard
[15:28:53] <stashbot>	 T414066: Download enterprise structured content snapshots in hdfs - https://phabricator.wikimedia.org/T414066
[15:28:53] <stashbot>	 T414070: Chunk, trim and generate passage embeddings from enterprise structured content snapshots - https://phabricator.wikimedia.org/T414070
[15:32:09] <dcausse>	 filed T414095 & T414091 as well
[15:32:09] <stashbot>	 T414095: Configure opensearch ML connectors/models - https://phabricator.wikimedia.org/T414095
[15:32:10] <stashbot>	 T414091: Import passage vectors into opensearch - https://phabricator.wikimedia.org/T414091
[17:17:36] <inflatador>	 lunch, back in ~60m
[18:15:05] * ebernhardson is writing docs for all our debug apis...it's way longer than i was expecting :P
[18:31:35] <inflatador>	 back
[18:34:15] <dcausse>	 I can imagine... thanks for taking care of this!
[18:34:51] <ebernhardson>	 i'm cheating, claude wrote the first draft. But it's still going to take hours to get it into a good place
[18:35:49] <ebernhardson>	 maybe hours is underselling :P
[18:41:03] <dcausse>	 :)
[18:48:25] <dcausse>	 dinner
[20:01:57] <inflatador>	 appointment, back in ~90
[21:04:02] <ebernhardson>	 hmm, after writing "All these debug features are publicly accessible. Wiki is a public resources, and these APIs don't expose any private data" i wonder...maybe we should have some config flag that turns them off? Our use case is that everything is public, but that might not be true of external cirrus users.
[21:04:22] <ebernhardson>	 but i dunno, it's been fine. It's just becoming better documented now
[22:05:09] <ebernhardson>	 getting close: https://phabricator.wikimedia.org/P86859