Similarity search using FAISS Index on Ray

1

I used Ray’s BatchPredictor to extract image features and save them as parquet files based on the partitions.
Now I want to use FAISS for similarity search. What would be the best way to use FAISS while leveraging Ray?

2

That’s a good question. HuggingFace has a library for FAISS embeddings. I wonder if you can leverage Ray to use Ray Data and Ray Task to load parquet files and use their HF API for searching the index.

cc: @Yard1 Will that work?

3

No luck so far. Would appreciate any help with this.