"Import and vectorize data" -- vector or hybrid index?

Barry Briggs 60

When using the "Import and Vectorize Data" option (say, on JSON documents), are the original text fields preserved in the index with vectors added (for example, "body:" and "bodyVector:") or is the document simply vectorized as a whole? If the latter, is it possible in some way to extract specific fields in Azure OpenAI for example for citations?

Laxman Reddy Revuri 1,130 Reputation points Microsoft Vendor

2024-12-23T00:46:24.9066667+00:00

Hi @Barry Briggs
Thanks for the question and using MS Q&A platform.
Hybrid Index: The explanation also stands as to how a hybrid index supports both original text fields e.g. body and vectorized versions e.g. bodyVector as storage constructs. This configuration makes it possible for the users to use raw text in keyword searches as well as vectorized fields in similarity searches.
https://learn.microsoft.com/en-us/azure/search/hybrid-search-overview
Vector-Only Index: The description of a vector-only index is also accurate. In this instance, however, all original text fields are lost as only the vector representation of the whole document is kept. This reduces the querying option to standard similarity searches based on the vectors but does not allow direct extraction of fields.
https://learn.microsoft.com/en-us/azure/search/vector-search-overview
Field Extraction for Citations: The suggested techniques for extracting certain fields likebody for the purpose of citations are reasonable. Indexing original fields for example separately, citing other forms of retrieval, such as a hybrid index, are also effective approaches to ensure particular text that needs to be searched and retrieved is available. https://learn.microsoft.com/en-us/azure/search/vector-search-how-to-configure-vectorizer
I hope this information is helpful.
Laxman Reddy Revuri 1,130 Reputation points Microsoft Vendor

2024-12-24T05:59:01.64+00:00

Hi @Barry Briggs
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Barry Briggs 60 Reputation points

2024-12-24T15:28:01.33+00:00

Not exactly. I'm curious what exactly happens when the wizard is selected. In my case I have a set of JSON documents with fields like title, author, body, etc. After the wizard indexing runs there is an index created -- is that a purely vector index or is it a hybrid index?

Empirically it seems it is a hybrid index -- when loaded into the Azure OpenAI chat playground the index shows as hybrid; and when used it seems that the text /keyword fields are preserved (which is good!).

Just looking for confirmation that my assumptions are correct. Thanks very much!

1 answer

Laxman Reddy Revuri 1,130 Reputation points Microsoft Vendor

2024-12-26T04:09:53.2833333+00:00

Hi @Barry Briggs
You are correct! Using the "Import and Vectorize Data" wizard in Azure Cognitive Search with a dataset like JSON documents containing fields such as title, author, and body results in the creation of a hybrid index.
Please accept as "Yes" if the answer provided is useful , so that you can help others in the community looking for remediation for similar issues.
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

"Import and vectorize data" -- vector or hybrid index?

1 answer

Your answer