adding fields to an existing index

Question

Hello, I hope you're having a great day.
I'm posting this inquiry as I have some questions.
Have a wonderful day!

[Project Service Requirements]

A single index should store data from multiple users, but search queries must only return results within the data uploaded by each respective user.

Different users should be able to use different embedding models, meaning the index must accommodate varying vector field dimensions.

[Usage Scenario]

Create an index with the following fields: id, title, content, partition, vector_768.

Store User A's data, ensuring appropriate values are stored in each field. The vector_768 field will contain vectorized data with a dimension of 768.

Add a new field (vector_3076) to the existing index.

Store User B's data, ensuring that the id, title, content, and partition fields are populated appropriately. The vector_3076 field will contain vectorized data with a dimension of 3076.

[Questions]

Is it possible to add fields to an existing index to store vectorized data generated by different embedding models?

In the given usage scenario, will values be correctly stored and persist across all fields?

When adding fields to an existing index (including previously added fields), should SearchIndexClient.create_or_update_index() be used?

(Additionally, please confirm whether the index does not undergo a rebuild when adding fields or storing data.)

Answer 1

Hi @이소영
1.No, Azure AI Search does not permit adding new fields to an existing index. If you need to support different vector dimensions for different users, it is essential to define all required fields upfront while creating the index. Should additional fields be required later, you would need to create a new index and reindex the data accordingly.

2.will values be correctly stored and persist across all fields?
Yes, but only if all required fields are defined at the time of index creation. If an index is initially created with only vector_768, it cannot be modified later to include vector_3076. Instead, a new index must be created. However, if the index is designed in advance to accommodate both vector_768 and vector_3076, values will be correctly stored and persist across all fields without any issues.

3.The SearchIndexClient.create_or_update_index() method does not support adding new fields to an existing index. In case additional fields are required, a complete index rebuilding is necessary, as Azure AI Search does not allow dynamic schema updates. Therefore, careful planning during the initial setup is recommended.

4.If you are just adding new documents without modifying the schema, the index remains unchanged. You can use SearchClient.upload_documents() to store data without triggering a rebuild.
references:
https://learn.microsoft.com/en-us/rest/api/searchservice/update-index?utm_source=chatgpt.com
https://learn.microsoft.com/en-us/azure/search/search-howto-reindex