[Azure AI Search] Index Management

Question

Hello! I have some questions about implementing a search service through Azure AI Search.

[Requirements for the service to be developed]

1. There are multiple categories (ex> HR, purchase, etc.) in each project, and multiple documents are managed by category
2. The metadata managed by category is different, and filtering, facet, etc. search conditions should be applied to each metadata item.
3. Each document can be embedded with a different embedding model
4. There are not many documents that use the same schema and embedding model.
5. The overall amount of data that needs to be embedded is not large.

**

[Inquiry]**

1. (Support for multiple schemas for the same index)

I would like to inquire about whether multiple schemas are supported in one index of Azure AI Search.

A field for storing metadata is also required in the schema,

and when searching, filtering and facet processing should be possible for each metadata field (ex> title, owner).

2. (Is it possible to manage vector data with various dimensions vectorized through various embedding models within the same index?)

I would like to ask how I can manage vector data with various dimensions vectorized through various embedding models within a single index.

I would like to ask if there is any other efficient way other than defining vector fields by dimension within a single index and storing/retrieving data in different vector fields by dimension.

3. (Flexible management of storage size and maximum number of indexes per service)

I would like to ask if there is a way to flexibly manage the allowed storage size and maximum number of indexes per service.

If it is not possible to manage flexibly and is only possible with a fixed number, I would like to ask how to manage it with a fixed number but support a smaller size and a larger number of indexes per storage. (Currently, the Azure AI Search in use is the Standard S1 plan, with 160GB of storage per service and a maximum index count of 50.

I want to use a smaller storage size and a larger maximum index per service. I would like to ask if there is another way other than moving to an expensive plan like Standard S3.)

4. In addition, if there is an efficient way to configure Azure AI Search to meet the service requirements to be developed above, please share it.

Answer 1

Hi @이소영
Thanks for the question and using MS Q&A platform.
1.Azure AI Search does not allow multiple schemas in one index. However, you can add all necessary metadata fields (like title, owner) and use filters and facets to search by them.

2.In Azure AI Search, you need to create separate vector fields for embeddings with different dimensions (e.g., one for 512 dimensions and another for 1024). You cannot store mixed dimensions in a single field. Additionally, hybrid search can combine vector and keyword queries, improving result comprehensiveness. When defining your schema, optimizing for retrieval patterns will enhance performance and accuracy.

3.The Standard S1 plan gives you 160GB of storage and allows up to 50 indexes. If you need more indexes but less storage, your only option is upgrading to a higher tier like Standard S3.
To manage smaller storage and more indexes

Explore other pricing tiers like Basic (up to 15 indexes) or Standard S3 for higher limits.

Consolidate data into fewer indexes to stay within current limits.

Monitor usage and adjust the plan if storage is underutilized.
Service limits in Azure AI Search
4.To meet your service requirements, consider structuring your index to accommodate the different categories and metadata. Ensure that you define appropriate vector fields for your embedding models and utilize filtering and faceting capabilities effectively. Additionally, Note the limitations of your current service tier when planning your index design.
references:
Choose an approach for optimizing vector storage and processing.
Vector storage in Azure AI Search
Vector index size and staying under limits.
Vectors in Azure AI Search.
Please accept as "Yes" if the answer provided is useful, so that you can help others in the community looking for remediation for similar issues.