How to transfer data between Azure AI Search (including vectorized data)

Question

I would like to inquire about how to transfer data (including vectorized data) stored in Azure AI Search stg to Azure AI Search prd.

In order to reduce costs, I would like to transfer data without re-vectorizing vectorized data.

The original document size is 1.5TB, so I would like to transfer a large amount of data.

1. I would like to inquire about an efficient data transfer method.
2. When transferring a large amount of data, I would like to inquire about a method that is easy to handle in parallel processing or in case of errors.
3. I would like to inquire about the cost.

I would like to inquire about how much it will cost to transfer data when 1.5TB of original data is vectorized and stored in AI Search.

(I would like to inquire about the approximate cost for each when exporting and importing data.)

4. The link below is a link to Azure AI Search data backup & restore index provided by MS that I found by Googling.

I would like to inquire about a better data transfer method other than this method, and if there is a better tool related to it.

https://learn.microsoft.com/en-us/samples/azure-samples/azure-search-dotnet-utilities/azure-search-backup-restore-index/

https://github.com/Azure/azure-search-vector-samples/blob/main/demo-python/code/index-backup-restore/azure-search-backup-and-restore.ipynb

Answer 1

Hi there 이소영

Thanks for using QandA plaftorm

My suggestion is using the Index Backup and Restore feature. This avoids re-vectorizing data and allows for easy migration of large datasets. You can also use Azure Data Factory (ADF) to create custom pipelines for transferring data with parallel processing capabilities, making it ideal for handling large volumes of data. Using Azure Functions can help automate workflows, handle retries, and ensure error-free transfers. If this helps kindly accept the answer thanks much.