How to create an index and indexer for RAG solution with Azure OpenAI and Azure AI Search in Python?

11-4688 211 Reputation points
2024-12-13T14:40:57.52+00:00

Hello,

I am an enthusiastic supporter of the potential that Azure solutions, particularly regarding RAG, bring to the table.

However, my current approach to index creation appears somewhat rudimentary. I currently upload my source documents to Azure Blob Storage and proceed as follows: I navigate to AI Foundry, run Chat Playground, choose "Add a data source," select "Azure Blob Storage (preview)," and set up the process as below:

User's image

User's image

It creates an index like below; I failed to replicate this result by clicking it out manually:

User's image Despite this, I have several concerns:

  1. Azure Blob Storage is still in preview, making me hesitant to rely on it for a production environment.
  2. The manual setup process is cumbersome and not scalable for multiple indexes.
  3. The current setup lacks flexibility—I cannot specify advanced features such as chunk overlapping or define vector profiles during deployment.

Could you recommend any automation tools, preferably a Python module, that would allow me to streamline this process without manual intervention?

Azure AI Search
Azure AI Search
An Azure search service with built-in artificial intelligence capabilities that enrich information to help identify and explore relevant content at scale.
1,119 questions
Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
3,003 questions
Azure OpenAI Service
Azure OpenAI Service
An Azure service that provides access to OpenAI’s GPT-3 models with enterprise capabilities.
3,452 questions
{count} votes

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.