Azure AI Agent Service file search tool

File search augments agents with knowledge from outside its model, such as proprietary product information or documents provided by your users.

Note

Using the standard agent setup, the improved file search tool ensures your files remain in your own storage, and your Azure AI Search resource is used to ingest them, ensuring you maintain complete control over your data.

File sources

• Upload local files
• Azure Blob Storage

Dependency on agent setup

Basic agent setup

The file search tool has the same functionality as Azure OpenAI Assistants. Microsoft managed search and storage resources are used.

• Uploaded files get stored in Microsoft managed storage
• A vector store is created using a Microsoft managed search resource

Standard agent setup

The file search tool uses the Azure AI Search and Azure Blob Storage resources you connected during agent setup.

• Uploaded files get stored in your connected Azure Blob Storage account
• Vector stores get created using your connected Azure AI Search resource
  
  

For both agent setups, OpenAI handles the entire ingestion process, which includes:

• Automatically parsing and chunking documents
• Generating and storing embeddings
• Utilizing both vector and keyword searches to retrieve relevant content for user queries.

There is no difference in the code between the two setups; the only variation is in where your files and created vector stores are stored.

How it works

The file search tool implements several retrieval best practices out of the box to help you extract the right data from your files and augment the model’s responses. The file search tool:

• Rewrites user queries to optimize them for search.
• Breaks down complex user queries into multiple searches it can run in parallel.
• Runs both keyword and semantic searches across both agent and thread vector stores.
• Reranks search results to pick the most relevant ones before generating the final response.
• By default, the file search tool uses the following settings:
  • Chunk size: 800 tokens
  • Chunk overlap: 400 tokens
  • Embedding model: text-embedding-3-large at 256 dimensions
  • Maximum number of chunks added to context: 20

Vector stores

Vector store objects give the file search tool the ability to search your files. Adding a file to a vector store automatically parses, chunks, embeds, and stores the file in a vector database that's capable of both keyword and semantic search. Each vector store can hold up to 10,000 files. Vector stores can be attached to both agents and threads. Currently you can attach at most one vector store to an agent and at most one vector store to a thread.

Similarly, these files can be removed from a vector store by either:

• Deleting the vector store file object or,
• By deleting the underlying file object, which removes the file it from all vector_store and code_interpreter configurations across all agents and threads in your organization

The maximum file size is 512 MB. Each file should contain no more than 5,000,000 tokens per file (computed automatically when you attach a file).

Ensuring vector store readiness before creating runs

We highly recommend that you ensure all files in a vector_store are fully processed before you create a run. This ensures that all the data in your vector store is searchable. You can check for vector store readiness by using the polling helpers in the SDKs, or by manually polling the vector store object to ensure the status is completed.

As a fallback, there's a 60-second maximum wait in the run object when the thread's vector store contains files that are still being processed. This is to ensure that any files your users upload in a thread a fully searchable before the run proceeds. This fallback wait does not apply to the agent's vector store.

Quickstart – Upload Local Files with file search

In this example, we use Azure AI Agent Service to create an agent that can help answer questions on information you upload from local files.

Prerequisites

1. Complete the agent setup.

2. Ensure that you have the role Storage Blob Data Contributor on your project's storage account.

3. Ensure that you have the role Azure AI Developer on your project.

Step 1: Create a project client

Create a client object that contains the connection string for connecting to your AI project and other resources.

Python

import os
from azure.ai.projects import AIProjectClient
from azure.ai.projects.models import FileSearchTool, MessageAttachment, FilePurpose
from azure.identity import DefaultAzureCredential


# Create an Azure AI Client from a connection string, copied from your AI Studio project.
# At the moment, it should be in the format "<HostName>;<AzureSubscriptionId>;<ResourceGroup>;<ProjectName>"
# Customer needs to login to Azure subscription via Azure CLI and set the environment variables

credential = DefaultAzureCredential()
project_client = AIProjectClient.from_connection_string(
    credential=credential, conn_str=os.environ["PROJECT_CONNECTION_STRING"] 
)

C#

using System;
using System.Collections.Generic;
using System.IO;
using System.Threading.Tasks;
using Azure.Core.TestFramework;
using NUnit.Framework;

// Create an Azure AI Client from a connection string, copied from your AI Studio project.
// At the moment, it should be in the format "<HostName>;<AzureSubscriptionId>;<ResourceGroup>;<ProjectName>"
// Customer needs to login to Azure subscription via Azure CLI and set the environment variables
var connectionString = TestEnvironment.AzureAICONNECTIONSTRING;
AgentsClient client = new AgentsClient(connectionString, new DefaultAzureCredential());

Step 2: Upload files and add them to a Vector Store

To access your files, the file search tool uses the vector store object. Upload your files and create a vector store. After creating the vector store, poll its status until all files are out of the in_progress state to ensure that all content is fully processing. The SDK provides helpers for uploading and polling.

Python

# We will upload the local file and will use it for vector store creation.

#upload a file
file = project_client.agents.upload_file_and_poll(file_path='./data/product_catelog.md', purpose=FilePurpose.AGENTS)
print(f"Uploaded file, file ID: {file.id}")

# create a vector store with the file you uploaded
vector_store = project_client.agents.create_vector_store_and_poll(file_ids=[file.id], name="my_vectorstore")
print(f"Created vector store, vector store ID: {vector_store.id}")

C#

// Upload a file and wait for it to be processed
File.WriteAllText(
    path: "sample_file_for_upload.txt",
    contents: "The word 'apple' uses the code 442345, while the word 'banana' uses the code 673457.");
Response<AgentFile> uploadAgentFileResponse = await client.UploadFileAsync(
    filePath: "sample_file_for_upload.txt",
    purpose: AgentFilePurpose.Agents);

AgentFile uploadedAgentFile = uploadAgentFileResponse.Value;

// Create a vector store with the file and wait for it to be processed.
// If you do not specify a vector store, create_message will create a vector store with a default expiration policy of seven days after they were last active
VectorStore vectorStore = await client.CreateVectorStoreAsync(
    fileIds:  new List<string> { uploadedAgentFile.Id },
    name: "my_vector_store");

Step 3: Create an agent and enable file search

To make the files accessible to your agent, create a FileSearchTool object with the vector_store ID, and attach tools and tool_resources to the agent.

Python

# create a file search tool
file_search_tool = FileSearchTool(vector_store_ids=[vector_store.id])

# notices that FileSearchTool as tool and tool_resources must be added or the agent will be unable to search the file
agent = project_client.agents.create_agent(
    model="gpt-4o-mini",
    name="my-agent",
    instructions="You are a helpful agent",
    tools=file_search_tool.definitions,
    tool_resources=file_search_tool.resources,
)
print(f"Created agent, agent ID: {agent.id}")

C#

FileSearchToolResource fileSearchToolResource = new FileSearchToolResource();
fileSearchToolResource.VectorStoreIds.Add(vectorStore.Id);

// Create an agent with toolResources and process assistant run
Response<Agent> agentResponse = await client.CreateAgentAsync(
        model: "gpt-4o-mini",
        name: "SDK Test Agent - Retrieval",
        instructions: "You are a helpful agent that can help fetch data from files you know about.",
        tools: new List<ToolDefinition> { new FileSearchToolDefinition() },
        toolResources: new ToolResources() { FileSearch = fileSearchToolResource });
Agent agent = agentResponse.Value;

Step 4: Create a thread

You can also attach files as Message attachments on your thread. Doing so creates another vector_store associated with the thread, or, if there's already a vector store attached to this thread, attaches the new files to the existing thread vector store. When you create a Run on this thread, the file search tool queries both the vector_store from your agent and the vector_store on the thread.

Python

# Create a thread
thread = project_client.agents.create_thread()
print(f"Created thread, thread ID: {thread.id}")

# Upload the user provided file as a messsage attachment
message_file = project_client.agents.upload_file_and_poll(file_path='product_info_1.md', purpose=FilePurpose.AGENTS)
print(f"Uploaded file, file ID: {message_file.id}")

# Create a message with the file search attachment
# Notice that vector store is created temporarily when using attachments with a default expiration policy of seven days.
attachment = MessageAttachment(file_id=message_file.id, tools=FileSearchTool().definitions)
message = project_client.agents.create_message(
    thread_id=thread.id, role="user", content="What feature does Smart Eyewear offer?", attachments=[attachment]
)
print(f"Created message, message ID: {message.id}")

C#

// Create thread for communication
Response<AgentThread> threadResponse = await client.CreateThreadAsync();
AgentThread thread = threadResponse.Value;

// Create message to thread
Response<ThreadMessage> messageResponse = await client.CreateMessageAsync(
    thread.Id,
    MessageRole.User,
    "Can you give me the documented codes for 'banana' and 'orange'?");
ThreadMessage message = messageResponse.Value;

Step 5: Create a run and check the output

Create a run and observe that the model uses the file search tool to provide a response to the user's question.

Python

run = project_client.agents.create_and_process_run(thread_id=thread.id, assistant_id=agent.id)
print(f"Created run, run ID: {run.id}")

project_client.agents.delete_vector_store(vector_store.id)
print("Deleted vector store")

project_client.agents.delete_agent(agent.id)
print("Deleted agent")

messages = project_client.agents.list_messages(thread_id=thread.id)
print(f"Messages: {messages}")

C#

// Run the agent
Response<ThreadRun> runResponse = await client.CreateRunAsync(thread, agent);

do
{
    await Task.Delay(TimeSpan.FromMilliseconds(500));
    runResponse = await client.GetRunAsync(thread.Id, runResponse.Value.Id);
}
while (runResponse.Value.Status == RunStatus.Queued
    || runResponse.Value.Status == RunStatus.InProgress);

Response<PageableList<ThreadMessage>> afterRunMessagesResponse
    = await client.GetMessagesAsync(thread.Id);
IReadOnlyList<ThreadMessage> messages = afterRunMessagesResponse.Value.Data;

// Note: messages iterate from newest to oldest, with the messages[0] being the most recent
foreach (ThreadMessage threadMessage in messages)
{
    Console.Write($"{threadMessage.CreatedAt:yyyy-MM-dd HH:mm:ss} - {threadMessage.Role,10}: ");
    foreach (MessageContent contentItem in threadMessage.ContentItems)
    {
        if (contentItem is MessageTextContent textItem)
        {
            Console.Write(textItem.Text);
        }
        else if (contentItem is MessageImageFileContent imageFileItem)
        {
            Console.Write($"<image from ID: {imageFileItem.FileId}");
        }
        Console.WriteLine();
    }
}

Quickstart – Use existing files in Azure Blob Storage with file search

In this example, we use Azure AI Agent Service to create an agent that can help answer questions on information from files in Azure Blob Storage.

Prerequisites

1. Complete the standard agent setup.

2. Ensure that you have the role Storage Blob Data Contributor on your project's storage account.

3. Ensure that you have the role Azure AI Developer on your project.

Important

File search using Blob storage is only supported by the standard agent setup.

Step 1: Create a project client

import os
from azure.ai.projects import AIProjectClient
from azure.ai.projects.models import FileSearchTool, VectorStoreDataSource, VectorStoreDataSourceAssetType
from azure.identity import DefaultAzureCredential


# Create an Azure AI Client from a connection string, copied from your AI Studio project.
# At the moment, it should be in the format "<HostName>;<AzureSubscriptionId>;<ResourceGroup>;<ProjectName>"
# Customer needs to login to Azure subscription via Azure CLI and set the environment variables

credential = DefaultAzureCredential()
project_client = AIProjectClient.from_connection_string(
    credential=credential, conn_str=os.environ["PROJECT_CONNECTION_STRING"]
)

Step 2: Upload local files to your project Azure Blob Storage container

Upload your local file to the project’s Azure Blob Storage container. This is the same storage account you connected to your agent during setup. When creating additional agents within the same project, you can reuse the asset URIs of any previously uploaded files that those agents need. This means you don't have to upload the same file repeatedly, as the asset URIs allow you to reference the files directly.

Then, create a vector store using the asset_uri, which is the location of your file in your project's datastore.

# We'll upload the local file to your project Azure Blob Storage container and will use it for vector store creation.
_, asset_uri = project_client.upload_file("sample_file_for_upload.md")
print(f"Uploaded file, asset URI: {asset_uri}")

# create a vector store with a file in blob storage and wait for it to be processed
ds = VectorStoreDataSource(asset_identifier=asset_uri, asset_type=VectorStoreDataSourceAssetType.URI_ASSET)
vector_store = project_client.agents.create_vector_store_and_poll(data_sources=[ds], name="sample_vector_store")
print(f"Created vector store, vector store ID: {vector_store.id}")

Step 3: Create an agent with access to the file search tool

# create a file search tool
file_search_tool = FileSearchTool(vector_store_ids=[vector_store.id])

# notices that FileSearchTool as tool and tool_resources must be added or the assistant unable to search the file
agent_1 = project_client.agents.create_agent(
    model="gpt-4o-mini",
    name="my-assistant",
    instructions="You are helpful assistant",
    tools=file_search_tool.definitions,
    tool_resources=file_search_tool.resources,
)
# [END upload_file_and_create_agent_with_file_search]
print(f"Created agent_1, agent_1 ID: {agent_1.id}")

thread = project_client.agents.create_thread()
print(f"Created thread, thread ID: {thread.id}")

message = project_client.agents.create_message(
    thread_id=thread.id, role="user", content="What feature does Smart Eyewear offer?"
)
print(f"Created message, message ID: {message.id}")

run = project_client.agents.create_and_process_run(thread_id=thread.id, assistant_id=agent_1.id)

project_client.agents.delete_vector_store(vector_store.id)
print("Deleted vector store")

project_client.agents.delete_agent(agent.id)
print("Deleted agent")

messages = project_client.agents.list_messages(thread_id=thread.id)
print(f"Messages: {messages}")

Step 4: Create second vector store using the previously uploaded file

Now, create a second vector store using the previously uploaded file. Using the asset_uri of a file already in Azure Blob Storage is useful if you have multiple agents that need access to the same files, as it eliminates the need to upload the same file multiple times.

# create a vector store with a previously uploaded file and wait for it to be processed
ds_2 = VectorStoreDataSource(asset_identifier=asset_uri, asset_type=VectorStoreDataSourceAssetType.URI_ASSET)
vector_store_2 = project_client.agents.create_vector_store_and_poll(data_sources=[ds_2], name="sample_vector_store_2")
print(f"Created vector store, vector store ID: {vector_store.id}")

Step 5: Create a second agent with access to the file search tool

file_search_tool_2 = FileSearchTool(vector_store_ids=[vector_store_2.id])
# notices that FileSearchTool as tool and tool_resources must be added or the assistant unable to search the file
agent_2 = project_client.agents.create_agent(
    model="gpt-4o-mini",
    name="my-assistant-2",
    instructions="You are helpful assistant",
    tools=file_search_tool_2.definitions,
    tool_resources=file_search_tool_2.resources,
)
# [END upload_file_and_create_agent_with_file_search]
print(f"Created agent, agent ID: {agent_2.id}")

Supported file types

Note

For text/ MIME types, the encoding must be either utf-8, utf-16, or ASCII.

File format	MIME Type
.c	text/x-c
.cs	text/x-csharp
.cpp	text/x-c++
.doc	application/msword
.docx	application/vnd.openxmlformats-officedocument.wordprocessingml.document
.html	text/html
.java	text/x-java
.json	application/json
.md	text/markdown
.pdf	application/pdf
.php	text/x-php
.pptx	application/vnd.openxmlformats-officedocument.presentationml.presentation
.py	text/x-python
.py	text/x-script.python
.rb	text/x-ruby
.tex	text/x-tex
.txt	text/plain
.css	text/css
.js	text/javascript
.sh	application/x-sh
.ts	application/typescript

Creating vector stores and adding files

Adding files to vector stores is an async operation. To ensure the operation is complete, we recommend that you use the 'create and poll' helpers in our official SDKs. If you're not using the SDKs, you can retrieve the vector_store object and monitor its file_counts property to see the result of the file ingestion operation.

Files can also be added to a vector store after it's created by creating vector store files.

# create a vector store with no file and wait for it to be processed
vector_store = project_client.agents.create_vector_store_and_poll(data_sources=[], name="sample_vector_store")
print(f"Created vector store, vector store ID: {vector_store.id}")

# add the file to the vector store or you can supply file ids in the vector store creation
vector_store_file_batch = project_client.agents.create_vector_store_file_batch_and_poll(
    vector_store_id=vector_store.id, file_ids=[file.id]
)
print(f"Created vector store file batch, vector store file batch ID: {vector_store_file_batch.id}")

Alternatively, you can add several files to a vector store by creating batches of up to 500 files.

batch = project_client.agents.create_vector_store_file_batch_and_poll(
  vector_store_id=vector_store.id,
  file_ids=[file_1.id, file_2.id, file_3.id, file_4.id, file_5.id]
)

Basic agent setup: Deleting files from vector stores

Files can be removed from a vector store by either:

• Deleting the vector store file object or,
• Deleting the underlying file object, which removes the file it from all vector_store and code_interpreter configurations across all agents and threads in your organization

The maximum file size is 512 MB. Each file should contain no more than 5,000,000 tokens per file (computed automatically when you attach a file).

Remove vector store

You can remove a vector store from the file search tool.

file_search_tool.remove_vector_store(vector_store.id)
print(f"Removed vector store from file search, vector store ID: {vector_store.id}")

project_client.agents.update_agent(
    assistant_id=agent.id, tools=file_search_tool.definitions, tool_resources=file_search_tool.resources
)
print(f"Updated agent, agent ID: {agent.id}")

Deleting vector stores

project_client.agents.delete_vector_store(vector_store.id)
print("Deleted vector store")

Managing costs with expiration policies

For basic agent setup, the file_search tool uses the vector_stores object as its resource and you're billed based on the size of the vector_store objects created. The size of the vector store object is the sum of all the parsed chunks from your files and their corresponding embeddings.

To help you manage the costs associated with these vector_store objects, we added support for expiration policies in the vector_store object. You can set these policies when creating or updating the vector_store object.

vector_store = project_client.agents.create_vector_store_and_poll(
  name="Product Documentation",
  file_ids=[file_1.id],
  expires_after={
      "anchor": "last_active_at",
      "days": 7
  }
)

Thread vector stores have default expiration policies

Vector stores created using thread helpers (like tool_resources.file_search.vector_stores in Threads or message.attachments in Messages) have a default expiration policy of seven days after they were last active (defined as the last time the vector store was part of a run).

When a vector store expires, the runs on that thread fail. To fix this issue, you can recreate a new vector_store with the same files and reattach it to the thread.