How can I use Azure AI search with Azure Agents?

Question

I've created an Azure index based on pdf files but when I connect it to the Azure Agent I can't get the expected responses

Message: "It seems that I am unable to retrieve information from the uploaded index regarding the "repair plan." Could you please check if the document is uploaded correctly or specify what details you are looking for?"

When I use the index from Microsoft Azure, I'm able to get responses

I've also followed this documentation, but I'm still not getting a response. Do you have any suggestions?

Answer 1

Hi @Junior Saravia ,

First, once again confirm that the agent is actually calling the search index. In the OpenAI Studio, there’s a Logs section in the Agent Playground where you can see if the agent is making requests to the knowledge source. If there’s no request happening, the agent might not be recognizing when to use the search index. You could try updating the system instructions to be more explicit, like saying, “When the user asks about repair procedures, retrieve information from the Azure Cognitive Search knowledge base.” That way, it knows to reach out to the index instead of relying on its internal knowledge.

Another thing to check is whether the search index is actually returning the right data when the agent queries it. Sometimes, the chunking of documents can affect retrieval. If your repair plan is part of a larger PDF, it might be split in a way that makes it harder to find. You might want to experiment with different chunk sizes to see if that improves results.

One more thing you can try is testing the search connection outside of the agent by using the Chat Playground in OpenAI Studio. There’s an option to "Add your data" where you can directly link your search index. If that works but the agent still doesn’t, then the issue is more likely with how the agent is set up to call the knowledge source.

Hope it helps!

---

Please do not forget to click "Accept the answer” and Yes wherever the information provided helps you, this can be beneficial to other community members.

If you have any other questions or still running into more issues, let me know in the "comments" and I would be happy to help you.