Azure AI - Inconsistent responses for the same question.

Question

Hello! I'm wondering if anyone knows if there is a reason for the chatbot to occasionally give an answer of "The requested information is not found in the retrieved data. Please try another query or topic." even though it has given an answer to the same question previously. Indexed files are the same.

This usually happens when I first open the chatbot, but after refreshing or giving it other multiple questions, it snaps out of it and gives the correct answer.

Answer 1

@Andrew Satanovsky Welcome to Microsoft Q&A Forum, Thank you for posting your query here!

Are you seeing this issue only from the chatbot (Webapp) or in the AI Studio -> Chat playground too ?

If you are seeing this only in the webapp chatbot, then Could you please navigate to your Azure Webapp and then within its Configuration setting enable the Always On setting as shown below:

Could you please disable the "limit responses to your data" flag as it is intended to restrict the model from responding to questions that don't pertain or have answers in the data, so it will give the generic message whenever it determines that your question is out of domain of the data.

Also note that turning off the  "Limit your response to your data", the model would try it best to answer the user question based on the retrieval documents and its internal knowledge. This would also be have higher possibility to let the model to provide answer to the user.

Which GPT model are you using ? Did you observe the same behavior while using GPT 4 or Using the gpt-35-turbo-16k ?

Could you please try changing your text input prompt and try to be more descriptive and check if that helps ?

Is your own data in pdf / docx / md file ? or Are you using Cognitive Search index with Blob Storage as data source ?
If you are using Cognitive search index, Could you please try to use the AI Search to give the same query and check if that gives the required response ? And check giving the same query in the chat works fine or not.

One possibility for model could not answer the question is due to the document is filtered by the service , this could be resolved by the strictness. Strictness is defined below in the link Using your data with Azure OpenAI Service - Azure OpenAI | Microsoft Learn.  Less strictness means filtering less documents.  

Try setting this value to 1 and check if that makes any difference.

Awaiting your reply.