Inconsistent results with AI Search and Open AI Models : Understanding the interaction between model and Azure AI Search

Question

While using Open AI model (gpt 3.5 Turbo) with Azure AI search via a python based web application and the search results are very disappointing and unreliable. I have setup a semantic configuration too.

I noticed when i search for something i.e Find all orders by XYZ user, the AI search is pulling data that is totally irrelevant to the query.

My assumption is Azure AI analyzes the query and finds out the key search terms such as "XYZ user" and search for this term in the configured index. But many times it fetches completely irrelevant documents from the dataset. I tried some possible thoughts from my knowledge to detect these selected keywords that is actually used for the search.

The intent and citations fields in the ChatCompletionMessage are completely irrelevant most of the items.

Why does this happen ? Its kind of a black box what happens once we send the query. Please explain and help me achieve consistent results with the RAG application.

My Understanding
user query -> azure ai search analyzes -> figures out key words from query -> searches the configured index -> picks relevant documents -> adds to model context and generates response.

PFB my application architecture.

1. A GPT 3.5 Turbo model deployed in Azure Open AI Studio
2. An array of json objects (simulating ecommerce orders) as dataset configured in Azure AI Search. Refer to the configuration json order-dataset-index.txt order-dataset-indexer.txt order-dataset-index.png
   Azure AI Indexer
   Data to extract : Content and Metadata Content and metadata JSON Array
3. Used the code segment provided in "view-code" option provided in Azure Open AI Studio with API key option.
4. System Message :
   "You are an e-commerce order and tracking assistant. Your role is to help users by answering questions about orders, products, delivery statuses, payment status and returns. You have access to an Azure AI Search Index named "order-dataset-index" containing the orders dataset. Your task is to analyze the dataset, find relevant information and provide clear, friendly responses ensuring accurate order details. If multiple orders match the search, summarize the data with all the key details."
   1. User Message :
      """Follow the instructions to complete the task

                               - Analyze the query wrapped in triple backticks (```) and find the intent. Store the intent in a variable and append to final response.
      
                               - Extract, Prioritize key search terms from the query for efficient document retrieval in the connected dataset. Store keywords in a list and append to final response.
      
                               - Fetch the relevant documents for the connected dataset using the prioritized keywords only. Store fetched documents, prioritized keywords in a list and append to final response.
      
                               - Use the retrieved documents to generate a summary in 300 words of the query. Store the summary in a variable and append to final response.
      
                               - Always check the final response for correctness and completeness in relation with the given query. If not satisfactory, rephrase the query and try again. 
      
                               --During successful summary generation, provide 3 relevant follow-up questions that the user might want to ask. Store follow-up questions in a list and append to final response.
      
                               --During failures, generate an appropriate response about the failure and ask the user politely to rephrase the query. Store the failure message in a variable and append to final response.
      
      
      
                               Provide the final output as valid JSON object only in the format mentioned below. 
      
                               {
      
                                   "model_response": "The summary generated by the model as response.",
      
                                   "follow_up_questions": ["Follow-up question 1", "Follow-up question 2", "Follow-up question 3"],
      
                                   "keywords" : ["keyword1", "keyword2", "keyword3"],
      
                                   "priority_keywords" : ["priority_keyword1", "priority_keyword2", "priority_keyword3"],
      
                                   "documents" : ["document1", "document2", "document3"],
      
                                   "intent" : "intent of the query",
      
                                   "failure": "Failure message if any",
      
                                   "summary": "Summary generated by the model for the given query"
      
                               }
      
                           """ 
      
                           + f"The user query is {delimiter} {user_message} {delimiter}"
      

5. Dataset (Json Array) Order_info_data_3 - Copy.txt
6. Model Configuration Parameters Model Configuration: {'max_tokens': 500, 'temperature': 0.3, 'top_p': 0.95, 'frequency_penalty': 0, 'presence_penalty': 0}
7. Extra Body Configuration for connecting model with AI Search Extra Body for Azure Search: { "data_sources":[

     {
   
        "type":"azure_search",
   
        "parameters":{
   
           "authentication":{
   
              "type":"api_key",
   
              "key":"
   

Answer 1

Azure AI Search uses relevance scoring to determine the most relevant documents.
I'd review the scoring profile configuration to ensure it prioritizes the correct fields and data types.

https://learn.microsoft.com/en-us/azure/search/index-similarity-and-scoring