LLM end to end evaluation

By the time you reach this phase, you have already generated your search index and determined what searches you want to perform. This phase addresses the process of evaluating of your Retrieval-Augmented Generation (RAG) solution from the perspective of evaluating expected user prompts containing the retrieved grounding data against the language model. Before you reach this phase, you should complete the preparation phase where you collect your test documents and queries, chunk your test documents, enrich the chunks, embed the chunks, create a search index, and implement a search strategy. You should evaluate each of these phases and be happy with the results. At this point, you should feel comfortable that your solution returns relevant grounding data for a user query.

This grounding data forms the context for the prompt that you send to the language model to address the user's query. Prompt engineering strategies are beyond the scope of this article. This article addresses the evaluation of the engineered call to the language model from the perspective of the grounding data. This article covers some common language model evaluation metrics, and some specific similarity and evaluation metrics that can be used in model evaluation calculations or as stand alone metrics.

This article doesn't attempt to provide an exhaustive list of either language model metrics or similarity and evaluation metrics. The number of these metrics are growing every day. What is important for you to take away from this article is that there are various metrics, each with their own distinct use case. You're the only one with a holistic understand your workload. You and your data scientists must determine what it is that you want to measure and which metrics help you accomplish that task.

This article is part of a series. Read the introduction.

Language model evaluation metrics

There are several metrics you can use to evaluate the language model's response, including groundedness, completeness, utilization, and relevancy.

Language model responses are non-deterministic, meaning the same prompt to a language model can and will often return different results. This is important to understand when using a language model as part of your evaluation process. Consider using a target range over a single target when evaluating using a language model.

Groundedness

Groundedness, sometimes referred to as faithfulness, measures whether the response is completely based on the context. It validates that the response isn't using information other than what exists in the context. A low groundedness metric indicates that the language model might be drifting into imaginative or nonsensical territory known as hallucinations.

Calculating

• Azure AI Content Safety Service (AACS) based groundedness is a custom model that uses Natural Language Inference (NLI) to determine whether claims, in this case chunks, are entailed or not entailed by a source document.
• Large language model based groundedness uses a language model to determine the level of groundedness of the response.
• Ragas faithfulness library
• MLflow faithfulness calculation

Evaluating

If groundedness is low, it indicates that the language model doesn't see the chunks as relevant. You should evaluate whether you need to add data to your corpus, adjust your chunking strategy or chunk size, or fine tune your prompt.

Completeness

Completeness measures whether the response is answering all parts of the query. Completeness helps you understand whether the chunks in the context are pertinent and directly related to the query and provide a complete answer.

Calculating

• AI-assisted: Retrieval Score prompting
• A language model can help you measure the quality of the language model response. You need the question, context, and generated answer to perform this measurement. The following outlines the high level process:
  1. Use the language model to rephrase, summarize, or simplify the question. This step identifies the intent.
  2. Ask the model to check if the intent or the answer to the intent is found or can be derived from retrieved documents where the answer can be "No," or "Yes" for each document. Answers that start with "Yes" indicate that the retrieved documents are relevant to the intent or answer to the intent.
  3. Calculate the ratio of the intents that have an answer beginning with "Yes".
  4. Square the score to highlight the errors.

Evaluating

If completeness is low, start by evaluating your embedding model. Compare the vocabulary in your content with the vocabulary in your chosen embedding model. Determine whether you need a domain specific embedding model or you need to fine-tune an existing model. As a next step, evaluate your chunking strategy. If you're using fixed length, consider increasing your chunk size. You can also evaluate whether your test data has enough data to completely address the question.

Utilization

Utilization measures the extent to which the response is made up of information from the chunks in the context. The goal is to determine the extent to which each chunk is part of the response. Low utilization indicates that your results might not be relevant to the query. Utilization should be evaluated along side completeness.

Calculating

You can use a language model to calculate the utilization. You can pass the response and the context containing the chunks to the language model. You can ask the language model to determine the number of chunks that entail the answer.

Evaluating

The following table provides guidance, taking both completeness and utilization together.

Relevance

Measures the extent to which the language model's response is pertinent and related to the query.

Calculating

• AI-assisted: Relevance in Azure AI Studio - You can use Azure AI Studio to perform the calculations, or use the guidance in this article to calculate relevance for yourself.
• Ragas answer relevancy library
• MLflow relevance calculation

Evaluating

When relevance is low, evaluate the following:

• Ensure that the chunks provided to the language model are relevant.
  • Determine whether there are viable chunks that are relevant that weren't returned. If there are, evaluate your embedding model.
  • If there aren't viable chunks, look to see whether relevant data exists. If it does, evaluate your chunking strategy.
• If relevant chunks were returned, evaluate your prompt.

Other evaluation methods like Completeness should be calculated and should yield similar scores to the ones observed in the relevance measure.

Similarity and evaluation metrics

As mentioned in the introduction, there are hundreds of similarity and evaluation metrics used in data science. Some algorithms are specific to a domain, such as speech to text or language to language translation. Each algorithm has a unique strategy for calculating its metric.

The data scientist determines what it is you want to measure and what metric or combination of metrics you can use to measure it. For example, in the area of language translation, the Bleu metric checks how many n-grams appear in both the machine translation and human translation to measure similarity based on using the same words. Cosine similarity uses embeddings between the machine and human translations to measure semantic similarity. If your goal was to have high semantic similarity and use similar words to the human translation, your goal would be a high Bleu score with high cosine similarity. If you only cared about semantic similarity, you would focus on cosine similarity.

The following list contains a small sample of common similarity and evaluation metrics. Notice that the similarity metrics listed are described as token based, sequence based, or edit based, illustrating how they use vastly different approaches to calculating similarity. Also note that the list contains three algorithms for evaluating the quality of text translation from one language to another.

• Longest common substring - Sequence based algorithm that finds the longest common substring between two strings. The longest common substring percentage takes the longest common substring and divides it by either the number of characters of the smaller or larger input string.
• Longest common subsequence (LCS) - Sequence based algorithm that finds the longest subsequence between two strings. LCS doesn't require the subsequences to be in consecutive order.
• Cosine similarity - Token based algorithm that calculates the cosine of the angle between the two vectors.
• Jaro Winkler - Edit based algorithm that counts the minimum number of steps to transform one string into another.
• Hamming - Edit based algorithm that measures the minimum number of substitutions that are required to transform one string to another.
• Jaccard - Token based algorithm that calculates similarity by dividing the intersection of two strings by the union of those strings.
• Levenshtein - Edit based algorithm that calculates similarity by determining the minimum number of single character edits required to transform one string to another.
• BLEU - Evaluates the quality of text that is the result of machine translation from one language to another. Bleu calculates the overlap of n-grams between a machine translation and a human quality translation to make this evaluation.
• ROUGE - Compare a machine translation of one language to another to a human created translation. There are several ROUGE variants that use the overlap of n-grams, skip-bigrams, or longest common subsequence.
• METEOR - Evaluates the quality of text that is the result of machine translation by looking at exact matches, matches after stemming, synonyms, paraphrasing, and alignment.

Refer to the following resources for common similarity and evaluation metrics:

• PyPi textdistance package
• Wikipedia list of similarity algorithms

Documentation, reporting, and aggregation

You should document both the hyperparameters you chose for an experiment and the resulting evaluation metrics so you can understand the impact of the hyperparameters on your results. You should document hyperparameters and results at granular levels like embedding or search evaluation and at a macro level, like testing the entire system end to end.

During design and development, you might be able to track the hyperparameters and results manually. However, while performing multiple evaluations against your entire test document and test query corpus might involve hundreds of evaluation runs and thousands of results. You should automate the persistence of parameters and results for your evaluations.

Once your hyperparameters and results are persisted, you should consider building charts and graphs to allow you to more easily visualize the effects the hyperparameter choices have on the metrics. Visualization helps you identify which choices lead to dips or spikes in performance.

It's important for you to understand that designing and evaluating your RAG solution isn't a one-time operation. Your corpus of documents will change over time. The questions your customers are asking will change over time and your understanding of the types of questions will evolve as you learn from production. You should revisit this process again and again. Maintaining documentation of past evaluations is critical for future design and evaluation efforts.

The RAG Experiment Accelerator

These articles walk you through all the phases and design choices involved in designing and evaluating a RAG solution. The articles focus on what you should do, not how to do it. An engineering team that works with Microsoft's top customers developed a tool called the RAG Experiment Accelerator. The RAG Experiment Accelerator is a state-of-the-art experimentation framework designed to optimize and enhance the development of Retrieval Augmented Generation (RAG) solutions. RAG Experiment Accelerator empowers researchers and developers to efficiently explore and fine-tune the critical components that drive RAG performance, ultimately leading to more accurate and coherent text generation.

With its CLI based interface, you can effortlessly experiment with various embedding models, refine chunking strategies, and evaluate different search approaches to unlock the full potential of your RAG system. It allows you to focus on the core aspects of RAG development while abstracting away the complexities of hyper-parameter tuning using simple configuration.

Moreover, the framework provides comprehensive support for language model configuration, enabling you to strike the perfect balance between model complexity and generation quality. This tool allows you to streamline the experimentation process, save valuable time, and significantly improve the performance of your RAG models.

Whether you're a seasoned researcher pushing the boundaries of natural language understanding or an industry professional seeking to enhance text generation capabilities, this experimentation framework is the ultimate solution to accelerate your RAG development journey. Embrace the future of RAG experimentation and unlock the true potential of your models with this cutting-edge tool.

RAG with Vision Application Framework

Much of the guidance in this article around working with media in your RAG solution came from another engineering team that works with Microsoft's top customers. This team wrote a framework called the RAG with Vision Application Framework. This framework provides a Python-based retrieval-augmented generation (RAG) pipeline that processes both textual and image content from MHTML documents.

The framework loads, chunks, and enriches both text and images from MHTML files and ingests the chunks to Azure Search. The framework implements caching for image enrichment for both processing and cost efficiency. The framework also incorporates evaluation as part of the pipeline.

Contributors

• Ritesh Modi
• Rob Bagby
• Ryan Pfalz
• Raouf Aliouat
• Randy Thurman
• Prabal Deb
• Mahdi Setayesh
• Soubhi Hadri
• Paul Butler

Next steps

Related resources

• Develop an evaluation flow in Azure AI Studio