View conversational language understanding model details

After model training is completed, you can view your model details and see how well it performs against the test set.

Note

Using the Automatically split the testing set from training data option may result in different model evaluation result every time you train a new model, as the test set is selected randomly from your utterances. To make sure that the evaulation is calcualted on the same test set every time you train a model, make sure to use the Use a manual split of training and testing data option when starting a training job and define your Testing set when add your utterances.

Prerequisites

Before viewing a model's evaluation, you need:

• A successfully created project.
• A successfully trained model.

See the project development lifecycle for more information.

Model details

Language studio

1. Go to your project page in Language Studio.

2. Select Model performance from the menu on the left side of the screen.

3. In this page you can only view the successfully trained models, F1 score of each model and model expiration date. You can select the model name for more details about its performance. Models only include evaluation details if there was test data selected while training the model.

Overview

• In this tab you can view the model's details such as: F1 score, precision, recall, date and time for the training job, total training time and number of training and testing utterances included in this training job. You can view details between intents or entities by selecting Model Type at the top.

  

• You will also see guidance on how to improve the model. When clicking on view details a side panel will open to give more guidance on how to improve the model.

  

Model type performance

• This is a snapshot of how your model performed during testing. The metrics here are static and tied to your model, so they won’t update until you train again.

• You can see for each intent or entity the precision, recall, F1 score, number of training and testing labels. Entities that do not include a learned component will show no training labels. A learned component is added only by adding labels in your training set.

  

Test set details

• Here you will see the utterances included in the test set and their intent or entity predictions. You can use the Show errors only toggle to show only the utterances where there are different predictions from their labels, or unselect the toggle to view all utterances in the test set. You can also toggle the view between Showing entity labels as the view for each utterance, or Showing entity predictions. Entity predictions show as dotted lines and labels show as solid lines.

• You can expand each row to view its intent or entity predictions, specified by the Model Type column. The Text column shows the text of the entity that was predicted or labeled. Each row has a Labeled as column to indicate the label in the test set, and Predicted as column to indicate the actual prediction. Also, you will see whether it is a true positive, false positive or false negative in the Result Type column.

  

Dataset distribution

This snapshot shows how intents or entities are distributed across your training and testing sets. This data is static and tied to your model, so it won’t update until you train again. Entities that do not include a learned component will show no training labels. A learned component is added only by adding labels in your training set.

Confusion matrix

A confusion matrix is an N x N matrix used for evaluating the performance of your model, where N is the number of target intents or entities. The matrix compares the expected labels with those predicted by the model to identify which intents or entities are being misclassified as other intents and entities. You can click into any cell of the confusion matrix to identify exactly which utterances contributed to the values in that cell.

You can view the intent confusion matrix in raw count or normalized view. Raw count is the actual number of utterances that have been predicted and labeled for a set of intents. Normalized value is the ratio, between 0 and 1, of the predicted and labeled utterances for a set of intents.

You can view the entity confusion matrix in character overlap count or normalized character overlap view. Character overlap count is the actual number of spans that have been predicted and labeled for a set of entities. Normalized character overlap is the ratio, between 0 and 1, of the predicted and labeled spans for a set of entities. Sometimes entities can be predicted or labeled partially, leading to decimal values in the confusion matrix.

• All values: Will show the confusion matrix for all intents or entities.

• Only errors: Will show the confusion matrix for intents or entities with errors only.

• Only matches: Will show the confusion matrix for intents or entities with correct predictions only.

REST APIs

Create a GET request using the following URL, headers, and JSON body to get the trained model evaluation summary.

Model Summary

This API returns the summary of your model's evaluation results, including the precision, recall, F1, and confusion matrix of your intents and entities.

Request URL

{ENDPOINT}/language/authoring/analyze-conversations/projects/{projectName}/models/{trainedModelLabel}/evaluation/summary-result?api-version={API-VERSION}

Placeholder	Value	Example
{ENDPOINT}	The endpoint for authenticating your API request.	https://<your-custom-subdomain>.cognitiveservices.azure.com
{PROJECT-NAME}	The name for your project. This value is case-sensitive.	myProject
{trainedModelLabel}	The name for your trained model. This value is case-sensitive.	Model1
{API-VERSION}	The version of the API you are calling.	2023-04-01

Headers

Use the following header to authenticate your request.

Key	Value
Ocp-Apim-Subscription-Key	The key to your resource. Used for authenticating your API requests.

Response Body

Once you send the request, you will get the following response.

{
  "entitiesEvaluation": {
    "confusionMatrix": {
      "additionalProp1": {
        "additionalProp1": {
          "normalizedValue": 0,
          "rawValue": 0
        },
        "additionalProp2": {
          "normalizedValue": 0,
          "rawValue": 0
        },
        "additionalProp3": {
          "normalizedValue": 0,
          "rawValue": 0
        }
      },
      "additionalProp2": {
        "additionalProp1": {
          "normalizedValue": 0,
          "rawValue": 0
        },
        "additionalProp2": {
          "normalizedValue": 0,
          "rawValue": 0
        },
        "additionalProp3": {
          "normalizedValue": 0,
          "rawValue": 0
        }
      },
      "additionalProp3": {
        "additionalProp1": {
          "normalizedValue": 0,
          "rawValue": 0
        },
        "additionalProp2": {
          "normalizedValue": 0,
          "rawValue": 0
        },
        "additionalProp3": {
          "normalizedValue": 0,
          "rawValue": 0
        }
      }
    },
    "entities": {
      "additionalProp1": {
        "f1": 0,
        "precision": 0,
        "recall": 0,
        "truePositivesCount": 0,
        "trueNegativesCount": 0,
        "falsePositivesCount": 0,
        "falseNegativesCount": 0
      },
      "additionalProp2": {
        "f1": 0,
        "precision": 0,
        "recall": 0,
        "truePositivesCount": 0,
        "trueNegativesCount": 0,
        "falsePositivesCount": 0,
        "falseNegativesCount": 0
      },
      "additionalProp3": {
        "f1": 0,
        "precision": 0,
        "recall": 0,
        "truePositivesCount": 0,
        "trueNegativesCount": 0,
        "falsePositivesCount": 0,
        "falseNegativesCount": 0
      }
    },
    "microF1": 0,
    "microPrecision": 0,
    "microRecall": 0,
    "macroF1": 0,
    "macroPrecision": 0,
    "macroRecall": 0
  },
  "intentsEvaluation": {
    "confusionMatrix": {
      "additionalProp1": {
        "additionalProp1": {
          "normalizedValue": 0,
          "rawValue": 0
        },
        "additionalProp2": {
          "normalizedValue": 0,
          "rawValue": 0
        },
        "additionalProp3": {
          "normalizedValue": 0,
          "rawValue": 0
        }
      },
      "additionalProp2": {
        "additionalProp1": {
          "normalizedValue": 0,
          "rawValue": 0
        },
        "additionalProp2": {
          "normalizedValue": 0,
          "rawValue": 0
        },
        "additionalProp3": {
          "normalizedValue": 0,
          "rawValue": 0
        }
      },
      "additionalProp3": {
        "additionalProp1": {
          "normalizedValue": 0,
          "rawValue": 0
        },
        "additionalProp2": {
          "normalizedValue": 0,
          "rawValue": 0
        },
        "additionalProp3": {
          "normalizedValue": 0,
          "rawValue": 0
        }
      }
    },
    "intents": {
      "additionalProp1": {
        "f1": 0,
        "precision": 0,
        "recall": 0,
        "truePositivesCount": 0,
        "trueNegativesCount": 0,
        "falsePositivesCount": 0,
        "falseNegativesCount": 0
      },
      "additionalProp2": {
        "f1": 0,
        "precision": 0,
        "recall": 0,
        "truePositivesCount": 0,
        "trueNegativesCount": 0,
        "falsePositivesCount": 0,
        "falseNegativesCount": 0
      },
      "additionalProp3": {
        "f1": 0,
        "precision": 0,
        "recall": 0,
        "truePositivesCount": 0,
        "trueNegativesCount": 0,
        "falsePositivesCount": 0,
        "falseNegativesCount": 0
      }
    },
    "microF1": 0,
    "microPrecision": 0,
    "microRecall": 0,
    "macroF1": 0,
    "macroPrecision": 0,
    "macroRecall": 0
  },
  "evaluationOptions": {
    "kind": "percentage",
    "trainingSplitPercentage": 0,
    "testingSplitPercentage": 0
  }
}

Model Results

This API returns the individual results for each utterance including their expected and actual predictions for intents and entities.

Request URL

{ENDPOINT}/language/authoring/analyze-conversations/projects/{projectName}/models/{trainedModelLabel}/evaluation/result?top={top}&skip={skip}&maxpagesize={maxpagesize}&api-version={API-VERSION}

Placeholder	Value	Example
{ENDPOINT}	The endpoint for authenticating your API request.	https://<your-custom-subdomain>.cognitiveservices.azure.com
{PROJECT-NAME}	The name for your project. This value is case-sensitive.	myProject
{trainedModelLabel}	The name for your trained model. This value is case-sensitive.	Model1
{API-VERSION}	The version of the API you are calling.	2023-04-01
{top}	The maximum number of utterances to return from the collection. Optional.	100
{skip}	An offset into the collection of the first utterance to be returned. Optional.	100
{maxpagesize}	The maximum number of utterances to include in a single response. Optional	100

Headers

Use the following header to authenticate your request.

Key	Value
Ocp-Apim-Subscription-Key	The key to your resource. Used for authenticating your API requests.

Response Body

Once you send the request, you will get the following response.

{
  "value": [
    {
      "text": "send the email",
      "language": "en-us",
      "entitiesResult": {
        "expectedEntities": [],
        "predictedEntities": []
      },
      "intentsResult": {
        "expectedIntent": "SendEmail",
        "predictedIntent": "SendEmail"
      }
    },
    {
      "text": "send a mail to daniel",
      "language": "en-us",
      "entitiesResult": {
        "expectedEntities": [
          {
            "category": "ContactName",
            "offset": 15,
            "length": 6
          }
        ],
        "predictedEntities": [
          {
            "category": "ContactName",
            "offset": 15,
            "length": 6
          }
        ]
      },
      "intentsResult": {
        "expectedIntent": "SendEmail",
        "predictedIntent": "SendEmail"
      }
    },
    {
      "text": "i forgot to add an important part to that email to james . please set it up to edit",
      "language": "en-us",
      "entitiesResult": {
        "expectedEntities": [
          {
            "category": "ContactName",
            "offset": 51,
            "length": 5
          }
        ],
        "predictedEntities": [
          {
            "category": "Category",
            "offset": 19,
            "length": 9
          },
          {
            "category": "ContactName",
            "offset": 51,
            "length": 5
          }
        ]
      },
      "intentsResult": {
        "expectedIntent": "AddMore",
        "predictedIntent": "AddMore"
      }
    },
    {
      "text": "send email to a and tian",
      "language": "en-us",
      "entitiesResult": {
        "expectedEntities": [
          {
            "category": "ContactName",
            "offset": 14,
            "length": 1
          },
          {
            "category": "ContactName",
            "offset": 20,
            "length": 4
          }
        ],
        "predictedEntities": [
          {
            "category": "ContactName",
            "offset": 14,
            "length": 1
          },
          {
            "category": "ContactName",
            "offset": 20,
            "length": 4
          }
        ]
      },
      "intentsResult": {
        "expectedIntent": "SendEmail",
        "predictedIntent": "SendEmail"
      }
    },
    {
      "text": "send thomas an email",
      "language": "en-us",
      "entitiesResult": {
        "expectedEntities": [
          {
            "category": "ContactName",
            "offset": 5,
            "length": 6
          }
        ],
        "predictedEntities": [
          {
            "category": "ContactName",
            "offset": 5,
            "length": 6
          }
        ]
      },
      "intentsResult": {
        "expectedIntent": "SendEmail",
        "predictedIntent": "SendEmail"
      }
    },
    {
      "text": "i need to add more to the email message i am sending to vincent",
      "language": "en-us",
      "entitiesResult": {
        "expectedEntities": [
          {
            "category": "ContactName",
            "offset": 56,
            "length": 7
          }
        ],
        "predictedEntities": [
          {
            "category": "ContactName",
            "offset": 56,
            "length": 7
          }
        ]
      },
      "intentsResult": {
        "expectedIntent": "AddMore",
        "predictedIntent": "AddMore"
      }
    },
    {
      "text": "send an email to lily roth and abc123@microsoft.com",
      "language": "en-us",
      "entitiesResult": {
        "expectedEntities": [
          {
            "category": "ContactName",
            "offset": 17,
            "length": 9
          }
        ],
        "predictedEntities": [
          {
            "category": "ContactName",
            "offset": 17,
            "length": 9
          }
        ]
      },
      "intentsResult": {
        "expectedIntent": "SendEmail",
        "predictedIntent": "SendEmail"
      }
    },
    {
      "text": "i need to add something else to my email to cheryl",
      "language": "en-us",
      "entitiesResult": {
        "expectedEntities": [
          {
            "category": "ContactName",
            "offset": 44,
            "length": 6
          }
        ],
        "predictedEntities": [
          {
            "category": "ContactName",
            "offset": 44,
            "length": 6
          }
        ]
      },
      "intentsResult": {
        "expectedIntent": "AddMore",
        "predictedIntent": "AddMore"
      }
    },
    {
      "text": "send an email to larry , joseph and billy larkson",
      "language": "en-us",
      "entitiesResult": {
        "expectedEntities": [
          {
            "category": "ContactName",
            "offset": 17,
            "length": 5
          },
          {
            "category": "ContactName",
            "offset": 25,
            "length": 6
          },
          {
            "category": "ContactName",
            "offset": 36,
            "length": 13
          }
        ],
        "predictedEntities": [
          {
            "category": "ContactName",
            "offset": 17,
            "length": 5
          },
          {
            "category": "ContactName",
            "offset": 25,
            "length": 6
          },
          {
            "category": "ContactName",
            "offset": 36,
            "length": 13
          }
        ]
      },
      "intentsResult": {
        "expectedIntent": "SendEmail",
        "predictedIntent": "SendEmail"
      }
    },
    {
      "text": "send mail to dorothy",
      "language": "en-us",
      "entitiesResult": {
        "expectedEntities": [
          {
            "category": "ContactName",
            "offset": 13,
            "length": 7
          }
        ],
        "predictedEntities": [
          {
            "category": "ContactName",
            "offset": 13,
            "length": 7
          }
        ]
      },
      "intentsResult": {
        "expectedIntent": "SendEmail",
        "predictedIntent": "SendEmail"
      }
    }
  ],
  "nextLink": "{Endpoint}/language/authoring/analyze-conversations/projects/{projectName}/models/{trainedModelLabel}/evaluation/result/?api-version=2022-10-01-preview&top={top}&skip={skip}&maxpagesize={maxpagesize}"
}

Load or export model data

Language studio

To load your model data:

1. Select any model in the model evaluation page.

2. Select the Load model data button.

3. Confirm that you do not have any unsaved changes you need to capture in window that appears, and select Load data.

4. Wait until your model data finishes loading back into your project. On completion, you'll be redirected back to the Schema design page.

To export your model data:

1. Select any model in the model evaluation page.

2. Select the Export model data button. Wait for the JSON snapshot of your model to be downloaded locally.

REST APIs

Load model data

Create a POST request using the following URL, headers, and JSON body to load your model data to your project.

Request URL

Use the following URL when creating your API request. Replace the placeholder values below with your own values.

{ENDPOINT}/language/authoring/analyze-conversations/projects/{PROJECT-NAME}/models/{MODEL-NAME}:load-snapshot?stringIndexType=Utf16CodeUnit&api-version={API-VERSION}

Placeholder	Value	Example
{ENDPOINT}	The endpoint for authenticating your API request.	https://<your-custom-subdomain>.cognitiveservices.azure.com
{PROJECT-NAME}	The name for your project. This value is case-sensitive.	EmailApp
{API-VERSION}	The version of the API you are calling.	2022-10-01-preview
{MODEL-NAME}	The name of your model. This value is case-sensitive.	v1

Headers

Use the following header to authenticate your request.

Key	Value
Ocp-Apim-Subscription-Key	The key to your resource. Used for authenticating your API requests.

Once you send your API request, you will receive a 202 response indicating success. In the response headers, extract the operation-location value. It will be formatted like this:

{ENDPOINT}/language/authoring/analyze-conversations/projects/{PROJECT-NAME}/models/{MODEL-NAME}/jobs/{JOB-ID}?api-version={API-VERSION}

JOB-ID is used to identify your request, since this operation is asynchronous. Use this URL to get the status of your model data loading, using the same authentication method.

Export model data

Create a POST request using the following URL, headers, and JSON body to export your model data.

Request URL

Use the following URL when creating your API request. Replace the placeholder values below with your own values.

{ENDPOINT}/language/authoring/analyze-conversations/projects/{PROJECT-NAME}/:export?stringIndexType=Utf16CodeUnit&api-version={API-VERSION}&trainedModelLabel={MODEL-NAME}

Placeholder	Value	Example
{ENDPOINT}	The endpoint for authenticating your API request.	https://<your-custom-subdomain>.cognitiveservices.azure.com
{PROJECT-NAME}	The name for your project. This value is case-sensitive.	EmailApp
{API-VERSION}	The version of the API you are calling.	2022-10-01-preview
{MODEL-NAME}	The name of your model. This value is case-sensitive.	v1

Headers

Use the following header to authenticate your request.

Key	Value
Ocp-Apim-Subscription-Key	The key to your resource. Used for authenticating your API requests.

Once you send your API request, you will receive a 202 response indicating success. In the response headers, extract the operation-location value. It will be formatted like this:

{ENDPOINT}/language/authoring/analyze-conversations/projects/{PROJECT-NAME}/jobs/{JOB-ID}?api-version={API-VERSION}

JOB-ID is used to identify your request, since this operation is asynchronous. Use this URL to get the exported project JSON, using the same authentication method.

Delete model

Language studio

To delete your model from within the Language Studio:

1. Select Model performance from the left side menu.

2. Select the model name you want to delete and select Delete from the top menu.

3. In the window that appears, select OK to delete the model.

REST APIs

Create a DELETE request using the following URL, headers, and JSON body to delete a model.

Request URL

{ENDPOINT}/language/authoring/analyze-conversations/projects/{projectName}/models/{trainedModelLabel}?api-version={API-VERSION}

Placeholder	Value	Example
{YOUR-ENDPOINT}	The endpoint for authenticating your API request.	https://<your-custom-subdomain>.cognitiveservices.azure.com
{PROJECT-NAME}	The name for your project. This value is case-sensitive.	myProject
{trainedModelLabel}	The name for your model name. This value is case-sensitive.	model1
{API-VERSION}	The version of the API you are calling.	2023-04-01

Headers

Use the following header to authenticate your request.

Key	Value
Ocp-Apim-Subscription-Key	The key to your resource. Used for authenticating your API requests.

Once you send your API request, you will receive a 204 response indicating success, which means your model has been deleted.

Next steps

• As you review your how your model performs, learn about the evaluation metrics that are used.
• If you're happy with your model performance, you can deploy your model