View the custom NER model's evaluation and details

After your model has finished training, you can view the model performance and see the extracted entities for the documents in the test set.

Note

Using the Automatically split the testing set from training data option may result in different model evaluation result every time you train a new model, as the test set is selected randomly from the data. To make sure that the evaulation is calcualted on the same test set every time you train a model, make sure to use the Use a manual split of training and testing data option when starting a training job and define your Test documents when labeling data.

Prerequisites

Before viewing model evaluation, you need:

• A successfully created project with a configured Azure blob storage account.
• Text data that has been uploaded to your storage account.
• Labeled data
• A successfully trained model

See the project development lifecycle for more information.

Model details

Language studio

1. Go to your project page in Language Studio.

2. Select Model performance from the menu on the left side of the screen.

3. In this page you can only view the successfully trained models, F1 score for each model and model expiration date. You can select the model name for more details about its performance.

Note

Entities that are neither labeled nor predicted in the test set will not be part of the displayed results.

Overview

• In this tab you can view the model's details such as: F1 score, precision, recall, date and time for the training job, total training time and number of training and testing documents included in this training job.

  

• You will also see guidance on how to improve the model. When clicking on view details a side panel will open to give more guidance on how to improve the model. In this example, BorrowerAddress and BorrowerName entities are confused with $none entity. By clicking on the confused entities, you will be taken to the data labeling page to label more data with the correct entity.

  

  Learn more about model guidance and confusion matrix in model performance concepts.

Entity type performance

• This is a snapshot of how your model performed during testing. The metrics here are static and tied to your model, so they won’t update until you train again.

• You can see for each entity, precision, recall, F1 score, number of training and testing labels.

  

Test set details

• Here you will see the documents included in the test set and the result entity type for each document. You can use the Show mismatches only toggle to show only documents with mismatches, or unselect the toggle to view all document in the test set.

• For each document, you can view: labeled text, its respective labeled entity type and what entity it was predicted with. Also, you will see whether it is a true positive, false positive or false negative.

  

Dataset distribution

This snapshot shows how entities are distributed across your training and testing sets. This data is static and tied to your model, so it won’t update until you train again.

• You can view the dataset distribution in graph or table view.

Graph view

• Documents with at least one label: This view will show for each entity, the number of occurrences for this entity across the training and testing sets.

• Total instances throughout documents: This view will show for each entity, the labeled occurrences across training and testing sets.

  

Table view

For each entity, you can view: tags per entity in training set, tagged documents in training set, tags per entity in testing set, tagged documents in testing set, tags per entity total and tagged documents total.

Confusion matrix

A confusion matrix is an N x N matrix used for evaluating the performance of a classification model, where N is the number of target entities. The matrix compares the actual target values with those predicted by the machine learning model to show how well the extraction model is performing and what kinds of errors it is making.

You can view the confusion matrix in normalized or raw count view.

• All values: Will show the confusion matrix for all entities.

• Only errors: Will show the confusion matrix for entities with errors only.

• Only matches: Will show the confusion matrix for entities with correct predictions only.

REST APIs

Submit a GET request using the following URL, headers, and JSON body to get trained model evaluation summary.

Request URL

{ENDPOINT}/language/authoring/analyze-text/projects/{PROJECT-NAME}/models/{trainedModelLabel}/evaluation/summary-result?api-version={API-VERSION}

Placeholder	Value	Example
{ENDPOINT}	The endpoint for authenticating your API request.	https://<your-custom-subdomain>.cognitiveservices.azure.com
{PROJECT-NAME}	The name for your project. This value is case-sensitive.	myProject
{trainedModelLabel}	The name for your trained model. This value is case-sensitive.	Model1
{API-VERSION}	The version of the API you are calling. See Model lifecycle to learn more about other available API versions.	2022-05-01

Headers

Use the following header to authenticate your request.

Key	Value
Ocp-Apim-Subscription-Key	The key to your resource. Used for authenticating your API requests.

Response Body

Once you send the request, you will get the following response.

{
  "projectKind": "CustomEntityRecognition",
  "customEntityRecognitionEvaluation": {
    "confusionMatrix": {
      "additionalProp1": {
        "additionalProp1": {
          "normalizedValue": 0,
          "rawValue": 0
        },
        "additionalProp2": {
          "normalizedValue": 0,
          "rawValue": 0
        },
        "additionalProp3": {
          "normalizedValue": 0,
          "rawValue": 0
        }
      },
      "additionalProp2": {
        "additionalProp1": {
          "normalizedValue": 0,
          "rawValue": 0
        },
        "additionalProp2": {
          "normalizedValue": 0,
          "rawValue": 0
        },
        "additionalProp3": {
          "normalizedValue": 0,
          "rawValue": 0
        }
      },
      "additionalProp3": {
        "additionalProp1": {
          "normalizedValue": 0,
          "rawValue": 0
        },
        "additionalProp2": {
          "normalizedValue": 0,
          "rawValue": 0
        },
        "additionalProp3": {
          "normalizedValue": 0,
          "rawValue": 0
        }
      }
    },
    "entities": {
      "additionalProp1": {
        "f1": 0,
        "precision": 0,
        "recall": 0,
        "truePositivesCount": 0,
        "trueNegativesCount": 0,
        "falsePositivesCount": 0,
        "falseNegativesCount": 0
      },
      "additionalProp2": {
        "f1": 0,
        "precision": 0,
        "recall": 0,
        "truePositivesCount": 0,
        "trueNegativesCount": 0,
        "falsePositivesCount": 0,
        "falseNegativesCount": 0
      },
      "additionalProp3": {
        "f1": 0,
        "precision": 0,
        "recall": 0,
        "truePositivesCount": 0,
        "trueNegativesCount": 0,
        "falsePositivesCount": 0,
        "falseNegativesCount": 0
      }
    },
    "microF1": 0,
    "microPrecision": 0,
    "microRecall": 0,
    "macroF1": 0,
    "macroPrecision": 0,
    "macroRecall": 0
  },
  "evaluationOptions": {
    "kind": "percentage",
    "trainingSplitPercentage": 0,
    "testingSplitPercentage": 0
  }
}

Load or export model data

Language studio

To load your model data:

1. Select any model in the model evaluation page.

2. Select the Load model data button.

3. Confirm that you do not have any unsaved changes you need to capture in window that appears, and select Load data.

4. Wait until your model data finishes loading back into your project. On completion, you'll be redirected back to the Schema design page.

To export your model data:

1. Select any model in the model evaluation page.

2. Select the Export model data button. Wait for the JSON snapshot of your model to be downloaded locally.

REST APIs

Load model data

Create a POST request using the following URL, headers, and JSON body to load your model data to your project.

Request URL

Use the following URL when creating your API request. Replace the placeholder values below with your own values.

{ENDPOINT}/language/authoring/analyze-text/projects/{PROJECT-NAME}/models/{MODEL-NAME}:load-snapshot?stringIndexType=Utf16CodeUnit&api-version={API-VERSION}

Placeholder	Value	Example
{ENDPOINT}	The endpoint for authenticating your API request.	https://<your-custom-subdomain>.cognitiveservices.azure.com
{PROJECT-NAME}	The name for your project. This value is case-sensitive.	EmailApp
{API-VERSION}	The version of the API you are calling.	2022-10-01-preview
{MODEL-NAME}	The name of your model. This value is case-sensitive.	v1

Headers

Use the following header to authenticate your request.

Key	Value
Ocp-Apim-Subscription-Key	The key to your resource. Used for authenticating your API requests.

Once you send your API request, you will receive a 202 response indicating success. In the response headers, extract the operation-location value. It will be formatted like this:

{ENDPOINT}/language/authoring/analyze-text/projects/{PROJECT-NAME}/models/{MODEL-NAME}/jobs/{JOB-ID}?api-version={API-VERSION}

JOB-ID is used to identify your request, since this operation is asynchronous. Use this URL to get the status of your model data loading, using the same authentication method.

Export model data

Create a POST request using the following URL, headers, and JSON body to export your model data.

Request URL

Use the following URL when creating your API request. Replace the placeholder values below with your own values.

{ENDPOINT}/language/authoring/analyze-text/projects/{PROJECT-NAME}/:export?stringIndexType=Utf16CodeUnit&api-version={API-VERSION}&trainedModelLabel={MODEL-NAME}

Placeholder	Value	Example
{ENDPOINT}	The endpoint for authenticating your API request.	https://<your-custom-subdomain>.cognitiveservices.azure.com
{PROJECT-NAME}	The name for your project. This value is case-sensitive.	EmailApp
{API-VERSION}	The version of the API you are calling.	2022-10-01-preview
{MODEL-NAME}	The name of your model. This value is case-sensitive.	v1

Headers

Use the following header to authenticate your request.

Key	Value
Ocp-Apim-Subscription-Key	The key to your resource. Used for authenticating your API requests.

Once you send your API request, you will receive a 202 response indicating success. In the response headers, extract the operation-location value. It will be formatted like this:

{ENDPOINT}/language/authoring/analyze-text/projects/{PROJECT-NAME}/jobs/{JOB-ID}?api-version={API-VERSION}

JOB-ID is used to identify your request, since this operation is asynchronous. Use this URL to get the exported project JSON, using the same authentication method.

Delete model

Language studio

To delete your model from within the Language Studio:

1. Select Model performance from the left side menu.

2. Select the model name you want to delete and select Delete from the top menu.

3. In the window that appears, select OK to delete the model.

REST APIs

Create a DELETE request using the following URL, headers, and JSON body to delete a trained model.

Request URL

{Endpoint}/language/authoring/analyze-text/projects/{PROJECT-NAME}/models/{trainedModelLabel}?api-version={API-VERSION}

Placeholder	Value	Example
{ENDPOINT}	The endpoint for authenticating your API request.	https://<your-custom-subdomain>.cognitiveservices.azure.com
{PROJECT-NAME}	The name for your project. This value is case-sensitive.	myProject
{trainedModelLabel}	The name for your model name. This value is case-sensitive.	model1
{API-VERSION}	The version of the API you are calling. The value referenced here is for the latest version released. See Model lifecycle to learn more about other available API versions.	2022-05-01

Headers

Use the following header to authenticate your request.

Key	Value
Ocp-Apim-Subscription-Key	The key to your resource. Used for authenticating your API requests.

Once you send your API request, you will receive a 204 response indicating success, which means your trained model has been deleted.

Next steps

• Deploy your model
• Learn about the metrics used in evaluation.