Text Analysis Authoring - Get Model Evaluation Summary

Service:

Azure AI Services

API Version:

2023-04-01

Gets the evaluation summary of a trained model. The summary includes high level performance measurements of the model e.g., F1, Precision, Recall, etc.

GET {Endpoint}/language/authoring/analyze-text/projects/{projectName}/models/{trainedModelLabel}/evaluation/summary-result?api-version=2023-04-01

URI Parameters

Name	In	Required	Type	Description
Endpoint	path	True	string url	Supported Cognitive Services endpoint (e.g., https://.api.cognitiveservices.azure.com).
projectName	path	True	string	The name of the project to use.
trainedModelLabel	path	True	string	The trained model label.
api-version	query	True	string	The API version to use for this operation.

Responses

Name	Type	Description
200 OK	TextAnalysisAuthoringEvaluationSummary: TextAnalysisAuthoringCustomEntityRecognitionEvaluationSummary TextAnalysisAuthoringCustomMultiLabelClassificationEvaluationSummary TextAnalysisAuthoringCustomSingleLabelClassificationEvaluationSummary	The request has succeeded.
Other Status Codes	ErrorResponse	An unexpected error response.

Security

Ocp-Apim-Subscription-Key

Type: apiKey
In: header

OAuth2Auth

Type: oauth2
Flow: accessCode
Authorization URL: https://login.microsoftonline.com/common/oauth2/authorize
Token URL: https://login.microsoftonline.com/common/oauth2/token

Scopes

Name	Description
https://cognitiveservices.azure.com/.default

Examples

SuccessfulGetModelEvaluationSummary

Sample request

HTTP

GET {Endpoint}/language/authoring/analyze-text/projects/LoanAgreements/models/model2/evaluation/summary-result?api-version=2023-04-01

Sample response

Status code:

200

{
  "projectKind": "CustomEntityRecognition",
  "customEntityRecognitionEvaluation": {
    "confusionMatrix": {
      "BorrowerAddress": {
        "BorrowerAddress": {
          "normalizedValue": 86.206894,
          "rawValue": 3.4482758
        },
        "$none": {
          "normalizedValue": 13.793103,
          "rawValue": 0.55172414
        }
      },
      "BorrowerCity": {
        "BorrowerCity": {
          "normalizedValue": 100,
          "rawValue": 4
        }
      },
      "BorrowerName": {
        "BorrowerName": {
          "normalizedValue": 100,
          "rawValue": 4
        }
      },
      "BorrowerState": {
        "BorrowerState": {
          "normalizedValue": 100,
          "rawValue": 4
        }
      },
      "Date": {
        "Date": {
          "normalizedValue": 100,
          "rawValue": 4
        }
      },
      "Interest": {
        "Interest": {
          "normalizedValue": 100,
          "rawValue": 4
        }
      },
      "LenderAddress": {
        "LenderAddress": {
          "normalizedValue": 100,
          "rawValue": 4
        }
      },
      "LenderCity": {
        "LenderCity": {
          "normalizedValue": 100,
          "rawValue": 4
        }
      },
      "LenderName": {
        "LenderName": {
          "normalizedValue": 100,
          "rawValue": 4
        }
      },
      "LenderState": {
        "LenderState": {
          "normalizedValue": 100,
          "rawValue": 4
        }
      },
      "LoanAmountNumbers": {
        "LoanAmountNumbers": {
          "normalizedValue": 100,
          "rawValue": 4
        }
      },
      "LoanAmountWords": {
        "LoanAmountWords": {
          "normalizedValue": 100,
          "rawValue": 4
        }
      },
      "$none": {
        "$none": {
          "normalizedValue": 99.81485,
          "rawValue": 51.90372
        },
        "BorrowerAddress": {
          "normalizedValue": 0.18315019,
          "rawValue": 0.0952381
        },
        "Interest": {
          "normalizedValue": 0.002005294,
          "rawValue": 0.0010427529
        }
      }
    },
    "entities": {
      "Date": {
        "f1": 1,
        "precision": 1,
        "recall": 1,
        "truePositiveCount": 4,
        "trueNegativeCount": 0,
        "falsePositiveCount": 0,
        "falseNegativeCount": 0
      },
      "BorrowerName": {
        "f1": 1,
        "precision": 1,
        "recall": 1,
        "truePositiveCount": 4,
        "trueNegativeCount": 0,
        "falsePositiveCount": 0,
        "falseNegativeCount": 0
      },
      "BorrowerAddress": {
        "f1": 0.6666666865348816,
        "precision": 0.6000000238418579,
        "recall": 0.75,
        "truePositiveCount": 3,
        "trueNegativeCount": 0,
        "falsePositiveCount": 2,
        "falseNegativeCount": 1
      },
      "BorrowerCity": {
        "f1": 1,
        "precision": 1,
        "recall": 1,
        "truePositiveCount": 4,
        "trueNegativeCount": 0,
        "falsePositiveCount": 0,
        "falseNegativeCount": 0
      },
      "BorrowerState": {
        "f1": 1,
        "precision": 1,
        "recall": 1,
        "truePositiveCount": 4,
        "trueNegativeCount": 0,
        "falsePositiveCount": 0,
        "falseNegativeCount": 0
      },
      "LenderName": {
        "f1": 1,
        "precision": 1,
        "recall": 1,
        "truePositiveCount": 4,
        "trueNegativeCount": 0,
        "falsePositiveCount": 0,
        "falseNegativeCount": 0
      },
      "LenderAddress": {
        "f1": 1,
        "precision": 1,
        "recall": 1,
        "truePositiveCount": 4,
        "trueNegativeCount": 0,
        "falsePositiveCount": 0,
        "falseNegativeCount": 0
      },
      "LenderCity": {
        "f1": 1,
        "precision": 1,
        "recall": 1,
        "truePositiveCount": 4,
        "trueNegativeCount": 0,
        "falsePositiveCount": 0,
        "falseNegativeCount": 0
      },
      "LenderState": {
        "f1": 1,
        "precision": 1,
        "recall": 1,
        "truePositiveCount": 4,
        "trueNegativeCount": 0,
        "falsePositiveCount": 0,
        "falseNegativeCount": 0
      },
      "LoanAmountWords": {
        "f1": 1,
        "precision": 1,
        "recall": 1,
        "truePositiveCount": 4,
        "trueNegativeCount": 0,
        "falsePositiveCount": 0,
        "falseNegativeCount": 0
      },
      "LoanAmountNumbers": {
        "f1": 1,
        "precision": 1,
        "recall": 1,
        "truePositiveCount": 4,
        "trueNegativeCount": 0,
        "falsePositiveCount": 0,
        "falseNegativeCount": 0
      },
      "Interest": {
        "f1": 0.75,
        "precision": 0.75,
        "recall": 0.75,
        "truePositiveCount": 3,
        "trueNegativeCount": 0,
        "falsePositiveCount": 1,
        "falseNegativeCount": 1
      }
    },
    "microF1": 0.94845366,
    "microPrecision": 0.93877554,
    "microRecall": 0.9583333,
    "macroF1": 0.9513889,
    "macroPrecision": 0.9458334,
    "macroRecall": 0.9583333
  },
  "evaluationOptions": {
    "kind": "percentage",
    "trainingSplitPercentage": 80,
    "testingSplitPercentage": 20
  }
}

Definitions

Name	Description
Error	The error object.
ErrorCode	Human-readable error code.
ErrorResponse	Error response.
EvaluationKind
EvaluationOptions	Represents the options used running the evaluation.
InnerErrorCode	Human-readable error code.
InnerErrorModel	An object containing more specific information about the error. As per Microsoft One API guidelines - https://github.com/Microsoft/api-guidelines/blob/vNext/Guidelines.md#7102-error-condition-responses.
ProjectKind
TextAnalysisAuthoringCustomEntityRecognitionEvaluationSummary	Represents the evaluation summary for a custom entity recognition project.
TextAnalysisAuthoringCustomMultiLabelClassificationEvaluationSummary	Represents the evaluation summary for a custom multi-label classification project.
TextAnalysisAuthoringCustomSingleLabelClassificationEvaluationSummary	Represents the evaluation summary for a custom single-label classification project.
TextAnalysisAuthoringEntityEvaluationSummary	Represents the evaluation summary for an entity.
TextAnalysisAuthoringEntityRecognitionEvaluationSummary	Represents the evaluation summary for a custom entity recognition project.
TextAnalysisAuthoringMultiLabelClassEvaluationSummary	Represents the evaluation summary of a class in a multi-label classification project.
TextAnalysisAuthoringMultiLabelClassificationEvaluationSummary	Represents the evaluation summary for a multi-label classification project.
TextAnalysisAuthoringSingleLabelClassEvaluationSummary	Represents the evaluation summary for a class in a single-label classification project.
TextAnalysisAuthoringSingleLabelClassificationEvaluationSummary	Represents the evaluation summary for a custom single-label classification project.

Error

The error object.

Name	Type	Description
code	ErrorCode	One of a server-defined set of error codes.
details	Error[]	An array of details about specific errors that led to this reported error.
innererror	InnerErrorModel	An object containing more specific information than the current object about the error.
message	string	A human-readable representation of the error.
target	string	The target of the error.

ErrorCode

Human-readable error code.

Name	Type	Description
AzureCognitiveSearchIndexLimitReached	string
AzureCognitiveSearchIndexNotFound	string
AzureCognitiveSearchNotFound	string
AzureCognitiveSearchThrottling	string
Conflict	string
Forbidden	string
InternalServerError	string
InvalidArgument	string
InvalidRequest	string
NotFound	string
OperationNotFound	string
ProjectNotFound	string
QuotaExceeded	string
ServiceUnavailable	string
Timeout	string
TooManyRequests	string
Unauthorized	string
Warning	string

ErrorResponse

Error response.

Name	Type	Description
error	Error	The error object.

EvaluationKind

Name	Type	Description
manual	string	Split the data according to the chosen dataset for every example in the data.
percentage	string	Split the data into training and test sets according to user-defined percentages.

EvaluationOptions

Represents the options used running the evaluation.

Name	Type	Description
kind	EvaluationKind	Represents the evaluation kind. By default, the evaluation kind is set to percentage.
testingSplitPercentage	integer	Represents the testing dataset split percentage. Only needed in case the evaluation kind is percentage.
trainingSplitPercentage	integer	Represents the training dataset split percentage. Only needed in case the evaluation kind is percentage.

InnerErrorCode

Human-readable error code.

Name	Type	Description
AzureCognitiveSearchNotFound	string
AzureCognitiveSearchThrottling	string
EmptyRequest	string
ExtractionFailure	string
InvalidCountryHint	string
InvalidDocument	string
InvalidDocumentBatch	string
InvalidParameterValue	string
InvalidRequest	string
InvalidRequestBodyFormat	string
KnowledgeBaseNotFound	string
MissingInputDocuments	string
ModelVersionIncorrect	string
UnsupportedLanguageCode	string

InnerErrorModel

An object containing more specific information about the error. As per Microsoft One API guidelines - https://github.com/Microsoft/api-guidelines/blob/vNext/Guidelines.md#7102-error-condition-responses.

Name	Type	Description
code	InnerErrorCode	One of a server-defined set of error codes.
details	object	Error details.
innererror	InnerErrorModel	An object containing more specific information than the current object about the error.
message	string	Error message.
target	string	Error target.

ProjectKind

Name	Type	Description
CustomEntityRecognition	string	For building an extraction model to identify your domain categories using your own data.
CustomMultiLabelClassification	string	For building a classification model to classify text using your own data. Each file can have one or many labels. For example, file 1 is classified as A, B, and C and file 2 is classified as B and C.
CustomSingleLabelClassification	string	For building a classification model to classify text using your own data. Each file will have only one label. For example, file 1 is classified as A and file 2 is classified as B.

TextAnalysisAuthoringCustomEntityRecognitionEvaluationSummary

Represents the evaluation summary for a custom entity recognition project.

Name	Type	Description
customEntityRecognitionEvaluation	TextAnalysisAuthoringEntityRecognitionEvaluationSummary	Contains the data related to extraction evaluation.
evaluationOptions	EvaluationOptions	Represents the options used running the evaluation.
projectKind	string: CustomEntityRecognition	Represents the project type that the evaluation ran on.

TextAnalysisAuthoringCustomMultiLabelClassificationEvaluationSummary

Represents the evaluation summary for a custom multi-label classification project.

Name	Type	Description
customMultiLabelClassificationEvaluation	TextAnalysisAuthoringMultiLabelClassificationEvaluationSummary	Contains the data related to multi label classification evaluation.
evaluationOptions	EvaluationOptions	Represents the options used running the evaluation.
projectKind	string: CustomMultiLabelClassification	Represents the project type that the evaluation ran on.

TextAnalysisAuthoringCustomSingleLabelClassificationEvaluationSummary

Represents the evaluation summary for a custom single-label classification project.

Name	Type	Description
customSingleLabelClassificationEvaluation	TextAnalysisAuthoringSingleLabelClassificationEvaluationSummary	Contains the data related to single label classification evaluation.
evaluationOptions	EvaluationOptions	Represents the options used running the evaluation.
projectKind	string: CustomSingleLabelClassification	Represents the project type that the evaluation ran on.

TextAnalysisAuthoringEntityEvaluationSummary

Represents the evaluation summary for an entity.

Name	Type	Description
f1	number	Represents the model precision
falseNegativeCount	integer	Represents the count of false negative
falsePositiveCount	integer	Represents the count of false positive
precision	number	Represents the model recall
recall	number	Represents the model F1 score
trueNegativeCount	integer	Represents the count of true negative
truePositiveCount	integer	Represents the count of true positive

TextAnalysisAuthoringEntityRecognitionEvaluationSummary

Represents the evaluation summary for a custom entity recognition project.

Name	Type	Description
confusionMatrix	object	Represents the confusion matrix between two entities (the two entities can be the same). The matrix is between the entity that was labelled and the entity that was predicted.
entities	<string,  TextAnalysisAuthoringEntityEvaluationSummary>	Represents the entities evaluation
macroF1	number	Represents the macro F1. Expected value is a float between 0 and 1 inclusive.
macroPrecision	number	Represents the macro precision. Expected value is a float between 0 and 1 inclusive.
macroRecall	number	Represents the macro recall. Expected value is a float between 0 and 1 inclusive.
microF1	number	Represents the micro F1. Expected value is a float between 0 and 1 inclusive.
microPrecision	number	Represents the micro precision. Expected value is a float between 0 and 1 inclusive.
microRecall	number	Represents the micro recall. Expected value is a float between 0 and 1 inclusive.

TextAnalysisAuthoringMultiLabelClassEvaluationSummary

Represents the evaluation summary of a class in a multi-label classification project.

Name	Type	Description
f1	number	Represents the model precision
falseNegativeCount	integer	Represents the count of false negative
falsePositiveCount	integer	Represents the count of false positive
precision	number	Represents the model recall
recall	number	Represents the model F1 score
trueNegativeCount	integer	Represents the count of true negative
truePositiveCount	integer	Represents the count of true positive

TextAnalysisAuthoringMultiLabelClassificationEvaluationSummary

Represents the evaluation summary for a multi-label classification project.

Name	Type	Description
classes	<string,  TextAnalysisAuthoringMultiLabelClassEvaluationSummary>	Represents the classes evaluation
macroF1	number	Represents the macro F1. Expected value is a float between 0 and 1 inclusive.
macroPrecision	number	Represents the macro precision. Expected value is a float between 0 and 1 inclusive.
macroRecall	number	Represents the macro recall. Expected value is a float between 0 and 1 inclusive.
microF1	number	Represents the micro F1. Expected value is a float between 0 and 1 inclusive.
microPrecision	number	Represents the micro precision. Expected value is a float between 0 and 1 inclusive.
microRecall	number	Represents the micro recall. Expected value is a float between 0 and 1 inclusive.

TextAnalysisAuthoringSingleLabelClassEvaluationSummary

Represents the evaluation summary for a class in a single-label classification project.

Name	Type	Description
f1	number	Represents the model precision
falseNegativeCount	integer	Represents the count of false negative
falsePositiveCount	integer	Represents the count of false positive
precision	number	Represents the model recall
recall	number	Represents the model F1 score
trueNegativeCount	integer	Represents the count of true negative
truePositiveCount	integer	Represents the count of true positive

TextAnalysisAuthoringSingleLabelClassificationEvaluationSummary

Represents the evaluation summary for a custom single-label classification project.

Name	Type	Description
classes	<string,  TextAnalysisAuthoringSingleLabelClassEvaluationSummary>	Represents the classes evaluation
confusionMatrix	object	Represents the confusion matrix between two classes (the two classes can be the same). The matrix is between the class that was labelled and the class that was predicted.
macroF1	number	Represents the macro F1. Expected value is a float between 0 and 1 inclusive.
macroPrecision	number	Represents the macro precision. Expected value is a float between 0 and 1 inclusive.
macroRecall	number	Represents the macro recall. Expected value is a float between 0 and 1 inclusive.
microF1	number	Represents the micro F1. Expected value is a float between 0 and 1 inclusive.
microPrecision	number	Represents the micro precision. Expected value is a float between 0 and 1 inclusive.
microRecall	number	Represents the micro recall. Expected value is a float between 0 and 1 inclusive.