Text Analysis Authoring - Get Model Evaluation Summary

Service:

Azure AI Services

API Version:

2022-05-01

Gets the evaluation summary of a trained model. The summary includes high level performance measurements of the model e.g., F1, Precision, Recall, etc.

GET {Endpoint}/language/authoring/analyze-text/projects/{projectName}/models/{trainedModelLabel}/evaluation/summary-result?api-version=2022-05-01

URI Parameters

Name	In	Required	Type	Description
Endpoint	path	True	string	Supported Cognitive Services endpoint (e.g., https://.api.cognitiveservices.azure.com).
projectName	path	True	string maxLength: 100	The name of the project to use.
trainedModelLabel	path	True	string	The trained model label.
api-version	query	True	string	Client API version.

Request Header

Name	Required	Type	Description
Ocp-Apim-Subscription-Key	True	string	A subscription key for a Language service resource.

Responses

Name	Type	Description
200 OK	TextAnalysisAuthoringEvaluationSummary: TextAnalysisAuthoringCustomEntityRecognitionEvaluationSummary TextAnalysisAuthoringCustomMultiLabelClassificationEvaluationSummary TextAnalysisAuthoringCustomSingleLabelClassificationEvaluationSummary	List of all evaluation results.
Other Status Codes	ErrorResponse	Error response.

Security

Ocp-Apim-Subscription-Key

A subscription key for a Language service resource.

Type: apiKey
In: header

Examples

Successful Get Model Evaluation Summary

Sample request

HTTP

GET {Endpoint}/language/authoring/analyze-text/projects/LoanAgreements/models/model2/evaluation/summary-result?api-version=2022-05-01

Sample response

Status code:

200

{
  "projectKind": "CustomEntityRecognition",
  "customEntityRecognitionEvaluation": {
    "confusionMatrix": {
      "BorrowerAddress": {
        "BorrowerAddress": {
          "normalizedValue": 86.206894,
          "rawValue": 3.4482758
        },
        "$none": {
          "normalizedValue": 13.793103,
          "rawValue": 0.55172414
        }
      },
      "BorrowerCity": {
        "BorrowerCity": {
          "normalizedValue": 100,
          "rawValue": 4
        }
      },
      "BorrowerName": {
        "BorrowerName": {
          "normalizedValue": 100,
          "rawValue": 4
        }
      },
      "BorrowerState": {
        "BorrowerState": {
          "normalizedValue": 100,
          "rawValue": 4
        }
      },
      "Date": {
        "Date": {
          "normalizedValue": 100,
          "rawValue": 4
        }
      },
      "Interest": {
        "Interest": {
          "normalizedValue": 100,
          "rawValue": 4
        }
      },
      "LenderAddress": {
        "LenderAddress": {
          "normalizedValue": 100,
          "rawValue": 4
        }
      },
      "LenderCity": {
        "LenderCity": {
          "normalizedValue": 100,
          "rawValue": 4
        }
      },
      "LenderName": {
        "LenderName": {
          "normalizedValue": 100,
          "rawValue": 4
        }
      },
      "LenderState": {
        "LenderState": {
          "normalizedValue": 100,
          "rawValue": 4
        }
      },
      "LoanAmountNumbers": {
        "LoanAmountNumbers": {
          "normalizedValue": 100,
          "rawValue": 4
        }
      },
      "LoanAmountWords": {
        "LoanAmountWords": {
          "normalizedValue": 100,
          "rawValue": 4
        }
      },
      "$none": {
        "$none": {
          "normalizedValue": 99.81485,
          "rawValue": 51.90372
        },
        "BorrowerAddress": {
          "normalizedValue": 0.18315019,
          "rawValue": 0.0952381
        },
        "Interest": {
          "normalizedValue": 0.002005294,
          "rawValue": 0.0010427529
        }
      }
    },
    "entities": {
      "Date": {
        "f1": 1,
        "precision": 1,
        "recall": 1,
        "truePositiveCount": 4,
        "trueNegativeCount": 0,
        "falsePositiveCount": 0,
        "falseNegativeCount": 0
      },
      "BorrowerName": {
        "f1": 1,
        "precision": 1,
        "recall": 1,
        "truePositiveCount": 4,
        "trueNegativeCount": 0,
        "falsePositiveCount": 0,
        "falseNegativeCount": 0
      },
      "BorrowerAddress": {
        "f1": 0.6666666865348816,
        "precision": 0.6000000238418579,
        "recall": 0.75,
        "truePositiveCount": 3,
        "trueNegativeCount": 0,
        "falsePositiveCount": 2,
        "falseNegativeCount": 1
      },
      "BorrowerCity": {
        "f1": 1,
        "precision": 1,
        "recall": 1,
        "truePositiveCount": 4,
        "trueNegativeCount": 0,
        "falsePositiveCount": 0,
        "falseNegativeCount": 0
      },
      "BorrowerState": {
        "f1": 1,
        "precision": 1,
        "recall": 1,
        "truePositiveCount": 4,
        "trueNegativeCount": 0,
        "falsePositiveCount": 0,
        "falseNegativeCount": 0
      },
      "LenderName": {
        "f1": 1,
        "precision": 1,
        "recall": 1,
        "truePositiveCount": 4,
        "trueNegativeCount": 0,
        "falsePositiveCount": 0,
        "falseNegativeCount": 0
      },
      "LenderAddress": {
        "f1": 1,
        "precision": 1,
        "recall": 1,
        "truePositiveCount": 4,
        "trueNegativeCount": 0,
        "falsePositiveCount": 0,
        "falseNegativeCount": 0
      },
      "LenderCity": {
        "f1": 1,
        "precision": 1,
        "recall": 1,
        "truePositiveCount": 4,
        "trueNegativeCount": 0,
        "falsePositiveCount": 0,
        "falseNegativeCount": 0
      },
      "LenderState": {
        "f1": 1,
        "precision": 1,
        "recall": 1,
        "truePositiveCount": 4,
        "trueNegativeCount": 0,
        "falsePositiveCount": 0,
        "falseNegativeCount": 0
      },
      "LoanAmountWords": {
        "f1": 1,
        "precision": 1,
        "recall": 1,
        "truePositiveCount": 4,
        "trueNegativeCount": 0,
        "falsePositiveCount": 0,
        "falseNegativeCount": 0
      },
      "LoanAmountNumbers": {
        "f1": 1,
        "precision": 1,
        "recall": 1,
        "truePositiveCount": 4,
        "trueNegativeCount": 0,
        "falsePositiveCount": 0,
        "falseNegativeCount": 0
      },
      "Interest": {
        "f1": 0.75,
        "precision": 0.75,
        "recall": 0.75,
        "truePositiveCount": 3,
        "trueNegativeCount": 0,
        "falsePositiveCount": 1,
        "falseNegativeCount": 1
      }
    },
    "microF1": 0.94845366,
    "microPrecision": 0.93877554,
    "microRecall": 0.9583333,
    "macroF1": 0.9513889,
    "macroPrecision": 0.9458334,
    "macroRecall": 0.9583333
  },
  "evaluationOptions": {
    "kind": "percentage",
    "trainingSplitPercentage": 80,
    "testingSplitPercentage": 20
  }
}

Definitions

Name	Description
Error	The error object.
ErrorCode	Human-readable error code.
ErrorResponse	Error response.
EvaluationKind	Represents the evaluation kind. By default, the evaluation kind is set to percentage.
InnerErrorCode	Human-readable error code.
InnerErrorModel	An object containing more specific information about the error. As per Microsoft One API guidelines - https://github.com/Microsoft/api-guidelines/blob/vNext/Guidelines.md#7102-error-condition-responses.
ProjectKind	The project kind.
TextAnalysisAuthoringCustomEntityRecognitionEvaluationSummary	Represents the evaluation summary for a custom entity recognition project.
TextAnalysisAuthoringCustomMultiLabelClassificationEvaluationSummary	Represents the evaluation summary for a custom multi-label classification project.
TextAnalysisAuthoringCustomSingleLabelClassificationEvaluationSummary	Represents the evaluation summary for a custom single-label classification project.
TextAnalysisAuthoringEntityEvaluationSummary	Represents the evaluation summary for an entity.
TextAnalysisAuthoringEntityRecognitionEvaluationSummary	Represents the evaluation summary for a custom entity recognition project.
TextAnalysisAuthoringEvaluationOptions	Represents the options used running the evaluation.
TextAnalysisAuthoringMultiLabelClassEvaluationSummary	Represents the evaluation summary of a class in a multi-label classification project.
TextAnalysisAuthoringMultiLabelClassificationEvaluationSummary	Represents the evaluation summary for a multi-label classification project.
TextAnalysisAuthoringSingleLabelClassEvaluationSummary	Represents the evaluation summary for a class in a single-label classification project.
TextAnalysisAuthoringSingleLabelClassificationEvaluationSummary	Represents the evaluation summary for a custom single-label classification project.

Error

Object

The error object.

Name	Type	Description
code	ErrorCode	One of a server-defined set of error codes.
details	Error[]	An array of details about specific errors that led to this reported error.
innererror	InnerErrorModel	An object containing more specific information than the current object about the error.
message	string	A human-readable representation of the error.
target	string	The target of the error.

ErrorCode

Enumeration

Human-readable error code.

Value	Description
AzureCognitiveSearchIndexLimitReached
AzureCognitiveSearchIndexNotFound
AzureCognitiveSearchNotFound
AzureCognitiveSearchThrottling
Conflict
Forbidden
InternalServerError
InvalidArgument
InvalidRequest
NotFound
OperationNotFound
ProjectNotFound
QuotaExceeded
ServiceUnavailable
Timeout
TooManyRequests
Unauthorized
Warning

ErrorResponse

Object

Error response.

Name	Type	Description
error	Error	The error object.

EvaluationKind

Enumeration

Represents the evaluation kind. By default, the evaluation kind is set to percentage.

Value	Description
manual	Split the data according to the chosen dataset for every example in the data.
percentage	Split the data into training and test sets according to user-defined percentages.

InnerErrorCode

Enumeration

Human-readable error code.

Value	Description
AzureCognitiveSearchNotFound
AzureCognitiveSearchThrottling
EmptyRequest
ExtractionFailure
InvalidCountryHint
InvalidDocument
InvalidDocumentBatch
InvalidParameterValue
InvalidRequest
InvalidRequestBodyFormat
KnowledgeBaseNotFound
MissingInputDocuments
ModelVersionIncorrect
UnsupportedLanguageCode

InnerErrorModel

Object

An object containing more specific information about the error. As per Microsoft One API guidelines - https://github.com/Microsoft/api-guidelines/blob/vNext/Guidelines.md#7102-error-condition-responses.

Name	Type	Description
code	InnerErrorCode	One of a server-defined set of error codes.
details	object	Error details.
innererror	InnerErrorModel	An object containing more specific information than the current object about the error.
message	string	Error message.
target	string	Error target.

ProjectKind

Enumeration

The project kind.

Value	Description
CustomEntityRecognition	For building an extraction model to identify your domain categories using your own data.
CustomMultiLabelClassification	For building a classification model to classify text using your own data. Each file can have one or many labels. For example, file 1 is classified as A, B, and C and file 2 is classified as B and C.
CustomSingleLabelClassification	For building a classification model to classify text using your own data. Each file will have only one label. For example, file 1 is classified as A and file 2 is classified as B.

TextAnalysisAuthoringCustomEntityRecognitionEvaluationSummary

Object

Represents the evaluation summary for a custom entity recognition project.

Name	Type	Description
customEntityRecognitionEvaluation	TextAnalysisAuthoringEntityRecognitionEvaluationSummary	Contains the data related to extraction evaluation.
evaluationOptions	TextAnalysisAuthoringEvaluationOptions	Represents the options used running the evaluation.
projectKind	string: CustomEntityRecognition	Represents the project type that the evaluation ran on.

TextAnalysisAuthoringCustomMultiLabelClassificationEvaluationSummary

Object

Represents the evaluation summary for a custom multi-label classification project.

Name	Type	Description
customMultiLabelClassificationEvaluation	TextAnalysisAuthoringMultiLabelClassificationEvaluationSummary	Contains the data related to multi label classification evaluation.
evaluationOptions	TextAnalysisAuthoringEvaluationOptions	Represents the options used running the evaluation.
projectKind	string: CustomMultiLabelClassification	Represents the project type that the evaluation ran on.

TextAnalysisAuthoringCustomSingleLabelClassificationEvaluationSummary

Object

Represents the evaluation summary for a custom single-label classification project.

Name	Type	Description
customSingleLabelClassificationEvaluation	TextAnalysisAuthoringSingleLabelClassificationEvaluationSummary	Contains the data related to single label classification evaluation.
evaluationOptions	TextAnalysisAuthoringEvaluationOptions	Represents the options used running the evaluation.
projectKind	string: CustomSingleLabelClassification	Represents the project type that the evaluation ran on.

TextAnalysisAuthoringEntityEvaluationSummary

Object

Represents the evaluation summary for an entity.

Name	Type	Description
f1	number (double)	Represents the model precision
falseNegativeCount	integer (int32)	Represents the count of false negative
falsePositiveCount	integer (int32)	Represents the count of false positive
precision	number (double)	Represents the model recall
recall	number (double)	Represents the model F1 score
trueNegativeCount	integer (int32)	Represents the count of true negative
truePositiveCount	integer (int32)	Represents the count of true positive

TextAnalysisAuthoringEntityRecognitionEvaluationSummary

Object

Represents the evaluation summary for a custom entity recognition project.

Name	Type	Description
confusionMatrix	object	Represents the confusion matrix between two entities (the two entities can be the same). The matrix is between the entity that was labelled and the entity that was predicted.
entities	<string,  TextAnalysisAuthoringEntityEvaluationSummary>	Represents the entities evaluation
macroF1	number (float)	Represents the macro F1
macroPrecision	number (float)	Represents the macro precision
macroRecall	number (float)	Represents the macro recall
microF1	number (float)	Represents the micro F1
microPrecision	number (float)	Represents the micro precision
microRecall	number (float)	Represents the micro recall

TextAnalysisAuthoringEvaluationOptions

Object

Represents the options used running the evaluation.

Name	Type	Description
kind	EvaluationKind	Represents the evaluation kind. By default, the evaluation kind is set to percentage.
testingSplitPercentage	integer (int32)	Represents the testing dataset split percentage. Only needed in case the evaluation kind is percentage.
trainingSplitPercentage	integer (int32)	Represents the training dataset split percentage. Only needed in case the evaluation kind is percentage.

TextAnalysisAuthoringMultiLabelClassEvaluationSummary

Object

Represents the evaluation summary of a class in a multi-label classification project.

Name	Type	Description
f1	number (double)	Represents the model precision
falseNegativeCount	integer (int32)	Represents the count of false negative
falsePositiveCount	integer (int32)	Represents the count of false positive
precision	number (double)	Represents the model recall
recall	number (double)	Represents the model F1 score
trueNegativeCount	integer (int32)	Represents the count of true negative
truePositiveCount	integer (int32)	Represents the count of true positive

TextAnalysisAuthoringMultiLabelClassificationEvaluationSummary

Object

Represents the evaluation summary for a multi-label classification project.

Name	Type	Description
classes	<string,  TextAnalysisAuthoringMultiLabelClassEvaluationSummary>	Represents the classes evaluation
macroF1	number (float)	Represents the macro F1
macroPrecision	number (float)	Represents the macro precision
macroRecall	number (float)	Represents the macro recall
microF1	number (float)	Represents the micro F1
microPrecision	number (float)	Represents the micro precision
microRecall	number (float)	Represents the micro recall

TextAnalysisAuthoringSingleLabelClassEvaluationSummary

Object

Represents the evaluation summary for a class in a single-label classification project.

Name	Type	Description
f1	number (double)	Represents the model precision
falseNegativeCount	integer (int32)	Represents the count of false negative
falsePositiveCount	integer (int32)	Represents the count of false positive
precision	number (double)	Represents the model recall
recall	number (double)	Represents the model F1 score
trueNegativeCount	integer (int32)	Represents the count of true negative
truePositiveCount	integer (int32)	Represents the count of true positive

TextAnalysisAuthoringSingleLabelClassificationEvaluationSummary

Object

Represents the evaluation summary for a custom single-label classification project.

Name	Type	Description
classes	<string,  TextAnalysisAuthoringSingleLabelClassEvaluationSummary>	Represents the classes evaluation
confusionMatrix	object	Represents the confusion matrix between two classes (the two classes can be the same). The matrix is between the class that was labelled and the class that was predicted.
macroF1	number (float)	Represents the macro F1
macroPrecision	number (float)	Represents the macro precision
macroRecall	number (float)	Represents the macro recall
microF1	number (float)	Represents the micro F1
microPrecision	number (float)	Represents the micro precision
microRecall	number (float)	Represents the micro recall