Reference: Image Embeddings | Azure Machine Learning

Creates an embedding vector representing the input image and text pair.

POST /images/embeddings?api-version=2024-04-01-preview

URI Parameters

Name In Required Type Description
api-version query True string The version of the API in the format "YYYY-MM-DD" or "YYYY-MM-DD-preview".

Request Header

Name Required Type Description
extra-parameters string The behavior of the API when extra parameters are indicated in the payload. Using pass-through makes the API to pass the parameter to the underlying model. Use this value when you want to pass parameters that you know the underlying model can support. Using ignore makes the API to drop any unsupported parameter. Use this value when you need to use the same payload across different models, but one of the extra parameters may make a model to error out if not supported. Using error makes the API to reject any extra parameter in the payload. Only parameters specified in this API can be indicated, or a 400 error is returned.
azureml-model-deployment string Name of the deployment you want to route the request to. Supported for endpoints that support multiple deployments.

Request Body

Name Required Type Description
model string The model name. This parameter is ignored if the endpoint serves only one model.
input True EmbeddingInput[] Input image to embed. To embed multiple inputs in a single request, pass an array. The input must not exceed the max input tokens for the model.
dimensions integer The number of dimensions the resulting output embeddings should have. Passing null causes the model to use its default value. Returns a 422 error if the model doesn't support the value or parameter.
encoding_format EmbeddingEncodingFormat The format to return the embeddings in. Either base64, float, int8, uint8, binary, or ubinary. Returns a 422 error if the model doesn't support the value or parameter.

Responses

Name Type Description
200 OK CreateEmbeddingResponse OK
401 Unauthorized UnauthorizedError Access token is missing or invalid

Headers

x-ms-error-code: string
404 Not Found NotFoundError Modality not supported by the model. Check the documentation of the model to see which routes are available.

Headers

x-ms-error-code: string
422 Unprocessable Entity UnprocessableContentError The request contains unprocessable content. The error is returned when the payload indicated is valid according to this specification. However, some of the instructions indicated in the payload are not supported by the underlying model. Use the details section to understand the offending parameter.

Headers

x-ms-error-code: string
429 Too Many Requests TooManyRequestsError You have hit your assigned rate limit and your request need to be paced.

Headers

x-ms-error-code: string
Other Status Codes ContentFilterError Bad request

Headers

x-ms-error-code: string

Security

Authorization

The token with the Bearer: prefix, e.g. Bearer abcde12345

Type: apiKey
In: header

AADToken

Azure Active Directory OAuth2 authentication

Type: oauth2
Flow: application
Token URL: https://login.microsoftonline.com/common/oauth2/v2.0/token

Examples

Creates an embedding vector representing the input text and image

Sample Request

POST /images/embeddings?api-version=2024-04-01-preview

{
  "input": [
    {
      "text": "A nice picture of a cat",
      "image": "data:image/jpeg;base64,iVBORw0KG..."
    }
  ],
  "encoding_format": "float",
  "dimensions": 1024
}

Sample Response

Status code: 200

{
  "data": [
    {
      "index": 0,
      "object": "embedding",
      "embedding": [
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0
      ]
    }
  ],
  "object": "list",
  "model": "DINO",
  "usage": {
    "prompt_tokens": 15,
    "total_tokens": 15
  }
}

Definitions

Name Description
ContentFilterError The API call fails when the prompt triggers a content filter as configured. Modify the prompt and try again.
CreateEmbeddingResponse
CreateImageEmbeddingRequest
Detail
Embedding Represents an image generated.
EmbeddingEncodingFormat The format to return the embeddings in. Either base64, float, int8, uint8, binary, or ubinary. Returns a 422 error if the model doesn't support the value or parameter.
EmbeddingObject The object type, which is always "embedding".
EmbeddingInput Represents an image with optional text.
ListObject The object type, which is always "list".
NotFoundError
TooManyRequestsError
UnauthorizedError
UnprocessableContentError The request contains unprocessable content. The error is returned when the payload indicated is valid according to this specification. However, some of the instructions indicated in the payload are not supported by the underlying model. Use the details section to understand the offending parameter.
Usage The usage information for the request.

ContentFilterError

The API call fails when the prompt triggers a content filter as configured. Modify the prompt and try again.

Name Type Description
code string The error code.
error string The error description.
message string The error message.
param string The parameter that triggered the content filter.
status integer The HTTP status code.

CreateEmbeddingResponse

Name Type Description
data Embedding[] The list of embeddings generated by the model.
model string The name of the model used to generate the embedding.
object ListObject The object type, which is always "list".
usage Usage The usage information for the request.

CreateImageEmbeddingRequest

Name Type Default Value Description
dimensions integer The number of dimensions the resulting output embeddings should have. Passing null causes the model to use its default value. Returns a 422 error if the model doesn't support the value or parameter.
encoding_format EmbeddingEncodingFormat float The format to return the embeddings in. Either base64, float, int8, uint8, binary, or ubinary. Returns a 422 error if the model doesn't support the value or parameter.
input EmbeddingInput[] Input image to embed. To embed multiple inputs in a single request, pass an array. The input must not exceed the max input tokens for the model.

Detail

Name Type Description
loc string[] The parameter causing the issue
value string The value passed to the parameter causing issues.

Embedding

Represents an image generated.

Name Type Description
embedding number[] The embedding vector, which is a list of floats. The length of vector depends on the model used.
index integer The index of the embedding in the list of embeddings.
object EmbeddingObject The object type, which is always "embedding".

EmbeddingEncodingFormat

The format to return the embeddings in. Either base64, float, int8, uint8, binary, or ubinary. Returns a 422 error if the model doesn't support the value or parameter.

Name Type Description
base64 string
binary string
float string
int8 string
ubinary string
uint8 string

EmbeddingObject

The object type, which is always "embedding".

Name Type Description
embedding string

EmbeddingInput

Represents an image with optional text.

Name Type Description
image string The input image encoded in base64 string as a data URL. Example: data:image/{format};base64,{data}.
text string Optional. The text input to feed into the model (like DINO, CLIP). Returns a 422 error if the model doesn't support the value or parameter.

ListObject

The object type, which is always "list".

Name Type Description
list string

NotFoundError

Name Type Description
error string The error description.
message string The error message.
status integer The HTTP status code.

TooManyRequestsError

Name Type Description
error string The error description.
message string The error message.
status integer The HTTP status code.

UnauthorizedError

Name Type Description
error string The error description.
message string The error message.
status integer The HTTP status code.

UnprocessableContentError

The request contains unprocessable content. The error is returned when the payload indicated is valid according to this specification. However, some of the instructions indicated in the payload are not supported by the underlying model. Use the details section to understand the offending parameter.

Name Type Description
code string The error code.
detail Detail
error string The error description.
message string The error message.
status integer The HTTP status code.

Usage

The usage information for the request.

Name Type Description
prompt_patches integer The number of image patches used by the image prompt.
prompt_tokens integer The number of tokens used by the prompt.
total_patches integer The total number of patches used by the request.
total_tokens integer The total number of tokens used by the request.