Reference: Image Embeddings | Azure AI Foundry
Important
Items marked (preview) in this article are currently in public preview. This preview is provided without a service-level agreement, and we don't recommend it for production workloads. Certain features might not be supported or might have constrained capabilities. For more information, see Supplemental Terms of Use for Microsoft Azure Previews.
Creates an embedding vector representing the input image and text pair.
POST /images/embeddings?api-version=2024-04-01-preview
URI Parameters
| Name | In | Required | Type | Description |
|---|---|---|---|---|
| api-version | query | True | string | The version of the API in the format "YYYY-MM-DD" or "YYYY-MM-DD-preview". |
Request Header
| Name | Required | Type | Description |
|---|---|---|---|
| extra-parameters | string | The behavior of the API when extra parameters are indicated in the payload. Using pass-through makes the API to pass the parameter to the underlying model. Use this value when you want to pass parameters that you know the underlying model can support. Using ignore makes the API to drop any unsupported parameter. Use this value when you need to use the same payload across different models, but one of the extra parameters may make a model to error out if not supported. Using error makes the API to reject any extra parameter in the payload. Only parameters specified in this API can be indicated, or a 400 error is returned. |
|
| azureml-model-deployment | string | Name of the deployment you want to route the request to. Supported for endpoints that support multiple deployments. |
Request Body
| Name | Required | Type | Description |
|---|---|---|---|
| input | True | EmbeddingInput[] | Input image to embed. To embed multiple inputs in a single request, pass an array. The input must not exceed the max input tokens for the model. |
| dimensions | integer | The number of dimensions the resulting output embeddings should have. Passing null causes the model to use its default value. Returns a 422 error if the model doesn't support the value or parameter. | |
| encoding_format | EmbeddingEncodingFormat | The format to return the embeddings in. Either base64, float, int8, uint8, binary, or ubinary. Returns a 422 error if the model doesn't support the value or parameter. |
Responses
| Name | Type | Description |
|---|---|---|
| 200 OK | CreateEmbeddingResponse | OK |
| 401 Unauthorized | UnauthorizedError | Access token is missing or invalid Headers x-ms-error-code: string |
| 404 Not Found | NotFoundError | Modality not supported by the model. Check the documentation of the model to see which routes are available. Headers x-ms-error-code: string |
| 422 Unprocessable Entity | UnprocessableContentError | The request contains unprocessable content. The error is returned when the payload indicated is valid according to this specification. However, some of the instructions indicated in the payload are not supported by the underlying model. Use the details section to understand the offending parameter.Headers x-ms-error-code: string |
| 429 Too Many Requests | TooManyRequestsError | You have hit your assigned rate limit and your request need to be paced. Headers x-ms-error-code: string |
| Other Status Codes | ContentFilterError | Bad request Headers x-ms-error-code: string |
Security
Authorization
The token with the Bearer: prefix, e.g. Bearer abcde12345
Type: apiKey
In: header
AADToken
Azure Active Directory OAuth2 authentication
Type: oauth2
Flow: application
Token URL: https://login.microsoftonline.com/common/oauth2/v2.0/token
Examples
Creates an embedding vector representing the input text and image
Sample Request
POST /images/embeddings?api-version=2024-04-01-preview
{
"input": [
{
"text": "A nice picture of a cat",
"image": "data:image/jpeg;base64,iVBORw0KG..."
}
],
"encoding_format": "float",
"dimensions": 1024
}
Sample Response
Status code: 200
{
"data": [
{
"index": 0,
"object": "embedding",
"embedding": [
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
]
}
],
"object": "list",
"model": "DINO",
"usage": {
"prompt_tokens": 15,
"total_tokens": 15
}
}
Definitions
| Name | Description |
|---|---|
| ContentFilterError | The API call fails when the prompt triggers a content filter as configured. Modify the prompt and try again. |
| CreateEmbeddingResponse | |
| CreateImageEmbeddingRequest | |
| Detail | |
| Embedding | Represents an image generated. |
| EmbeddingEncodingFormat | The format to return the embeddings in. Either base64, float, int8, uint8, binary, or ubinary. Returns a 422 error if the model doesn't support the value or parameter. |
| EmbeddingObject | The object type, which is always "embedding". |
| EmbeddingInput | Represents an image with optional text. |
| ListObject | The object type, which is always "list". |
| NotFoundError | |
| TooManyRequestsError | |
| UnauthorizedError | |
| UnprocessableContentError | The request contains unprocessable content. The error is returned when the payload indicated is valid according to this specification. However, some of the instructions indicated in the payload are not supported by the underlying model. Use the details section to understand the offending parameter. |
| Usage | The usage information for the request. |
ContentFilterError
The API call fails when the prompt triggers a content filter as configured. Modify the prompt and try again.
| Name | Type | Description |
|---|---|---|
| code | string | The error code. |
| error | string | The error description. |
| message | string | The error message. |
| param | string | The parameter that triggered the content filter. |
| status | integer | The HTTP status code. |
CreateEmbeddingResponse
| Name | Type | Description |
|---|---|---|
| data | Embedding[] | The list of embeddings generated by the model. |
| model | string | The name of the model used to generate the embedding. |
| object | ListObject | The object type, which is always "list". |
| usage | Usage | The usage information for the request. |
CreateImageEmbeddingRequest
| Name | Type | Default Value | Description |
|---|---|---|---|
| dimensions | integer | The number of dimensions the resulting output embeddings should have. Passing null causes the model to use its default value. Returns a 422 error if the model doesn't support the value or parameter. | |
| encoding_format | EmbeddingEncodingFormat | float | The format to return the embeddings in. Either base64, float, int8, uint8, binary, or ubinary. Returns a 422 error if the model doesn't support the value or parameter. |
| input | EmbeddingInput[] | Input image to embed. To embed multiple inputs in a single request, pass an array. The input must not exceed the max input tokens for the model. |
Detail
| Name | Type | Description |
|---|---|---|
| loc | string[] | The parameter causing the issue |
| value | string | The value passed to the parameter causing issues. |
Embedding
Represents an image generated.
| Name | Type | Description |
|---|---|---|
| embedding | number[] | The embedding vector, which is a list of floats. The length of vector depends on the model used. |
| index | integer | The index of the embedding in the list of embeddings. |
| object | EmbeddingObject | The object type, which is always "embedding". |
EmbeddingEncodingFormat
The format to return the embeddings in. Either base64, float, int8, uint8, binary, or ubinary. Returns a 422 error if the model doesn't support the value or parameter.
| Name | Type | Description |
|---|---|---|
| base64 | string | |
| binary | string | |
| float | string | |
| int8 | string | |
| ubinary | string | |
| uint8 | string |
EmbeddingObject
The object type, which is always "embedding".
| Name | Type | Description |
|---|---|---|
| embedding | string |
EmbeddingInput
Represents an image with optional text.
| Name | Type | Description |
|---|---|---|
| image | string | The input image encoded in base64 string as a data URL. Example: data:image/{format};base64,{data}. |
| text | string | Optional. The text input to feed into the model (like DINO, CLIP). Returns a 422 error if the model doesn't support the value or parameter. |
ListObject
The object type, which is always "list".
| Name | Type | Description |
|---|---|---|
| list | string |
NotFoundError
| Name | Type | Description |
|---|---|---|
| error | string | The error description. |
| message | string | The error message. |
| status | integer | The HTTP status code. |
TooManyRequestsError
| Name | Type | Description |
|---|---|---|
| error | string | The error description. |
| message | string | The error message. |
| status | integer | The HTTP status code. |
UnauthorizedError
| Name | Type | Description |
|---|---|---|
| error | string | The error description. |
| message | string | The error message. |
| status | integer | The HTTP status code. |
UnprocessableContentError
The request contains unprocessable content. The error is returned when the payload indicated is valid according to this specification. However, some of the instructions indicated in the payload are not supported by the underlying model. Use the details section to understand the offending parameter.
| Name | Type | Description |
|---|---|---|
| code | string | The error code. |
| detail | Detail | |
| error | string | The error description. |
| message | string | The error message. |
| status | integer | The HTTP status code. |
Usage
The usage information for the request.
| Name | Type | Description |
|---|---|---|
| prompt_patches | integer | The number of image patches used by the image prompt. |
| prompt_tokens | integer | The number of tokens used by the prompt. |
| total_patches | integer | The total number of patches used by the request. |
| total_tokens | integer | The total number of tokens used by the request. |