Edit

Share via

Quickstart: Use images in your AI chats

  • Article

Get started using GPT-4 Turbo with images with the Azure OpenAI Service.

Note

Model choice

The latest vision-capable models are gpt-4o and gpt-4o mini. These are in public preview. The latest available GA model is gpt-4 version turbo-2024-04-09.

Start exploring GPT-4 Turbo with Vision capabilities with a no-code approach through Azure AI Studio.

Prerequisites

Note

It is currently not supported to turn off content filtering for the GPT-4 Turbo with Vision model.

Go to Azure AI Studio

Browse to Azure AI Studio and sign in with the credentials associated with your Azure OpenAI resource. During or after the sign-in workflow, select the appropriate directory, Azure subscription, and Azure OpenAI resource.

Under Management select Deployments and Create a GPT-4 Turbo with Vision deployment by selecting model name: "gpt-4" and model version "vision-preview". For more information about model deployment, see the resource deployment guide.

Under the Playground section select Chat.

Playground

From this page, you can quickly iterate and experiment with the model's capabilities.

For general help with assistant setup, chat sessions, settings, and panels, refer to the Chat quickstart.

Start a chat session to analyze images or video

In this chat session, you're instructing the assistant to aid in understanding images that you input.

  1. To start, select your GPT-4 Turbo with Vision deployment from the dropdown.

  2. In the Assistant setup pane, provide a System Message to guide the assistant. The default System Message is: "You are an AI assistant that helps people find information." You can tailor the System Message to the image or scenario that you're uploading.

    Note

    It is recommended to update the System Message to be specific to the task in order to avoid unhelpful responses from the model.

  3. Save your changes, and when prompted to confirm updating the system message, select Continue.

  4. In the Chat session pane, enter a text prompt like "Describe this image," and upload an image with the attachment button. You can use a different text prompt for your use case. Then select Send.

  5. Observe the output provided. Consider asking follow-up questions related to the analysis of your image to learn more.

Screenshot of AI studio chat playground.

Clean up resources

If you want to clean up and remove an Azure OpenAI resource, you can delete the resource or resource group. Deleting the resource group also deletes any other resources associated with it.

Use this article to get started using the Azure OpenAI REST APIs to deploy and use the GPT-4 Turbo with Vision model.

Prerequisites

Note

It is currently not supported to turn off content filtering for the GPT-4 Turbo with Vision model.

Retrieve key and endpoint

To successfully call the Azure OpenAI APIs, you need the following information about your Azure OpenAI resource:

Variable Name Value
Endpoint api_base The endpoint value is located under Keys and Endpoint for your resource in the Azure portal. You can also find the endpoint via the Deployments page in Azure AI Studio. An example endpoint is: https://docs-test-001.openai.azure.com/.
Key api_key The key value is also located under Keys and Endpoint for your resource in the Azure portal. Azure generates two keys for your resource. You can use either value.

Go to your resource in the Azure portal. On the navigation pane, select Keys and Endpoint under Resource Management. Copy the Endpoint value and an access key value. You can use either the KEY 1 or KEY 2 value. Having two keys allows you to securely rotate and regenerate keys without causing a service disruption.

Screenshot that shows the Keys and Endpoint page for an Azure OpenAI resource in the Azure portal.

Create a new Python application

Create a new Python file named quickstart.py. Open the new file in your preferred editor or IDE.

  1. Replace the contents of quickstart.py with the following code.

    # Packages required:
import requests 
import json 

api_base = '<your_azure_openai_endpoint>' 
deployment_name = '<your_deployment_name>'
API_KEY = '<your_azure_openai_key>'

base_url = f"{api_base}openai/deployments/{deployment_name}" 
headers = {   
    "Content-Type": "application/json",   
    "api-key": API_KEY 
} 

# Prepare endpoint, headers, and request body 
endpoint = f"{base_url}/chat/completions?api-version=2023-12-01-preview" 
data = { 
    "messages": [ 
        { "role": "system", "content": "You are a helpful assistant." }, 
        { "role": "user", "content": [  
            { 
                "type": "text", 
                "text": "Describe this picture:" 
            },
            { 
                "type": "image_url",
                "image_url": {
                    "url": "<image URL>"
                }
            }
        ] } 
    ], 
    "max_tokens": 2000 
}   

# Make the API call   
response = requests.post(endpoint, headers=headers, data=json.dumps(data))   

print(f"Status Code: {response.status_code}")   
print(response.text)

  2. Make the following changes:

    1. Enter your endpoint URL and key in the appropriate fields.
    2. Enter your GPT-4 Turbo with Vision deployment name in the appropriate field.
    3. Change the value of the "image" field to the URL of your image.

      Tip

      You can also use a base 64 encoded image data instead of a URL. For more information, see the GPT-4 Turbo with Vision how-to guide.

  3. Run the application with the python command:

    python quickstart.py

Clean up resources

If you want to clean up and remove an Azure OpenAI resource, you can delete the resource or resource group. Deleting the resource group also deletes any other resources associated with it.

Use this article to get started using the Azure OpenAI Python SDK to deploy and use the GPT-4 Turbo with Vision model.

Library source code | Package (PyPi) |

Prerequisites

Set up

Install the OpenAI Python client library with:

pip install openai

Note

This library is maintained by OpenAI. Refer to the release history to track the latest updates to the library.

Retrieve key and endpoint

To successfully make a call against Azure OpenAI, you need an endpoint and a key.

Variable name Value
ENDPOINT The service endpoint can be found in the Keys & Endpoint section when examining your resource from the Azure portal. Alternatively, you can find the endpoint via the Deployments page in Azure AI Studio. An example endpoint is: https://docs-test-001.openai.azure.com/.
API-KEY This value can be found in the Keys & Endpoint section when examining your resource from the Azure portal. You can use either KEY1 or KEY2.

Go to your resource in the Azure portal. The Keys & Endpoint section can be found in the Resource Management section. Copy your endpoint and access key as you'll need both for authenticating your API calls. You can use either KEY1 or KEY2. Always having two keys allows you to securely rotate and regenerate keys without causing a service disruption.

Screenshot of the overview UI for an Azure OpenAI resource in the Azure portal with the endpoint and access keys location circled in red.

Environment variables

Create and assign persistent environment variables for your key and endpoint.

Important

If you use an API key, store it securely somewhere else, such as in Azure Key Vault. Don't include the API key directly in your code, and never post it publicly.

For more information about AI services security, see Authenticate requests to Azure AI services.

setx AZURE_OPENAI_API_KEY "REPLACE_WITH_YOUR_KEY_VALUE_HERE" 
setx AZURE_OPENAI_ENDPOINT "REPLACE_WITH_YOUR_ENDPOINT_HERE"

Create a new Python application

Create a new Python file named quickstart.py. Open the new file in your preferred editor or IDE.

  1. Replace the contents of quickstart.py with the following code.

    from openai import AzureOpenAI

api_base = os.getenv("AZURE_OPENAI_ENDPOINT")
api_key= os.getenv("AZURE_OPENAI_API_KEY")
deployment_name = '<your_deployment_name>'
api_version = '2023-12-01-preview' # this might change in the future

client = AzureOpenAI(
    api_key=api_key,  
    api_version=api_version,
    base_url=f"{api_base}/openai/deployments/{deployment_name}"
)

response = client.chat.completions.create(
    model=deployment_name,
    messages=[
        { "role": "system", "content": "You are a helpful assistant." },
        { "role": "user", "content": [  
            { 
                "type": "text", 
                "text": "Describe this picture:" 
            },
            { 
                "type": "image_url",
                "image_url": {
                    "url": "<image URL>"
                }
            }
        ] } 
    ],
    max_tokens=2000 
)

print(response)

  2. Make the following changes:

    1. Enter the name of your GPT-4 Turbo with Vision deployment in the appropriate field.
    2. Change the value of the "url" field to the URL of your image.

      Tip

      You can also use a base 64 encoded image data instead of a URL. For more information, see the GPT-4 Turbo with Vision how-to guide.

  3. Run the application with the python command:

    python quickstart.py

Clean up resources

If you want to clean up and remove an Azure OpenAI resource, you can delete the resource or resource group. Deleting the resource group also deletes any other resources associated with it.

Use this article to get started using the OpenAI JavaScript SDK to deploy and use the GPT-4 Turbo with Vision model.

This SDK is provided by OpenAI with Azure specific types provided by Azure.

Reference documentation | Library source code | Package (npm) | Samples

Prerequisites

Note

This library is maintained by OpenAI. Refer to the release history to track the latest updates to the library.

Retrieve key and endpoint

To successfully make a call against Azure OpenAI, you need an endpoint and a key.

Variable name Value
ENDPOINT The service endpoint can be found in the Keys & Endpoint section when examining your resource from the Azure portal. Alternatively, you can find the endpoint via the Deployments page in Azure AI Studio. An example endpoint is: https://docs-test-001.openai.azure.com/.
API-KEY This value can be found in the Keys & Endpoint section when examining your resource from the Azure portal. You can use either KEY1 or KEY2.

Go to your resource in the Azure portal. The Keys & Endpoint section can be found in the Resource Management section. Copy your endpoint and access key as you'll need both for authenticating your API calls. You can use either KEY1 or KEY2. Always having two keys allows you to securely rotate and regenerate keys without causing a service disruption.

Screenshot of the overview UI for an Azure OpenAI resource in the Azure portal with the endpoint and access keys location circled in red.

Environment variables

Create and assign persistent environment variables for your key and endpoint.

Important

If you use an API key, store it securely somewhere else, such as in Azure Key Vault. Don't include the API key directly in your code, and never post it publicly.

For more information about AI services security, see Authenticate requests to Azure AI services.

setx AZURE_OPENAI_API_KEY "REPLACE_WITH_YOUR_KEY_VALUE_HERE" 
setx AZURE_OPENAI_ENDPOINT "REPLACE_WITH_YOUR_ENDPOINT_HERE"

Create a Node application

In a console window (such as cmd, PowerShell, or Bash), create a new directory for your app, and navigate to it. Then run the npm init command to create a node application with a package.json file.

npm init

Install the client library

Install the client libraries with:

npm install openai @azure/identity

Your app's package.json file will be updated with the dependencies.

Create a new JavaScript application for image prompts

Select an image from the azure-samples/cognitive-services-sample-data-files and set the URL for an image in the environment variables.

  1. Replace the contents of quickstart.js with the following code.

    const AzureOpenAI = require('openai').AzureOpenAI;
const { 
    DefaultAzureCredential, 
    getBearerTokenProvider 
} = require('@azure/identity');

// You will need to set these environment variables or edit the following values
const endpoint = process.env["AZURE_OPENAI_ENDPOINT"] || "<endpoint>";
const imageUrl = process.env["IMAGE_URL"] || "<image url>";

// Required Azure OpenAI deployment name and API version
const apiVersion = "2024-07-01-preview";
const deploymentName = "gpt-4-with-turbo";

// keyless authentication    
const credential = new DefaultAzureCredential();
const scope = "https://cognitiveservices.azure.com/.default";
const azureADTokenProvider = getBearerTokenProvider(credential, scope);

function getClient(): AzureOpenAI {
  return new AzureOpenAI({
    endpoint,
    azureADTokenProvider,
    apiVersion,
    deployment: deploymentName,
  });
}
function createMessages() {
  return {
    messages: [
      { role: "system", content: "You are a helpful assistant." },
      {
        role: "user",
        content: [
          {
            type: "text",
            text: "Describe this picture:",
          },
          {
            type: "image_url",
            image_url: {
              url: imageUrl,
            },
          },
        ],
      },
    ],
    model: "",
    max_tokens: 2000,
  };
}
async function printChoices(completion) {
  for (const choice of completion.choices) {
    console.log(choice.message);
  }
}
export async function main() {
  console.log("== Get GPT-4 Turbo with vision Sample ==");

  const client = getClient();
  const messages = createMessages();
  const completion = await client.chat.completions.create(messages);
  await printChoices(completion);
}

main().catch((err) => {
  console.error("Error occurred:", err);
});

  2. Make the following changes:

    1. Enter the name of your GPT-4 Turbo with Vision deployment in the appropriate field.
    2. Change the value of the "url" field to the URL of your image.

      Tip

      You can also use a base 64 encoded image data instead of a URL. For more information, see the GPT-4 Turbo with Vision how-to guide.

  3. Run the application with the following command:

    node quickstart.js

Clean up resources

If you want to clean up and remove an Azure OpenAI resource, you can delete the resource or resource group. Deleting the resource group also deletes any other resources associated with it.

Use this article to get started using the OpenAI JavaScript SDK to deploy and use the GPT-4 Turbo with Vision model.

This SDK is provided by OpenAI with Azure specific types provided by Azure.

Reference documentation | Library source code | Package (npm) | Samples

Prerequisites

Note

This library is maintained by OpenAI. Refer to the release history to track the latest updates to the library.

Retrieve key and endpoint

To successfully make a call against Azure OpenAI, you need an endpoint and a key.

Variable name Value
ENDPOINT The service endpoint can be found in the Keys & Endpoint section when examining your resource from the Azure portal. Alternatively, you can find the endpoint via the Deployments page in Azure AI Studio. An example endpoint is: https://docs-test-001.openai.azure.com/.
API-KEY This value can be found in the Keys & Endpoint section when examining your resource from the Azure portal. You can use either KEY1 or KEY2.

Go to your resource in the Azure portal. The Keys & Endpoint section can be found in the Resource Management section. Copy your endpoint and access key as you'll need both for authenticating your API calls. You can use either KEY1 or KEY2. Always having two keys allows you to securely rotate and regenerate keys without causing a service disruption.

Screenshot of the overview UI for an Azure OpenAI resource in the Azure portal with the endpoint and access keys location circled in red.

Environment variables

Create and assign persistent environment variables for your key and endpoint.

Important

If you use an API key, store it securely somewhere else, such as in Azure Key Vault. Don't include the API key directly in your code, and never post it publicly.

For more information about AI services security, see Authenticate requests to Azure AI services.

setx AZURE_OPENAI_API_KEY "REPLACE_WITH_YOUR_KEY_VALUE_HERE" 
setx AZURE_OPENAI_ENDPOINT "REPLACE_WITH_YOUR_ENDPOINT_HERE"

Create a Node application

In a console window (such as cmd, PowerShell, or Bash), create a new directory for your app, and navigate to it. Then run the npm init command to create a node application with a package.json file.

npm init

Install the client library

Install the client libraries with:

npm install openai @azure/identity

Your app's package.json file will be updated with the dependencies.

Create a new JavaScript application for image prompts

Select an image from the azure-samples/cognitive-services-sample-data-files and set the URL for an image in the environment variables.

  1. Create a quickstart.ts and paste in the following code.

    import { AzureOpenAI } from "openai";
import { 
    DefaultAzureCredential, 
    getBearerTokenProvider 
} from "@azure/identity";
import type {
  ChatCompletion,
  ChatCompletionCreateParamsNonStreaming,
} from "openai/resources/index";

// You will need to set these environment variables or edit the following values
const endpoint = process.env["AZURE_OPENAI_ENDPOINT"] || "<endpoint>";
const imageUrl = process.env["IMAGE_URL"] || "<image url>";

// Required Azure OpenAI deployment name and API version
const apiVersion = "2024-07-01-preview";
const deploymentName = "gpt-4-with-turbo";

// keyless authentication    
const credential = new DefaultAzureCredential();
const scope = "https://cognitiveservices.azure.com/.default";
const azureADTokenProvider = getBearerTokenProvider(credential, scope);

function getClient(): AzureOpenAI {
  return new AzureOpenAI({
    endpoint,
    azureADTokenProvider,
    apiVersion,
    deployment: deploymentName,
  });
}
function createMessages(): ChatCompletionCreateParamsNonStreaming {
  return {
    messages: [
      { role: "system", content: "You are a helpful assistant." },
      {
        role: "user",
        content: [
          {
            type: "text",
            text: "Describe this picture:",
          },
          {
            type: "image_url",
            image_url: {
              url: imageUrl,
            },
          },
        ],
      },
    ],
    model: "",
    max_tokens: 2000,
  };
}
async function printChoices(completion: ChatCompletion): Promise<void> {
  for (const choice of completion.choices) {
    console.log(choice.message);
  }
}
export async function main() {
  console.log("== Get GPT-4 Turbo with vision Sample ==");

  const client = getClient();
  const messages = createMessages();
  const completion = await client.chat.completions.create(messages);
  await printChoices(completion);
}

main().catch((err) => {
  console.error("Error occurred:", err);
});

  2. Make the following changes:

    1. Enter the name of your GPT-4 Turbo with Vision deployment in the appropriate field.
    2. Change the value of the "url" field to the URL of your image.

      Tip

      You can also use a base 64 encoded image data instead of a URL. For more information, see the GPT-4 Turbo with Vision how-to guide.

  3. Build the application with the following command:

    tsc

  4. Run the application with the following command:

    node quickstart.js

Clean up resources

If you want to clean up and remove an Azure OpenAI resource, you can delete the resource or resource group. Deleting the resource group also deletes any other resources associated with it.

Use this article to get started using the Azure OpenAI .NET SDK to deploy and use the GPT-4 Turbo with Vision model.

Prerequisites

Set up

Retrieve key and endpoint

To successfully make a call against Azure OpenAI, you need an endpoint and a key.

Variable name Value
AZURE_OPENAI_ENDPOINT The service endpoint can be found in the Keys & Endpoint section when examining your resource from the Azure portal. Alternatively, you can find the endpoint via the Deployments page in Azure AI Studio. An example endpoint is: https://docs-test-001.openai.azure.com/.
AZURE_OPENAI_API_KEY This value can be found in the Keys & Endpoint section when examining your resource from the Azure portal. You can use either KEY1 or KEY2.

Go to your resource in the Azure portal. The Endpoint and Keys can be found in the Resource Management section. Copy your endpoint and access key as you need both for authenticating your API calls. You can use either KEY1 or KEY2. Always having two keys allows you to securely rotate and regenerate keys without causing a service disruption.

Screenshot of the overview UI for an Azure OpenAI resource in the Azure portal with the endpoint & access keys location highlighted.

Create the .NET app

  1. Create a .NET app using the dotnet new command:

    dotnet new console -n OpenAISpeech

  2. Change into the directory of the new app:

    cd OpenAISpeech

Install the client library

Install the Azure.OpenAI client library:

dotnet add package Azure.AI.OpenAI

Passwordless authentication is more secure than key-based alternatives and is the recommended approach for connecting to Azure services. If you choose to use Passwordless authentication, you'll need to complete the following:

  1. Add the Azure.Identity package.

    dotnet add package Azure.Identity

  2. Assign the Cognitive Services User role to your user account. This can be done in the Azure portal on your OpenAI resource under Access control (IAM) > Add role assignment.

  3. Sign-in to Azure using Visual Studio or the Azure CLI via az login.

Update the app code

  1. Replace the contents of program.cs with the following code and update the placeholder values with your own.

    using Azure;
using Azure.AI.OpenAI;
using Azure.Identity;
using OpenAI.Chat; // Required for Passwordless auth

var endpoint = new Uri("YOUR_AZURE_OPENAI_ENDPOINT");
var credentials = new AzureKeyCredential("YOUR_AZURE_OPENAI_KEY");
// var credentials = new DefaultAzureCredential(); // Use this line for Passwordless auth
var deploymentName = "gpt-4"; // Default name, update with your own if needed

var openAIClient = new AzureOpenAIClient(endpoint, credentials);
var chatClient = openAIClient.GetChatClient(deploymentName);

var imageUri = "YOUR_IMAGE_URL";

List<ChatMessage> messages = [
    new UserChatMessage(
        ChatMessageContentPart.CreateTextMessageContentPart("Please describe the following image:"),
        ChatMessageContentPart.CreateImageMessageContentPart(new Uri(imageUri), "image/png"))
];

ChatCompletion chatCompletion = await chatClient.CompleteChatAsync(messages);

Console.WriteLine($"[ASSISTANT]:");
Console.WriteLine($"{chatCompletion.Content[0].Text}");

    Important

    For production, store and access your credentials using a secure method, such as Azure Key Vault. For more information about credential security, see Azure AI services security.

  2. Run the application using the dotnet run command or the run button at the top of Visual Studio:

    dotnet run

The app generates an audio file at the location you specified for the speechFilePath variable. Play the file on your device to hear the generated audio.

Clean up resources

If you want to clean up and remove an Azure OpenAI resource, you can delete the resource or resource group. Deleting the resource group also deletes any other resources associated with it.

Next steps