Tutorial: Connect to Azure OpenAI Service in AKS using a connection string (preview)

In this tutorial, you learn how to create a pod in an Azure Kubernetes (AKS) cluster, which talks to Azure OpenAI Service using a connection string. You complete the following tasks:

  • Create an AKS cluster and Azure OpenAI Service with gpt-4 model deployment.
  • Create a connection between the AKS cluster and Azure OpenAI with Service Connector.
  • Clone a sample application that will talk to the OpenAI service from an AKS cluster.
  • Deploy the application to a pod in the AKS cluster and test the connection.
  • Clean up resources.

Warning

Microsoft recommends that you use the most secure authentication flow available. The authentication flow described in this procedure requires a very high degree of trust in the application, and carries risks that are not present in other flows. You should only use this flow when other more secure flows, such as managed identities, aren't viable. See the tutorial using a managed identity.

Prerequisites

Create Azure resources

  1. Create a resource group for this tutorial.

    az group create \
        --name MyResourceGroup \
        --location eastus
    
  2. Create an AKS cluster with the following command, or by referring to the AKS quickstart. This cluster is where we create the service connection and pod definition and deploy the sample application.

    az aks create \
        --resource-group MyResourceGroup \
        --name MyAKSCluster \
        --enable-managed-identity \
        --node-count 1 \
       --generate-ssh-keys
    
  3. Connect to the cluster using the az aks get-credentials command.

    az aks get-credentials \
        --resource-group MyResourceGroup \
        --name MyAKSCluster
    
  4. Create an Azure OpenAI Service resource using the az cognitiveservices account create command. Optionally refer to this tutorial for more instructions. Azure OpenAI Service is the target service that we'll connect to the AKS cluster.

    az cognitiveservices account create \
        --resource-group MyResourceGroup \
        --name MyOpenAIService \
        --location eastus \
        --kind OpenAI \
        --sku s0 \
        --custom-domain myopenaiservice \
        --subscription <SubscriptionID>
    
  5. Deploy a model with the az cognitiveservices deployment create command. The model is used in the sample application to test the connection.

    az cognitiveservices account deployment create \
        --resource-group MyResourceGroup \
        --name MyOpenAIService
        --deployment-name MyModel \
        --model-name gpt-4 \
        --model-version 0613 \
        --model-format OpenAI \
        --sku-name "Standard"
        --capacity 1
    
  6. Create an Azure Container Registry (ACR) resource with the az acr create command, or referring to this tutorial. The registry hosts the container image of the sample application, which the AKS pod definition consumes.

    az acr create \
        --resource-group MyResourceGroup \
        --name myregistry \
        --sku Standard
    
  7. Enable anonymous pull using az acr update command so that the AKS cluster can consume the images in the registry.

    az acr update \
        --resource-group MyResourceGroup \
        --name myregistry \
        --anonymous-pull-enabled
    

Create a service connection in AKS with Service Connector (preview)

Create a service connection between an AKS cluster and Azure OpenAI Service in the Azure portal or the Azure CLI.

Refer to the AKS service connection quickstart for instructions to create a new connection and fill in the settings referring to the examples in the following table. Leave all other settings with their default values.

  1. Basics tab:

    Setting Example value Description
    Kubernetes namespace default The Kubernetes namespace.
    Service type OpenAI Service The target service type.
    Connection name openai_conn Use the connection name provided by Service Connector or choose your own connection name.
    Subscription <MySubscription> The subscription used for Azure OpenAI Service.
    OpenAI <MyOpenAIService> The target Azure OpenAI service you want to connect to.
    Client type Python The code language or framework you use to connect to the target service.
  2. Authentication tab:

    Authentication Setting Example value Description
    Authentication type Connection String Service Connector authentication type.

Once the connection has been created, you can view its details in the Service Connector pane.

Clone sample application

  1. Clone the sample repository:

    git clone https://github.com/Azure-Samples/serviceconnector-aks-samples.git
    
  2. Go to the repository's sample folder for Azure OpenAI:

    cd serviceconnector-aks-samples/azure-openai-connection-string
    
  3. Replace the <MyModel> placeholder in the app.py file with the model name we deployed.

Build and push container images

  1. Build and push the images to your container registry using the Azure CLI az acr build command.

    az acr build --registry <MyRegistry> --image sc-demo-openai-connstr:latest ./
    
  2. View the images in your container registry using the az acr repository list command.

    az acr repository list --name <MyRegistry> --output table
    

Run application and test connection

  1. Replace the placeholders in the pod.yaml file in the azure-openai-connection-string folder.

    • Replace <YourContainerImage> with the name of the image we built earlier. For example: <MyRegistry>.azurecr.io/sc-demo-openai-connstr:latest.
    • Replace <SecretCreatedByServiceConnector> with the secret created by Service Connector. You may check the secret name in the Azure portal, in the Service Connector pane.
  2. Deploy the pod to your cluster with the kubectl apply command. Install kubectl locally using the az aks install-cli command if it isn't installed. The command creates a pod named sc-demo-openai-connstr in the default namespace of your AKS cluster.

    kubectl apply -f pod.yaml
    
  3. Check if the deployment is successful by viewing the pod with kubectl.

    kubectl get pod/sc-demo-openai-connstr
    
  4. Check that the connection is working by viewing the logs with kubectl.

    kubectl logs pod/sc-demo-openai-connstr
    

Clean up resources

If you don't need these resources anymore, clean up the Azure resources created in this tutorial by deleting the resource group.

az group delete \
    --resource-group MyResourceGroup

Next steps

Read the following articles to learn more about Service Connector concepts and how it helps AKS connect to Azure services.