Serverless code interpreter sessions in Azure Container Apps

Artikkeli
11/07/2024

Azure Container Apps dynamic sessions provides fast and scalable access to a code interpreter. Each code interpreter session is fully isolated by a Hyper-V boundary and is designed to run untrusted code.

Uses for code interpreter sessions

Code interpreter sessions are ideal for scenarios where you need to run code that is potentially malicious or could cause harm to the host system or other users, such as:

Code generated by a large language model (LLM).
Code submitted by an end user in a web or SaaS application.

For popular LLM frameworks such as LangChain, LlamaIndex, or Semantic Kernel, you can use tools and plugins to integrate AI apps with code interpreter sessions.

Your applications can also integrate with code interpreter session using a REST API. The API allows you to execute code in a session and retrieve results. You can also upload and download files to and from the session. You can upload and download executable code files, or data files that your code can process.

The built-in code interpreter sessions support the most common code execution scenarios without the need to manage infrastructure or containers. If you need full control over the code execution environment or have a different scenario that requires isolated sandboxes, you can use custom code interpreter sessions.

Code interpreter session pool

To use code interpreter sessions, you need an Azure resource called a session pool that defines the configuration for code interpreter sessions. In the session pool, you can specify settings such as the maximum number of concurrent sessions and how long a session can be idle before the session is terminated.

You can create a session pool using the Azure portal, Azure CLI, or Azure Resource Manager templates. After you create a session pool, you can use the pool's management API endpoints to manage and execute code inside a session.

Create a session pool with Azure CLI

To create a code interpreter session pool using the Azure CLI, ensure you have the latest versions of the Azure CLI and the Azure Container Apps extension with the following commands:

# Upgrade the Azure CLI
az upgrade

# Install or upgrade the Azure Container Apps extension
az extension add --name containerapp --upgrade --allow-preview true -y

Use the az containerapps sessionpool create command to create the pool. The following example creates a Python code interpreter session pool named my-session-pool. Make sure to replace <RESOURCE_GROUP> with your resource group name before you run the command.

az containerapp sessionpool create \
    --name my-session-pool \
    --resource-group <RESOURCE_GROUP> \
    --location westus2 \
    --container-type PythonLTS \
    --max-sessions 100 \
    --cooldown-period 300 \
    --network-status EgressDisabled

You can define the following settings when you create a session pool:

Setting	Description
`--container-type`	The type of code interpreter to use. The only supported value is `PythonLTS`.
`--max-sessions`	The maximum number of allocated sessions allowed concurrently. The maximum value is `600`.
`--cooldown-period`	The number of allowed idle seconds before termination. The idle period is reset each time the session's API is called. The allowed range is between `300` and `3600`.
`--network-status`	Designates whether outbound network traffic is allowed from the session. Valid values are `EgressDisabled` (default) and `EgressEnabled`.

Important

If you enable egress, code running in the session can access the internet. Use caution when the code is untrusted as it can be used to perform malicious activities such as denial-of-service attacks.

Get the pool management API endpoint with Azure CLI

To use code interpreter sessions with LLM framework integrations or by calling the management API endpoints directly, you need the pool's management API endpoint. The endpoint is in the format https://<REGION>.dynamicsessions.io/subscriptions/<SUBSCRIPTION_ID>/resourceGroups/<RESOURCE_GROUP>/sessionPools/<SESSION_POOL_NAME>.

To retrieve the management API endpoint for a session pool, use the az containerapps sessionpool show command. Make sure to replace <RESOURCE_GROUP> with your resource group name before you run the command.

az containerapp sessionpool show \
    --name my-session-pool \
    --resource-group <RESOURCE_GROUP> \
    --query 'properties.poolManagementEndpoint' -o tsv

Code execution in a session

After you create a session pool, your application can interact with sessions in the pool using an integration with an LLM framework or by using the pool's management API endpoints directly.

Session identifiers

Important

The session identifier is sensitive information which requires you to use a secure process to manage its value. Part of this process requires that your application ensures each user or tenant only has access to their own sessions. Failure to secure access to sessions may result in misuse or unauthorized access to data stored in your users' sessions. For more information, see Session identifiers

When you interact with sessions in a pool, you use a session identifier to reference each session A session identifier is a string that you define that is unique within the session pool. If you're building a web application, you can use the user's ID. If you're building a chatbot, you can use the conversation ID.

If there's a running session with the identifier, the session is reused. If there's no running session with the identifier, a new session is automatically created.

To learn more about session identifiers, see Sessions overview.

Authentication

Authentication is handled using Microsoft Entra (formerly Azure Active Directory) tokens. Valid Microsoft Entra tokens are generated by an identity belonging to the Azure ContainerApps Session Executor and Contributor roles on the session pool.

If you're using an LLM framework integration, the framework handles the token generation and management for you. Ensure that the application is configured with a managed identity with the necessary role assignments on the session pool.

If you're using the pool's management API endpoints directly, you must generate a token and include it in the Authorization header of your HTTP requests. In addition to the role assignments previously mentioned, token needs to contain an audience (aud) claim with the value https://dynamicsessions.io.

To learn more, see Authentication.

LLM framework integrations

Instead of using the session pool management API directly, the following LLM frameworks provide integrations with code interpreter sessions:

Framework	Package	Tutorial
LangChain	Python: `langchain-azure-dynamic-sessions`	Tutorial
LlamaIndex	Python: `llama-index-tools-azure-code-interpreter`	Tutorial
Semantic Kernel	Python: `semantic-kernel` (version 0.9.8-b1 or later)	Tutorial

Management API endpoints

If you're not using an LLM framework integration, you can interact with the session pool directly using the management API endpoints.

The following endpoints are available for managing sessions in a pool:

Endpoint path	Method	Description
`code/execute`	`POST`	Execute code in a session.
`files/upload`	`POST`	Upload a file to a session.
`files/content/{filename}`	`GET`	Download a file from a session.
`files`	`GET`	List the files in a session.

Build the full URL for each endpoint by concatenating the pool's management API endpoint with the endpoint path. The query string must include an identifier parameter containing the session identifier, and an api-version parameter with the value 2024-02-02-preview.

For example: https://<REGION>.dynamicsessions.io/subscriptions/<SUBSCRIPTION_ID>/resourceGroups/<RESOURCE_GROUP>/sessionPools/<SESSION_POOL_NAME>/code/execute?api-version=2024-02-02-preview&identifier=<IDENTIFIER>

Execute code in a session

To execute code in a session, send a POST request to the code/execute endpoint with the code to run in the request body. This example prints "Hello, world!" in Python.

Before you send the request, replace the placeholders between the <> brackets with the appropriate values for your session pool and session identifier.

POST https://<REGION>.dynamicsessions.io/subscriptions/<SUBSCRIPTION_ID>/resourceGroups/<RESOURCE_GROUP>/sessionPools/<SESSION_POOL_NAME>/code/execute?api-version=2024-02-02-preview&identifier=<SESSION_ID>
Content-Type: application/json
Authorization: Bearer <token>

{
    "properties": {
        "codeInputType": "inline",
        "executionType": "synchronous",
        "code": "print('Hello, world!')"
    }
}

To reuse a session, specify the same session identifier in subsequent requests.

Upload a file to a session

To upload a file to a session, send a POST request to the uploadFile endpoint in a multipart form data request. Include the file data in the request body. The file must include a filename.

Uploaded files are stored in the session's file system under the /mnt/data directory.

Before you send the request, replace the placeholders between the <> brackets with values specific to your request.

POST https://<REGION>.dynamicsessions.io/subscriptions/<SUBSCRIPTION_ID>/resourceGroups/<RESOURCE_GROUP>/sessionPools/<SESSION_POOL_NAME>/files/upload?api-version=2024-02-02-preview&identifier=<SESSION_ID>
Content-Type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW
Authorization: Bearer <token>

------WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Disposition: form-data; name="file"; filename="<FILE_NAME_AND_EXTENSION>"
Content-Type: application/octet-stream

(data)
------WebKitFormBoundary7MA4YWxkTrZu0gW--

Download a file from a session

To download a file from a session's /mnt/data directory, send a GET request to the file/content/{filename} endpoint. The response includes the file data.

Before you send the request, replace the placeholders between the <> brackets with values specific to your request.

GET https://<REGION>.dynamicsessions.io/subscriptions/<SUBSCRIPTION_ID>/resourceGroups/<RESOURCE_GROUP>/sessionPools/<SESSION_POOL_NAME>/files/content/<FILE_NAME_AND_EXTENSION>?api-version=2024-02-02-preview&identifier=<SESSION_ID>
Authorization: Bearer <TOKEN>

List the files in a session

To list the files in a session's /mnt/data directory, send a GET request to the files endpoint.

Before you send the request, replace the placeholders between the <> brackets with values specific to your request.

GET https://<REGION>.dynamicsessions.io/subscriptions/<SUBSCRIPTION_ID>/resourceGroups/<RESOURCE_GROUP>/sessionPools/<SESSION_POOL_NAME>/files?api-version=2024-02-02-preview&identifier=<SESSION_ID>
Authorization: Bearer <TOKEN>

The response contains a list of files in the session.

The following listing shows a sample of the type of response you can expect from requesting session contents.

{
    "$id": "1",
    "value": [
        {
            "$id": "2",
            "properties": {
                "$id": "3",
                "filename": "test1.txt",
                "size": 16,
                "lastModifiedTime": "2024-05-02T07:21:07.9922617Z"
            }
        },
        {
            "$id": "4",
            "properties": {
                "$id": "5",
                "filename": "test2.txt",
                "size": 17,
                "lastModifiedTime": "2024-05-02T07:21:08.8802793Z"
            }
        }
    ]
}

Preinstalled packages

Python code interpreter sessions include popular Python packages such as NumPy, pandas, and scikit-learn.

To output the list of preinstalled packages, call the code/execute endpoint with the following code.

Before you send the request, replace the placeholders between the <> brackets with values specific to your request.

POST https://<REGION>.dynamicsessions.io/subscriptions/<SUBSCRIPTION_ID>/resourceGroups/<RESOURCE_GROUP>/sessionPools/<SESSION_POOL_NAME>/identifier/<SESSION_ID>/code/execute?api-version=2024-02-02-preview&identifier=<SESSION_ID>
Content-Type: application/json
Authorization: Bearer <TOKEN>

{
    "properties": {
        "codeInputType": "inline",
        "executionType": "synchronous",
        "code": "import pkg_resources\n[(d.project_name, d.version) for d in pkg_resources.working_set]"
    }
}

Logging

Code interpreter sessions don't support logging directly. Your application that's interacting with the sessions can log requests to the session pool management API and its responses.

Billing

Code interpreter sessions are billed based on the duration of each session. See Billing for more information.

Next steps

Quickstart: use code interpreter sessions with LangChain

Jaa

Serverless code interpreter sessions in Azure Container Apps

Uses for code interpreter sessions

Code interpreter session pool

Create a session pool with Azure CLI

Get the pool management API endpoint with Azure CLI

Code execution in a session

Session identifiers

Authentication

LLM framework integrations

Management API endpoints

Execute code in a session

Upload a file to a session

Download a file from a session

List the files in a session

Preinstalled packages

Logging

Billing

Next steps

Palaute

Lisäresursseja