Edit

Share via

Configure a private endpoint for an Azure Machine Learning workspace

  • Article

APPLIES TO: Azure CLI ml extension v2 (current)

In this document, you learn how to configure a private endpoint for your Azure Machine Learning workspace. For information on creating a virtual network for Azure Machine Learning, see Virtual network isolation and privacy overview.

Azure Private Link enables you to restrict connections to your workspace to an Azure Virtual Network. You restrict a workspace to only accept connections from a virtual network by creating a private endpoint. The private endpoint is a set of private IP addresses within your virtual network. You can then limit access to your workspace to only occur over the private IP addresses. A private endpoint helps reduce the risk of data exfiltration. To learn more about private endpoints, see the Azure Private Link article.

Warning

Securing a workspace with private endpoints does not ensure end-to-end security by itself. You must secure all of the individual components of your solution. For example, if you use a private endpoint for the workspace, but your Azure Storage Account is not behind the VNet, traffic between the workspace and storage does not use the VNet for security.

For more information on securing resources used by Azure Machine Learning, see the following articles:

Prerequisites

  • You must have an existing virtual network to create the private endpoint in.

    Warning

    Do not use the 172.17.0.0/16 IP address range for your VNet. This is the default subnet range used by the Docker bridge network, and will result in errors if used for your VNet. Other ranges may also conflict depending on what you want to connect to the virtual network. For example, if you plan to connect your on premises network to the VNet, and your on-premises network also uses the 172.16.0.0/16 range. Ultimately, it is up to you to plan your network infrastructure.

  • Disable network policies for private endpoints before adding the private endpoint.

Limitations

  • If you enable public access for a workspace secured with private endpoint and use Azure Machine Learning studio over the public internet, some features such as the designer might fail to access your data. This problem happens when the data is stored on a service that is secured behind the virtual network. For example, an Azure Storage Account.

  • If you're using Mozilla Firefox, you might encounter problems trying to access the private endpoint for your workspace. This problem might be related to DNS over HTTPS in Mozilla Firefox. We recommend using Microsoft Edge or Google Chrome.

  • Using a private endpoint doesn't affect Azure control plane (management operations) such as deleting the workspace or managing compute resources. For example, creating, updating, or deleting a compute target. These operations are performed over the public Internet as normal. Data plane operations, such as using Azure Machine Learning studio, APIs (including published pipelines), or the SDK use the private endpoint.

  • When you create a compute instance or compute cluster in a workspace with a private endpoint, the compute instance and compute cluster must be in the same Azure region as the workspace.

  • When you attach an Azure Kubernetes Service cluster to a workspace with a private endpoint, the cluster must be in the same region as the workspace.

  • When you use a workspace with multiple private endpoints, one of the private endpoints must be in the same virtual network as the following dependency services:

    • Azure Storage Account that provides the default storage for the workspace
    • Azure Key Vault for the workspace
    • Azure Container Registry for the workspace.

    For example, one virtual network ('services') would contain a private endpoint for the dependency services and the workspace. This configuration allows the workspace to communicate with the services. Another virtual network ('clients') might only contain a private endpoint for the workspace, and be used only for communication between client development machines and the workspace.

Create a workspace that uses a private endpoint

Use one of the following methods to create a workspace with a private endpoint. Each of these methods requires an existing virtual network:

Tip

If you'd like to create a workspace, private endpoint, and virtual network at the same time, see Use an Azure Resource Manager template to create a workspace for Azure Machine Learning.

APPLIES TO: Azure CLI ml extension v2 (current)

When you use the Azure CLI extension 2.0 CLI for machine learning, a YAML document is used to configure the workspace. The following example demonstrates creating a new workspace using a YAML configuration:

Tip

When you use a private link, your workspace cannot use Azure Container Registry tasks compute for image building. Instead, the workspace defaults to using a serverless compute cluster to build images. This works only when the workspace-deependent resources such as the storage account and container registry are not under any network restrictions (private endpoint). If your workspace dependencies are under network restrictions, use the image_build_compute property to specify a compute cluster to use for image building. The image_build_compute property in this configuration specifies a CPU compute cluster name to use for Docker image environment building. You can also specify whether the private link workspace should be accessible over the internet using the public_network_access property.

In this example, the compute referenced by image_build_compute will need to be created before building images.

$schema: https://azuremlschemas.azureedge.net/latest/workspace.schema.json
name: mlw-privatelink-prod
location: eastus
display_name: Private Link endpoint workspace-example
description: When using private link, you must set the image_build_compute property to a cluster name to use for Docker image environment building. You can also specify whether the workspace should be accessible over the internet.
image_build_compute: cpu-compute
public_network_access: Disabled
tags:
  purpose: demonstration

az ml workspace create \
    -g <resource-group-name> \
    --file privatelink.yml

After creating the workspace, use the Azure networking CLI commands to create a private link endpoint for the workspace.

az network private-endpoint create \
    --name <private-endpoint-name> \
    --vnet-name <vnet-name> \
    --subnet <subnet-name> \
    --private-connection-resource-id "/subscriptions/<subscription>/resourceGroups/<resource-group-name>/providers/Microsoft.MachineLearningServices/workspaces/<workspace-name>" \
    --group-id amlworkspace \
    --connection-name workspace -l <location>

To create the private DNS zone entries for the workspace, use the following commands:

# Add privatelink.api.azureml.ms
az network private-dns zone create \
    -g <resource-group-name> \
    --name privatelink.api.azureml.ms

az network private-dns link vnet create \
    -g <resource-group-name> \
    --zone-name privatelink.api.azureml.ms \
    --name <link-name> \
    --virtual-network <vnet-name> \
    --registration-enabled false

az network private-endpoint dns-zone-group create \
    -g <resource-group-name> \
    --endpoint-name <private-endpoint-name> \
    --name myzonegroup \
    --private-dns-zone privatelink.api.azureml.ms \
    --zone-name privatelink.api.azureml.ms

# Add privatelink.notebooks.azure.net
az network private-dns zone create \
    -g <resource-group-name> \
    --name privatelink.notebooks.azure.net

az network private-dns link vnet create \
    -g <resource-group-name> \
    --zone-name privatelink.notebooks.azure.net \
    --name <link-name> \
    --virtual-network <vnet-name> \
    --registration-enabled false

az network private-endpoint dns-zone-group add \
    -g <resource-group-name> \
    --endpoint-name <private-endpoint-name> \
    --name myzonegroup \
    --private-dns-zone privatelink.notebooks.azure.net \
    --zone-name privatelink.notebooks.azure.net

Add a private endpoint to a workspace

Use one of the following methods to add a private endpoint to an existing workspace:

Warning

If you have any existing compute targets associated with this workspace, and they are not behind the same virtual network that the private endpoint is created in, they will not work.

APPLIES TO: Azure CLI ml extension v2 (current)

When using the Azure CLI extension 2.0 CLI for machine learning, use the Azure networking CLI commands to create a private link endpoint for the workspace.

az network private-endpoint create \
    --name <private-endpoint-name> \
    --vnet-name <vnet-name> \
    --subnet <subnet-name> \
    --private-connection-resource-id "/subscriptions/<subscription>/resourceGroups/<resource-group-name>/providers/Microsoft.MachineLearningServices/workspaces/<workspace-name>" \
    --group-id amlworkspace \
    --connection-name workspace -l <location>

To create the private DNS zone entries for the workspace, use the following commands:

# Add privatelink.api.azureml.ms
az network private-dns zone create \
    -g <resource-group-name> \
    --name 'privatelink.api.azureml.ms'

az network private-dns link vnet create \
    -g <resource-group-name> \
    --zone-name 'privatelink.api.azureml.ms' \
    --name <link-name> \
    --virtual-network <vnet-name> \
    --registration-enabled false

az network private-endpoint dns-zone-group create \
    -g <resource-group-name> \
    --endpoint-name <private-endpoint-name> \
    --name myzonegroup \
    --private-dns-zone 'privatelink.api.azureml.ms' \
    --zone-name 'privatelink.api.azureml.ms'

# Add privatelink.notebooks.azure.net
az network private-dns zone create \
    -g <resource-group-name> \
    --name 'privatelink.notebooks.azure.net'

az network private-dns link vnet create \
    -g <resource-group-name> \
    --zone-name 'privatelink.notebooks.azure.net' \
    --name <link-name> \
    --virtual-network <vnet-name> \
    --registration-enabled false

az network private-endpoint dns-zone-group add \
    -g <resource-group-name> \
    --endpoint-name <private-endpoint-name> \
    --name myzonegroup \
    --private-dns-zone 'privatelink.notebooks.azure.net' \
    --zone-name 'privatelink.notebooks.azure.net'

Remove a private endpoint

You can remove one or all private endpoints for a workspace. Removing a private endpoint removes the workspace from the virtual network that the endpoint was associated with. Removing the private endpoint might prevent the workspace from accessing resources in that virtual network, or resources in the virtual network from accessing the workspace. For example, if the virtual network doesn't allow access to or from the public internet.

Warning

Removing the private endpoints for a workspace doesn't make it publicly accessible. To make the workspace publicly accessible, use the steps in the Enable public access section.

To remove a private endpoint, use the following information:

APPLIES TO: Azure CLI ml extension v2 (current)

When using the Azure CLI extension 2.0 CLI for machine learning, use the following command to remove the private endpoint:

az network private-endpoint delete \
    --name <private-endpoint-name> \
    --resource-group <resource-group-name> \

Enable public access

In some situations, you might want to allow someone to connect to your secured workspace over a public endpoint, instead of through the virtual network. Or you might want to remove the workspace from the virtual network and re-enable public access.

Important

Enabling public access doesn't remove any private endpoints that exist. All communications between components behind the VNet that the private endpoint(s) connect to are still secured. It enables public access only to the workspace, in addition to the private access through any private endpoints.

Warning

When connecting over the public endpoint while the workspace uses a private endpoint to communicate with other resources:

  • Some features of studio will fail to access your data. This problem happens when the data is stored on a service that is secured behind the VNet. For example, an Azure Storage Account. To resolve this problem, add your client device's IP address to the Azure Storage Account's firewall.
  • Using Jupyter, JupyterLab, RStudio, or Posit Workbench (formerly RStudio Workbench) on a compute instance, including running notebooks, is not supported.

To enable public access, use the following steps:

Tip

There are two possible properties that you can configure:

  • allow_public_access_when_behind_vnet - used by the Python SDK v1
  • public_network_access - used by the CLI and Python SDK v2 Each property overrides the other. For example, setting public_network_access will override any previous setting to allow_public_access_when_behind_vnet.

Microsoft recommends using public_network_access to enable or disable public access to a workspace.

APPLIES TO: Azure CLI ml extension v2 (current)

When using the Azure CLI extension 2.0 CLI for machine learning, use the az ml update command to enable public_network_access for the workspace:

az ml workspace update \
    --set public_network_access=Enabled \
    -n <workspace-name> \
    -g <resource-group-name>

You can also enable public network access by using a YAML file. For more information, see the workspace YAML reference.

Enable Public Access only from internet IP ranges

You can use IP network rules to allow access to your workspace and endpoint from specific public internet IP address ranges by creating IP network rules. Each Azure Machine Learning workspace supports up to 200 rules. These rules grant access to specific internet-based services and on-premises networks and block general internet traffic.

Warning

APPLIES TO: Azure CLI ml extension v2 (current)

Use the az ml workspace network-rule Azure CLI command to manage public access from an IP address or address range:

Tip

The configurations for the selected IP addresses are stored in the workspace's properties, under network_acls:

properties:
  # ...
  network_acls:
    description: "The network ACLS for this workspace, enforced when public_network_access is set to Enabled."
    $ref: "3/defintions/networkAcls"
  • List IP network rules: az ml workspace network-rule list --resource-group "myresourcegroup" --workspace-name "myWS" --query ipRules
  • Add a rule for a single IP address: az ml workspace network-rule add --resource-group "myresourcegroup" --workspace-name "myWS" --ip-address "16.17.18.19"
  • Add a rule for an IP address range: az ml workspace network-rule add --resource-group "myresourcegroup" --workspace-name "myWS" --ip-address "16.17.18.0/24"
  • Remove a rule for a single IP address: az ml workspace network-rule remove --resource-group "myresourcegroup" --workspace-name "myWS" --ip-address "16.17.18.19"
  • Remove a rule for an IP address range: az ml workspace network-rule remove --resource-group "myresourcegroup" --workspace-name "myWS" --ip-address "16.17.18.0/24"

You can also use the Workspace class from the Azure Machine Learning Python SDK to define which IP addresses are allowed inbound access:

Workspace( 
  public_network_access = "Enabled", 
  network_rule_set = NetworkRuleSet(default_action = "Allow", bypass = "AzureServices", resource_access_rules = None, ip_rules = yourIPAddress,)

Restrictions for IP network rules

The following restrictions apply to IP address ranges:

  • IP network rules are allowed only for public internet IP addresses.

    Reserved IP address ranges aren't allowed in IP rules such as private addresses that start with 10, 172.16 to 172.31, and 192.168.

  • You must provide allowed internet address ranges by using CIDR notation in the form 16.17.18.0/24 or as individual IP addresses like 16.17.18.19.

  • Only IPv4 addresses are supported for configuration of storage firewall rules.

  • When this feature is enabled, you can test public endpoints using any client tool such as Curl, but the Endpoint Test tool in the portal isn't supported.

  • You can only set the IP addresses for the workspace after the workspace has been created.

Securely connect to your workspace

To connect to a workspace that's secured behind a VNet, use one of the following methods:

  • Azure VPN gateway - Connects on-premises networks to the VNet over a private connection. Connection is made over the public internet. There are two types of VPN gateways that you might use:

    • Point-to-site: Each client computer uses a VPN client to connect to the VNet.
    • Site-to-site: A VPN device connects the VNet to your on-premises network.

  • ExpressRoute - Connects on-premises networks into the cloud over a private connection. Connection is made using a connectivity provider.

  • Azure Bastion - In this scenario, you create an Azure Virtual Machine (sometimes called a jump box) inside the VNet. You then connect to the VM using Azure Bastion. Bastion allows you to connect to the VM using either an RDP or SSH session from your local web browser. You then use the jump box as your development environment. Since it is inside the VNet, it can directly access the workspace. For an example of using a jump box, see Tutorial: Create a secure workspace.

Important

When using a VPN gateway or ExpressRoute, you will need to plan how name resolution works between your on-premises resources and those in the VNet. For more information, see Use a custom DNS server.

If you have problems connecting to the workspace, see Troubleshoot secure workspace connectivity.

Multiple private endpoints

Azure Machine Learning supports multiple private endpoints for a workspace. Multiple private endpoints are often used when you want to keep different environments separate. The following are some scenarios that are enabled by using multiple private endpoints:

  • Client development environments in a separate virtual network.

  • An Azure Kubernetes Service (AKS) cluster in a separate virtual network.

  • Other Azure services in a separate virtual network. For example, Azure Synapse and Azure Data Factory can use a Microsoft managed virtual network. In either case, a private endpoint for the workspace can be added to the managed virtual network used by those services. For more information on using a managed virtual network with these services, see the following articles:

    Important

    Synapse's data exfiltration protection is not supported with Azure Machine Learning.

Important

Each VNet that contains a private endpoint for the workspace must also be able to access the Azure Storage Account, Azure Key Vault, and Azure Container Registry used by the workspace. For example, you might create a private endpoint for the services in each VNet.

Adding multiple private endpoints uses the same steps as described in the Add a private endpoint to a workspace section.

Scenario: Isolated clients

If you want to isolate the development clients, so they don't have direct access to the compute resources used by Azure Machine Learning, use the following steps:

Note

These steps assume that you have an existing workspace, Azure Storage Account, Azure Key Vault, and Azure Container Registry. Each of these services has a private endpoints in an existing VNet.

  1. Create another virtual network for the clients. This virtual network might contain Azure Virtual Machines that act as your clients, or it might contain a VPN Gateway used by on-premises clients to connect to the virtual network.
  2. Add a new private endpoint for the Azure Storage Account, Azure Key Vault, and Azure Container Registry used by your workspace. These private endpoints should exist in the client virtual network.
  3. If you have another storage that is used by your workspace, add a new private endpoint for that storage. The private endpoint should exist in the client virtual network and have private DNS zone integration enabled.
  4. Add a new private endpoint to your workspace. This private endpoint should exist in the client virtual network and have private DNS zone integration enabled.
  5. To enable Azure Machine Learning studio to access the storage accounts, visit the studio in a virtual network article.

The following diagram illustrates this configuration. The Workload virtual network contains compute resources created by the workspace for training & deployment. The Client virtual network contains clients or client ExpressRoute/VPN connections. Both VNets contain private endpoints for the workspace, Azure Storage Account, Azure Key Vault, and Azure Container Registry.

Diagram of isolated client VNet

Scenario: Isolated Azure Kubernetes Service

If you want to create an isolated Azure Kubernetes Service used by the workspace, use the following steps:

Note

These steps assume that you have an existing workspace, Azure Storage Account, Azure Key Vault, and Azure Container Registry. Each of these services has a private endpoints in an existing VNet.

  1. Create an Azure Kubernetes Service instance. During creation, AKS creates a virtual network that contains the AKS cluster.
  2. Add a new private endpoint for the Azure Storage Account, Azure Key Vault, and Azure Container Registry used by your workspace. These private endpoints should exist in the client virtual network.
  3. If you have other storage that is used by your workspace, add a new private endpoint for that storage. The private endpoint should exist in the client virtual network and have private DNS zone integration enabled.
  4. Add a new private endpoint to your workspace. This private endpoint should exist in the client virtual network and have private DNS zone integration enabled.
  5. Attach the AKS cluster to the Azure Machine Learning workspace. For more information, see Create and attach an Azure Kubernetes Service cluster.

Diagram of isolated AKS VNet

Scenario: Managed online endpoints with access from selected IP addresses

Enabling inbound access from selected IP addresses is affected by the ingress setting on your managed online endpoints. If public ingress is enabled on your managed online endpoint, then you can't enable selected IP addresses on our workspace.

The following table shows the possible configurations for your workspace and managed online endpoint network configurations, and how it affects both. For more information, see Network isolation with managed online endpoints.

Workspace
public network access		 Managed online endpoint
public network access		 Does the workspace
respect the selected IPs?		 Does the online endpoint
respect the selected IPs?
Disabled Disabled No (all public traffic rejected) No
Disabled Enabled No (all public traffic rejected) Yes
Enabled from selected IPs Disabled Yes No
Enabled from selected IPs Enabled Yes Yes

Note

If the workspace public network access configuration is changed from selected IPs to disabled, the managed online enedpoints will continue to respect the selected IPs. If you do not want thee selected IPs applied to your online endpoints, remove the addresses before selecting Disabled for the workspace in the Azure portal. The Python SDK and Azure CLI support this change after or before.

Scenario: Batch endpoints with access from selected IP addresses

The selected IP's configuration isn't supported for batch endpoints. There's no public network access flag on batch endpoints. If the Azure Machine Learning workspace is disabled, and private link enabled, the batch endpoint is private as well. If the workspace's public network access is changed from disabled to enabled, the batch endpoints stay private and don't become public. For more information, see Securing batch endpoints.

Related content