Share via

Configure private connectivity from serverless compute

  • Article

This article describes how to configure private connectivity from serverless compute using the Azure Databricks account console UI. You can also use the Network Connectivity Configurations API.

If you configure your Azure resource to only accept connections from private endpoints, any connection to the resource from your classic Databricks compute resources also needs to use private endpoints.

To configure an Azure Storage firewall for serverless compute access using subnets, instead see Configure a firewall for serverless compute access. To manage existing private endpoint rules, see Manage private endpoint rules.

Important

Starting December 4, 2024, Databricks will begin charging for networking costs on serverless workloads that connect to external resources. Billing will be implemented gradually, and you might not be charged until after December 4, 2024. You won’t be charged retroactively for usage before billing is enabled. After billing is enabled, you may be charged for:

  • Private connectivity to your resources over Private Link. Data processing charges for private connectivity to your resources over Private Link are waived indefinitely. Per-hour charges will apply.
  • Public connectivity to your resources over NAT gateway.
  • Data transfer charges incurred, such as when serverless compute and the target resource are in different regions.

Overview of private connectivity for serverless compute

Serverless network connectivity is managed with network connectivity configurations (NCCs). Account admins create NCCs in the account console and an NCC can be attached to one or more workspaces

When you add a private endpoint in an NCC, Azure Databricks creates a private endpoint request to your Azure resource. Once the request is accepted on the resource side, the private endpoint is used to access resources from the serverless compute plane. The private endpoint is dedicated to your Azure Databricks account and accessible only from authorized workspaces.

NCC private endpoints are supported from SQL warehouses, jobs, notebooks, Delta Live Tables, and model serving endpoints.

Note

NCC private endpoints are only supported for data sources that you manage. For connection to the workspace storage account, contact your Azure Databricks account team.

Note

Model serving uses Azure blob storage path to download model artifacts, so create a private endpoint for your sub-resource ID blob. You will need DFS to log models in Unity Catalog from serverless notebooks.

For more information on NCCs, see What is a network connectivity configuration (NCC)?.

Requirements

  • Your workspace must be on the Premium plan.
  • You must be an Azure Databricks account admin.
  • Each Azure Databricks account can have up to 10 NCCs per region.
  • Each region can have 100 private endpoints, distributed as needed across 1-10 NCCs.
  • Each NCC can be attached to up to 50 workspaces.

Step 1: Create a network connectivity configuration

Databricks recommends sharing an NCC among workspaces within the same business unit and those sharing the same region connectivity properties. For example, if some workspaces use Private Link and other workspaces use firewall enablement, use separate NCCs for those use cases.

  1. As an account admin, go to the account console.
  2. In the sidebar, click Cloud Resources.
  3. Click Network Connectivity Configs.
  4. Click Add Network Connectivity Config.
  5. Type a name for the NCC.
  6. Choose the region. This must match your workspace region.
  7. Click Add.

Step 2: Attach an NCC to a workspace

  1. In the account console sidebar, click Workspaces.
  2. Click your workspace’s name.
  3. Click Update workspace.
  4. In the Network Connectivity Config field, select your NCC. If it’s not visible, confirm that you’ve selected the same Azure region for both the workspace and the NCC.
  5. Click Update.
  6. Wait 10 minutes for the change to take effect.
  7. Restart any running serverless services in the workspace.

Step 3: Create private endpoint rules

You must create a private endpoint rule in your NCC for each Azure resource.

  1. Get a list of Azure resource IDs for all your destinations.
    1. In another browser tab, use the Azure portal navigate to your data source’s Azure services.
    2. On its Overview page, look in the Essentials section.
    3. Click the JSON View link. The resource ID for the service is displayed at the top of the page.
    4. Copy that resource ID to another location. Repeat for all destinations. For more information about finding your resource ID, see Azure Private Endpoint private DNS zone values.
  2. Switch back to your account console browser tab.
  3. In the sidebar, click Cloud Resources.
  4. Click Network Connectivity Configs.
  5. Select the NCC that you created in step 1.
  6. In Private endpoint rules, click Add private endpoint rule.
  7. In the Destination Azure resource ID field, paste the resource ID for your resource.
  8. In the Azure subresource ID field, specify the destination ID and subresource type. Each private endpoint rule must use a different subresource ID. For a list of supported subresource types, see Azure Private Link availability.
  9. Click Add.
  10. Wait a few minutes until all endpoint rules have the status PENDING.

Step 4: Approve the new private endpoints on your resources

The endpoints do not take effect until an admin with rights on the resource approves the new private endpoint. To approve a private endpoint using the Azure portal, do the following:

  1. In the Azure portal, navigate to your resource.

  2. In the sidebar, click Networking.

  3. Click Private endpoint connections.

  4. Click the Private access tab.

  5. Under the Private endpoint connections, review the list of private endpoints.

  6. Click the checkbox next to each one to approve, and click the Approve button above the list.

  7. Return to your NCC in Azure Databricks and refresh the browser page until all endpoint rules have the status ESTABLISHED.

    Private endpoint list

(Optional) Step 5: Set your storage account to disallow public network access

If you haven’t already limited access to the Azure storage account to only allow-listed networks, you can choose to do this.

  1. Go to the Azure portal.
  2. Navigate to your storage account for the data source.
  3. In the sidebar, click Networking.
  4. In the field Public network access, check the value. By default, the value is Enabled from all networks. Change this to Disabled

Step 6: Restart serverless compute plane resources and test the connection

  1. After the previous step, wait five additional minutes for the changes to propagate.
  2. Restart any running serverless compute plane resources in the workspaces that your NCC is attached to. If you don’t have any running serverless compute plane resources, start one now.
  3. Confirm that all resources start successfully.
  4. Run at least one query on your data source to confirm that the serverless SQL warehouse can reach your data source.