Κοινή χρήση μέσω

Networking

  • Άρθρο

This article introduces networking configurations for the deployment and management of Azure Databricks accounts and workspaces.

Important

Effective December 4, 2024, Azure Databricks began charging for networking costs associated with serverless workloads connecting to customer resources. You are currently billed for private endpoint per-hour charges to your resources. Data processing charges for Private Link connections are waived indefinitely. Billing for other networking costs will be rolled out gradually, including:

  • Public connectivity to your resources, like over a NAT gateway.
  • Data transfer charges, such as when serverless compute and the target resource reside in different regions.

Charges will not be applied retroactively.

Azure Databricks architecture overview

Azure Databricks operates out of a control plane and a compute plane.

  • The control plane includes the backend services that Azure Databricks manages in your Azure Databricks account. The web application is in the control plane.
  • The compute plane is where your data is processed. There are two types of compute planes depending on the compute that you are using.
    • For classic Azure Databricks compute, the compute resources are in your Azure subscription in what is called the classic compute plane. This refers to the network in your Azure subscription and its resources. Classic compute plan resources are in the region that your workspace is in.
    • For serverless compute, the serverless compute resources run in a serverless compute plane in your Azure Databricks account. Serverless compute plan resources are in the same cloud region as your workspace’s classic compute plane. You select this region when creating a workspace.

To learn more about classic compute and serverless compute, see Types of compute. For additional architecture information, see Azure Databricks architecture overview.

Secure network connectivity

Azure Databricks provides a secure networking environment by default, but if your organization has additional needs, you can configure network connectivity features between the different networking connections shown in the diagram below.

Network connectivity overview diagram

  1. Users and applications to Azure Databricks: You can configure features to control access and provide private connectivity between users and their Azure Databricks workspaces. See Users to Azure Databricks networking.
  2. The control plane and the classic compute plane: Classic compute resources, such as clusters, are deployed in are in your Azure subscription and connect to the control plane. You can use classic network connectivity features to deploy classic compute plane resources in your own virtual networks and to enable private connectivity from the clusters to the control plane. See Classic compute plane networking.
  3. The serverless compute plane and storage: You can configure private and dedicated connections from serverless compute to storage. See Serverless compute plane networking.

You can configure Azure storage networking features, such as private endpoints to secure the connection between the classic compute plane and your Azure resources. See Grant your Azure Databricks workspace access to Azure Data Lake Storage Gen2 and Networking recommendations for Lakehouse Federation.

You can also enable firewall support for the workspace storage account to limit access to the account from authorized networks and connections. See Enable firewall support for your workspace storage account.

Connectivity between the control plane and the serverless compute plane is always over the Azure network backbone and not the public internet.