Classic compute plane networking
This article introduces features to customize network access between the Azure Databricks control plane and the classic compute plane. Connectivity between the control plane and the serverless compute plane is always over the cloud network backbone and not the public internet.
To learn more about the control plane and the compute plane, see Azure Databricks architecture overview.
To learn more about classic compute and serverless compute, see Types of compute.
The features in this section focus on establishing and securing the connection between the Azure Databricks control plane and classic compute plane. This connection is labeled as 2 the diagram below:
For more information on configuring Azure networking features between Azure Databricks and Azure storage, see Grant your Azure Databricks workspace access to Azure Data Lake Storage Gen2.
Enable secure cluster connectivity
Databricks recommends that you enable secure cluster connectivity on your Azure Databricks workspaces. When secure cluster connectivity is enabled, compute resources in the classic compute plane connect to the control plane through a relay. This means customer virtual networks have no open ports and compute plane resources have no public IP addresses. This simplifies network administration by removing the need to configure ports on security groups or network peering. To learn more about deploying a workspace with secure cluster connectivity, see Enable secure cluster connectivity.
Deploy a workspace in your own virtual network
By default, every Azure Databricks deployment creates a locked virtual network (VNet) in your Azure subscription. Classic compute resources are created in that virtual network. You can choose to create a new workspace in your own customer-managed virtual network (also known as VNet injection) instead, enabling you to:
- Secure the connection from Azure Databricks to Azure storage using service endpoints or private endpoints. See Grant your Azure Databricks workspace access to Azure Data Lake Storage Gen2.
- Restrict outbound traffic from your virtual network using network security group rules.
- Secure the connection to an on-premises network from Azure Databricks, taking advantage of user-defined routes. See Connect your Azure Databricks workspace to your on-premises network and User-defined route settings for Azure Databricks.
To deploy a workspace in your own virtual network, see Deploy Azure Databricks in your Azure virtual network (VNet injection). You can also peer the Azure Databricks virtual network with another Azure virtual network, see Peer virtual networks.
Enable private connectivity from the control plane to the classic compute plane
Azure Private Link provides private connectivity from Azure VNets and on-premises networks to Azure services without exposing the traffic to the public network. You can enable private connectivity from the classic compute plane to Azure Databricks workspace’s core services in the control plane by enabling Azure Private Link.
For more information, see Enable Azure Private Link back-end and front-end connections.