Connection Challenges with Azure Databricks and SQL Server On VM in Serverless compute.

Question

We successfully configured VNet peering between the SQL VM's VNet and the Databricks VNet, allowing access to the SQL Server on VMs. Testing the connection from Databricks using a job cluster yielded successful results, confirming that everything functions as intended. However, when attempting to run the same notebook on a serverless compute, we encountered connectivity issues, preventing us from establishing a connection. This limitation compels us to rely on the shared job cluster, which requires time for scaling and initiating instances when none are active.

Answer 1

Hi Vetrivel,

 

I encountered a similar issue with a customer who faced connectivity challenges between Databricks serverless compute and SQL resources within a VNet. They had VNet peering set up successfully, allowing job clusters within Databricks to connect to SQL VMs without issue. However, when they attempted to use serverless compute for the same purpose, connectivity failed due to serverless compute's inability to access resources within peered VNets. 

Here’s how we tackled it:

Private Link Configuration: We enabled Azure Private Link on their Databricks workspace, which allowed the serverless environment to connect to SQL Server through a secure, private endpoint. This approach bypassed the VNet peering limitation by creating a direct path to SQL resources.

Job Clusters Optimization: To address latency and scaling delays in job clusters, we optimized the auto-scaling settings and implemented a schedule for cluster startup times. This approach minimized idle times while ensuring clusters were available when needed.

Gateway Solution: In some cases, we also explored using a gateway or proxy. This intermediary allowed the serverless compute to route traffic to SQL resources within the VNet, effectively acting as a bridge between environments.

These strategies proved effective in overcoming the serverless compute limitation and ensuring seamless connectivity. 

Let me know if any of these would work for your setup, or if you’d like further details on implementing them.