Connection Challenges with Azure Databricks and SQL Server On VM in Serverless compute.

Vetrivel Dhakshinamoorthy 20 Reputation points
2024-10-28T05:19:04+00:00

We successfully configured VNet peering between the SQL VM's VNet and the Databricks VNet, allowing access to the SQL Server on VMs. Testing the connection from Databricks using a job cluster yielded successful results, confirming that everything functions as intended. However, when attempting to run the same notebook on a serverless compute, we encountered connectivity issues, preventing us from establishing a connection. This limitation compels us to rely on the shared job cluster, which requires time for scaling and initiating instances when none are active.

SQL Server on Azure Virtual Machines
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,221 questions
{count} votes

Accepted answer
  1. RevelinoB 3,505 Reputation points
    2024-10-28T05:29:54.6933333+00:00

    Hi Vetrivel,

     

    I encountered a similar issue with a customer who faced connectivity challenges between Databricks serverless compute and SQL resources within a VNet. They had VNet peering set up successfully, allowing job clusters within Databricks to connect to SQL VMs without issue. However, when they attempted to use serverless compute for the same purpose, connectivity failed due to serverless compute's inability to access resources within peered VNets. 

    Here’s how we tackled it:

    Private Link Configuration: We enabled Azure Private Link on their Databricks workspace, which allowed the serverless environment to connect to SQL Server through a secure, private endpoint. This approach bypassed the VNet peering limitation by creating a direct path to SQL resources.

    Job Clusters Optimization: To address latency and scaling delays in job clusters, we optimized the auto-scaling settings and implemented a schedule for cluster startup times. This approach minimized idle times while ensuring clusters were available when needed.

    Gateway Solution: In some cases, we also explored using a gateway or proxy. This intermediary allowed the serverless compute to route traffic to SQL resources within the VNet, effectively acting as a bridge between environments.

    These strategies proved effective in overcoming the serverless compute limitation and ensuring seamless connectivity. 

    Let me know if any of these would work for your setup, or if you’d like further details on implementing them.


0 additional answers

Sort by: Most helpful

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.