Hi Vetrivel,
I encountered a similar issue with a customer who faced connectivity challenges between Databricks serverless compute and SQL resources within a VNet. They had VNet peering set up successfully, allowing job clusters within Databricks to connect to SQL VMs without issue. However, when they attempted to use serverless compute for the same purpose, connectivity failed due to serverless compute's inability to access resources within peered VNets.
Here’s how we tackled it:
Private Link Configuration: We enabled Azure Private Link on their Databricks workspace, which allowed the serverless environment to connect to SQL Server through a secure, private endpoint. This approach bypassed the VNet peering limitation by creating a direct path to SQL resources.
Job Clusters Optimization: To address latency and scaling delays in job clusters, we optimized the auto-scaling settings and implemented a schedule for cluster startup times. This approach minimized idle times while ensuring clusters were available when needed.
Gateway Solution: In some cases, we also explored using a gateway or proxy. This intermediary allowed the serverless compute to route traffic to SQL resources within the VNet, effectively acting as a bridge between environments.
These strategies proved effective in overcoming the serverless compute limitation and ensuring seamless connectivity.
Let me know if any of these would work for your setup, or if you’d like further details on implementing them.