Unable to Enable Serverless Clusters in Azure Databricks Workspace

Question

I've been trying to activate serverless clusters in Azure Databricks, but I can't find the feature enabled in the workspace console settings. My understanding is that this feature should be available in all regions; I'm currently working in US East. Could this feature be restricted in the Sponsor subscription, or am I missing some configuration? I am an Azure and workspace admin. Additionally, does the cost of these types of clusters also get covered by the Sponsor Credits?

Answer 1

It sounds like you’re encountering some limitations in enabling serverless clusters in Azure Databricks. Let me break this down and address your questions:

1. Availability of Serverless Clusters

While serverless clusters are designed to be available in all Azure regions, there could be limitations based on:

• Subscription Type: Some features, including serverless clusters, may not be available for certain types of Azure subscriptions, such as a Sponsor subscription. This is often due to restrictions imposed on certain SKUs or billing setups.
• Workspace Configuration: Ensure that the workspace is set up with Premium Tier Databricks. Serverless clusters are not supported on the Standard Tier.

2. Steps to Check/Enable

Here’s what you can do to confirm the issue and potentially resolve it:

• Navigate to your Databricks Workspace > Admin Console > Cluster Policies and check for the “Enable Serverless Compute” option.
• If the option is not visible, verify that the Databricks Premium Plan is active in your workspace settings.
• Confirm that your Azure region (US East) supports Databricks Serverless. It should, but it’s good to double-check in the Azure Regional Availability Guide.

3. Sponsor Subscription Restrictions

Yes, certain features may be restricted under a Sponsor subscription, including those that generate significant or variable costs. This is especially true for serverless clusters, as they are billed based on usage. Reach out to your subscription administrator or Azure Support to confirm if serverless clusters are restricted in your current subscription type.

4. Sponsor Credits for Serverless Clusters

Azure Sponsor Credits typically cover all eligible Azure services, including Databricks clusters (standard and serverless). However, since serverless clusters may not yet be enabled in your workspace, confirm whether the billing rules for your subscription include these clusters. Contact your Azure Sponsor program administrator or check your Azure Cost Management settings to ensure compatibility.

Next Steps:

1. Verify your Databricks Premium Plan and check for serverless compute settings.
2. Confirm whether your Sponsor subscription includes serverless cluster capabilities.
3. Reach out to Azure Support for specific guidance on feature enablement and billing coverage.

---