Azure Databricks

Question

Azure Databricks

Hemanth Kumar 0

We have our production Azure Databricks resources configured with two subnets: one for the host and one for the container, both using a /27 CIDR block. As we plan to execute multiple jobs and only have 16 IPs remaining, we have decided to expand the subnets from /27 to /24. How can we achieve this without disrupting our existing resources? Can the Databricks support team assist us with this process?

Chandra Boorla 9,985 Reputation points Microsoft External Staff

2025-03-12T19:26:16.52+00:00

@Hemanth Kumar

Just checking in to see if the below answer provided by @ Alex Burlachenko helped.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

1 answer

Your answer

Chandra Boorla 9,985 Reputation points Microsoft External Staff

2025-03-12T19:26:16.52+00:00

@Hemanth Kumar

Just checking in to see if the below answer provided by @ Alex Burlachenko helped.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Answer 1

Dear Hemanth,

Thank you for reaching out regarding your Azure Databricks subnet expansion. Expanding subnets from /27 to /24 is a significant change, and it’s important to ensure that this process is carried out without disrupting your existing resources.

Plan the Subnet Expansion

Review your current network configuration and ensure that the new /24 CIDR block does not overlap with other subnets in your virtual network (VNet).
Confirm that the expanded subnet size aligns with your future scalability needs.

Create New Subnets

In your Azure portal, create new subnets with the /24 CIDR block in the same VNet where your Databricks workspace is deployed.
Ensure that the new subnets have the necessary configurations (e.g., NSGs, route tables) to match the existing setup.

Update Azure Databricks Workspace Configuration

Azure Databricks does not support directly modifying the CIDR block of existing subnets. Instead, you will need to:
- Create a new workspace with the updated subnet configuration (/24 CIDR block).
- Migrate your existing workloads, clusters, and configurations to the new workspace.
Refer to the official documentation for guidance on migrating workspaces.

Test the New Workspace

Before decommissioning the old workspace, thoroughly test the new workspace to ensure that all jobs, clusters, and configurations are functioning as expected.

Decommission the Old Workspace

Once the new workspace is fully operational and validated, you can decommission the old workspace to free up resources.

Yes, the Azure Databricks support team can provide guidance and best practices for subnet expansion and workspace migration. If you encounter any issues during the process, you can reach out to them via the Azure Support portal.

Downtime, oh hoh, migrating to a new workspace may involve some downtime, so plan this during a maintenance window.

Data and Configuration Backup

Ensure that all data, notebooks, and configurations are backed up before starting the migration.

Networking Dependencies

Verify that any dependencies (e.g., private endpoints, VNet peering) are updated to reflect the new subnet configuration.

Create new subnets with the /24 CIDR block.

Set up a new Databricks workspace with the updated subnets.

Migrate your workloads and test the new workspace.

Reach out to Azure Databricks support if you need assistance.

Best regards,

Alex p.s.

If you found the answer helpful, please click on Upvote and Accept Answer. This will help other community members.

Hemanth Kumar 0 Reputation points

2025-03-11T13:40:11.0066667+00:00

Hi @Anonymous , Thanks for the response. I understood this from the online resources and the documentation in the internet whatever you explained here. However, my point is to

Use the same workspace. I cannot delete the exisiting databricks workspace

use the subnet with the same name but a different cidr which is /24 for my container subnet.

My host subnet is /27 at the moment. Do i need to change my host subnet to /24 along with my container subnet

How about all the permissions and the access at the metastore level if we migrate to a new workspace?
Alex Burlachenko 1,755 Reputation points

2025-03-11T13:49:19.9066667+00:00

No, you do not need to change your host subnet to /24 if it is not causing IP exhaustion. You can keep the host subnet at /27 while expanding only the container subnet to /24. However, since Azure Databricks does not allow modifying existing subnets, you will need to create new subnets for both the host and container (even if the host subnet size remains the same) and recreate the workspace.

Best regards,
Alex
p.s. If you found the answer helpful, please click on Upvote and Accept Answer. This will help other community members.
Alex Burlachenko 1,755 Reputation points

2025-03-11T13:55:08.27+00:00

Attach the existing metastore to the new workspace. Permissions persist automatically. See Unity Catalog docs.

Export metadata and recreate permissions manually in the new workspace. Refer to the migration guide.

Best regards, Alex
Alex Burlachenko 1,755 Reputation points

2025-03-11T13:58:57.3933333+00:00

and last one what i get - unfortunately, Azure Databricks does not support modifying the subnets (e.g., changing the CIDR block) of an existing workspace. You must create a new workspace with the desired subnet configuration and migrate your workloads. The existing workspace cannot be updated to use new subnets.
rgds,

Alex

P.S. If you found the answer helpful, please click on Upvote and Accept Answer. This will help other community members.

Share via

Azure Databricks

1 answer

Your answer