Need for volumes in Databricks

Dhruv Singla 105

Why do we need Volumes when we can access the location using external locations? The doc says it is to add governance, but we can already govern using external locations. So, why add another layer of governance? I am guessing that instead of giving access to the entire external location, we can provide access to a specific subfolder of the location using volumes.

Smaran Thoomu 20,935 Reputation points Microsoft External Staff

2024-09-25T19:54:29.8466667+00:00

Hi @Dhruv Singla

Welcome to the Microsoft Q&A platform.

Volumes in Databricks provide a way to manage data access and governance in a more granular way than external locations. While external locations allow you to access data stored outside of Databricks, volumes provide a way to manage access to specific subfolders within those external locations.Volumes can be used to provide access to specific data sets or subfolders within an external location, while restricting access to other data sets or subfolders. This can help to ensure that users only have access to the data they need, while maintaining governance and security.

In addition, volumes provide a way to manage data access and governance within Databricks itself, rather than relying on external systems. This can help to simplify data management and ensure that data is governed consistently across all Databricks workloads.

Overall, volumes provide a way to manage data access and governance in a more granular and flexible way than external locations, while also providing a way to manage data access and governance within Databricks itself.

Reference: https://docs.databricks.com/en/files/volumes.html

https://docs.databricks.com/en/sql/language-manual/sql-ref-volumes.html

I hope this helps. Please let me know if you have any questions!
Dhruv Singla 105 Reputation points

2024-09-25T20:55:12.0833333+00:00

I have some doubt. Will put it in a comment
Dhruv Singla 105 Reputation points

2024-09-25T20:56:27.6433333+00:00

Can you add pictures showing an id that could not access a storage location (the pic with error). And then the id was able to access the location due to it being given access by volumes and still not having storage location access.

I wanted to verify by my end, but am unable to create a unity catalog enabled databricks workspace even on my pay-as-you-go subscription.
Smaran Thoomu 20,935 Reputation points Microsoft External Staff

2024-09-27T20:59:19.49+00:00
@Dhruv Singla Thank you for your follow-up question.

I can provide you with a conceptual explanation and some code examples to illustrate how Volumes can be used to control access in Databricks, even when direct access to the storage location is restricted.

Conceptual Explanation:

Storage Account: Imagine a cloud storage account (like Azure Blob Storage or AWS S3) containing your data.

Restricted Access: A Databricks user or a cluster doesn't have direct permissions to access this storage account.

Volumes as a Bridge: A Databricks administrator creates a Volume. This Volume:

Points to a specific subfolder within the restricted storage account.

Is granted the necessary permissions to access that subfolder.

Controlled Access: Now, the administrator can grant specific users or clusters permission to access the data through the Volume. This allows them to read and write data within the designated subfolder, even though they lack direct access to the storage account.

Illustrative Code Example (PySpark):

# Let's assume a Volume named "my_volume" has been created, # pointing to a subfolder within a restricted storage location. # Access data within the Volume (without direct storage access) df = spark.read.format("csv").load("dbfs:/Volumes/my_volume/data.csv") df.show() # Write data to the Volume df.write.format("parquet").save("dbfs:/Volumes/my_volume/processed_data/")

Key Points

Security: Volumes enhance security by restricting access to only the data users need.

Data Governance: They streamline data governance by centralizing access control within Databricks.

Abstraction: Volumes abstract away the complexities of underlying storage permissions from end-users.

I hope this explanation is helpful! Let me know if you have any further questions.
Dhruv Singla 105 Reputation points

2024-09-28T14:29:57.4933333+00:00

Thanks for the conceptual explanation, but I want to see it. I know the theoretical part but can't believe it until I test it. I don't have a unity catalog environment by myself otherwise wouldn't have asked for it. Can you try to provide screenshots?
Smaran Thoomu 20,935 Reputation points Microsoft External Staff

2024-09-30T18:27:08.28+00:00

@Dhruv Singla I apologize for the inconvenience caused. Unfortunately, I cannot provide you with screenshots as I do not have access to the unity catalog external environment.

1 answer

Paras Vora 0 Reputation points

2025-03-03T13:54:55.6+00:00

Hello,

At the risk of over simplification, Volumes in the Unity Catalog are primarily created to access the files or unstructured data which may not be supported by Delta lake protocols. If you have tabular data as external tables, you don't really need to use Volumes. There are a certain types of compute clusters through which you can either access the Unity Catalog features or the traditional / classic features like DBUtils and data access from mount points directly. Here, using volumes, you can do both i.e. use the Unity Catalog features and be able to read files from the external storage

For example, with a 'shared' access cluster using which you can access Unity catalog features but you cannot read files from mount points /mnt/filestore/ . you will simply get an access denied error.

Hope this helps,
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Need for volumes in Databricks

1 answer

Your answer