Issue with Specifying Cluster Log Path During Databricks Unity Catalog Upgrade

Question

Hi Team,

As part of our upgrade to Unity Catalog, we are removing all mount point usages. During this process, we encountered an issue while specifying the cluster log destination in the Linked Service configuration. We noticed that DBFS paths (with mount points) are being used for cluster logs, and we attempted to specify an ABFSS path (e.g., abfss://******@ABCDadls.dfs.core.windows.net/Logs/ClusterLogs) or a Volume path, but both resulted in the following error:

dbfs destination (abfss://******@ABCDadls.dfs.core.windows.net/Logs/ClusterLogs) is not valid. Use the format 'dbfs:/my/dbfs/path'.

It seems that the Linked Service for cluster logs only accepts DBFS paths. In Unity Catalog, we aim to remove mount points and should instead reference paths directly from cloud storage, but this issue is blocking us from using the desired external storage.

Could you please advise on how we can correctly configure the cluster log path in the Linked Service for Unity Catalog?

Thank you for your support!

Best regards

Answer 1

@Sharukh Kundagol

Greetings & Welcome to Microsoft Q&A forum! Thanks for posting your query!

It appears that the issue you are facing is due to the requirement for cluster log paths to be specified using DBFS paths in the Linked Service configuration. Unity Catalog does not currently support specifying paths using ABFSS or Volume paths for cluster logs, which is why you are encountering the error message.

Issue Overview

Current Configuration - You are trying to specify a log destination for your Databricks cluster using an ABFSS path or a volume path.

Error Message - The error indicates that the system only accepts DBFS paths in the format dbfs:/my/dbfs/path, which is problematic since you are trying to eliminate mount points in favor of direct cloud storage references.

Understanding the Query

Unity Catalog Upgrade - As part of your upgrade to Unity Catalog, you are removing mount points, which is a good practice for better data governance and management.

Cluster Log Destination - The cluster log destination is crucial for monitoring and debugging, and it seems that the current configuration does not support the desired external storage paths.

Possible Solutions

Use DBFS Paths - If the system only accepts DBFS paths, you may need to create a temporary mount point those points to your desired external storage location. This would allow you to use a DBFS path while still accessing the external storage. However, this is a workaround and may not align with your goal of removing mount points.

Appreciate if you could share the feedback on our Microsoft Feedback Channel. Which would be open for the user community to upvote & comment on. This allows our product teams to effectively prioritize your request against our existing feature backlog and gives insight into the potential impact of implementing the suggested feature.

For more details, please refer the following documents:

Best practices for DBFS and Unity Catalog

How do paths work for data managed by Unity Catalog?

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

Thank you.