Error "Invalid configuration value detected for fs.azure.account.keyInvalid configuration value detected for fs.azure.account.key" when listing files stored in an Azure Storage account using "dbutils.fs.ls"

Question

Hi,

I want to get a list of files stored in an Azure Storage account using "dbutils.fs.ls" command in Databricks. But, I get the following error.

Failure to initialize configuration for storage account AAAAAA.dfs.core.windows.net: Invalid configuration value detected for fs.azure.account.keyInvalid configuration value detected for fs.azure.account.key

Databricks workspace:

Unity catalog is enabled.

There is a catalog named Exploration. User has "ALL PRIVILEGES" grant on this catalog.

Storage Account:

Azure storage account: ADSL Gen2

User has "Read/Write/Execute" to the root folder "Exploration". Also, user has the same access to the sub-folder "test".

What do I need to do to be able to list the files?

Notes:

• I am able to create sub-folders in the root folder Exploration.
• I can upload files to the Azure storage account.
  When I try to list files in the root folder, I get another error:

Answer 1

@Mohammad Saber - Thanks for the question and using MS Q&A platform.

Regarding the first error message: Failure to initialize configuration for storage account AAAAAA.dfs.core.windows.net: Invalid configuration value detected for fs.azure.account.keyInvalid configuration value detected for fs.azure.account.key

This is a known issue that - you can't use storage account access key to access data using the abfss protocol.

Reason: You can't use storage account SAS Token to access data using the abfss protocol. You need to provide more configuration options if you want to use abfss - it's all described in documentation.

For more details, refer to the below links:

https://stackoverflow.com/questions/69825487/error-invalid-configuration-value-detected-for-fs-azure-account-key
https://medium.com/@kyle.hale/troubleshooting-invalid-configuration-value-detected-for-fs-azure-account-key-6c6fcc67b217

Regarding the second error message: Input path url xxxxxxxxxx overlaps with managed storage.

Running the list command on a managed directory is not supported in Unity Catalog. Catalog/schema storage locations are reserved for managed storage.

For more details, refer to the below links:
https://kb.databricks.com/en_US/unity-catalog/invalid_parameter_valuelocation_overlap-overlaps-with-managed-storage-error

https://community.databricks.com/s/question/0D58Y00009wikefSAA/dbutilsfsls-doesnt-work-on-unity-catalog-managed-location

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Answer 2

Change your databricks cluster to 12.2 LTS and above to resolve this error.