Data factory Copy activity fails when copy the data unity catalog table from databricks to storage account gen2

Question

Data factory Copy activity fails when copy the data unity catalog table from databricks to storage account gen2

How to fix the connectivity issue even after providing all required access on catalog schema and tables level.

error details: ErrorCode=AzureDatabricksCommandError,Hit an error when running the command in Azure Databricks. Error details: [INSUFFICIENT_PERMISSIONS] Insufficient privileges:

User does not have permission SELECT on any file. SQLSTATE: 42501.

Answer 1

@vinodh

Greetings & Welcome to Microsoft Q&A forum! Thanks for posting your query!

The error you are encountering indicates that the user does not have the necessary permissions to perform the SELECT operation on the files in the Unity Catalog table. To resolve this issue, you should ensure that the user or service principal executing the Data Factory Copy activity has been granted the appropriate permissions.

Here are some steps to troubleshoot and fix the connectivity issue:

Check Permissions - Verify that the user has been granted the SELECT permission on the specific files or tables in the Unity Catalog. You may need to explicitly grant these permissions if they are not already set.

Catalog and Schema Access - Ensure that the user has the necessary permissions at both the catalog and schema levels. This includes checking if the user has access to the catalog that contains the tables.

Databricks Workspace Configuration - Confirm that the Databricks workspace is properly configured to allow access from Azure Data Factory. This may involve checking network settings and ensuring that the integration runtime can communicate with Databricks.

Review Access Policies - If you are using access control policies, ensure that they are correctly configured to allow the necessary access for the user.

For additional information, please refer: User does not have permission SELECT on ANY File

I hope this information helps. Please do let us know if you have any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.