Cannot create Azure Key Vault backed Databricks Secret Scope with Service Principal (AAD Enterprise App)

Question

In an end-to-end Databricks deployment, orchestrated by Terraform using a Service Principal (AAD Enterprise App), it is not possible to create an Azure Key Vault backed Databricks Secret Scope. You have to do this with a regular user identity. This is preventing full automation of the deployment (e.g., secret management, storage mounting, etc.).

I asked this on StackOverflow and was advised that this is a well-known and well-documented architectural problem with Azure, that I should raise with Microsoft Support for prioritisation. The exact problem that needs to be solved on the Azure side is not clear: If that alone could be described, then I can at least go to Microsoft with more useful information.

Is there a known ticket ID with Azure for this issue, or can the underlying issue be made clear?

Answer 1

Hello @Christopher Harrison ,

Thanks for the question and using MS Q&A platform.

This is a known limitation with Azure Databricks. For more details, please do checkout GitHub issue which addressing same issue.

Unfortunately, this feature is not yet available. Our product team is currently working on this feature.

At this moment we don't have exact ETA on when will this feature will be available. And we will update this thread once it's available.

Appreciate if you could share the feedback on our Azure Databricks feedback channel. Which would be open for the user community to upvote & comment on. This allows our product teams to effectively prioritize your request against our existing feature backlog and gives insight into the potential impact of implementing the suggested feature.

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators