Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,345 questions
1 answer
how to disbale serverless for azure databricks notebook
I want to restrict the serverless in notebook in my dev workspace and i want to disable the feature . i am not getting option where to disable it .
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(208, 28.999999999999996%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
1 answer
How to allow Databricks to only read from one storage account and read/write from a different storage account?
RBAC setup: Under the same subscription I have resource group 1 and resource group 2. There is a Databricks instance created in resource group 1 in which a service principal was created. Within the resource group 1 storage account the service principal…
asked 2025-02-27T22:29:11.89+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(224.00000000000003, 37%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBD%3C/text%3E%3C/svg%3E)
Brian Dietrich
0
Reputation points
answered 2025-02-28T03:46:02.0966667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(252.8, 99%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJN%3C/text%3E%3C/svg%3E)
J N S S Kasyap
270
Reputation points Microsoft Vendor
1 answer
Retrieving data from Azure storage account archive tier through Azure Databricks
Hi Community, Recently we are studying the practice to perform data retention in Azure. We are using Databricks and Storage Account as data lake. We are thinking of using Databricks to extract part of the data from delta table and write them to the…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(25.6, 87%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EB%3C/text%3E%3C/svg%3E)
0 answers
How to connect On prem tableau server to Azure Databricks
Trying to establish a connection between on prem tableau server to Azure Databricks. Do we have to create App registration for the tableau server?
asked 2025-02-27T20:07:20.87+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(246.4, 88%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERV%3C/text%3E%3C/svg%3E)
Rajagopalan, Valli (She/Her/Hers) (MNIT)
0
Reputation points
commented 2025-02-27T22:11:01.9733333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 18%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGG%3C/text%3E%3C/svg%3E)
Ganesh Gurram
4,340
Reputation points Microsoft Vendor
1 answer
Your Role: Global Administrator and 1 other roles does not work with Databricks
Hi, Your Role: Global Administrator and 1 other roles. This I can see In Roles and Administrators. But I cannot - from Databricks - get to Manage Account from email on top right, which means I am not Global Administrator. What is the problem? Seems…
asked 2025-02-20T19:46:32.8233333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 15%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGA%3C/text%3E%3C/svg%3E)
Ged Alexander
0
Reputation points
commented 2025-02-27T18:09:25.5633333+00:00
Ged Alexander
0
Reputation points
1 answer
Queries regarding Serverless compute in Azure Databricks
We have received a service health advisory from Microsoft with ID "PTC3-9V8" regarding the Serverless compute transition on Azure Databricks by 24 March 2025. After going through the advisory, we have few queries which we want to…
asked 2025-02-26T23:22:03.45+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(284.8, 39%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECS%3C/text%3E%3C/svg%3E)
Chakraborty, Shubham
60
Reputation points
answered 2025-02-27T09:54:06.7433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
20,630
Reputation points Microsoft Vendor
1 answer
How can I prohibit ordinary users to execute notebooks of shared folder in Azure Databricks?
Hello, How can I prohibit ordinary users to create, modify, and execute notebooks of shared folder in Azure Databricks? For security reasons, I want to prevent ordinary users from accidentally sharing the results of SELECT sensitive data in the Shared…
asked 2025-02-25T04:03:49.2733333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(208, 50%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPB%3C/text%3E%3C/svg%3E)
Power BI Test User 2
0
Reputation points
commented 2025-02-27T04:09:25.1933333+00:00
J N S S Kasyap
270
Reputation points Microsoft Vendor
2 answers
One of the answers was accepted by the question author.
Error "Invalid configuration value detected for fs.azure.account.keyInvalid configuration value detected for fs.azure.account.key" when listing files stored in an Azure Storage account using "dbutils.fs.ls"
Hi, I want to get a list of files stored in an Azure Storage account using "dbutils.fs.ls" command in Databricks. But, I get the following error. Failure to initialize configuration for storage account AAAAAA.dfs.core.windows.net: Invalid…
asked 2023-06-21T11:09:15.6666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(16, 5%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMS%3C/text%3E%3C/svg%3E)
Mohammad Saber
591
Reputation points
commented 2025-02-26T21:05:32.96+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(259.20000000000005, 22%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EFR%3C/text%3E%3C/svg%3E)
Fredy Rivera
0
Reputation points Microsoft Employee
1 answer
Azure Databricks Too Many Requests errors
We are getting many errors with loading notebooks and also now running jobs on clusters due to Databricks saying it has too many requests. For example, getting the below error message: run failed with error message Cluster '0724-103023-f2llqh3p' was…
asked 2024-07-24T10:51:26.46+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 19%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
Matthieu Marshall
6
Reputation points
commented 2025-02-26T08:48:51.29+00:00
Matthieu Marshall
6
Reputation points
1 answer
One of the answers was accepted by the question author.
Unity Catalog is not enabled Please contact your workspace administrator to enable Unity Catalog to use Genie.
Unity Catalog is not enabled Please contact your workspace administrator to enable Unity Catalog to use Genie. Can not find how to check above.
asked 2025-02-22T14:16:08.28+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(150.4, 41%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETK%3C/text%3E%3C/svg%3E)
TSUKAMOTO, KUNITAKA
45
Reputation points
accepted 2025-02-26T03:25:55.9766667+00:00
TSUKAMOTO, KUNITAKA
45
Reputation points
2 answers
Error when creating table in Unity Catalog enabled cluster with 14.3 version
I am trying to create a table in schema using dbutils.fs.ls("abfss://****@storageaccount.dfs.core.windows.net/fullpath/") in Unity Catalog enabled cluster with 14.3 version,** but I keep getting this error message: "AnalysisException:…
asked 2024-09-09T14:25:36.2433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(51.2, 77%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAG%3C/text%3E%3C/svg%3E)
Ashwini Gaikwad
130
Reputation points
commented 2025-02-25T14:36:31.2433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(166.4, 45%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ER%3C/text%3E%3C/svg%3E)
rookie_pookie
20
Reputation points
1 answer
One of the answers was accepted by the question author.
Databricks Simba Spark ODBC .NET8 C# Driver Parameters in SQL Queries
Hello, I'm using Simba ODBC driver v2.8.0 in order to query data from my azure databrick sql warehouse into a .net 8 Asp.net Api App. The ODBC driver works fine using plain text query but i need to parametrize the query. Searching around I found that it…
asked 2024-06-03T15:30:12.4666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(265.6, 63%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELN%3C/text%3E%3C/svg%3E)
Luigi Navarra
30
Reputation points
commented 2025-02-25T12:30:17.1+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(160, 66%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPK%3C/text%3E%3C/svg%3E)
Pawan Kumar
0
Reputation points
1 answer
Basic SKU public IP retirement Databricks query
With the Retirement Announcement - Basic SKU Public IP addresses will be retired on 30 September 2025 is there any change that needs to be made to Azure Databricks workers? Currently we've got a number of workers that auto scale which all use Basic SKU.…
asked 2024-11-26T15:41:12.72+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(3.2, 94%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMB%3C/text%3E%3C/svg%3E)
Matty Burn
0
Reputation points
edited an answer 2025-02-25T09:18:03.63+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 65%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECB%3C/text%3E%3C/svg%3E)
Chandra Boorla
9,120
Reputation points Microsoft Vendor
1 answer
Automate Lineage into azure purview from Azure Databricks Unity Catalog
Hi Team, In my last question someone from Microsoft team suggested to create lineage tables separately and scan the databricks unity catalog source and then lineage automatically came to azure purview. But when I tried this solution, I'm not able to get…
asked 2025-02-13T08:15:02.3066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(297.6, 5%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESL%3C/text%3E%3C/svg%3E)
Sri Lakshman Velugubantla
20
Reputation points
commented 2025-02-25T04:37:42.1+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
phemanth
14,220
Reputation points Microsoft Vendor
1 answer
Does Databricks permit an enterprise to maintain multiple Databricks accounts within a single region?
Does Databricks permit an enterprise to maintain multiple Databricks accounts within a single region in Azure? We need two isolated Unity Catalog metastores in the same region.
asked 2025-02-20T21:34:28.15+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(124.80000000000001, 10%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESD%3C/text%3E%3C/svg%3E)
Sanju Dabi
0
Reputation points
commented 2025-02-25T01:20:04.49+00:00
Chandra Boorla
9,120
Reputation points Microsoft Vendor
1 answer
How to set up secrets in Azure Databricks Cluster configs?
Hi there, I am currently working on setting up Spark configurations within our Databricks cluster to access Azure Data Lake Storage (ADLS) using OAuth and a service principal. Our goal is to configure these settings at the cluster level, so that we can…
asked 2025-02-19T16:41:50.09+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(140.8, 63%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
Swathi
0
Reputation points
commented 2025-02-25T01:08:09.47+00:00
Chandra Boorla
9,120
Reputation points Microsoft Vendor
0 answers
I need to download files from databricks onto my local machine
My attempt at using the url method returns a 401 error "credential was not sent or was of an unsupported type for this API" My attempt as using the cdi is also unsuccessful. I do not have the download option available in the dbfs, I have the…
asked 2025-02-20T21:48:05.9466667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(60.8, 9%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJM%3C/text%3E%3C/svg%3E)
Jaden Materi
0
Reputation points
commented 2025-02-25T00:34:11.9966667+00:00
J N S S Kasyap
270
Reputation points Microsoft Vendor
2 answers
Unexpected Azure Databricks VM Pricing – Matches RI Cost Without RI Purchase
I have an Azure Databricks customer with both Dev and Prod environments. The concern is regarding the VM cost associated with Databricks. The VM type is D4ads_v5 Linux, located in Southeast Asia. According to the Azure Pricing Calculator, the expected…
asked 2025-02-19T13:20:43.8466667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(214.4, 67%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESM%3C/text%3E%3C/svg%3E)
Sundarakannan M
61
Reputation points
commented 2025-02-24T18:51:30.8733333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(54.400000000000006, 73%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
anashetty
2,295
Reputation points Microsoft Vendor
1 answer
Unexpected Azure Databricks VM Pricing – Matches RI Cost Without RI Purchase
I have an Azure Databricks customer with both Dev and Prod environments. The concern is regarding the VM cost associated with Databricks. The VM type is D4ads_v5 Linux, located in Southeast Asia. According to the Azure Pricing Calculator, the expected…
asked 2025-02-19T13:34:20+00:00
Sundarakannan M
61
Reputation points
commented 2025-02-21T18:43:32.74+00:00
phemanth
14,220
Reputation points Microsoft Vendor
2 answers
I’ve been using the following code for years to determine the latest table version using Databricks’ Time Travel feature without any issues. However, after adding a new row to the table recently, I encountered the following error:
Cannot time travel Delta table to version 1. Available versions: [3, 23]. This behavior is unexpected, as the code has worked reliably until now. Here is the code I’m using: from delta.tables import DeltaTable import pyspark.sql.functions dt =…
asked 2025-02-19T12:37:44.52+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 32%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
Stephen01
125
Reputation points
commented 2025-02-21T14:36:24.9666667+00:00
Ganesh Gurram
4,340
Reputation points Microsoft Vendor