Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,338 questions
0 answers
I need to download files from databricks onto my local machine
My attempt at using the url method returns a 401 error "credential was not sent or was of an unsupported type for this API" My attempt as using the cdi is also unsuccessful. I do not have the download option available in the dbfs, I have the…
asked 2025-02-20T21:48:05.9466667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(60.8, 9%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJM%3C/text%3E%3C/svg%3E)
Jaden Materi
0
Reputation points
commented 2025-02-21T04:58:23.84+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(252.8, 99%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJN%3C/text%3E%3C/svg%3E)
J N S S Kasyap
80
Reputation points Microsoft Vendor
1 answer
Automate Lineage into azure purview from Azure Databricks Unity Catalog
Hi Team, In my last question someone from Microsoft team suggested to create lineage tables separately and scan the databricks unity catalog source and then lineage automatically came to azure purview. But when I tried this solution, I'm not able to get…
asked 2025-02-13T08:15:02.3066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(297.6, 5%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESL%3C/text%3E%3C/svg%3E)
Sri Lakshman Velugubantla
20
Reputation points
commented 2025-02-21T04:39:30.73+00:00
Sri Lakshman Velugubantla
20
Reputation points
1 answer
Does Databricks permit an enterprise to maintain multiple Databricks accounts within a single region?
Does Databricks permit an enterprise to maintain multiple Databricks accounts within a single region in Azure? We need two isolated Unity Catalog metastores in the same region.
asked 2025-02-20T21:34:28.15+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(124.80000000000001, 10%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESD%3C/text%3E%3C/svg%3E)
Sanju Dabi
0
Reputation points
answered 2025-02-21T04:33:49.2333333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 65%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECB%3C/text%3E%3C/svg%3E)
Chandra Boorla
8,795
Reputation points Microsoft Vendor
1 answer
How to set up secrets in Azure Databricks Cluster configs?
Hi there, I am currently working on setting up Spark configurations within our Databricks cluster to access Azure Data Lake Storage (ADLS) using OAuth and a service principal. Our goal is to configure these settings at the cluster level, so that we can…
asked 2025-02-19T16:41:50.09+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(140.8, 63%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
Swathi
0
Reputation points
commented 2025-02-21T02:53:15.0766667+00:00
Chandra Boorla
8,795
Reputation points Microsoft Vendor
2 answers
I’ve been using the following code for years to determine the latest table version using Databricks’ Time Travel feature without any issues. However, after adding a new row to the table recently, I encountered the following error:
Cannot time travel Delta table to version 1. Available versions: [3, 23]. This behavior is unexpected, as the code has worked reliably until now. Here is the code I’m using: from delta.tables import DeltaTable import pyspark.sql.functions dt =…
asked 2025-02-19T12:37:44.52+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 32%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
Stephen01
125
Reputation points
commented 2025-02-20T21:05:21.2033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 18%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGG%3C/text%3E%3C/svg%3E)
Ganesh Gurram
4,335
Reputation points Microsoft Vendor
1 answer
Your Role: Global Administrator and 1 other roles does not work with Databricks
Hi, Your Role: Global Administrator and 1 other roles. This I can see In Roles and Administrators. But I cannot - from Databricks - get to Manage Account from email on top right, which means I am not Global Administrator. What is the problem? Seems…
asked 2025-02-20T19:46:32.8233333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 15%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGA%3C/text%3E%3C/svg%3E)
Ged Alexander
0
Reputation points
answered 2025-02-20T20:56:38.3433333+00:00
Jose Benjamin Solis Nolasco
76
Reputation points
1 answer
Unexpected Azure Databricks VM Pricing – Matches RI Cost Without RI Purchase
I have an Azure Databricks customer with both Dev and Prod environments. The concern is regarding the VM cost associated with Databricks. The VM type is D4ads_v5 Linux, located in Southeast Asia. According to the Azure Pricing Calculator, the expected…
asked 2025-02-19T13:34:20+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(214.4, 67%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESM%3C/text%3E%3C/svg%3E)
Sundarakannan M
61
Reputation points
commented 2025-02-20T19:29:45.37+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
phemanth
13,900
Reputation points Microsoft Vendor
2 answers
Unexpected Azure Databricks VM Pricing – Matches RI Cost Without RI Purchase
I have an Azure Databricks customer with both Dev and Prod environments. The concern is regarding the VM cost associated with Databricks. The VM type is D4ads_v5 Linux, located in Southeast Asia. According to the Azure Pricing Calculator, the expected…
asked 2025-02-19T13:20:43.8466667+00:00
Sundarakannan M
61
Reputation points
commented 2025-02-20T06:00:34.03+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(54.400000000000006, 73%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
anashetty
2,275
Reputation points Microsoft Vendor
1 answer
Troubleshooting Azure Databricks to On-premise MS SQL Server VPN tunnel instability
We have a VPN tunnel that connects Azure Databricks to an on-premise server. The tunnel is however instable, and the phase 2 selector goes down daily at random times. We did an audit and it does seem to be mostly after working hours, varying from 7pm to…
asked 2025-02-14T10:35:45.9233333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 26%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EN%3C/text%3E%3C/svg%3E)
Nathalie
20
Reputation points
commented 2025-02-19T10:10:54.7+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 15%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPB%3C/text%3E%3C/svg%3E)
Praveen Bandaru
515
Reputation points Microsoft Vendor
1 answer
Azure Databricks - Database Error
I created the following database in Azure Databricks - CREATE DATABASE IF NOT EXISTS f1_processed LOCATION "/mnt/<Container_Name>/<Folder_Name>" Database got created under "hive_metastore" Then I'm trying to create a…
asked 2025-02-13T04:55:01.34+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(131.20000000000002, 57.99999999999999%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERM%3C/text%3E%3C/svg%3E)
Ramanathan Murugan Veerabagu
0
Reputation points
commented 2025-02-18T12:50:28.17+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
20,305
Reputation points Microsoft Vendor
1 answer
Pyspark dataframe is taking too long to save on ADLS from Databricks.
I'm running a notebook on Azure databricks using a multinode cluster with 1 driver and 1-8 workers(each with 16 cores and 56 gb ram). Reading the source data from Azure ADLS which has 30K records. Notebook is consist of few transformation steps, also…
asked 2022-11-29T10:09:42.24+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(179.20000000000002, 57.00000000000001%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPR%3C/text%3E%3C/svg%3E)
Pratik Roy
1
Reputation point
commented 2025-02-18T04:26:33.81+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 36%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMT%3C/text%3E%3C/svg%3E)
Madhur Tyagi
0
Reputation points
2 answers
Can't access the Azure Databricks Accounts Console
Hi there, When I tried to log on to https://accounts.azuredatabricks.net/login, I encountered an error. It says: Selected user account does not exist in tenant 'Microsoft Services' and cannot access the application…
asked 2024-07-30T12:22:12.19+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 39%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EZZ%3C/text%3E%3C/svg%3E)
Ziwei Zeng
10
Reputation points
commented 2025-02-17T16:22:40.1333333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(211.20000000000002, 78%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERK%3C/text%3E%3C/svg%3E)
Ramu Kamath
5
Reputation points
1 answer
Getting error while creating database
In Azure Databricks, I'm getting error while creating Database with the following syntax : - CREATE DATABASE IF NOT EXISTS <DB_Name> LOCATION "/mnt/<Container_Name>/<Folder_Name>" error is 1 "CREATE SCHEMA in Unity…
asked 2025-02-13T04:09:46.5566667+00:00
Ramanathan Murugan Veerabagu
0
Reputation points
commented 2025-02-17T12:16:32.76+00:00
Smaran Thoomu
20,305
Reputation points Microsoft Vendor
1 answer
Unable to Find 'Manage Accounts' Menu for Unity Catalog Metastore Setup
I'm trying to set up the Unity Catalog Metastore, but I can't find the 'Manage Accounts' menu in the 'Accounts' section, so I can't access the account console. Can anyone help me with this issue?"
asked 2025-02-10T18:33:16.54+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 96%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDS%3C/text%3E%3C/svg%3E)
deepak shetty
0
Reputation points
edited a comment 2025-02-12T15:41:47.8066667+00:00
Ganesh Gurram
4,335
Reputation points Microsoft Vendor
2 answers
Is Azure Synapse being discontinued?
Hello everyone! I recently heard that Microsoft might be discontinuing the development of Azure Synapse, and I’d like to know more about it if that’s the case. If it is being deprecated, is there an alternative to Serverless SQL Pool? I know Databricks…
asked 2025-02-10T12:10:09.0033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(249.60000000000002, 40%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EES%3C/text%3E%3C/svg%3E)
EDUARDO SOARES AZZOLIN
0
Reputation points
edited the question 2025-02-12T10:49:31.9+00:00
Smaran Thoomu
20,305
Reputation points Microsoft Vendor
3 answers
How to specify a custom catalog name for Azure Databricks Delta Lake Dataset in ADF
Hello, I am creating an Azure Databricks Delta Lake Dataset in ADF and I am only able to choose the database name that links to Databricks's hive_metastore. How can I specify a custom catalog name that I created in Databricks instead of…
asked 2024-01-04T06:25:53.11+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(249.60000000000002, 17%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETY%3C/text%3E%3C/svg%3E)
Tom Young
0
Reputation points
commented 2025-02-12T09:26:43.0333333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 48%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMS%3C/text%3E%3C/svg%3E)
Milankumar Soni, Meetkumar
0
Reputation points
1 answer
Unable to connect to Azure Databricks repo with Azure DevOps server 2020
We are establishing the connection with Azure Databricks repo with the On-premise Azure DevOps Server 2020. We created a service principal and added to the databricks. we created a PAT from On-premise Azure DevOps server and added in the linked…
asked 2025-02-10T07:20:19.6933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(233.6, 38%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECV%3C/text%3E%3C/svg%3E)
Chunduru, Vamsee Krishna
0
Reputation points
commented 2025-02-12T01:36:37.4066667+00:00
Chandra Boorla
8,795
Reputation points Microsoft Vendor
1 answer
Vnet Injected Databricks Workspace with IP ACL gives 403 while executing notebook using compute cluster
I have an IP access list enabled on my Vnet injected databricks workspace in Azure. I can access my workspace as my VPN ip is enabled on the workspace IP list. However, when I try executing my notebook by attaching a cluster, it gives below error- 403:…
asked 2025-02-07T08:42:14.1133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(22.400000000000002, 64%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPP%3C/text%3E%3C/svg%3E)
Pratima Patil
30
Reputation points
commented 2025-02-11T12:16:51.06+00:00
Smaran Thoomu
20,305
Reputation points Microsoft Vendor
1 answer
Azure Databricks and Key vault key versioning
We are building a Data encryption solution and are using pyspark AES encrypt and AES Decrypt function to encrypt the PII columns in the Databricks tables on Unity catalog. We have created views on top of these tables and using the same encryption key we…
asked 2025-01-28T07:07:43.85+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 34%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHK%3C/text%3E%3C/svg%3E)
Hemanth Kumar
0
Reputation points
commented 2025-02-11T09:21:50.0066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(188.79999999999998, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EN%3C/text%3E%3C/svg%3E)
Navya
15,555
Reputation points Microsoft Vendor
2 answers
How to use databricks ai to auto generate data definitions for all the tables in my database?
I know we can go to the catalog in databricks and generate data definitions for columns inside of our database using ai, but is there a way of automatically generating these definitions without have to manually generate them and click accept on every…
asked 2024-03-06T21:31:37.0066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 49%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBZ%3C/text%3E%3C/svg%3E)
Bradley, Zack
0
Reputation points
commented 2025-02-10T18:05:50.6533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 93%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMC%3C/text%3E%3C/svg%3E)
Mauricio Caro
0
Reputation points