2,340 questions with Azure Databricks tags

Sort by: Updated
0 answers

Unity Catalog is not enabled Please contact your workspace administrator to enable Unity Catalog to use Genie.

Unity Catalog is not enabled Please contact your workspace administrator to enable Unity Catalog to use Genie. Can not find how to check above.

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,340 questions
asked 2025-02-22T14:16:08.28+00:00
TSUKAMOTO, KUNITAKA 25 Reputation points
1 answer

how to disbale serverless for azure databricks notebook

I want to restrict the serverless in notebook in my dev workspace and i want to disable the feature . i am not getting option where to disable it .

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,340 questions
asked 2025-02-22T08:03:22.1233333+00:00
Anuj Sen 0 Reputation points
answered 2025-02-22T10:09:46.22+00:00
Vinodh247 28,371 Reputation points MVP
1 answer

Unexpected Azure Databricks VM Pricing – Matches RI Cost Without RI Purchase

I have an Azure Databricks customer with both Dev and Prod environments. The concern is regarding the VM cost associated with Databricks. The VM type is D4ads_v5 Linux, located in Southeast Asia. According to the Azure Pricing Calculator, the expected…

Azure Cost Management
Azure Cost Management
A Microsoft offering that enables tracking of cloud usage and expenditures for Azure and other cloud providers.
3,047 questions
Azure Virtual Network
Azure Virtual Network
An Azure networking service that is used to provision private networks and optionally to connect to on-premises datacenters.
2,646 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,340 questions
asked 2025-02-19T13:34:20+00:00
Sundarakannan M 61 Reputation points
commented 2025-02-21T18:43:32.74+00:00
phemanth 13,905 Reputation points Microsoft Vendor
2 answers

I’ve been using the following code for years to determine the latest table version using Databricks’ Time Travel feature without any issues. However, after adding a new row to the table recently, I encountered the following error:

Cannot time travel Delta table to version 1. Available versions: [3, 23]. This behavior is unexpected, as the code has worked reliably until now. Here is the code I’m using: from delta.tables import DeltaTable import pyspark.sql.functions dt =…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,340 questions
asked 2025-02-19T12:37:44.52+00:00
Stephen01 125 Reputation points
commented 2025-02-21T14:36:24.9666667+00:00
Ganesh Gurram 4,335 Reputation points Microsoft Vendor
0 answers

I need to download files from databricks onto my local machine

My attempt at using the url method returns a 401 error "credential was not sent or was of an unsupported type for this API" My attempt as using the cdi is also unsuccessful. I do not have the download option available in the dbfs, I have the…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,340 questions
asked 2025-02-20T21:48:05.9466667+00:00
Jaden Materi 0 Reputation points
commented 2025-02-21T04:58:23.84+00:00
J N S S Kasyap 85 Reputation points Microsoft Vendor
1 answer

Automate Lineage into azure purview from Azure Databricks Unity Catalog

Hi Team, In my last question someone from Microsoft team suggested to create lineage tables separately and scan the databricks unity catalog source and then lineage automatically came to azure purview. But when I tried this solution, I'm not able to get…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,340 questions
Microsoft Purview
Microsoft Purview
A Microsoft data governance service that helps manage and govern on-premises, multicloud, and software-as-a-service data. Previously known as Azure Purview.
1,406 questions
asked 2025-02-13T08:15:02.3066667+00:00
Sri Lakshman Velugubantla 20 Reputation points
commented 2025-02-21T04:39:30.73+00:00
Sri Lakshman Velugubantla 20 Reputation points
1 answer

Does Databricks permit an enterprise to maintain multiple Databricks accounts within a single region?

Does Databricks permit an enterprise to maintain multiple Databricks accounts within a single region in Azure? We need two isolated Unity Catalog metastores in the same region.

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,340 questions
asked 2025-02-20T21:34:28.15+00:00
Sanju Dabi 0 Reputation points
answered 2025-02-21T04:33:49.2333333+00:00
Chandra Boorla 8,870 Reputation points Microsoft Vendor
1 answer

How to set up secrets in Azure Databricks Cluster configs?

Hi there, I am currently working on setting up Spark configurations within our Databricks cluster to access Azure Data Lake Storage (ADLS) using OAuth and a service principal. Our goal is to configure these settings at the cluster level, so that we can…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,340 questions
asked 2025-02-19T16:41:50.09+00:00
Swathi 0 Reputation points
commented 2025-02-21T02:53:15.0766667+00:00
Chandra Boorla 8,870 Reputation points Microsoft Vendor
1 answer

Your Role: Global Administrator and 1 other roles does not work with Databricks

Hi, Your Role: Global Administrator and 1 other roles. This I can see In Roles and Administrators. But I cannot - from Databricks - get to Manage Account from email on top right, which means I am not Global Administrator. What is the problem? Seems…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,340 questions
asked 2025-02-20T19:46:32.8233333+00:00
Ged Alexander 0 Reputation points
answered 2025-02-20T20:56:38.3433333+00:00
Jose Benjamin Solis Nolasco 236 Reputation points
2 answers

Unexpected Azure Databricks VM Pricing – Matches RI Cost Without RI Purchase

I have an Azure Databricks customer with both Dev and Prod environments. The concern is regarding the VM cost associated with Databricks. The VM type is D4ads_v5 Linux, located in Southeast Asia. According to the Azure Pricing Calculator, the expected…

Azure Virtual Machines
Azure Virtual Machines
An Azure service that is used to provision Windows and Linux virtual machines.
8,406 questions
Azure Cost Management
Azure Cost Management
A Microsoft offering that enables tracking of cloud usage and expenditures for Azure and other cloud providers.
3,047 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,340 questions
asked 2025-02-19T13:20:43.8466667+00:00
Sundarakannan M 61 Reputation points
commented 2025-02-20T06:00:34.03+00:00
anashetty 2,275 Reputation points Microsoft Vendor
1 answer

Troubleshooting Azure Databricks to On-premise MS SQL Server VPN tunnel instability

We have a VPN tunnel that connects Azure Databricks to an on-premise server. The tunnel is however instable, and the phase 2 selector goes down daily at random times. We did an audit and it does seem to be mostly after working hours, varying from 7pm to…

Azure Virtual Network
Azure Virtual Network
An Azure networking service that is used to provision private networks and optionally to connect to on-premises datacenters.
2,646 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,340 questions
asked 2025-02-14T10:35:45.9233333+00:00
Nathalie 20 Reputation points
commented 2025-02-19T10:10:54.7+00:00
Praveen Bandaru 515 Reputation points Microsoft Vendor
1 answer

Azure Databricks - Database Error

I created the following database in Azure Databricks - CREATE DATABASE IF NOT EXISTS f1_processed LOCATION "/mnt/<Container_Name>/<Folder_Name>" Database got created under "hive_metastore" Then I'm trying to create a…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,340 questions
asked 2025-02-13T04:55:01.34+00:00
Ramanathan Murugan Veerabagu 0 Reputation points
commented 2025-02-18T12:50:28.17+00:00
Smaran Thoomu 20,395 Reputation points Microsoft Vendor
1 answer

Pyspark dataframe is taking too long to save on ADLS from Databricks.

I'm running a notebook on Azure databricks using a multinode cluster with 1 driver and 1-8 workers(each with 16 cores and 56 gb ram). Reading the source data from Azure ADLS which has 30K records. Notebook is consist of few transformation steps, also…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,541 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,340 questions
asked 2022-11-29T10:09:42.24+00:00
Pratik Roy 1 Reputation point
commented 2025-02-18T04:26:33.81+00:00
Madhur Tyagi 0 Reputation points
2 answers

Can't access the Azure Databricks Accounts Console

Hi there, When I tried to log on to https://accounts.azuredatabricks.net/login, I encountered an error. It says: Selected user account does not exist in tenant 'Microsoft Services' and cannot access the application…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,340 questions
asked 2024-07-30T12:22:12.19+00:00
Ziwei Zeng 10 Reputation points
commented 2025-02-17T16:22:40.1333333+00:00
Ramu Kamath 5 Reputation points
1 answer

Getting error while creating database

In Azure Databricks, I'm getting error while creating Database with the following syntax : - CREATE DATABASE IF NOT EXISTS <DB_Name> LOCATION "/mnt/<Container_Name>/<Folder_Name>" error is 1 "CREATE SCHEMA in Unity…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,340 questions
asked 2025-02-13T04:09:46.5566667+00:00
Ramanathan Murugan Veerabagu 0 Reputation points
commented 2025-02-17T12:16:32.76+00:00
Smaran Thoomu 20,395 Reputation points Microsoft Vendor
1 answer

Unable to Find 'Manage Accounts' Menu for Unity Catalog Metastore Setup

I'm trying to set up the Unity Catalog Metastore, but I can't find the 'Manage Accounts' menu in the 'Accounts' section, so I can't access the account console. Can anyone help me with this issue?"

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,340 questions
asked 2025-02-10T18:33:16.54+00:00
deepak shetty 0 Reputation points
edited a comment 2025-02-12T15:41:47.8066667+00:00
Ganesh Gurram 4,335 Reputation points Microsoft Vendor
2 answers

Is Azure Synapse being discontinued?

Hello everyone! I recently heard that Microsoft might be discontinuing the development of Azure Synapse, and I’d like to know more about it if that’s the case. If it is being deprecated, is there an alternative to Serverless SQL Pool? I know Databricks…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,200 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,340 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,274 questions
asked 2025-02-10T12:10:09.0033333+00:00
EDUARDO SOARES AZZOLIN 0 Reputation points
edited the question 2025-02-12T10:49:31.9+00:00
Smaran Thoomu 20,395 Reputation points Microsoft Vendor
3 answers

How to specify a custom catalog name for Azure Databricks Delta Lake Dataset in ADF

Hello, I am creating an Azure Databricks Delta Lake Dataset in ADF and I am only able to choose the database name that links to Databricks's hive_metastore. How can I specify a custom catalog name that I created in Databricks instead of…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,541 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,340 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,274 questions
asked 2024-01-04T06:25:53.11+00:00
Tom Young 0 Reputation points
commented 2025-02-12T09:26:43.0333333+00:00
Milankumar Soni, Meetkumar 0 Reputation points
1 answer

Unable to connect to Azure Databricks repo with Azure DevOps server 2020

We are establishing the connection with Azure Databricks repo with the On-premise Azure DevOps Server 2020. We created a service principal and added to the databricks. we created a PAT from On-premise Azure DevOps server and added in the linked…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,340 questions
asked 2025-02-10T07:20:19.6933333+00:00
Chunduru, Vamsee Krishna 0 Reputation points
commented 2025-02-12T01:36:37.4066667+00:00
Chandra Boorla 8,870 Reputation points Microsoft Vendor
1 answer

Vnet Injected Databricks Workspace with IP ACL gives 403 while executing notebook using compute cluster

I have an IP access list enabled on my Vnet injected databricks workspace in Azure. I can access my workspace as my VPN ip is enabled on the workspace IP list. However, when I try executing my notebook by attaching a cluster, it gives below error- 403:…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,340 questions
asked 2025-02-07T08:42:14.1133333+00:00
Pratima Patil 30 Reputation points
commented 2025-02-11T12:16:51.06+00:00
Smaran Thoomu 20,395 Reputation points Microsoft Vendor