2,218 questions with Azure Databricks tags

Sort by: Updated
0 answers

How to read zip file in Azure Synapse Notebook with extension of .zip.

Hi Team, Could you please advise the pyspark code, how to read zip files in synapse notebook. Example: File_Name - samplefile.zip Note: In the samplefile.zip we have multiple csv files, please let us know. The file Extension is only with .zip. …

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,997 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,218 questions
asked 2024-11-08T16:44:52.83+00:00
edited a comment 2024-11-08T18:31:27.7233333+00:00
phemanth 11,280 Reputation points Microsoft Vendor
0 answers

understanding how to use mergeSchema=True

In COPY INTO under FORMAT_OPTIONS generally i use mergeSchema=True but in few queries i am seeing that it is also metioned under copy_options why ? and I have also seen mergeSchema=true given under options while read or readStream, i have seen that in…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,218 questions
asked 2024-11-07T09:17:43.31+00:00
Mantri Sai Ashish 0 Reputation points
commented 2024-11-08T15:11:28.77+00:00
Ganesh Gurram 1,110 Reputation points Microsoft Vendor
1 answer One of the answers was accepted by the question author.

How to list secret scopes and identify key vaults

What is the best way to view all existing secret scopes within a Databricks workspace? Additionally, how can we identify the specific Azure Key Vault linked to each of these secret scopes? I’d like to understand the steps or API calls needed to retrieve…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,218 questions
asked 2024-11-06T11:50:02.7566667+00:00
Gabriel-2005 325 Reputation points
accepted 2024-11-08T11:25:54.35+00:00
Gabriel-2005 325 Reputation points
1 answer

Databricks vs Batch pool

Hi, I have Googled the difference between using Azure Databrick and Batch pool to run python pipelines and ETL. But I haven’t found a clear difference between them two. Based on what I have Googled, Databrick can become very expensive, so cost matters a…

Azure Batch
Azure Batch
An Azure service that provides cloud-scale job scheduling and compute management.
339 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,218 questions
asked 2024-11-07T14:10:41.1766667+00:00
Renan 40 Reputation points
commented 2024-11-08T09:20:32.0266667+00:00
Renan 40 Reputation points
1 answer

getting error while creating mount point in data bricks

Hi, I am getting the following error when trying to create mount point. py4j.security.Py4JSecurityException: Method public com.databricks.backend.daemon.dbutils.DBUtilsCore$Result…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,218 questions
asked 2024-10-24T09:28:58.9066667+00:00
Lakshmi Narayana Sarma Bhamidipati 90 Reputation points
commented 2024-11-08T08:01:37.0633333+00:00
Lakshmi Narayana Sarma Bhamidipati 90 Reputation points
1 answer

Azure Databricks Workspace: Users are being added to the admins group on their own

We have an Azure Databricks workspace with Unity Catalog enabled. We are using SCIM provisioning to sync some Entra security groups to the workspace so that we can control access to the catalog items. We have noticed that some users are being added…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,218 questions
asked 2024-11-01T10:04:34.28+00:00
Chris O'Brien 0 Reputation points
commented 2024-11-08T02:18:39.42+00:00
Chandra Boorla 2,915 Reputation points Microsoft Vendor
2 answers

Want to know my subscription and billing details for azure account

Hi, I had created azure free trail subscription account on sep, 6th, 2024 and cancelled the subscription as well but today i got a mail regarding billing. when i proctored the billing information in the account there were no subscriptions listed. i…

Azure Cost Management
Azure Cost Management
A Microsoft offering that enables tracking of cloud usage and expenditures for Azure and other cloud providers.
2,697 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,218 questions
asked 2024-11-06T06:01:05.6933333+00:00
Naveen Korra 0 Reputation points
commented 2024-11-08T01:15:31.19+00:00
Rahul Podila 195 Reputation points Microsoft Vendor
1 answer One of the answers was accepted by the question author.

Availability of Serverless Compute in South African North region

Are there any plans and dates to bring Databricks of Serverless Compute to the South African North region? Thanks, Andreas.

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,218 questions
asked 2024-11-07T02:51:10.8433333+00:00
Andreas Cambitsis 66 Reputation points
accepted 2024-11-07T06:43:48.1566667+00:00
Andreas Cambitsis 66 Reputation points
1 answer One of the answers was accepted by the question author.

copy tables from sql server on-prem to .parquet format in delta lake

Currently our copy process for 10 tables is running in pipeline in ADF, our source is ON-PREM SQL Server and target is in BLOB Storage GEN2. it copies data(tables) in parquet file format in HOT storage Account --- StorageV2 (general purpose v2) but I…

Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
3,224 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,218 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,843 questions
asked 2024-10-29T19:03:53.5233333+00:00
Zaeemul Haque 45 Reputation points
accepted 2024-11-06T16:58:39.44+00:00
Zaeemul Haque 45 Reputation points
1 answer One of the answers was accepted by the question author.

Azure Databricks Compute Instance Quota Issue

Unable to create a compute instance in Azure Databricks due to a quota issue. The error message received is: This account may not have enough CPU cores to start a cluster. Contact your administrator to increase the limits. Additionally, it is noted…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,218 questions
asked 2024-10-30T17:40:56.9433333+00:00
Dmitriy Khodyrev 20 Reputation points
accepted 2024-11-06T13:30:19.3333333+00:00
Dmitriy Khodyrev 20 Reputation points
1 answer

Unable to access account console in daatabrics

Hello guys I created a storage account through my pay as u go subscription and lauched a premium tier workspace for databricks but i was trying to connect my workspace to metastore in unity catalog but i dont see my manage account button to access…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,218 questions
asked 2024-10-30T11:19:15.72+00:00
Mantri Sai Ashish 0 Reputation points
commented 2024-11-06T13:23:25.7366667+00:00
PRADEEPCHEEKATLA 90,226 Reputation points
2 answers

I don't have access to admin console of databricks even though I am admin user

I created azure account and launched databricks, I don't have access to admin console of databricks even though I am the only admin user

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,218 questions
asked 2024-11-01T03:49:21.9833333+00:00
Ramya 0 Reputation points
commented 2024-11-06T00:54:38.9+00:00
Chandra Boorla 2,915 Reputation points Microsoft Vendor
1 answer

Error occurred when using ODBC driver with Workload Identity to connect to Azure Databricks Workspace from an AKS Pod deployment

Hello everybody. As per the title, I am trying to achieve the followings: Connect to an Azure Databricks Workspace, in order to execute an SQL Query, from an Azure Function running on an AKS Pod. Pod deployment is made using CI/CD pipelines with…

Azure Functions
Azure Functions
An Azure service that provides an event-driven serverless compute platform.
5,109 questions
ASP.NET Core
ASP.NET Core
A set of technologies in the .NET Framework for building web applications and XML web services.
4,612 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,218 questions
Azure Kubernetes Service (AKS)
Azure Kubernetes Service (AKS)
An Azure service that provides serverless Kubernetes, an integrated continuous integration and continuous delivery experience, and enterprise-grade security and governance.
2,146 questions
Microsoft Entra ID
Microsoft Entra ID
A Microsoft Entra identity service that provides identity management and access control capabilities. Replaces Azure Active Directory.
22,116 questions
asked 2024-11-04T15:55:37.3633333+00:00
Dragos Ionita 0 Reputation points
commented 2024-11-05T16:03:46.5266667+00:00
Sina Salam 12,011 Reputation points
1 answer One of the answers was accepted by the question author.

Azure Databricks is not sending email when Databricks Alert condition is met

Azure Databricks is not sending an email to the destination email address when Databricks Alert condition is met, Sometimes this is working and sometimes not.

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,218 questions
asked 2024-11-01T09:50:13.5066667+00:00
Mekala, Vamshi 20 Reputation points
commented 2024-11-05T08:58:24.0766667+00:00
Mekala, Vamshi 20 Reputation points
0 answers

Autoloader in Synapse

I want to load data incrementally using Azure Synapse Analytics Notebook(dedicated spark pool) there is an option on databricks to load data incrementally which is autoloader so my question was can we use this option in Azure Synapse Analytics…

Azure SQL Database
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,997 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,218 questions
SQL Server
SQL Server
A family of Microsoft relational database management and analysis systems for e-commerce, line-of-business, and data warehousing solutions.
13,998 questions
asked 2024-01-24T11:33:32.82+00:00
Aditya Singh 160 Reputation points
edited the question 2024-11-05T03:52:58.6933333+00:00
Sumarigo-MSFT 47,021 Reputation points Microsoft Employee
2 answers

Error when creating Access Connector for Azure Databricks

So, just created storage, and would like to setup the access connector, and faces the below error. Anyone seen this before ? All insights are very much appreciated

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,218 questions
asked 2024-10-31T10:38:45.89+00:00
Anders Thøgersen 0 Reputation points
commented 2024-11-05T01:13:43.6766667+00:00
Chandra Boorla 2,915 Reputation points Microsoft Vendor
1 answer

I dont see Account Console in Databricks. Please advise

I have personal account and when I log into databricks I dont see account console to admin stuff

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,218 questions
asked 2024-10-31T13:46:23.6+00:00
Gokhan Erkus 0 Reputation points
commented 2024-11-04T15:19:15.6933333+00:00
Smaran Thoomu 16,890 Reputation points Microsoft Vendor
3 answers

Azure Databricks Compute error: This account may not have enough CPU cores to satisfy this request Estimated available: 0, requested: 4

I have created Azure Free Tier Subscription with a credit $200 a day ago and deployed Azure Databricks for 14 days free as well. In Quotas I have 4CPU free: But when I'm trying to create Compte in Azure Databricks it says This account may not have…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,218 questions
asked 2024-10-27T20:06:04.87+00:00
Sergei Varaksin 0 Reputation points
answered 2024-11-04T06:15:09.8533333+00:00
Ganesh Gurram 1,110 Reputation points Microsoft Vendor
1 answer One of the answers was accepted by the question author.

Automating conversion of python scripts to databricks notebook

I'm looking to convert a Python .py file into a format compatible with Databricks, where the file can be split into multiple cells without using the Databricks GUI. Eventually, I’d like to automate this process. What’s the minimal code needed to create a…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,218 questions
asked 2024-10-29T14:45:29.6666667+00:00
Stephen-5002 40 Reputation points
edited an answer 2024-11-04T04:08:48.58+00:00
Smaran Thoomu 16,890 Reputation points Microsoft Vendor
1 answer

is there a spark connector for Azure Cosmos DB MongoDB API RU?

I'm using Azure Databricks to develop a proof of concept. Is there support for a Spark Connector for Azure Cosmos DB MongoDB API? If yes, what versions are supported. Is both batch and stream operations supported?

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,218 questions
asked 2024-10-30T16:53:46.0733333+00:00
Lum, Joey 0 Reputation points
commented 2024-11-04T01:27:45.2166667+00:00
Chandra Boorla 2,915 Reputation points Microsoft Vendor