How to read zip file in Azure Synapse Notebook with extension of .zip.
Hi Team, Could you please advise the pyspark code, how to read zip files in synapse notebook. Example: File_Name - samplefile.zip Note: In the samplefile.zip we have multiple csv files, please let us know. The file Extension is only with .zip. …
understanding how to use mergeSchema=True
In COPY INTO under FORMAT_OPTIONS generally i use mergeSchema=True but in few queries i am seeing that it is also metioned under copy_options why ? and I have also seen mergeSchema=true given under options while read or readStream, i have seen that in…
How to list secret scopes and identify key vaults
What is the best way to view all existing secret scopes within a Databricks workspace? Additionally, how can we identify the specific Azure Key Vault linked to each of these secret scopes? I’d like to understand the steps or API calls needed to retrieve…
Databricks vs Batch pool
Hi, I have Googled the difference between using Azure Databrick and Batch pool to run python pipelines and ETL. But I haven’t found a clear difference between them two. Based on what I have Googled, Databrick can become very expensive, so cost matters a…
getting error while creating mount point in data bricks
Hi, I am getting the following error when trying to create mount point. py4j.security.Py4JSecurityException: Method public com.databricks.backend.daemon.dbutils.DBUtilsCore$Result…
Azure Databricks Workspace: Users are being added to the admins group on their own
We have an Azure Databricks workspace with Unity Catalog enabled. We are using SCIM provisioning to sync some Entra security groups to the workspace so that we can control access to the catalog items. We have noticed that some users are being added…
Want to know my subscription and billing details for azure account
Hi, I had created azure free trail subscription account on sep, 6th, 2024 and cancelled the subscription as well but today i got a mail regarding billing. when i proctored the billing information in the account there were no subscriptions listed. i…
Availability of Serverless Compute in South African North region
Are there any plans and dates to bring Databricks of Serverless Compute to the South African North region? Thanks, Andreas.
copy tables from sql server on-prem to .parquet format in delta lake
Currently our copy process for 10 tables is running in pipeline in ADF, our source is ON-PREM SQL Server and target is in BLOB Storage GEN2. it copies data(tables) in parquet file format in HOT storage Account --- StorageV2 (general purpose v2) but I…
Azure Databricks Compute Instance Quota Issue
Unable to create a compute instance in Azure Databricks due to a quota issue. The error message received is: This account may not have enough CPU cores to start a cluster. Contact your administrator to increase the limits. Additionally, it is noted…
Unable to access account console in daatabrics
Hello guys I created a storage account through my pay as u go subscription and lauched a premium tier workspace for databricks but i was trying to connect my workspace to metastore in unity catalog but i dont see my manage account button to access…
I don't have access to admin console of databricks even though I am admin user
I created azure account and launched databricks, I don't have access to admin console of databricks even though I am the only admin user
Error occurred when using ODBC driver with Workload Identity to connect to Azure Databricks Workspace from an AKS Pod deployment
Hello everybody. As per the title, I am trying to achieve the followings: Connect to an Azure Databricks Workspace, in order to execute an SQL Query, from an Azure Function running on an AKS Pod. Pod deployment is made using CI/CD pipelines with…
Azure Databricks is not sending email when Databricks Alert condition is met
Azure Databricks is not sending an email to the destination email address when Databricks Alert condition is met, Sometimes this is working and sometimes not.
Autoloader in Synapse
I want to load data incrementally using Azure Synapse Analytics Notebook(dedicated spark pool) there is an option on databricks to load data incrementally which is autoloader so my question was can we use this option in Azure Synapse Analytics…
Error when creating Access Connector for Azure Databricks
So, just created storage, and would like to setup the access connector, and faces the below error. Anyone seen this before ? All insights are very much appreciated
I dont see Account Console in Databricks. Please advise
I have personal account and when I log into databricks I dont see account console to admin stuff
Azure Databricks Compute error: This account may not have enough CPU cores to satisfy this request Estimated available: 0, requested: 4
I have created Azure Free Tier Subscription with a credit $200 a day ago and deployed Azure Databricks for 14 days free as well. In Quotas I have 4CPU free: But when I'm trying to create Compte in Azure Databricks it says This account may not have…
Automating conversion of python scripts to databricks notebook
I'm looking to convert a Python .py file into a format compatible with Databricks, where the file can be split into multiple cells without using the Databricks GUI. Eventually, I’d like to automate this process. What’s the minimal code needed to create a…
is there a spark connector for Azure Cosmos DB MongoDB API RU?
I'm using Azure Databricks to develop a proof of concept. Is there support for a Spark Connector for Azure Cosmos DB MongoDB API? If yes, what versions are supported. Is both batch and stream operations supported?