Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,373 questions
2 answers
Error with Create Table USING DELTA LOCATION in training exercise
In the exercise https://microsoftlearning.github.io/mslearn-databricks/Instructions/Exercises/03-Delta-lake-in-Azure-Databricks.html the line of code spark.sql("CREATE TABLE AdventureWorks.ProductsExternal USING DELTA LOCATION…
Azure Databricks
Azure Training
Azure Training
Azure: A cloud computing platform and infrastructure for building, deploying and managing applications and services through a worldwide network of Microsoft-managed datacenters.Training: Instruction to develop new skills.
2,257 questions
asked 2024-05-01T13:00:09.32+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(6.4, 31%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJM%3C/text%3E%3C/svg%3E)
James Mitchell
10
Reputation points
commented 2025-03-21T23:27:55.01+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(240, 72%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELB%3C/text%3E%3C/svg%3E)
Leonardo Blandim
0
Reputation points
2 answers
Error in Prod but not in STG, Dev for Hilbert indexing can only be used on 9 or fewer columns
We have a very Big Fact Table with around 215 Million Rows We have optimize on this Big Fact Table with 11 columns This is working fine in Dev, STG, but fails in Prod giving: Hilbert indexing can only be used on 9 or fewer columns Our Cluster…
asked 2025-03-19T19:56:58.7333333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(240, 59%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EIN%3C/text%3E%3C/svg%3E)
Immaneni, Nanda
0
Reputation points
commented 2025-03-21T21:47:16.2366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 65%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECB%3C/text%3E%3C/svg%3E)
Chandra Boorla
10,410
Reputation points Microsoft External Staff
1 answer
vnet to vnet NSG entry
All, I setup Azure Databricks on my subscription and it is working. However, I'm trying to understand networking aspects of it better. Below is a screenshot of the NSG and I have a question. For the Inbound Security Rules, the 100 row has source…
asked 2022-02-06T05:55:21.333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 51%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EG%3C/text%3E%3C/svg%3E)
grajee
371
Reputation points
commented 2025-03-21T18:45:23.3766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(153.6, 63%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERG%3C/text%3E%3C/svg%3E)
Rohit Ghadshi
0
Reputation points
1 answer
Locked Out of Azure Databricks Due to Incorrect IP Access List
I mistakenly added a local IP address to the IP Access List in my Azure Databricks workspace. As a result, I can no longer access the workspace from the internet. Is there any way to remove or modify the IP access list without accessing the Databricks…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(310.4, 37%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3E1%3C/text%3E%3C/svg%3E)
0 answers
Databricks Serverless for DLT in UK South
I'm trying to create a DLT pipeline in my workspace but the Serverless option does not exist. It looks like this: But it should look like this: The workspace is brand new - so this is not a legacy issue. And according to the docs the region is enabled…
asked 2025-03-14T09:31:33.1433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(227.2, 43%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESD%3C/text%3E%3C/svg%3E)
Simon D'Morias
5
Reputation points
commented 2025-03-21T09:37:51.27+00:00
Simon D'Morias
5
Reputation points
2 answers
One of the answers was accepted by the question author.
Failure starting repl. Try detaching and re-attaching the notebook.
I( have been facing issues related to cluster. the moment i run a command it fails giving me the following error message "Failure starting repl. Try detaching and re-attaching the notebook." Please help. I tried restarting and changing the…
asked 2023-10-12T08:45:55.79+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(131.20000000000002, 39%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESV%3C/text%3E%3C/svg%3E)
S V Ravi Chandra Kolluri
35
Reputation points
edited an answer 2025-03-21T08:48:10.7066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(96, 9%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMN%3C/text%3E%3C/svg%3E)
maryam nkhaili
0
Reputation points
1 answer
Databricks AI_QUERY fails with certain OpenAI models
We have been using successfully using the AI_QUERY function in Azure Databricks to interact with o1-mini and text-embedding-ada-002 models. However in the past few days, we have started to encounter errors. Calling our o1-mini model as usual…
asked 2025-03-12T17:41:44.71+00:00
James Bentley
10
Reputation points
edited a comment 2025-03-20T08:49:26.3533333+00:00
James Bentley
10
Reputation points
1 answer
Databricks: Insufficient Permission to Add Public IP Address and NAT Gateway
Hello everyone, I have a Azure Databricks resource group that pre-dates the configuration of a NAT Gateway and Standard Public IP address by default. I have tried to create a Public IP/Nat Gateway in my databricks resource group, but DENY permissions are…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(128, 55.00000000000001%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDC%3C/text%3E%3C/svg%3E)
2 answers
Data Truncation Issue When Using COPY INTO in Databricks for Binary Column
Issue: When loading a long string value containing trailing null bytes (e.g., \x12\x34\x56\x00\x00\x00...) into a binary column using the COPY INTO statement, the data gets truncated, storing only the first few characters (\x12\x34\x56 → EjRW in…
asked 2025-03-14T18:51:59.73+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(275.2, 6%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Suwetha Sivakumar
0
Reputation points
answered 2025-03-19T07:51:54.5266667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
phemanth
14,800
Reputation points Microsoft External Staff
1 answer
How do I use the azure databricks dlt pipeline to consume azure Event Hub data
Hi there, How do I use the azure databricks dlt pipeline to consume azure Event Hub data? EH_NAME = "myeventhub" TOPIC = "myeventhub" KAFKA_BROKER = "{EH_NAMESPACE}.servicebus.windows.net:9093" GROUP_ID =…
asked 2025-03-10T03:52:50.48+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(64, 53%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EZ%3C/text%3E%3C/svg%3E)
zmsoft
385
Reputation points
commented 2025-03-19T07:35:17.6166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 18%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGG%3C/text%3E%3C/svg%3E)
Ganesh Gurram
5,135
Reputation points Microsoft External Staff
1 answer
Run query against delta table using ADF managed identity, but Insufficient privileges error
In ADF, I created a web activity to run query using rest api against deltalake table. Managed Identity is Authentication type table path is like this: catalogA.schemaB.tableC. catalogA permission is controlled by AD Group groupD. ADF managed…
asked 2025-03-13T19:32:40.5233333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(38.4, 66%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENS%3C/text%3E%3C/svg%3E)
Ni, Shaojun SITI-PTIY/BAC
0
Reputation points
commented 2025-03-18T18:52:09.2666667+00:00
Chandra Boorla
10,410
Reputation points Microsoft External Staff
1 answer
Help needed to sign onto Azure databricks
I recently created account to access Azure Databricks service. I created the service and the resource group and tried to access the databricks workspace using the private link (https://adb-4253786588898730.10.azuredatabricks.net). But I enter my…
asked 2025-03-13T19:24:48.88+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(115.19999999999999, 8%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
SM
0
Reputation points
commented 2025-03-18T18:48:39.49+00:00
Chandra Boorla
10,410
Reputation points Microsoft External Staff
1 answer
Databrick: pros and cons
Hi, Is it possible to highlight five pros and five cons of using Databricks for data engineering? This would help me make a more informed decision about the tool
asked 2025-03-13T10:55:40.2666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(112.00000000000001, 59%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ER%3C/text%3E%3C/svg%3E)
Renan
60
Reputation points
commented 2025-03-18T18:38:04.5533333+00:00
Chandra Boorla
10,410
Reputation points Microsoft External Staff
2 answers
One of the answers was accepted by the question author.
How can i migrate jobs from one workspace to other workspace (Dev to QA)
How can i migrate jobs from one workspace to other workspace (Dev to QA) .
asked 2024-11-15T05:44:35.18+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(156.8, 77%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELN%3C/text%3E%3C/svg%3E)
Lakshmi Narayana Sarma Bhamidipati
160
Reputation points
commented 2025-03-17T11:52:36.5466667+00:00
Chandra Boorla
10,410
Reputation points Microsoft External Staff
0 answers
Connecting Spark to Azure SQL
Hi, I'm following this article to connect Spark to Azure SQL https://learn.microsoft.com/en-us/azure/azure-sql/database/spark-connector?view=azuresql&source=docs It says that I need to get the Spark connector and it has a link to a maven repo that…
asked 2025-03-12T10:06:58.9866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(188.79999999999998, 36%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EYB%3C/text%3E%3C/svg%3E)
Yaish, Batel
0
Reputation points
commented 2025-03-17T09:57:22.75+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(252.8, 99%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJN%3C/text%3E%3C/svg%3E)
J N S S Kasyap
955
Reputation points Microsoft External Staff
2 answers
One of the answers was accepted by the question author.
Need to connect to IMB DB2 from Azure databricks workspace
Need to connect to IMB DB2 (DB version IBM DB2 for i 7.3) form azure databricks workspace, the DB2 is hosted is AS400 server and DB port 8471. Please suggest on the step by step procedure to connect to DB and query a table, like installing required…
asked 2023-08-22T04:58:34.3733333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(265.6, 96%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVR%3C/text%3E%3C/svg%3E)
veerabhadra reddy kovvuri
181
Reputation points
edited a comment 2025-03-17T04:03:35.1966667+00:00
veerabhadra reddy kovvuri
181
Reputation points
1 answer
Azure Databricks Workspace: Users are being added to the admins group on their own
We have an Azure Databricks workspace with Unity Catalog enabled. We are using SCIM provisioning to sync some Entra security groups to the workspace so that we can control access to the catalog items. We have noticed that some users are being added…
asked 2024-11-01T10:04:34.28+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 72%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECO%3C/text%3E%3C/svg%3E)
Chris O'Brien
0
Reputation points
commented 2025-03-14T16:47:56.91+00:00
Chandra Boorla
10,410
Reputation points Microsoft External Staff
5 answers
Allow only a specific Azure Databricks instance to connect to keyvault
I want to connect to Azure Keyvault using "Azure Key Vault-backed scopes" in Databricks. Now on the Keyvault side, the way we allow an application to connect is via: RBAC - We are using the RBAC permission model. Add an identity in RBAC and…
asked 2022-01-11T01:21:28.393+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(147.20000000000002, 87%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENS%3C/text%3E%3C/svg%3E)
Nidhi Singh
6
Reputation points
answered 2025-03-14T11:21:01.1533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(112.00000000000001, 78%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
Martyna Magdalena Sikorska
0
Reputation points
1 answer
One of the answers was accepted by the question author.
Serverless not showing for my notebooks in Databricks
Hi. Serverless option is not showing for my notebooks and I cannot even see it in the feature enablement in accounts administration. In the attached screenshots, you can see that I have unity catalog enable and that serverless does not appear in feature…
asked 2025-01-26T12:10:14.4433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(128, 39%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMA%3C/text%3E%3C/svg%3E)
Mohammed Amine Mostefai
20
Reputation points
commented 2025-03-14T09:23:37.76+00:00
Simon D'Morias
5
Reputation points
1 answer
One of the answers was accepted by the question author.
How to create a Materialized View in Azure Databricks stored in a Managed Catalog, based on data loaded from a foreign catalog
I am currently facing a problem with creating a materialized view in Catalog_2.Schema_B (Managed_Catalog) based on data I have connected from postgre SQL external source loaded into Catalog1_Schema_A (Foreign_Catalog). The error I get only says…
asked 2025-03-07T05:03:46.0333333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(48, 49%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKK%3C/text%3E%3C/svg%3E)
Kohdai Kurihara
40
Reputation points
accepted 2025-03-14T08:28:35.7166667+00:00
Kohdai Kurihara
40
Reputation points