Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,495 questions
0 answers
We are looking for a spark structured based streaming to consume data from activeMQ
We are looking for a spark structured based streaming to consume data from activeMQ. However, we dont see any default connector or support as such available for the same. Can you please suggest, is there any roadmap in place to support this ?
asked 2024-11-21T15:47:57.1766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(121.6, 83%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPC%3C/text%3E%3C/svg%3E)
Prashant Chouhan
0
Reputation points
asked 2024-11-21T15:47:57.1766667+00:00
Prashant Chouhan
0
Reputation points
1 answer
Failure starting repl. Try detaching and re-attaching the notebook.
I am using Azure Databricks for a couple of months already. I didn't change any code or configuration recently but all notebook are failed to run. The error is : Failure starting repl. Try detaching and re-attaching the notebook. It failed for simple…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(150.4, 90%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EML%3C/text%3E%3C/svg%3E)
0 answers
I am not able to access the delta path after creating the delta table in the Azure databricks.
I am using Pay-as-you-go subscription and I am working on Azure databricks. In Databricks, when Delta table crated , we could see the storage location from the catalog explorer. But, when trying to see the details by accessing path, I am coming across…
asked 2024-11-21T10:56:01.49+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 57.99999999999999%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESA%3C/text%3E%3C/svg%3E)
Sudarsan Akula
0
Reputation points
edited the question 2024-11-21T11:25:24.3033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(57.599999999999994, 31%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVK%3C/text%3E%3C/svg%3E)
Vinod Kumar Reddy Chilupuri
1,125
Reputation points Microsoft Vendor
1 answer
Init scripts stored in volumes are not getting executed via Job Shared cluster via ADF
Hello Team, We have stored our init script in Azure databricks managed volumes path and we want to execute notebook via Job Shared cluster, the pipeline we are triggering is via ADF but the job cluster gives an error as We are using Assigned Manged…
asked 2024-11-18T11:49:41.7666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(51.2, 77%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAG%3C/text%3E%3C/svg%3E)
Ashwini Gaikwad
130
Reputation points
commented 2024-11-21T07:39:13.45+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
17,520
Reputation points Microsoft Vendor
0 answers
Read AppServiceConsoleLogs and push to Blob Storage container/table
Hello Team, we have a webapp python Django based. We are generating custom logs for user actions using code below. Settings.py: LOGGING = { "version": 1, "disable_existing_loggers": False, "formatters": { …
asked 2024-11-20T09:31:14.9233333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(272, 35%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
BalaKrishna Mahamkali
0
Reputation points
commented 2024-11-21T05:52:05.7+00:00
VenkateshDodda-MSFT
21,976
Reputation points Microsoft Employee
1 answer
Spark config through Databricks
Hi, I am trying to access storage account through databricks (thru notebook) with storage access key. I keep encountering error when I do spark.config.set spark.conf.set( …
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(67.2, 36%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EM%3C/text%3E%3C/svg%3E)
1 answer
Managing PCI-DSS Compliance and Access to Serverless Features in Azure Databricks
Hello Azure Community, I am currently using Azure Databricks with PCI-DSS compliance enabled in our workspace, as maintaining stringent security standards is crucial for our organization. However, I've discovered that once PCI-DSS compliance is turned…
asked 2024-11-19T13:37:14.5166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(144, 68%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EZA%3C/text%3E%3C/svg%3E)
Zohaib Altaf
0
Reputation points
commented 2024-11-21T03:09:29.6366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 65%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECB%3C/text%3E%3C/svg%3E)
Chandra Boorla
3,460
Reputation points Microsoft Vendor
1 answer
Auto loader
Hi, How to add new column automatically when it is not present in delta table
asked 2024-11-20T09:02:26.5633333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 13%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVS%3C/text%3E%3C/svg%3E)
Vineet S
1,230
Reputation points
answered 2024-11-21T02:17:24.28+00:00
Chandra Boorla
3,460
Reputation points Microsoft Vendor
1 answer
Still being billed for Azure after I cancelled it
I signed up for the free Azure and then I cancelled the subscription, but I'm still receiving bills. How do I go about removing everything off my name?
asked 2024-10-10T05:26:01.63+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(163.2, 93%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDJ%3C/text%3E%3C/svg%3E)
Desire Joffe
0
Reputation points
commented 2024-11-20T21:50:12.63+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(256, 20%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPR%3C/text%3E%3C/svg%3E)
Pranay Reddy Madireddy
700
Reputation points Microsoft Vendor
1 answer
How to Spark Stream into azure PostgreSQL
I am trying spark stream data into postgresql via databricks, with delta tables and change data feed enabled. As far as I understand I can use JDBC or the databricks connector as seen here:…
asked 2024-11-19T16:52:06.9+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(76.8, 21%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAL%3C/text%3E%3C/svg%3E)
Arik Levy
20
Reputation points
commented 2024-11-20T18:58:46.6633333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(246.4, 4%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Sina Salam
12,731
Reputation points
1 answer
Error with Create Table USING DELTA LOCATION in training exercise
In the exercise https://microsoftlearning.github.io/mslearn-databricks/Instructions/Exercises/03-Delta-lake-in-Azure-Databricks.html the line of code spark.sql("CREATE TABLE AdventureWorks.ProductsExternal USING DELTA LOCATION…
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,236 questions
Azure Training
Azure Training
Azure: A cloud computing platform and infrastructure for building, deploying and managing applications and services through a worldwide network of Microsoft-managed datacenters.Training: Instruction to develop new skills.
1,769 questions
asked 2024-05-01T13:00:09.32+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(6.4, 31%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJM%3C/text%3E%3C/svg%3E)
James Mitchell
5
Reputation points
commented 2024-11-20T15:43:17.59+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(153.6, 67%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESM%3C/text%3E%3C/svg%3E)
Sylvain MENEUST
0
Reputation points
1 answer
One of the answers was accepted by the question author.
Resolving GPU memory errors in databricks with PyTorch on g4dn.2xlarge instances
I'm encountering persistent memory issues while training and testing a PyTorch model on Databricks. In my Databricks job configuration, I’ve specified node_type_id and driver_node_type_id as g4dn.2xlarge. According to the documentation, this instance…
asked 2024-11-18T07:29:13.7+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(300.8, 15%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EG2%3C/text%3E%3C/svg%3E)
Gabriel-2005
365
Reputation points
accepted 2024-11-20T11:42:59.3566667+00:00
Gabriel-2005
365
Reputation points
1 answer
Table Access control option is not available in Azure Databricks Compute cluster
Hi, I am trying to enable Table Access control in one of my Azure Databricks Compute Cluster, the cluster was created around 02 years back with pricing tier selected as Standard, now we have upgraded the same cluster to Premium tier in order to…
asked 2024-11-15T15:16:40.88+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(124.80000000000001, 24%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESK%3C/text%3E%3C/svg%3E)
Satish Kumar
51
Reputation points
commented 2024-11-20T07:47:33.7033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 18%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGG%3C/text%3E%3C/svg%3E)
Ganesh Gurram
1,510
Reputation points Microsoft Vendor
1 answer
How to downgrade from TLS 1.3 to 1.2 in Azure Databricks env
Hi, As part of troubleshooting a current intermittent issue with connecting to an external API, I am being asked to downgrade the TLS version from 1.3 to 1.2. I am running a python notebook in our Azure Databricks environment and believe that TLS 1.3 is…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(6.4, 50%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3E_%3C/text%3E%3C/svg%3E)
2 answers
One of the answers was accepted by the question author.
How can i migrate jobs from one workspace to other workspace (Dev to QA)
How can i migrate jobs from one workspace to other workspace (Dev to QA) .
asked 2024-11-15T05:44:35.18+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(156.8, 77%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELN%3C/text%3E%3C/svg%3E)
Lakshmi Narayana Sarma Bhamidipati
110
Reputation points
accepted 2024-11-20T05:09:57.46+00:00
Lakshmi Narayana Sarma Bhamidipati
110
Reputation points
0 answers
Create views in Azure Databricks Unity catalog using service principal
Hello Team, Is there a possibility to Create views in Azure databricks Unity Catalog using service principal. and if there is how to update views based on tables added or deleted in schema. For e.g. I would like to create each views per tables in the…
asked 2024-11-13T16:51:42.7266667+00:00
Ashwini Gaikwad
130
Reputation points
commented 2024-11-19T17:16:07.06+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
phemanth
11,460
Reputation points Microsoft Vendor
1 answer
Connect to metastore table in databricks from other databricks notebook using service principal
I want to read table present in other databricks account in my databricks account using service principal. I am not able to find service principal related guide to connect to other databricks workspace. I can only see storage account related service…
asked 2024-11-14T10:50:15.56+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 15%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBR%3C/text%3E%3C/svg%3E)
Bhargav R
0
Reputation points
commented 2024-11-19T01:39:57.83+00:00
Chandra Boorla
3,460
Reputation points Microsoft Vendor
1 answer
Retrieving 'Person or Group' Fields from SharePoint List in Azure Data Factory or Synapse
I'm able to retrieve data from a SharePoint list using Azure Data Factory > Copy Data (via the SharePoint Connector), but the list contains 'Person or Group' type fields that return only 'Id' values in the results. These 'Id' values are not useful,…
asked 2024-11-12T14:14:03.32+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(115.19999999999999, 82%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESP%3C/text%3E%3C/svg%3E)
Sunny Punalkar
0
Reputation points
commented 2024-11-18T14:00:05.3166667+00:00
Smaran Thoomu
17,520
Reputation points Microsoft Vendor
1 answer
One of the answers was accepted by the question author.
Executing mutiple queries in a single transaction of a notebook on databricks
I'm working on executing multiple queries within a single transaction on Databricks using Golang. However, I'm encountering a 'not implemented' error. Upon reviewing the library code, I found the following: // Not supported in Databricks. func (c…
asked 2024-11-15T07:43:37.5033333+00:00
Gabriel-2005
365
Reputation points
accepted 2024-11-16T07:35:36.8066667+00:00
Gabriel-2005
365
Reputation points
1 answer
'manage account' > 'account console' option
db-q-1.JPGHello Team, I do have an access as Global Administrator, but I am not getting 'manage account' > 'account console' option in Azure Databricks @ top right corner. I need to create Unity Catalog metastore. Thanks
asked 2024-11-11T10:54:30.5666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(179.20000000000002, 23%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAV%3C/text%3E%3C/svg%3E)
Ankur Varshney
0
Reputation points
commented 2024-11-14T18:01:14.67+00:00
phemanth
11,460
Reputation points Microsoft Vendor