Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,233 questions
1 answer
How to access secrets from Key Vault from Synapse Spark Job Definition PySpark file
Hello Everyone I'm trying to read the data source credentials as a secrets from the Key Vault using the Python SDK on a PySpark Job. I've gone through many articles and used different ways to read the secrets from Key Vault, but nothing worked out.…
asked 2025-03-04T07:32:09.6066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(83.2, 22%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EM%3C/text%3E%3C/svg%3E)
manigandan
0
Reputation points
commented 2025-03-12T13:18:50.3+00:00
manigandan
0
Reputation points
1 answer
One of the answers was accepted by the question author.
What’s the best approach to create external tables from JSON logs using Serverless SQL Pool?.
Hello, I am currently using Azure Logging to monitor logs in real-time via Log Stream. The goal is to store these logs in a Storage Account Gen 2 container and create an external table in Azure Synapse Analytics for data analysis. However, I…
asked 2025-03-10T21:16:18.3+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 98%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESF%3C/text%3E%3C/svg%3E)
Saul Figueroa
20
Reputation points
accepted 2025-03-12T13:12:00.8933333+00:00
Saul Figueroa
20
Reputation points
1 answer
How to remove/unset Spark configs that are pre-set by Microsoft
Background Synapse Apache Spark configurations allow users to create a blueprint for a Spark configuration that can be used for customizing the Spark behavior. This blueprint can be assigned to a Spark Pool (and will be used as new default for new…
asked 2025-03-08T12:09:21.8333333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(297.6, 42%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMB%3C/text%3E%3C/svg%3E)
Martin B
126
Reputation points
commented 2025-03-12T09:12:30.75+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(6.4, 57.00000000000001%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERG%3C/text%3E%3C/svg%3E)
Rakesh Govindula
95
Reputation points Microsoft External Staff
1 answer
Update/Transform Headerless CSV Source
I am attempting to create a new dataflow to sink into a SQL table from a CSV source. My csv is missing it's header row though, so I'm stumbling on how to map the columns. It is delimited via pipe ('|'). Previous answers I have found reference a Headers…
asked 2025-03-10T23:28:23.6066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 33%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENF%3C/text%3E%3C/svg%3E)
Nestor Fregoso
0
Reputation points
commented 2025-03-12T08:13:36.95+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
phemanth
14,560
Reputation points Microsoft External Staff
1 answer
Azure Synapse Link Roles
Hi Team I have a Synapse Link With F&O now we are transitioning to Production, but all the UAT process has been made with SysAdmin permissions on the Power Platform Environment, I tried to find documentation about the roles but can't find anything,…
asked 2025-03-10T18:50:37.9866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(44.800000000000004, 99%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMA%3C/text%3E%3C/svg%3E)
Manuel Alejandro Estrada Doñes
20
Reputation points
edited a comment 2025-03-12T04:48:10.0833333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 18%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGG%3C/text%3E%3C/svg%3E)
Ganesh Gurram
4,965
Reputation points Microsoft External Staff
1 answer
Is It Possible to Parse JSON Data in an External Table with Serverless SQL Pool?
Hello, I’m currently working on creating an external table in Azure Synapse Serverless SQL Pool using a JSON Lines formatted file. However, I’ve encountered a limitation: Serverless SQL Pool does not natively support JSON file formats for external table…
asked 2025-03-11T20:37:10.5866667+00:00
Saul Figueroa
20
Reputation points
answered 2025-03-12T03:37:19.7666667+00:00
Nandan Hegde
34,351
Reputation points MVP
0 answers
Recommended CI/CD Setup for Azure Synapse
Most tutorials and recommendations for how to set up CICD in azure synapse suggest creating multiple workspaces (e.g. -dev, -qa, -uat, -prod) and using a devops pipeline to move artifacts between the environments. Including this video tutorial from the…
asked 2025-03-11T09:28:23.9766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(272, 28.999999999999996%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EFS%3C/text%3E%3C/svg%3E)
Finn Schmidt
86
Reputation points
edited a comment 2025-03-11T23:53:06.8466667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(252.8, 99%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJN%3C/text%3E%3C/svg%3E)
J N S S Kasyap
860
Reputation points Microsoft External Staff
2 answers
Unable to update datetime to sink via data flow
Hi, I want to pass in a parameter to the data flow. This variable is current UTC value. @formatDateTime(utcNow(), 'yyyy-MM-dd HH:mm:ss.fff') In my data flow, I have a derived column. My sink column is timestamp data…
asked 2025-03-05T06:01:45.7+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(89.60000000000001, 14.000000000000002%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EL%3C/text%3E%3C/svg%3E)
Lotus88
116
Reputation points
commented 2025-03-11T20:22:54+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 65%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECB%3C/text%3E%3C/svg%3E)
Chandra Boorla
9,910
Reputation points Microsoft External Staff
1 answer
Unable to read content of file despite able to see FileInfo. urlopen error [Errno 5] Input/output error
I have 2 notebooks in different accounts, Staging & Production. Both use Managed Identity, linkedService -> System-assigned managed identity, and mounted drive. Both notebooks use the exact same code: Both can see the FileInfo, name, size,…
asked 2025-03-07T05:28:18.5866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(67.2, 85%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDW%3C/text%3E%3C/svg%3E)
Daniel Wang
5
Reputation points
commented 2025-03-11T17:57:46.8366667+00:00
J N S S Kasyap
860
Reputation points Microsoft External Staff
1 answer
One of the answers was accepted by the question author.
Synapselink Audit table source files
Hi. We have Synapselink for dataverse that send audit table data. The link creates audit and audit_partitioned tables automatically. Because of need to update spark pool there was pause in automatic table syncing. Now Audit_partitioned is missing some…
asked 2025-03-07T10:45:39.65+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(28.799999999999997, 7.000000000000001%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJH%3C/text%3E%3C/svg%3E)
Jani Hämäläinen
105
Reputation points
commented 2025-03-11T15:55:24.2966667+00:00
Vinodh247
29,701
Reputation points MVP
3 answers
Connect Local Machine to Synapse Lake Database using Python
Hi, I am trying to develop tooling for my use case, where I am using PySpark/Spark SQL on Synapse Analytics to create tables in Lake database. Sample code below: spark.sql("CREATE TABLE abc AS SELECT * FROM view") The table abc is created in…
asked 2025-02-17T15:02:43.2566667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(310.4, 88%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHN%3C/text%3E%3C/svg%3E)
Haria, Neel
0
Reputation points
commented 2025-03-11T09:14:40.2633333+00:00
AnnuKumari-MSFT
34,271
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Synapse stuck in provisioning state
I deployed a Synapse Workspace a long time ago. It never finished deploying, and now I want to get rid of it, but I am unable to delete it. The deletion fails with message: "Workspace is in non-terminal state Provisioning and cannot be…
asked 2025-03-06T13:39:11.0166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 28.000000000000004%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERK%3C/text%3E%3C/svg%3E)
Rasmus Kromann
20
Reputation points
accepted 2025-03-11T07:13:02.38+00:00
Rasmus Kromann
20
Reputation points
0 answers
Unable to use Linked Service Connection Using "To File Path": The configuration is overwritten after Publishing and during usage
Configuring Access to Amazon S3 with Limited Permissions I am working with an Amazon S3 storage system where my access is restricted to a specific bucket/ directory, rather than having root-level access. This limited permission setup has led me to…
asked 2025-03-07T08:37:29.18+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(44.800000000000004, 55.00000000000001%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
sr24
0
Reputation points
commented 2025-03-11T03:22:20.6466667+00:00
AnnuKumari-MSFT
34,271
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
Process streaming data with Synapse Pipeline or Data Flow
Hi Team, May I know if I can use Synapse pipeline or Data Flow to connect Eventhub and process Streaming data? Pipeline: Data Flow: I try to use this visual programming to create a process to read streaming data from eventhub and join cosmos db etc.…
asked 2024-10-18T13:18:52.67+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(67.2, 71%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJZ%3C/text%3E%3C/svg%3E)
Jia Zhang
60
Reputation points
commented 2025-03-10T20:49:37.55+00:00
Chandra Boorla
9,910
Reputation points Microsoft External Staff
3 answers
One of the answers was accepted by the question author.
Azure ML Workspace - Unable to get access token for ADLS Gen2
Hello Microsoft Q&A, when running azure ml pipelines I got the following error: " permission denied when access stream. Reason: Some(This request is not authorized to perform this operation using this permission.) " When I checked the data…
asked 2023-04-11T15:48:31.0466667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(233.6, 35%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EL%3C/text%3E%3C/svg%3E)
Lukas
151
Reputation points
commented 2025-03-10T17:04:21.5766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(233.6, 55.00000000000001%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERA%3C/text%3E%3C/svg%3E)
Riyaz Ahmed Khaja Hussain
0
Reputation points
1 answer
Azure Synapse Analytics is not getting registered when working with Lab " Explore Azure Synapse Analytics"
getting error while trying the lab Explore Azure Synapse Analytics Registering resource providers... Microsoft.Synapse is not yet registered. Waiting for 30 seconds before rechecking... Microsoft.Synapse is not yet registered. Waiting for 30 seconds…
asked 2024-12-14T10:38:47.2766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(144, 66%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMB%3C/text%3E%3C/svg%3E)
Monish Bhanushali
15
Reputation points
commented 2025-03-08T07:26:15.71+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(140.8, 86%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESD%3C/text%3E%3C/svg%3E)
Shubham Dubey
0
Reputation points
1 answer
Synapse Link Dataverse F&O Multiple Files Partitions
Hi Team We have an Azure Synapse Link Dataverse with F&O, with a partition by year and the option appendOnly checked. In all my tables I just getting one file per year that make me sense, but i have one table that for some reason is generating…
asked 2025-03-05T15:46:08.8833333+00:00
Manuel Alejandro Estrada Doñes
20
Reputation points
commented 2025-03-07T18:11:23.0033333+00:00
Chandra Boorla
9,910
Reputation points Microsoft External Staff
1 answer
One of the answers was accepted by the question author.
Azure Synapse Apache Spark pool in Current state: DeleteError. How do I get it into a succeeded state?
Hi, I tried to delete an Apache Spark pool. Deletion failed. Jobs are still running, but I cannot make any changes to the configuration of the Spark pool. Neither can I delete it. I read that "unlinking all notebooks from the spark…
asked 2025-03-06T12:56:23.7466667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(278.4, 33%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKB%3C/text%3E%3C/svg%3E)
Kajetan Bocek
40
Reputation points
accepted 2025-03-07T09:43:08.8333333+00:00
Kajetan Bocek
40
Reputation points
1 answer
Failed to update table. Change in table plan is limited to once a week. Last update was yesteday.
For a table in Log analytics workspace we had changed to Basic plan. We need it to be changed to Analytics immediately to reduce impact. But receiving error failed to update Change in table plan is limited to once a week. Last update was yesteday What…
asked 2025-02-28T07:36:02.4733333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 13%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES5%3C/text%3E%3C/svg%3E)
SirilJoseph-5276
0
Reputation points
commented 2025-03-07T04:20:17.6933333+00:00
phemanth
14,560
Reputation points Microsoft External Staff
1 answer
One of the answers was accepted by the question author.
how to update apache spark pool version from3.3 to 3.4 if don't have any spark pools under the synapse workspace
i dont have spark pools configured under synapse analytics workspaces so how can i update syanpse spark pools to 3.4 version. is there any dependency validation required for this?
asked 2024-10-23T09:51:20.06+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(3.2, 98%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAG%3C/text%3E%3C/svg%3E)
ANKIT GHENGE
110
Reputation points
commented 2025-03-06T21:15:12.8666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(153.6, 89%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EOC%3C/text%3E%3C/svg%3E)
Ongweoweh Corp
0
Reputation points