5,233 questions with Azure Synapse Analytics tags

Sort by: Updated
0 answers

Recommended CI/CD Setup for Azure Synapse

Most tutorials and recommendations for how to set up CICD in azure synapse suggest creating multiple workspaces (e.g. -dev, -qa, -uat, -prod) and using a devops pipeline to move artifacts between the environments. Including this video tutorial from the…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,233 questions
asked 2025-03-11T09:28:23.9766667+00:00
Finn Schmidt 86 Reputation points
edited a comment 2025-03-11T23:53:06.8466667+00:00
J N S S Kasyap 860 Reputation points Microsoft External Staff
0 answers

Is It Possible to Parse JSON Data in an External Table with Serverless SQL Pool?

Hello, I’m currently working on creating an external table in Azure Synapse Serverless SQL Pool using a JSON Lines formatted file. However, I’ve encountered a limitation: Serverless SQL Pool does not natively support JSON file formats for external table…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,233 questions
asked 2025-03-11T20:37:10.5866667+00:00
Saul Figueroa 0 Reputation points
2 answers

Unable to update datetime to sink via data flow

Hi, I want to pass in a parameter to the data flow. This variable is current UTC value. @formatDateTime(utcNow(), 'yyyy-MM-dd HH:mm:ss.fff') In my data flow, I have a derived column. My sink column is timestamp data…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,233 questions
asked 2025-03-05T06:01:45.7+00:00
Lotus88 116 Reputation points
commented 2025-03-11T20:22:54+00:00
Chandra Boorla 9,910 Reputation points Microsoft External Staff
1 answer

Unable to read content of file despite able to see FileInfo. urlopen error [Errno 5] Input/output error

I have 2 notebooks in different accounts, Staging & Production. Both use Managed Identity, linkedService -> System-assigned managed identity, and mounted drive. Both notebooks use the exact same code: Both can see the FileInfo, name, size,…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,233 questions
asked 2025-03-07T05:28:18.5866667+00:00
Daniel Wang 5 Reputation points
commented 2025-03-11T17:57:46.8366667+00:00
J N S S Kasyap 860 Reputation points Microsoft External Staff
1 answer

How to remove/unset Spark configs that are pre-set by Microsoft

Background Synapse Apache Spark configurations allow users to create a blueprint for a Spark configuration that can be used for customizing the Spark behavior. This blueprint can be assigned to a Spark Pool (and will be used as new default for new…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,233 questions
asked 2025-03-08T12:09:21.8333333+00:00
Martin B 126 Reputation points
commented 2025-03-11T17:48:31.7166667+00:00
Martin B 126 Reputation points
1 answer One of the answers was accepted by the question author.

Synapselink Audit table source files

Hi. We have Synapselink for dataverse that send audit table data. The link creates audit and audit_partitioned tables automatically. Because of need to update spark pool there was pause in automatic table syncing. Now Audit_partitioned is missing some…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,233 questions
asked 2025-03-07T10:45:39.65+00:00
Jani Hämäläinen 105 Reputation points
commented 2025-03-11T15:55:24.2966667+00:00
Vinodh247 29,701 Reputation points MVP
0 answers

What’s the best approach to create external tables from JSON logs using Serverless SQL Pool?.

Hello, I am currently using Azure Logging to monitor logs in real-time via Log Stream. The goal is to store these logs in a Storage Account Gen 2 container and create an external table in Azure Synapse Analytics for data analysis. However, I…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,233 questions
asked 2025-03-10T21:16:18.3+00:00
Saul Figueroa 0 Reputation points
edited a comment 2025-03-11T13:26:17.9333333+00:00
Venkat Reddy Navari 0 Reputation points Microsoft External Staff
1 answer

Update/Transform Headerless CSV Source

I am attempting to create a new dataflow to sink into a SQL table from a CSV source. My csv is missing it's header row though, so I'm stumbling on how to map the columns. It is delimited via pipe ('|'). Previous answers I have found reference a Headers…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,233 questions
asked 2025-03-10T23:28:23.6066667+00:00
Nestor Fregoso 0 Reputation points
edited a comment 2025-03-11T09:20:59.5333333+00:00
phemanth 14,560 Reputation points Microsoft External Staff
3 answers

Connect Local Machine to Synapse Lake Database using Python

Hi, I am trying to develop tooling for my use case, where I am using PySpark/Spark SQL on Synapse Analytics to create tables in Lake database. Sample code below: spark.sql("CREATE TABLE abc AS SELECT * FROM view") The table abc is created in…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,233 questions
asked 2025-02-17T15:02:43.2566667+00:00
Haria, Neel 0 Reputation points
commented 2025-03-11T09:14:40.2633333+00:00
AnnuKumari-MSFT 34,271 Reputation points Microsoft Employee
1 answer One of the answers was accepted by the question author.

Synapse stuck in provisioning state

I deployed a Synapse Workspace a long time ago. It never finished deploying, and now I want to get rid of it, but I am unable to delete it. The deletion fails with message: "Workspace is in non-terminal state Provisioning and cannot be…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,233 questions
asked 2025-03-06T13:39:11.0166667+00:00
Rasmus Kromann 0 Reputation points
accepted 2025-03-11T07:13:02.38+00:00
Rasmus Kromann 0 Reputation points
0 answers

Azure Synapse Link Roles

Hi Team I have a Synapse Link With F&O now we are transitioning to Production, but all the UAT process has been made with SysAdmin permissions on the Power Platform Environment, I tried to find documentation about the roles but can't find anything,…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,233 questions
asked 2025-03-10T18:50:37.9866667+00:00
Manuel Alejandro Estrada Doñes 20 Reputation points
edited a comment 2025-03-11T05:50:54.5266667+00:00
Ganesh Gurram 4,965 Reputation points Microsoft External Staff
0 answers

Unable to use Linked Service Connection Using "To File Path": The configuration is overwritten after Publishing and during usage

Configuring Access to Amazon S3 with Limited Permissions I am working with an Amazon S3 storage system where my access is restricted to a specific bucket/ directory, rather than having root-level access. This limited permission setup has led me to…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,233 questions
asked 2025-03-07T08:37:29.18+00:00
sr24 0 Reputation points
commented 2025-03-11T03:22:20.6466667+00:00
AnnuKumari-MSFT 34,271 Reputation points Microsoft Employee
1 answer

How to access secrets from Key Vault from Synapse Spark Job Definition PySpark file

Hello Everyone I'm trying to read the data source credentials as a secrets from the Key Vault using the Python SDK on a PySpark Job. I've gone through many articles and used different ways to read the secrets from Key Vault, but nothing worked out.…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,233 questions
asked 2025-03-04T07:32:09.6066667+00:00
manigandan 0 Reputation points
commented 2025-03-10T20:52:15.0333333+00:00
Chandra Boorla 9,910 Reputation points Microsoft External Staff
2 answers One of the answers was accepted by the question author.

Process streaming data with Synapse Pipeline or Data Flow

Hi Team, May I know if I can use Synapse pipeline or Data Flow to connect Eventhub and process Streaming data? Pipeline: Data Flow: I try to use this visual programming to create a process to read streaming data from eventhub and join cosmos db etc.…

Azure Event Hubs
Azure Event Hubs
An Azure real-time data ingestion service.
698 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,233 questions
asked 2024-10-18T13:18:52.67+00:00
Jia Zhang 60 Reputation points
commented 2025-03-10T20:49:37.55+00:00
Chandra Boorla 9,910 Reputation points Microsoft External Staff
3 answers One of the answers was accepted by the question author.

Azure ML Workspace - Unable to get access token for ADLS Gen2

Hello Microsoft Q&A, when running azure ml pipelines I got the following error: " permission denied when access stream. Reason: Some(This request is not authorized to perform this operation using this permission.) " When I checked the data…

Azure Machine Learning
Azure Machine Learning
An Azure machine learning service for building and deploying models.
3,178 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,233 questions
asked 2023-04-11T15:48:31.0466667+00:00
Lukas 151 Reputation points
commented 2025-03-10T17:04:21.5766667+00:00
Riyaz Ahmed Khaja Hussain 0 Reputation points
1 answer

Azure Synapse Analytics is not getting registered when working with Lab " Explore Azure Synapse Analytics"

getting error while trying the lab Explore Azure Synapse Analytics Registering resource providers... Microsoft.Synapse is not yet registered. Waiting for 30 seconds before rechecking... Microsoft.Synapse is not yet registered. Waiting for 30 seconds…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,233 questions
asked 2024-12-14T10:38:47.2766667+00:00
Monish Bhanushali 15 Reputation points
commented 2025-03-08T07:26:15.71+00:00
Shubham Dubey 0 Reputation points
1 answer

Synapse Link Dataverse F&O Multiple Files Partitions

Hi Team We have an Azure Synapse Link Dataverse with F&O, with a partition by year and the option appendOnly checked. In all my tables I just getting one file per year that make me sense, but i have one table that for some reason is generating…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,233 questions
asked 2025-03-05T15:46:08.8833333+00:00
Manuel Alejandro Estrada Doñes 20 Reputation points
commented 2025-03-07T18:11:23.0033333+00:00
Chandra Boorla 9,910 Reputation points Microsoft External Staff
1 answer One of the answers was accepted by the question author.

Azure Synapse Apache Spark pool in Current state: DeleteError. How do I get it into a succeeded state?

Hi, I tried to delete an Apache Spark pool. Deletion failed. Jobs are still running, but I cannot make any changes to the configuration of the Spark pool. Neither can I delete it. I read that "unlinking all notebooks from the spark…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,233 questions
asked 2025-03-06T12:56:23.7466667+00:00
Kajetan Bocek 40 Reputation points
accepted 2025-03-07T09:43:08.8333333+00:00
Kajetan Bocek 40 Reputation points
1 answer

Failed to update table. Change in table plan is limited to once a week. Last update was yesteday.

For a table in Log analytics workspace we had changed to Basic plan. We need it to be changed to Analytics immediately to reduce impact. But receiving error failed to update Change in table plan is limited to once a week. Last update was yesteday What…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,233 questions
asked 2025-02-28T07:36:02.4733333+00:00
SirilJoseph-5276 0 Reputation points
commented 2025-03-07T04:20:17.6933333+00:00
phemanth 14,560 Reputation points Microsoft External Staff
1 answer One of the answers was accepted by the question author.

how to update apache spark pool version from3.3 to 3.4 if don't have any spark pools under the synapse workspace

i dont have spark pools configured under synapse analytics workspaces so how can i update syanpse spark pools to 3.4 version. is there any dependency validation required for this?

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,233 questions
asked 2024-10-23T09:51:20.06+00:00
ANKIT GHENGE 110 Reputation points
commented 2025-03-06T21:15:12.8666667+00:00
Ongweoweh Corp 0 Reputation points