11,274 questions with Azure Data Factory tags

Sort by: Updated
1 answer

facing this below error how can i solve this

ErrorCode=UserErrorColumnMappingNotAllowed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Column mapping is not allowed for copy between 'AzureBlobStorage(DelimitedTextFormat)' and…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,274 questions
asked 2025-02-22T08:07:46.3166667+00:00
varalakshmi 0 Reputation points
answered 2025-02-22T09:58:56.84+00:00
Vinodh247 28,371 Reputation points MVP
0 answers

ADF Dataflow issue with source API query

am trying to setup an Azure Data Factory Dataflow. The source is a REST API from Monday.com which uses GraphQL. The API requires POST method, with 3 headers Content-Type "application/json", API-Version "2025-01 " and Authorization:…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,274 questions
asked 2025-02-22T03:15:01.8166667+00:00
Daniel Baird 0 Reputation points
0 answers

Miscellaneous error encountered while downloading one of the specified Azure Blob(s) during Azure Batch job in ADF

I am encountering a Miscellaneous error encountered while downloading one of the specified Azure Blob(s) error when I run a Python script in Azure Data Factory in a Subscription A while everything reproduced from scratch works on a Subscription…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,274 questions
asked 2025-02-21T11:31:27.7766667+00:00
11-4688 211 Reputation points
commented 2025-02-21T20:02:42.77+00:00
phemanth 13,905 Reputation points Microsoft Vendor
0 answers

Azure Data Factory Web activity calling child pipeline across ADF instance

I have child pipeline in ADF Instance 1 which invoke rest api POST method through web activity Parent pipeline is in ADF instance 2 which call above child pipeline also through web activity. I am able to successfully call child pipeline from parent…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,274 questions
asked 2025-02-19T15:19:04.2833333+00:00
Pankaj Joshi 371 Reputation points
commented 2025-02-21T19:37:02.33+00:00
phemanth 13,905 Reputation points Microsoft Vendor
1 answer

Azure Data factory: Linked service parameterization default values and CI/CD pipelines

I am struggling to understand how ADF parameterization functions when deploying from one environment to another when using DevOps pipelines. Our linked services may be set up with parameters such as "name":…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,274 questions
asked 2025-02-18T15:07:37.1933333+00:00
commented 2025-02-21T19:15:54.65+00:00
phemanth 13,905 Reputation points Microsoft Vendor
2 answers

Azure Data Factory pipeline to extract data from Snowflake and load it into Azure Blob Storage

Hi Team, Azure Data Factory pipeline to extract data from Snowflake and load it into Azure Blob Storage. However, I’m facing some challenges with the data copy process and could use some assistance. I have several Tables in my snowflake schema. Need…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,274 questions
asked 2025-02-19T16:44:10.35+00:00
Nalini Bhavaraju 20 Reputation points
commented 2025-02-21T18:35:41.9266667+00:00
phemanth 13,905 Reputation points Microsoft Vendor
1 answer

Error DF-EXPR-010 in ADF data flow, Column 'body' used in expression is unavailable or invalid

I need to take the results of a REST API query and transform them to upsert them into a table. The source returns this JSON: The Projection step ALWAYS puts the content in context of a body field. I can't figure out how to feed in the expected schema…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,274 questions
asked 2025-02-20T16:00:50.67+00:00
Altobelli, Marie K 0 Reputation points
commented 2025-02-21T18:22:02.52+00:00
phemanth 13,905 Reputation points Microsoft Vendor
1 answer

toDecimal function in dataflow of ADF/Azure Synapse Analytics not working correctly

I have input string field with name COST with below values. COST 2753000 217400 500 252567552 I am using toDecimal function in derived Column transformation in dataflow as below. z_COST= toDecimal(trim(COST)) Output is as…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,274 questions
asked 2025-02-20T15:15:57.9966667+00:00
Peruka, Sikander reddy 110 Reputation points
commented 2025-02-21T17:17:40.0066667+00:00
Peruka, Sikander reddy 110 Reputation points
1 answer

Azure Data Factory - Copy data from AWS Redshift to Azure Synapse - Failed to connect error

Hi, I am trying to copy a table of data from AWS Redshift into Azure Synapse, below is the JSON code from the data factory: { "name": "pipeline1", "properties": { "activities": [ { …

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,200 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,274 questions
asked 2025-02-18T22:50:02.35+00:00
Vinny Prajapati 0 Reputation points
commented 2025-02-21T17:17:00.0766667+00:00
phemanth 13,905 Reputation points Microsoft Vendor
0 answers

Based on wildcard (part-*.csv) I want to pick multiple part files and insert into SQL Server

I have a blob path where spark is writing multiple part files . Now I want to load all these part file data to SQL Server . I'm trying using wildcard character *.csv in ADF but it's not working. I tried using listing the files via GetMetaData but still…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,274 questions
asked 2025-02-20T11:17:48.35+00:00
Deepak Singh 0 Reputation points
edited a comment 2025-02-21T16:02:47.7033333+00:00
Deepak Singh 0 Reputation points
0 answers

Azure Data Factory Copy Activity fails to auto create table when copying from REST source to Azure SQL database

I am using a Copy Activity in Azure Data Factory. I have a REST source/data set created and a Azure SQL Database sink/data set created. I have the Copy Activity set to auto create and the table does not exist in the database before running the pipeline…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,274 questions
asked 2025-02-20T15:54:23.29+00:00
Tina Eilert(ARM) 0 Reputation points
commented 2025-02-21T14:41:48.7466667+00:00
Ganesh Gurram 4,335 Reputation points Microsoft Vendor
0 answers

Insert error from synapse data flow to SQL server DB

Hi, I am trying to capture an error throw by synapse data flow and insert to SQL DB table. But I keep getting update error. I think the error contain single quote ', is there a way to insert the error to the table? Below is my code in Script…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,200 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,274 questions
asked 2025-02-19T06:30:20.1366667+00:00
Lotus88 66 Reputation points
commented 2025-02-21T13:06:27.4933333+00:00
Smaran Thoomu 20,395 Reputation points Microsoft Vendor
0 answers

XML to JSON conversion in ADF missing out 'lang' tag details

I am converting an xml from html source to a json file using ADF. However, for one of the attributes, the 'lang' element is not getting translated. The xml data in file is as follows: ''' <displayname xml:lang="deAT">…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,274 questions
asked 2025-02-20T08:47:03.38+00:00
Alefiya Tikiwala 0 Reputation points
commented 2025-02-21T13:01:13.51+00:00
Smaran Thoomu 20,395 Reputation points Microsoft Vendor
0 answers

Removed trigger still triggers

Hi! I had a trigger in Azure Data Factory with a related pipeline. The trigger has been deleted (and the change published) but it is still triggering over 12 hours later. The trigger is not in the GUI, not connected to the pipeline and not in Git.…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,274 questions
asked 2025-02-19T09:54:08.86+00:00
JE 0 Reputation points
commented 2025-02-21T12:59:04.6633333+00:00
Smaran Thoomu 20,395 Reputation points Microsoft Vendor
3 answers One of the answers was accepted by the question author.

Issue with Bicep module output for ADF identity

Hello I am currently using Bicep version 0.4.1318 to deploy an Azure Data Factory instance. In my main Bicep module, I have the following code that calls a sub-module for provisioning the Data Factory: bicepRunCopy code1var adfName =…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,274 questions
asked 2025-02-19T08:17:02.44+00:00
Gabriel-2005 465 Reputation points
accepted 2025-02-21T12:49:16.2166667+00:00
Gabriel-2005 465 Reputation points
1 answer

I run a Pipeline runs at the adf and keep reporting an error Job in progress

I see that the error type prompted is user configuration. The problem occurred in the later link after it was run once before, but it will always be stuck in the second link after it is run again. How can I solve this problem?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,274 questions
asked 2025-02-20T03:09:46.0766667+00:00
lhwill 0 Reputation points
commented 2025-02-21T12:21:44.2433333+00:00
Chandra Boorla 8,795 Reputation points Microsoft Vendor
0 answers

copy data gives error NullReferenceException: Object reference not set to an instance of an object

I'm copying from Azure Blob Storage .csv files into Snowflake, and receiving the error ErrorCode=SessionLogFailToWriteLog,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Write log is failed with error message:…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,274 questions
asked 2025-02-03T23:51:04.0566667+00:00
MarkDG 0 Reputation points
commented 2025-02-21T11:20:32.6566667+00:00
MarkDG 0 Reputation points
2 answers

How to copy data from CSV file that has one column with line breaks

My source CSV file has two columns that have huge chunks of texts with lots of line breaks. When I use Copy activity to read this file in ADF, it breaks the rows whenever it sees an line break instead of actually parsing the complete row. …

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,274 questions
asked 2022-02-22T15:10:41.64+00:00
Mohsen Alam 11 Reputation points
commented 2025-02-21T09:17:02.29+00:00
Tim S 0 Reputation points
3 answers

Pipline Failure

Failure happened on 'Source' side. ErrorCode=SqlOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=A database operation failed with the following error: 'Conversion failed when converting date and/or time from…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,274 questions
asked 2025-02-18T06:32:39.91+00:00
Ext.Chandan Kumar 0 Reputation points
edited a comment 2025-02-21T06:28:28.85+00:00
Chandra Boorla 8,795 Reputation points Microsoft Vendor
1 answer

error with ssis package in azure Failed get parameter info of parameter sql password

There is an SSIS package running on a schedule in Azure. The SQL database is also in Azure. The SSIS package moves data from SQL to D365. I am getting this error when the package starts to run: error with ssis package in azure Failed get parameter…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,274 questions
asked 2025-02-19T17:33:28.59+00:00
Al Levy 0 Reputation points
commented 2025-02-21T06:11:05.8733333+00:00
Chandra Boorla 8,795 Reputation points Microsoft Vendor