11,335 questions with Azure Data Factory tags

Sort by: Updated
0 answers

ADF Dataflow issue with source API query

am trying to setup an Azure Data Factory Dataflow. The source is a REST API from Monday.com which uses GraphQL. The API requires POST method, with 3 headers Content-Type "application/json", API-Version "2025-01 " and Authorization:…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,335 questions
asked 2025-02-22T03:15:01.8166667+00:00
Daniel Baird 0 Reputation points
commented 2025-03-12T13:53:09.7566667+00:00
Daniel Baird 0 Reputation points
2 answers

adf_publish an Main branch in ADF

hi, I would like to understand about adf_publish branch. Wat is this branch used for? what code it contains? Main branch is the final or producation ready branch which contains the latest and working copy of all pipelines, dataflows and datasets. Why we…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,335 questions
asked 2024-03-13T05:07:46.6733333+00:00
Santhi Dhanuskodi 325 Reputation points
commented 2025-03-12T13:42:42.38+00:00
Siddharth Choudhary 20 Reputation points
1 answer

iif condition in Expression builder in ADF Alter row

Hi Team, In my source file, i have country column there i have list of countries. In my source data country name Spain is now changed to "Espana". for this i want to write update condition in AlterRow condition. I tried following…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,335 questions
asked 2025-03-12T10:28:32.18+00:00
Nemade, Pradnya 0 Reputation points
answered 2025-03-12T13:14:02.6733333+00:00
Amira Bedhiafi 29,711 Reputation points
0 answers

Netezza - Linked service has an error Payload [Unrecognized payload received from server]

Good Day, A pipeline that is using a Netezza connector stopped working. [ERROR [HY000] [Microsoft][Netezza] (2000) Unrecognized payload received from server. Opcode: 100 ERROR [HY000] [Microsoft][Netezza] (2000) Unrecognized payload received from…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,335 questions
asked 2025-03-12T08:53:20.2166667+00:00
Bongo 1 Reputation point
edited the question 2025-03-12T11:35:28.1233333+00:00
simo-k 4,020 Reputation points
1 answer

ADF Google Ads Linked Service using Service Authentication as the Authentication Type ERROR pkcs8PrivateKey

Trying to access Google Ads data using a Google Ads Service Account, and the ADF Google Ads Linked Service. The linked service "Authentication type" set to "Service authentication".  We generated a private key for this Service…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,335 questions
asked 2025-03-04T21:37:57.02+00:00
mamadou bhoye sow 5 Reputation points
answered 2025-03-12T10:04:52.18+00:00
AnnuKumari-MSFT 34,271 Reputation points Microsoft Employee
1 answer

Failure happened on 'Sink' side. ErrorCode=UserErrorKeyConflict,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Key Content-Type conflicts in AuthHeaders/AdditionalHeaders,Source=Microsoft.DataTransfer.ClientLibrary,'

I am trying to post file using rest connector, but it is not allowing to add additional headers for Content-Type - as it is by default applying content-type as application/json, what if, i need to use different content-type. I believe it should be…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,335 questions
asked 2025-03-08T18:31:17.22+00:00
Chourasia, Sanoj K 0 Reputation points
edited a comment 2025-03-12T09:51:23.0266667+00:00
phemanth 14,560 Reputation points Microsoft External Staff
0 answers

Error while Unzipping .tar.gz files using Copy activity - Failing for csv file having 8GB of data

Hi, I am trying to uncompress .tar.gz files using Copy Activity and it was working fine till yesterday. Today we have received a file with more data and it got failed while copying one file with 8GB of data with following error…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,335 questions
asked 2025-03-11T06:51:23.57+00:00
Nagarajan Arumugam 0 Reputation points
commented 2025-03-12T09:15:55.23+00:00
Venkat Reddy Navari 0 Reputation points Microsoft External Staff
0 answers

How to parametrize in the arm-template-parameters-definition.json the referenceName for Linked Service Reference in the Datasets part.

I would like to parametrize in the arm-template-parameters-definition.json the "referenceName" for linked service in the datasets part of the file. You can have a look at the below image the part I want to change in order the production…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,335 questions
asked 2025-03-10T07:51:01.0166667+00:00
Dina 0 Reputation points
commented 2025-03-12T08:30:22.4266667+00:00
Ganesh Gurram 4,965 Reputation points Microsoft External Staff
1 answer

Data Factory - Unable to read zip files from Amazon S3. Error: Central Directory corrupt. Unable to read beyond the end of the stream.

I am using Azure Data Factory's Copy activity to read csv containing in zip files in Amazon S3. I am experiencing problems since yesterday about reading only zip files from Amazon S3, other types of files were read all fine (or at least pure csv files).…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,335 questions
asked 2025-03-03T15:41:25.5666667+00:00
Quyen Dang 21 Reputation points
commented 2025-03-12T06:35:58.4066667+00:00
Rakesh Govindula 95 Reputation points Microsoft External Staff
1 answer

Get file names from different folders and combine them into one variable in ADF

I have 100 folders: f1-f100, each folders have many different csv files. How to get all the csv files' names together,in a list, like results in one variable? Thanks a lot source in a storage blob container like below: f1: test1.csv, test2.csv …

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,335 questions
asked 2025-03-10T17:16:24.1966667+00:00
Zhu, Yueli YZ [NC] 255 Reputation points
commented 2025-03-12T06:22:58.1666667+00:00
Shabarinath Chede 0 Reputation points
1 answer

Copy Data Activity with Upsert Operation to MongoDB Not Updating Records

I am trying to use Azure Data Factory's Copy Data Activity to copy data from a CSV file into a MongoDB collection using the upsert operation. However, I am facing an issue where the pipeline run indicates that 4 rows were written, but no records are…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,335 questions
asked 2025-03-06T20:28:21.6766667+00:00
Ramakrishna Vaitla 0 Reputation points
commented 2025-03-12T04:35:05.8+00:00
phemanth 14,560 Reputation points Microsoft External Staff
1 answer One of the answers was accepted by the question author.

How to fix "Record not found.." error while trying to insert new row into Dataverse entity using ADF

I have a data factory pipeline that contains a Copy activity with Source as a SQL Server table and Sink as Dataverse entity. The pipeline reads new or existing rows from the table and Upserts into dataverse. However for a new row that does not exists in…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,335 questions
asked 2025-03-07T19:59:03.8666667+00:00
Mohsin Patel 0 Reputation points Microsoft Employee
accepted 2025-03-12T03:34:53.74+00:00
Mohsin Patel 0 Reputation points Microsoft Employee
1 answer

How to express wildcard character(s) in Azure Logic Apps

I have following file name in Azure File Storage: I was trying to use following Expression to grab today's file in Azure Data Factory: But, somehow, I am getting error saying it would not recognize the file: I am wondering how to express wildcard…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,335 questions
asked 2025-03-11T21:26:32.22+00:00
King Java 690 Reputation points
edited an answer 2025-03-12T00:42:07.95+00:00
J N S S Kasyap 860 Reputation points Microsoft External Staff
0 answers

Unable to connect to private ADB cluster from ADB connector with Azure IR

I'm trying to create a new linked service in Azure Data Factory with ADB connector, and unable to connect to private ADB cluster from ADB connector with Azure IR - I'm getting "Loading failed" message, however with SHIR it works.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,335 questions
asked 2025-03-11T09:06:16.3166667+00:00
Eddie Schneider 0 Reputation points Microsoft Employee
commented 2025-03-11T23:09:28.6066667+00:00
Chandra Boorla 9,910 Reputation points Microsoft External Staff
1 answer

Unable to Create Ondemand Hdinsight Hadoop Clusters in Data factory.

I am trying to create a linked service of HDInsight hadoop Clusters. With Cluster size 4, 1 hour as time to live, Provided service principal Id, Azure Key vault Secret Key, Azure Storage account and a SQL Server in Hcatalog. But I am unable to move…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,335 questions
asked 2025-03-05T07:23:18.68+00:00
Vishesh Agarwal 0 Reputation points
commented 2025-03-11T20:24:56.9666667+00:00
Chandra Boorla 9,910 Reputation points Microsoft External Staff
0 answers

ADF Pipeline Issue from Snowflake to Azure SQL data copy

Hi Team, Im trying to create function in Azure SQL using ADF Pipeline,Some times My pipeline is getting success and sometimes its failing with below error.Please provide me solution for the below issue. operation on target ITERATE_TABLES_LOAD failed:…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,335 questions
asked 2025-03-10T07:07:41.4833333+00:00
PK PK 0 Reputation points
commented 2025-03-11T18:09:00.91+00:00
J N S S Kasyap 860 Reputation points Microsoft External Staff
1 answer

Azure Data Factory ForEach/If - Force Next Iteration

Our organization receives and loads files into an on-prem SQL Server using SSIS packages written in Visual Studio. As part of the file loads, we run a series of validations on the files to insure that certain things are correct - file name, is the file…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,335 questions
asked 2025-03-06T16:40:32.72+00:00
Corey Livermore 20 Reputation points
commented 2025-03-11T17:51:36.3033333+00:00
J N S S Kasyap 860 Reputation points Microsoft External Staff
1 answer

Microsoft Fabric API Authentication & Deployment Issue | Azure DevOps Pipeline

Dear All, We have configured Microsoft Fabric in our environment, with dedicated workspaces for Dev, UAT, and PROD. Instead of using Deployment Pipelines, we have integrated our workspaces with an Azure DevOps Git repository branch for version…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,335 questions
asked 2025-03-11T13:50:24.8933333+00:00
Suki Azure 21 Reputation points
edited an answer 2025-03-11T17:04:08.2166667+00:00
Smaran Thoomu 21,320 Reputation points Microsoft External Staff
1 answer

Within Azure Data Factory Studio, I get a Failed Connection to my SQL Database

Below is the error I keep getting, I've read all documentation and tried all steps to ensure identification is all there. I'm stuck and have a client deliverable. I would happily pay extra to get over this hurdle as I need to use the data warehouse…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,335 questions
asked 2025-02-14T00:58:25.4166667+00:00
Derek Lilley 20 Reputation points
edited the question 2025-03-11T14:50:59.33+00:00
Givary-MSFT 35,456 Reputation points Microsoft Employee
1 answer

Data flow gives different output in debug and trigger.

I have a few dataflows which uses sorter and aggregator. I sort my data on a column, then drop duplicates using aggregate transformation. I am using last($$) to select the last occurrence of the primary key. On using data preview or executing pipeline…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,335 questions
asked 2025-03-11T13:16:38.3566667+00:00
Bansal, Nimish 60 Reputation points
answered 2025-03-11T14:19:51.73+00:00
Alex Burlachenko 1,750 Reputation points