Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,333 questions
0 answers
How to parametrize in the arm-template-parameters-definition.json the referenceName for Linked Service Reference in the Datasets part.
I would like to parametrize in the arm-template-parameters-definition.json the "referenceName" for linked service in the datasets part of the file. You can have a look at the below image the part I want to change in order the production…
asked 2025-03-10T07:51:01.0166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(300.8, 30%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ED%3C/text%3E%3C/svg%3E)
Dina
0
Reputation points
commented 2025-03-12T08:30:22.4266667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 18%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGG%3C/text%3E%3C/svg%3E)
Ganesh Gurram
4,965
Reputation points Microsoft External Staff
1 answer
Data Factory - Unable to read zip files from Amazon S3. Error: Central Directory corrupt. Unable to read beyond the end of the stream.
I am using Azure Data Factory's Copy activity to read csv containing in zip files in Amazon S3. I am experiencing problems since yesterday about reading only zip files from Amazon S3, other types of files were read all fine (or at least pure csv files).…
asked 2025-03-03T15:41:25.5666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(16, 0%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EQD%3C/text%3E%3C/svg%3E)
Quyen Dang
21
Reputation points
commented 2025-03-12T06:35:58.4066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(6.4, 57.00000000000001%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERG%3C/text%3E%3C/svg%3E)
Rakesh Govindula
95
Reputation points Microsoft External Staff
1 answer
Get file names from different folders and combine them into one variable in ADF
I have 100 folders: f1-f100, each folders have many different csv files. How to get all the csv files' names together,in a list, like results in one variable? Thanks a lot source in a storage blob container like below: f1: test1.csv, test2.csv …
asked 2025-03-10T17:16:24.1966667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(214.4, 32%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EZY%3C/text%3E%3C/svg%3E)
Zhu, Yueli YZ [NC]
255
Reputation points
commented 2025-03-12T06:22:58.1666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(195.2, 88%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESC%3C/text%3E%3C/svg%3E)
Shabarinath Chede
0
Reputation points
1 answer
Copy Data Activity with Upsert Operation to MongoDB Not Updating Records
I am trying to use Azure Data Factory's Copy Data Activity to copy data from a CSV file into a MongoDB collection using the upsert operation. However, I am facing an issue where the pipeline run indicates that 4 rows were written, but no records are…
asked 2025-03-06T20:28:21.6766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(48, 4%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERV%3C/text%3E%3C/svg%3E)
Ramakrishna Vaitla
0
Reputation points
commented 2025-03-12T04:35:05.8+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
phemanth
14,560
Reputation points Microsoft External Staff
1 answer
One of the answers was accepted by the question author.
How to fix "Record not found.." error while trying to insert new row into Dataverse entity using ADF
I have a data factory pipeline that contains a Copy activity with Source as a SQL Server table and Sink as Dataverse entity. The pipeline reads new or existing rows from the table and Upserts into dataverse. However for a new row that does not exists in…
asked 2025-03-07T19:59:03.8666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(294.40000000000003, 91%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMP%3C/text%3E%3C/svg%3E)
Mohsin Patel
0
Reputation points Microsoft Employee
accepted 2025-03-12T03:34:53.74+00:00
Mohsin Patel
0
Reputation points Microsoft Employee
1 answer
How to express wildcard character(s) in Azure Logic Apps
I have following file name in Azure File Storage: I was trying to use following Expression to grab today's file in Azure Data Factory: But, somehow, I am getting error saying it would not recognize the file: I am wondering how to express wildcard…
asked 2025-03-11T21:26:32.22+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 39%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKJ%3C/text%3E%3C/svg%3E)
King Java
690
Reputation points
edited an answer 2025-03-12T00:42:07.95+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(252.8, 99%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJN%3C/text%3E%3C/svg%3E)
J N S S Kasyap
860
Reputation points Microsoft External Staff
0 answers
Unable to connect to private ADB cluster from ADB connector with Azure IR
I'm trying to create a new linked service in Azure Data Factory with ADB connector, and unable to connect to private ADB cluster from ADB connector with Azure IR - I'm getting "Loading failed" message, however with SHIR it works.
asked 2025-03-11T09:06:16.3166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(128, 45%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EES%3C/text%3E%3C/svg%3E)
Eddie Schneider
0
Reputation points Microsoft Employee
commented 2025-03-11T23:09:28.6066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 65%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECB%3C/text%3E%3C/svg%3E)
Chandra Boorla
9,910
Reputation points Microsoft External Staff
1 answer
Unable to Create Ondemand Hdinsight Hadoop Clusters in Data factory.
I am trying to create a linked service of HDInsight hadoop Clusters. With Cluster size 4, 1 hour as time to live, Provided service principal Id, Azure Key vault Secret Key, Azure Storage account and a SQL Server in Hcatalog. But I am unable to move…
asked 2025-03-05T07:23:18.68+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(169.60000000000002, 46%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVA%3C/text%3E%3C/svg%3E)
Vishesh Agarwal
0
Reputation points
commented 2025-03-11T20:24:56.9666667+00:00
Chandra Boorla
9,910
Reputation points Microsoft External Staff
0 answers
ADF Google Ads Linked Service using Service Authentication as the Authentication Type ERROR pkcs8PrivateKey
Trying to access Google Ads data using a Google Ads Service Account, and the ADF Google Ads Linked Service. The linked service "Authentication type" set to "Service authentication". We generated a private key for this Service…
asked 2025-03-04T21:37:57.02+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 12%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMB%3C/text%3E%3C/svg%3E)
mamadou bhoye sow
5
Reputation points
commented 2025-03-11T18:55:38.45+00:00
mamadou bhoye sow
5
Reputation points
0 answers
ADF Pipeline Issue from Snowflake to Azure SQL data copy
Hi Team, Im trying to create function in Azure SQL using ADF Pipeline,Some times My pipeline is getting success and sometimes its failing with below error.Please provide me solution for the below issue. operation on target ITERATE_TABLES_LOAD failed:…
asked 2025-03-10T07:07:41.4833333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(291.2, 23%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPP%3C/text%3E%3C/svg%3E)
PK PK
0
Reputation points
commented 2025-03-11T18:09:00.91+00:00
J N S S Kasyap
860
Reputation points Microsoft External Staff
1 answer
Azure Data Factory ForEach/If - Force Next Iteration
Our organization receives and loads files into an on-prem SQL Server using SSIS packages written in Visual Studio. As part of the file loads, we run a series of validations on the files to insure that certain things are correct - file name, is the file…
asked 2025-03-06T16:40:32.72+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(92.8, 68%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECL%3C/text%3E%3C/svg%3E)
Corey Livermore
20
Reputation points
commented 2025-03-11T17:51:36.3033333+00:00
J N S S Kasyap
860
Reputation points Microsoft External Staff
0 answers
Error while Unzipping .tar.gz files using Copy activity - Failing for csv file having 8GB of data
Hi, I am trying to uncompress .tar.gz files using Copy Activity and it was working fine till yesterday. Today we have received a file with more data and it got failed while copying one file with 8GB of data with following error…
asked 2025-03-11T06:51:23.57+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(64, 21%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENA%3C/text%3E%3C/svg%3E)
Nagarajan Arumugam
0
Reputation points
commented 2025-03-11T17:50:22.41+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(195.2, 50%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVR%3C/text%3E%3C/svg%3E)
Venkat Reddy Navari
0
Reputation points Microsoft External Staff
1 answer
Microsoft Fabric API Authentication & Deployment Issue | Azure DevOps Pipeline
Dear All, We have configured Microsoft Fabric in our environment, with dedicated workspaces for Dev, UAT, and PROD. Instead of using Deployment Pipelines, we have integrated our workspaces with an Azure DevOps Git repository branch for version…
asked 2025-03-11T13:50:24.8933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(211.20000000000002, 67%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESA%3C/text%3E%3C/svg%3E)
Suki Azure
21
Reputation points
edited an answer 2025-03-11T17:04:08.2166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
21,320
Reputation points Microsoft External Staff
1 answer
Within Azure Data Factory Studio, I get a Failed Connection to my SQL Database
Below is the error I keep getting, I've read all documentation and tried all steps to ensure identification is all there. I'm stuck and have a client deliverable. I would happily pay extra to get over this hurdle as I need to use the data warehouse…
asked 2025-02-14T00:58:25.4166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(307.2, 28.000000000000004%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDL%3C/text%3E%3C/svg%3E)
Derek Lilley
20
Reputation points
edited the question 2025-03-11T14:50:59.33+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(281.6, 17%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGM%3C/text%3E%3C/svg%3E)
Givary-MSFT
35,456
Reputation points Microsoft Employee
1 answer
Data flow gives different output in debug and trigger.
I have a few dataflows which uses sorter and aggregator. I sort my data on a column, then drop duplicates using aggregate transformation. I am using last($$) to select the last occurrence of the primary key. On using data preview or executing pipeline…
asked 2025-03-11T13:16:38.3566667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(179.20000000000002, 66%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBN%3C/text%3E%3C/svg%3E)
Bansal, Nimish
60
Reputation points
answered 2025-03-11T14:19:51.73+00:00
Alex Burlachenko
1,750
Reputation points
1 answer
One of the answers was accepted by the question author.
Copy Activity Issue
Hello Team, Already there is 1 table called Account in Azure SQL Database and data is getting loaded properly from SAP. I need to add 3 columns into the Account table and insert the data into 3 columns from the salesforce. So i am using the copy…
asked 2025-03-10T07:38:54.2833333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 13%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERK%3C/text%3E%3C/svg%3E)
Rohit Kulkarni
711
Reputation points
accepted 2025-03-11T11:33:39.83+00:00
Rohit Kulkarni
711
Reputation points
1 answer
XML to JSON conversion in ADF missing out 'lang' tag details
I am converting an xml from html source to a json file using ADF. However, for one of the attributes, the 'lang' element is not getting translated. The xml data in file is as follows: ''' <displayname xml:lang="deAT">…
asked 2025-02-20T08:47:03.38+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 26%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAT%3C/text%3E%3C/svg%3E)
Alefiya Tikiwala
0
Reputation points
edited a comment 2025-03-11T10:11:03.93+00:00
Alefiya Tikiwala
0
Reputation points
3 answers
ADF Power Query Sink Settings UI Missing Options
Why would my data factory not have the same UI as I see in training videos and QA answers on the Power Query data wrangling part of ADF? I can only see the delete button next to the dataset in the Power Query activity Sink…
asked 2025-01-23T15:50:49.8533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(272, 24%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EES%3C/text%3E%3C/svg%3E)
Elisa Schrader
20
Reputation points
commented 2025-03-11T09:26:21.9733333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(102.4, 36%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EZC%3C/text%3E%3C/svg%3E)
Zac Connolly
5
Reputation points
1 answer
How to delete a dev branch?
How to delete my dev branch?
asked 2025-03-10T18:58:57.3566667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(105.60000000000001, 86%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHZ%3C/text%3E%3C/svg%3E)
Henry Zhu
0
Reputation points Microsoft Employee
answered 2025-03-11T08:38:54.4266667+00:00
Smaran Thoomu
21,320
Reputation points Microsoft External Staff
1 answer
Using Managed Virtual Network Integration Runtime in Data Flows
I have created a Managed Virtual Network Integrated Runtime (MVNet IR) and successfully connected it to Azure SQL. I can access the database through datasets using this runtime without any issues. However, when creating Data Flows using the same datasets…
asked 2025-03-07T12:20:00.07+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(22.400000000000002, 73%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPS%3C/text%3E%3C/svg%3E)
Paila, S (Sreenu)
0
Reputation points
commented 2025-03-11T08:10:14.2066667+00:00
Smaran Thoomu
21,320
Reputation points Microsoft External Staff