Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,344 questions
1 answer
Rest call failed with client error in Source ADF , status code 404 NotFound in Azure Cloud
Hi, Trying to get data from the rest API data to Azure Data Factory here using Copy activity in source taking the rest dataset, but unable to get from that URL, Facing an Issue with Rest call failing with a client error in Source ADF, status code 404…
asked 2024-04-16T12:36:14.1433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(233.6, 83%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETR%3C/text%3E%3C/svg%3E)
Tejasree RACHURI
0
Reputation points
edited a comment 2025-03-14T23:58:26.9166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(128, 90%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKT%3C/text%3E%3C/svg%3E)
Kiran Talagana
0
Reputation points
2 answers
[ADF Deployments ] Client IP not authorized to access the API. Please ensure you are on corpnet, or that your IP is on an allowlist for the activities in your pipeline.
Status: Failed Error: Code: ClientIPNotAuthorized Message: Client IP not authorized to access the API. Please ensure you are on corpnet, or that your IP is on an allowlist for the activities in your pipeline. How can I resolve this issue?
asked 2022-12-20T19:01:52.357+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(19.2, 96%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ED%3C/text%3E%3C/svg%3E)
Dru
11
Reputation points Microsoft Employee
commented 2025-03-14T20:24:09.3166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(214.4, 34%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESP%3C/text%3E%3C/svg%3E)
Sheshnarayan Patel (TATA CONSULTANCY SERVICES LTD)
0
Reputation points Microsoft External Staff
1 answer
Azure Data Factory ForEach/If - Force Next Iteration
Our organization receives and loads files into an on-prem SQL Server using SSIS packages written in Visual Studio. As part of the file loads, we run a series of validations on the files to insure that certain things are correct - file name, is the file…
asked 2025-03-06T16:40:32.72+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(92.8, 68%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECL%3C/text%3E%3C/svg%3E)
Corey Livermore
20
Reputation points
commented 2025-03-14T18:30:45.6833333+00:00
Corey Livermore
20
Reputation points
0 answers
Incomplete Files from Copy Data Command in Azure Data Factory pipeline when uploading data from Snowflake
I am experiencing an issue where the file-sink of the Copy Data command (SnowflakeExportCopyCommand) is producing incomplete files when uploading data from Snowflake to Azure Blob Storage in our Azure Data Factory pipeline. Observations: The number of…
asked 2025-03-12T19:07:37.29+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 18%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESR%3C/text%3E%3C/svg%3E)
Susan Rakers
0
Reputation points
commented 2025-03-14T17:56:26.1966667+00:00
Susan Rakers
0
Reputation points
1 answer
Data Factory - Unable to read zip files from Amazon S3. Error: Central Directory corrupt. Unable to read beyond the end of the stream.
I am using Azure Data Factory's Copy activity to read csv containing in zip files in Amazon S3. I am experiencing problems since yesterday about reading only zip files from Amazon S3, other types of files were read all fine (or at least pure csv files).…
asked 2025-03-03T15:41:25.5666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(16, 0%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EQD%3C/text%3E%3C/svg%3E)
Quyen Dang
21
Reputation points
commented 2025-03-14T16:34:17.41+00:00
Quyen Dang
21
Reputation points
1 answer
One of the answers was accepted by the question author.
ADF Google Ads Linked Service using Service Authentication as the Authentication Type ERROR pkcs8PrivateKey
Trying to access Google Ads data using a Google Ads Service Account, and the ADF Google Ads Linked Service. The linked service "Authentication type" set to "Service authentication". We generated a private key for this Service…
asked 2025-03-04T21:37:57.02+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 12%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMB%3C/text%3E%3C/svg%3E)
mamadou bhoye sow
30
Reputation points
commented 2025-03-14T16:29:32.7766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(169.60000000000002, 63%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETS%3C/text%3E%3C/svg%3E)
Terry Shelton
0
Reputation points
0 answers
I am using a copy activity in ADF to copy 4million dicoms from one storage account to another. It throws 409 error :
I am trying to copy 4 million images from one storage account to another. It throws this error: Failure happened on 'Sink' side. ErrorCode=AzureBlobWriteOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Blob…
asked 2025-03-14T13:34:56.1966667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(48, 87%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJC%3C/text%3E%3C/svg%3E)
Jyothi Chaganty
6
Reputation points
commented 2025-03-14T14:30:57.3233333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(96, 92%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDD%3C/text%3E%3C/svg%3E)
Dilip Divakaran
81
Reputation points
0 answers
How to parametrize in the arm-template-parameters-definition.json the referenceName for Linked Service Reference in the Datasets part.
I would like to parametrize in the arm-template-parameters-definition.json the "referenceName" for linked service in the datasets part of the file. You can have a look at the below image the part I want to change in order the production…
asked 2025-03-10T07:51:01.0166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(300.8, 30%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ED%3C/text%3E%3C/svg%3E)
Dina
0
Reputation points
commented 2025-03-14T06:52:05.57+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(6.4, 57.00000000000001%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERG%3C/text%3E%3C/svg%3E)
Rakesh Govindula
175
Reputation points Microsoft External Staff
0 answers
Unable to connect to private ADB cluster from ADB connector with Azure IR
I'm trying to create a new linked service in Azure Data Factory with ADB connector, and unable to connect to private ADB cluster from ADB connector with Azure IR - I'm getting "Loading failed" message, however with SHIR it works.
asked 2025-03-11T09:06:16.3166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(128, 45%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EES%3C/text%3E%3C/svg%3E)
Eddie Schneider
0
Reputation points Microsoft Employee
commented 2025-03-13T20:56:08.85+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 65%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECB%3C/text%3E%3C/svg%3E)
Chandra Boorla
9,995
Reputation points Microsoft External Staff
1 answer
Data flow gives different output in debug and trigger.
I have a few dataflows which uses sorter and aggregator. I sort my data on a column, then drop duplicates using aggregate transformation. I am using last($$) to select the last occurrence of the primary key. On using data preview or executing pipeline…
asked 2025-03-11T13:16:38.3566667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(179.20000000000002, 66%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBN%3C/text%3E%3C/svg%3E)
Bansal, Nimish
60
Reputation points
commented 2025-03-13T20:38:03.7+00:00
Chandra Boorla
9,995
Reputation points Microsoft External Staff
1 answer
iif condition in Expression builder in ADF Alter row
Hi Team, In my source file, i have country column there i have list of countries. In my source data country name Spain is now changed to "Espana". for this i want to write update condition in AlterRow condition. I tried following…
asked 2025-03-12T10:28:32.18+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(246.4, 17%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENP%3C/text%3E%3C/svg%3E)
Nemade, Pradnya
0
Reputation points
commented 2025-03-13T20:35:05.4233333+00:00
Chandra Boorla
9,995
Reputation points Microsoft External Staff
4 answers
One of the answers was accepted by the question author.
Get file names from different folders and combine them into one variable in ADF
I have 100 folders: f1-f100, each folders have many different csv files. How to get all the csv files' names together,in a list, like results in one variable? Thanks a lot source in a storage blob container like below: f1: test1.csv, test2.csv …
asked 2025-03-10T17:16:24.1966667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(214.4, 32%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EZY%3C/text%3E%3C/svg%3E)
Zhu, Yueli YZ [NC]
280
Reputation points
accepted 2025-03-13T19:34:58.7733333+00:00
Zhu, Yueli YZ [NC]
280
Reputation points
1 answer
ErrorCode=InvalidParameter,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The value of the property 'columns' is invalid: 'Value cannot be null. Parameter name: columns'.,Source=,''Type=System.ArgumentNullException,Message=Valu
Hi, Am new to Postgresql, am trying to fetch the column list and their respective data types for a Postgresql table using a lookup activity within a foreach container in ADF . Query is working fine and fetching the expected output when executed in Azure…
asked 2025-03-12T18:01:16.0466667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(144, 22%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETS%3C/text%3E%3C/svg%3E)
TUMMALAPELLY, Sandeep
0
Reputation points
edited a comment 2025-03-13T19:26:24.99+00:00
Chandra Boorla
9,995
Reputation points Microsoft External Staff
2 answers
Azure Data Factory Copy Activity - MongoDB Upsert Failing to Update Existing Documents
I'm using Azure Data Factory's Copy Data activity to load data from a CSV file into a MongoDB collection. My goal is to use the "upsert" operation to either insert new documents or update existing ones based on the _id field. However, I'm…
asked 2025-03-13T09:59:04.5833333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(48, 4%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERV%3C/text%3E%3C/svg%3E)
Ramakrishna Vaitla
5
Reputation points
answered 2025-03-13T18:19:26.1466667+00:00
Chandra Boorla
9,995
Reputation points Microsoft External Staff
1 answer
Suddenly my Pipelines that are sync to an Azure SQL are failing
Hello, suddenly my Pipelines that connects fron an on prem SQL to Azure SQL are failing. The connections works on boths SQL, I can test and see the data in Debug in the Pipeline, but when I execute the pipeline it says the…
asked 2025-03-13T16:50:52.78+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(188.79999999999998, 1%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESP%3C/text%3E%3C/svg%3E)
Silvano Paxia
101
Reputation points
answered 2025-03-13T17:45:25.1133333+00:00
Chandra Boorla
9,995
Reputation points Microsoft External Staff
1 answer
One of the answers was accepted by the question author.
How to express wildcard character(s) in Azure Logic Apps
I have following file name in Azure File Storage: I was trying to use following Expression to grab today's file in Azure Data Factory: But, somehow, I am getting error saying it would not recognize the file: I am wondering how to express wildcard…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 39%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKJ%3C/text%3E%3C/svg%3E)
1 answer
ADF Dataflow issue with source API query
am trying to setup an Azure Data Factory Dataflow. The source is a REST API from Monday.com which uses GraphQL. The API requires POST method, with 3 headers Content-Type "application/json", API-Version "2025-01 " and Authorization:…
asked 2025-02-22T03:15:01.8166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 69%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDB%3C/text%3E%3C/svg%3E)
Daniel Baird
0
Reputation points
commented 2025-03-13T12:34:13.4866667+00:00
Daniel Baird
0
Reputation points
1 answer
How to delete a dev branch?
How to delete my dev branch?
asked 2025-03-10T18:58:57.3566667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(105.60000000000001, 86%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHZ%3C/text%3E%3C/svg%3E)
Henry Zhu
0
Reputation points Microsoft Employee
commented 2025-03-13T10:49:18.5933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
21,410
Reputation points Microsoft External Staff
1 answer
Microsoft Fabric API Authentication & Deployment Issue | Azure DevOps Pipeline
Dear All, We have configured Microsoft Fabric in our environment, with dedicated workspaces for Dev, UAT, and PROD. Instead of using Deployment Pipelines, we have integrated our workspaces with an Azure DevOps Git repository branch for version…
asked 2025-03-11T13:50:24.8933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(211.20000000000002, 67%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESA%3C/text%3E%3C/svg%3E)
Suki Azure
21
Reputation points
commented 2025-03-13T10:10:03.68+00:00
Smaran Thoomu
21,410
Reputation points Microsoft External Staff
1 answer
Ambiguity in loop variable #item when using contains within reduce in Data Flow Expressions.
I am trying to use a contains function within a reduce function. If I were to perform some comparison on the two items how would I accomplish this? Would the #item loop variable within the contain override the #item loop variable in the reduce…
asked 2025-03-12T22:37:55.83+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(96, 26%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESL%3C/text%3E%3C/svg%3E)
Sathyajit Loganathan
0
Reputation points
commented 2025-03-13T10:06:40.4233333+00:00
Sathyajit Loganathan
0
Reputation points