Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,339 questions
0 answers
Incomplete Files from Copy Data Command in Azure Data Factory pipeline when uploading data from Snowflake
I am experiencing an issue where the file-sink of the Copy Data command (SnowflakeExportCopyCommand) is producing incomplete files when uploading data from Snowflake to Azure Blob Storage in our Azure Data Factory pipeline. Observations: The number of…
asked 2025-03-12T19:07:37.29+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 18%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESR%3C/text%3E%3C/svg%3E)
Susan Rakers
0
Reputation points
asked 2025-03-12T19:07:37.29+00:00
Susan Rakers
0
Reputation points
0 answers
Azure Data Factory: additional column in copy data activity reuse column values in a function?
I am struggling with an Azure Data Factory pipeline action which is a copy data. I have an input CSV file which I need to copy over to a blob storage as a CSV file but one of the column's values must get URLEncoded. Problem which I currently have is that…
asked 2025-03-12T19:02:33.76+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(147.20000000000002, 39%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBV%3C/text%3E%3C/svg%3E)
Bram Vanhautte
0
Reputation points
asked 2025-03-12T19:02:33.76+00:00
Bram Vanhautte
0
Reputation points
0 answers
How to express wildcard character(s) in Azure Logic Apps
I have following file name in Azure File Storage: I was trying to use following Expression to grab today's file in Azure Data Factory: But, somehow, I am getting error saying it would not recognize the file: I am wondering how to express wildcard…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 39%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKJ%3C/text%3E%3C/svg%3E)
0 answers
What Azure Service to use to connect to ZOHO CRM to work with Azure AI backend (SaaS product) we are building
Looking to connect and fetch user data from ZOHO CRM to respond to customer queries via chat interface. Like Copilot type of service. What is the best way to connect to ZOHO CRM API via Azure AI service?
asked 2025-03-12T07:41:04.4633333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(262.40000000000003, 93%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDS%3C/text%3E%3C/svg%3E)
Deniz Subasi
0
Reputation points
edited the question 2025-03-12T18:10:21.4433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 76%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EV%3C/text%3E%3C/svg%3E)
VarunTha
13,035
Reputation points Microsoft External Staff
0 answers
ErrorCode=InvalidParameter,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The value of the property 'columns' is invalid: 'Value cannot be null. Parameter name: columns'.,Source=,''Type=System.ArgumentNullException,Message=Valu
Hi, Am new to Postgresql, am trying to fetch the column list and their respective data types for a Postgresql table using a lookup activity within a foreach container in ADF . Query is working fine and fetching the expected output when executed in Azure…
asked 2025-03-12T18:01:16.0466667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(144, 22%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETS%3C/text%3E%3C/svg%3E)
TUMMALAPELLY, Sandeep
0
Reputation points
asked 2025-03-12T18:01:16.0466667+00:00
TUMMALAPELLY, Sandeep
0
Reputation points
2 answers
One of the answers was accepted by the question author.
Trigger in ADF
I am working on an ADF pipeline that pulls data from an external SQL database. Instead of running the pipeline on a fixed schedule, I want it to be triggered automatically whenever new or updated data is available in a specific table within the SQL…
asked 2024-08-27T09:01:39.1533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(233.6, 53%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ED%3C/text%3E%3C/svg%3E)
DataCoder
280
Reputation points
edited a comment 2025-03-12T17:54:02.1333333+00:00
Jaliya Udagedara
2,826
Reputation points MVP
1 answer
Get file names from different folders and combine them into one variable in ADF
I have 100 folders: f1-f100, each folders have many different csv files. How to get all the csv files' names together,in a list, like results in one variable? Thanks a lot source in a storage blob container like below: f1: test1.csv, test2.csv …
asked 2025-03-10T17:16:24.1966667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(214.4, 32%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EZY%3C/text%3E%3C/svg%3E)
Zhu, Yueli YZ [NC]
260
Reputation points
commented 2025-03-12T17:50:24.3366667+00:00
Zhu, Yueli YZ [NC]
260
Reputation points
1 answer
Azure Data Factory ForEach/If - Force Next Iteration
Our organization receives and loads files into an on-prem SQL Server using SSIS packages written in Visual Studio. As part of the file loads, we run a series of validations on the files to insure that certain things are correct - file name, is the file…
asked 2025-03-06T16:40:32.72+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(92.8, 68%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECL%3C/text%3E%3C/svg%3E)
Corey Livermore
20
Reputation points
edited a comment 2025-03-12T17:25:16.5933333+00:00
Corey Livermore
20
Reputation points
1 answer
One of the answers was accepted by the question author.
ADF Google Ads Linked Service using Service Authentication as the Authentication Type ERROR pkcs8PrivateKey
Trying to access Google Ads data using a Google Ads Service Account, and the ADF Google Ads Linked Service. The linked service "Authentication type" set to "Service authentication". We generated a private key for this Service…
asked 2025-03-04T21:37:57.02+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 12%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMB%3C/text%3E%3C/svg%3E)
mamadou bhoye sow
30
Reputation points
edited a comment 2025-03-12T17:11:57.95+00:00
mamadou bhoye sow
30
Reputation points
0 answers
Netezza - Linked service has an error Payload [Unrecognized payload received from server]
Good Day, A pipeline that is using a Netezza connector stopped working. [ERROR [HY000] [Microsoft][Netezza] (2000) Unrecognized payload received from server. Opcode: 100 ERROR [HY000] [Microsoft][Netezza] (2000) Unrecognized payload received from…
asked 2025-03-12T08:53:20.2166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(217.60000000000002, 17%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EB%3C/text%3E%3C/svg%3E)
Bongo
1
Reputation point
edited a comment 2025-03-12T16:58:31.77+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(195.2, 50%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVR%3C/text%3E%3C/svg%3E)
Venkat Reddy Navari
80
Reputation points Microsoft External Staff
0 answers
ADF Pipeline Issue from Snowflake to Azure SQL data copy
Hi Team, Im trying to create function in Azure SQL using ADF Pipeline,Some times My pipeline is getting success and sometimes its failing with below error.Please provide me solution for the below issue. operation on target ITERATE_TABLES_LOAD failed:…
asked 2025-03-10T07:07:41.4833333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(291.2, 23%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPP%3C/text%3E%3C/svg%3E)
PK PK
0
Reputation points
commented 2025-03-12T16:37:30.8966667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(252.8, 99%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJN%3C/text%3E%3C/svg%3E)
J N S S Kasyap
865
Reputation points Microsoft External Staff
0 answers
How to parametrize in the arm-template-parameters-definition.json the referenceName for Linked Service Reference in the Datasets part.
I would like to parametrize in the arm-template-parameters-definition.json the "referenceName" for linked service in the datasets part of the file. You can have a look at the below image the part I want to change in order the production…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(300.8, 30%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ED%3C/text%3E%3C/svg%3E)
1 answer
Microsoft Fabric API Authentication & Deployment Issue | Azure DevOps Pipeline
Dear All, We have configured Microsoft Fabric in our environment, with dedicated workspaces for Dev, UAT, and PROD. Instead of using Deployment Pipelines, we have integrated our workspaces with an Azure DevOps Git repository branch for version…
asked 2025-03-11T13:50:24.8933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(211.20000000000002, 67%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESA%3C/text%3E%3C/svg%3E)
Suki Azure
21
Reputation points
commented 2025-03-12T14:11:24.12+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
21,325
Reputation points Microsoft External Staff
1 answer
How to delete a dev branch?
How to delete my dev branch?
asked 2025-03-10T18:58:57.3566667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(105.60000000000001, 86%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHZ%3C/text%3E%3C/svg%3E)
Henry Zhu
0
Reputation points Microsoft Employee
commented 2025-03-12T14:10:54.81+00:00
Smaran Thoomu
21,325
Reputation points Microsoft External Staff
0 answers
ADF Dataflow issue with source API query
am trying to setup an Azure Data Factory Dataflow. The source is a REST API from Monday.com which uses GraphQL. The API requires POST method, with 3 headers Content-Type "application/json", API-Version "2025-01 " and Authorization:…
asked 2025-02-22T03:15:01.8166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 69%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDB%3C/text%3E%3C/svg%3E)
Daniel Baird
0
Reputation points
commented 2025-03-12T13:53:09.7566667+00:00
Daniel Baird
0
Reputation points
2 answers
adf_publish an Main branch in ADF
hi, I would like to understand about adf_publish branch. Wat is this branch used for? what code it contains? Main branch is the final or producation ready branch which contains the latest and working copy of all pipelines, dataflows and datasets. Why we…
asked 2024-03-13T05:07:46.6733333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(19.2, 54%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESD%3C/text%3E%3C/svg%3E)
Santhi Dhanuskodi
325
Reputation points
commented 2025-03-12T13:42:42.38+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(80, 36%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESC%3C/text%3E%3C/svg%3E)
Siddharth Choudhary
20
Reputation points
1 answer
iif condition in Expression builder in ADF Alter row
Hi Team, In my source file, i have country column there i have list of countries. In my source data country name Spain is now changed to "Espana". for this i want to write update condition in AlterRow condition. I tried following…
asked 2025-03-12T10:28:32.18+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(246.4, 17%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENP%3C/text%3E%3C/svg%3E)
Nemade, Pradnya
0
Reputation points
answered 2025-03-12T13:14:02.6733333+00:00
Amira Bedhiafi
29,711
Reputation points
1 answer
Failure happened on 'Sink' side. ErrorCode=UserErrorKeyConflict,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Key Content-Type conflicts in AuthHeaders/AdditionalHeaders,Source=Microsoft.DataTransfer.ClientLibrary,'
I am trying to post file using rest connector, but it is not allowing to add additional headers for Content-Type - as it is by default applying content-type as application/json, what if, i need to use different content-type. I believe it should be…
asked 2025-03-08T18:31:17.22+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(236.8, 76%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECS%3C/text%3E%3C/svg%3E)
Chourasia, Sanoj K
0
Reputation points
edited a comment 2025-03-12T09:51:23.0266667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
phemanth
14,560
Reputation points Microsoft External Staff
0 answers
Error while Unzipping .tar.gz files using Copy activity - Failing for csv file having 8GB of data
Hi, I am trying to uncompress .tar.gz files using Copy Activity and it was working fine till yesterday. Today we have received a file with more data and it got failed while copying one file with 8GB of data with following error…
asked 2025-03-11T06:51:23.57+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(64, 21%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENA%3C/text%3E%3C/svg%3E)
Nagarajan Arumugam
0
Reputation points
commented 2025-03-12T09:15:55.23+00:00
Venkat Reddy Navari
80
Reputation points Microsoft External Staff
1 answer
Data Factory - Unable to read zip files from Amazon S3. Error: Central Directory corrupt. Unable to read beyond the end of the stream.
I am using Azure Data Factory's Copy activity to read csv containing in zip files in Amazon S3. I am experiencing problems since yesterday about reading only zip files from Amazon S3, other types of files were read all fine (or at least pure csv files).…
asked 2025-03-03T15:41:25.5666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(16, 0%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EQD%3C/text%3E%3C/svg%3E)
Quyen Dang
21
Reputation points
commented 2025-03-12T06:35:58.4066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(6.4, 57.00000000000001%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERG%3C/text%3E%3C/svg%3E)
Rakesh Govindula
95
Reputation points Microsoft External Staff