Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,271 questions
2 answers
How to copy data from CSV file that has one column with line breaks
My source CSV file has two columns that have huge chunks of texts with lots of line breaks. When I use Copy activity to read this file in ADF, it breaks the rows whenever it sees an line break instead of actually parsing the complete row. …
asked 2022-02-22T15:10:41.64+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(208, 42%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMA%3C/text%3E%3C/svg%3E)
Mohsen Alam
11
Reputation points
commented 2025-02-21T09:17:02.29+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(297.6, 43%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETS%3C/text%3E%3C/svg%3E)
Tim S
0
Reputation points
0 answers
Removed trigger still triggers
Hi! I had a trigger in Azure Data Factory with a related pipeline. The trigger has been deleted (and the change published) but it is still triggering over 12 hours later. The trigger is not in the GUI, not connected to the pipeline and not in Git.…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 73%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ%3C/text%3E%3C/svg%3E)
3 answers
Pipline Failure
Failure happened on 'Source' side. ErrorCode=SqlOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=A database operation failed with the following error: 'Conversion failed when converting date and/or time from…
asked 2025-02-18T06:32:39.91+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(147.20000000000002, 65%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EEC%3C/text%3E%3C/svg%3E)
Ext.Chandan Kumar
0
Reputation points
edited a comment 2025-02-21T06:28:28.85+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 65%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECB%3C/text%3E%3C/svg%3E)
Chandra Boorla
8,795
Reputation points Microsoft Vendor
1 answer
error with ssis package in azure Failed get parameter info of parameter sql password
There is an SSIS package running on a schedule in Azure. The SQL database is also in Azure. The SSIS package moves data from SQL to D365. I am getting this error when the package starts to run: error with ssis package in azure Failed get parameter…
asked 2025-02-19T17:33:28.59+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(304, 52%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAL%3C/text%3E%3C/svg%3E)
Al Levy
0
Reputation points
commented 2025-02-21T06:11:05.8733333+00:00
Chandra Boorla
8,795
Reputation points Microsoft Vendor
1 answer
One of the answers was accepted by the question author.
About copying in ADF/Synapse sucessfuly
Hi I'm copying CSV file (on Prem) to a Parquet file on ADLS2. Just copying one single file. If the Copy activity ends sucessfuly (Succeded state in the activity) ... is that a garanty that the file was copied well?? Regards
asked 2025-02-20T20:19:24.54+00:00
Jona
615
Reputation points
commented 2025-02-21T02:20:42.47+00:00
Chandra Boorla
8,795
Reputation points Microsoft Vendor
0 answers
Azure Devops error invalid parameters
Hi, Invalid parameters error in ADF azure devops armtemplate deployment what can be issue in armtemplate { "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#", "contentVersion":…
asked 2025-02-19T19:12:18.9666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 13%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVS%3C/text%3E%3C/svg%3E)
Vineet S
1,370
Reputation points
commented 2025-02-21T01:40:15.8733333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(252.8, 99%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJN%3C/text%3E%3C/svg%3E)
J N S S Kasyap
80
Reputation points Microsoft Vendor
0 answers
copy data gives error NullReferenceException: Object reference not set to an instance of an object
I'm copying from Azure Blob Storage .csv files into Snowflake, and receiving the error ErrorCode=SessionLogFailToWriteLog,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Write log is failed with error message:…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(163.2, 36%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EM%3C/text%3E%3C/svg%3E)
1 answer
toDecimal function in dataflow of ADF/Azure Synapse Analytics not working correctly
I have input string field with name COST with below values. COST 2753000 217400 500 252567552 I am using toDecimal function in derived Column transformation in dataflow as below. z_COST= toDecimal(trim(COST)) Output is as…
asked 2025-02-20T15:15:57.9966667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 23%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPS%3C/text%3E%3C/svg%3E)
Peruka, Sikander reddy
110
Reputation points
commented 2025-02-20T21:36:38.93+00:00
Peruka, Sikander reddy
110
Reputation points
1 answer
Why does ADF copy activity to Oracle RDBMS configured with concurrent connections and degree of parallelism always fail?
Configuring ADF copy activity to read from Azure Blob Storage and concurrently write to Oracle database (to speed up the copy activity) fails always with ORA-00054 "Resource busy and acquire with no wait error.....". DBA says ADF had X active…
asked 2025-02-18T16:49:47.3166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(256, 76%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAD%3C/text%3E%3C/svg%3E)
Ajay Devulapalli
0
Reputation points
commented 2025-02-20T21:03:34.43+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 18%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGG%3C/text%3E%3C/svg%3E)
Ganesh Gurram
4,335
Reputation points Microsoft Vendor
1 answer
Sudden slowness on Azure Datafactory pipelines while fetching data from oracle database
Hello, We have been running ADF pipelines to ingest data from Oracle database. Suddenly it started taking 7-8 hours to finish whereas earlier it was 2-3 hours. No changes on Network end. Self hosted integration runtime was upgraded on 18th Jan-2025 where…
asked 2025-02-18T09:52:26.3766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(275.2, 49%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHD%3C/text%3E%3C/svg%3E)
Hiranya Deka
0
Reputation points
commented 2025-02-20T20:59:01.36+00:00
Ganesh Gurram
4,335
Reputation points Microsoft Vendor
0 answers
Azure Data Factory Copy Activity fails to auto create table when copying from REST source to Azure SQL database
I am using a Copy Activity in Azure Data Factory. I have a REST source/data set created and a Azure SQL Database sink/data set created. I have the Copy Activity set to auto create and the table does not exist in the database before running the pipeline…
asked 2025-02-20T15:54:23.29+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(252.8, 99%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETE%3C/text%3E%3C/svg%3E)
Tina Eilert(ARM)
0
Reputation points
commented 2025-02-20T20:53:01.03+00:00
Ganesh Gurram
4,335
Reputation points Microsoft Vendor
0 answers
Azure Data factory: Linked service parameterization default values and CI/CD pipelines
I am struggling to understand how ADF parameterization functions when deploying from one environment to another when using DevOps pipelines. Our linked services may be set up with parameters such as "name":…
asked 2025-02-18T15:07:37.1933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(166.4, 84%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELA%3C/text%3E%3C/svg%3E)
Lockyer, Aaron (SGRE COG B MS SYS BSPM)
0
Reputation points
commented 2025-02-20T19:57:46.7333333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
phemanth
13,900
Reputation points Microsoft Vendor
0 answers
Azure Data Factory Web activity calling child pipeline across ADF instance
I have child pipeline in ADF Instance 1 which invoke rest api POST method through web activity Parent pipeline is in ADF instance 2 which call above child pipeline also through web activity. I am able to successfully call child pipeline from parent…
asked 2025-02-19T15:19:04.2833333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(92.8, 35%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPJ%3C/text%3E%3C/svg%3E)
Pankaj Joshi
371
Reputation points
commented 2025-02-20T19:51:46.9766667+00:00
phemanth
13,900
Reputation points Microsoft Vendor
0 answers
delta file Vaccuum Immediately dataflow
Hi , I am using Azure data factory Data Flow source is parquet file sink is Delta file . I wanted run the vacuum immediately keeping no history . do we have any option for that please . Sink is Delta File in the Dataflow
asked 2025-02-19T17:04:39.2466667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(304, 54%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVR%3C/text%3E%3C/svg%3E)
Vignesh Rajendran
1
Reputation point
commented 2025-02-20T19:37:40.3666667+00:00
phemanth
13,900
Reputation points Microsoft Vendor
0 answers
Based on wildcard (part-*.csv) I want to pick multiple part files and insert into SQL Server
I have a blob path where spark is writing multiple part files . Now I want to load all these part file data to SQL Server . I'm trying using wildcard character *.csv in ADF but it's not working. I tried using listing the files via GetMetaData but still…
asked 2025-02-20T11:17:48.35+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 93%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDS%3C/text%3E%3C/svg%3E)
Deepak Singh
0
Reputation points
commented 2025-02-20T19:33:35.12+00:00
phemanth
13,900
Reputation points Microsoft Vendor
1 answer
Error DF-EXPR-010 in ADF data flow, Column 'body' used in expression is unavailable or invalid
I need to take the results of a REST API query and transform them to upsert them into a table. The source returns this JSON: The Projection step ALWAYS puts the content in context of a body field. I can't figure out how to feed in the expected schema…
asked 2025-02-20T16:00:50.67+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(230.39999999999998, 28.000000000000004%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAM%3C/text%3E%3C/svg%3E)
Altobelli, Marie K
0
Reputation points
edited a comment 2025-02-20T19:08:30.7266667+00:00
phemanth
13,900
Reputation points Microsoft Vendor
0 answers
XML to JSON conversion in ADF missing out 'lang' tag details
I am converting an xml from html source to a json file using ADF. However, for one of the attributes, the 'lang' element is not getting translated. The xml data in file is as follows: ''' <displayname xml:lang="deAT">…
asked 2025-02-20T08:47:03.38+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 26%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAT%3C/text%3E%3C/svg%3E)
Alefiya Tikiwala
0
Reputation points
commented 2025-02-20T17:25:50.98+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
20,305
Reputation points Microsoft Vendor
1 answer
Running MongoDB Aggregation Operation in Azure Data Factory.
Hi, I want to query my MongoDB Atlas (Not Cosmos DB) to fetch data and process it in Azure Data Factory. I have successfully created a connection to MongoDB and can run filter operations. I want to perform aggregation operations like $lookup with the…
asked 2025-02-18T05:05:42.0433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 68%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
Anish Sapkota
5
Reputation points
commented 2025-02-20T15:33:31.26+00:00
J N S S Kasyap
80
Reputation points Microsoft Vendor
1 answer
NAT Gateway for Managed VNet IR
How can I make Azure Managed VNet IR to make requests through a static public I.P? I want to pull data from 'xyz.com/books' api. This API provider doesn't allow requests from everyone. They only allow requests from IPs which are whitelisted. I have…
asked 2025-02-10T22:46:31.83+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(54.400000000000006, 49%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
Amrale, Siddhesh
0
Reputation points
edited an answer 2025-02-20T15:33:18.6333333+00:00
AnnuKumari-MSFT
34,016
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
ADF Copy Activity for JSON filesfails to Locate Blob Despite Successful Metadata Retrieval
I’m currently working on an Azure Data Factory (ADF) pipeline that consolidates files from multiple source containers into a single target container. The goal is to gather one file type (lod2) from various resource-based subfolders and versioned folders…
asked 2025-02-17T23:18:32.66+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(153.6, 10%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDG%3C/text%3E%3C/svg%3E)
Denktas, Gökalp Yigit
20
Reputation points
commented 2025-02-20T13:36:42.29+00:00
Denktas, Gökalp Yigit
20
Reputation points