Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,323 questions
1 answer
Failure happened on 'Sink' side. ErrorCode=UserErrorKeyConflict,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Key Content-Type conflicts in AuthHeaders/AdditionalHeaders,Source=Microsoft.DataTransfer.ClientLibrary,'
I am trying to post file using rest connector, but it is not allowing to add additional headers for Content-Type - as it is by default applying content-type as application/json, what if, i need to use different content-type. I believe it should be…
asked 2025-03-08T18:31:17.22+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(236.8, 76%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECS%3C/text%3E%3C/svg%3E)
Chourasia, Sanoj K
0
Reputation points
commented 2025-03-08T18:33:25.1633333+00:00
Chourasia, Sanoj K
0
Reputation points
1 answer
One of the answers was accepted by the question author.
Azure Data Factory and DataBricks end-to-end project
my goal is to copy data from ADLS using ADF(copy activity) and connecting that data to databricks to process and analyse the data and after completion I want save the processed data back to the storage account so, I have my dataset in my local system so…
asked 2025-03-06T12:19:52.7433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(28.799999999999997, 72%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPS%3C/text%3E%3C/svg%3E)
paul sudarshan
20
Reputation points
accepted 2025-03-08T09:12:08.8966667+00:00
paul sudarshan
20
Reputation points
0 answers
How to fetch the child items within a folder located on azure blob
I am trying to fetch child items of a folder for recursive deletion, so my expectation is to delete the folders within the folder, like if folder A has subfolder B with file b and another subfolder AB has subfolder D, then deleting all items from…
asked 2025-02-26T18:58:40.5766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(144, 0%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAC%3C/text%3E%3C/svg%3E)
Ajay Chowdary Kandula
0
Reputation points
commented 2025-03-08T00:21:25.14+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(252.8, 99%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJN%3C/text%3E%3C/svg%3E)
J N S S Kasyap
860
Reputation points Microsoft External Staff
1 answer
Data Factory - Unable to read zip files from Amazon S3. Error: Central Directory corrupt. Unable to read beyond the end of the stream.
I am using Azure Data Factory's Copy activity to read csv containing in zip files in Amazon S3. I am experiencing problems since yesterday about reading only zip files from Amazon S3, other types of files were read all fine (or at least pure csv files).…
asked 2025-03-03T15:41:25.5666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(16, 0%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EQD%3C/text%3E%3C/svg%3E)
Quyen Dang
21
Reputation points
commented 2025-03-07T22:42:26.9966667+00:00
J N S S Kasyap
860
Reputation points Microsoft External Staff
0 answers
How to Call a ORACLE DATABASE PACKAGE from Azure data factory and receive the output of that package.
Team I need to call a ORACLE DATABASE PACKAGE from azure data factory and use its output in other activities but i am not able to call a PACKAGE. Inside the package there are many stored procedures written so i need to call a specific stored procedure…
asked 2025-03-05T11:24:06.85+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(256, 83%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
ADFUser
1
Reputation point
commented 2025-03-07T22:02:15.2833333+00:00
J N S S Kasyap
860
Reputation points Microsoft External Staff
1 answer
How to fix "Record not found.." error while trying to insert new row into Dataverse entity using ADF
I have a data factory pipeline that contains a Copy activity with Source as a SQL Server table and Sink as Dataverse entity. The pipeline reads new or existing rows from the table and Upserts into dataverse. However for a new row that does not exists in…
asked 2025-03-07T19:59:03.8666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(294.40000000000003, 91%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMP%3C/text%3E%3C/svg%3E)
Mohsin Patel
0
Reputation points Microsoft Employee
answered 2025-03-07T21:28:23.8966667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 65%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECB%3C/text%3E%3C/svg%3E)
Chandra Boorla
9,685
Reputation points Microsoft External Staff
1 answer
Unable to Create Ondemand Hdinsight Hadoop Clusters in Data factory.
I am trying to create a linked service of HDInsight hadoop Clusters. With Cluster size 4, 1 hour as time to live, Provided service principal Id, Azure Key vault Secret Key, Azure Storage account and a SQL Server in Hcatalog. But I am unable to move…
asked 2025-03-05T07:23:18.68+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(169.60000000000002, 46%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVA%3C/text%3E%3C/svg%3E)
Vishesh Agarwal
0
Reputation points
commented 2025-03-07T18:38:26.69+00:00
Chandra Boorla
9,685
Reputation points Microsoft External Staff
1 answer
How to upload the snapshot json into the storage account with Ev2 ARM template?
I'm trying to use the ARM template and EV2 to upload the snapshot json into the blob storage account. Is there any existing template or examples to support this action?
asked 2024-10-07T20:46:50.2366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(176, 67%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EN%3C/text%3E%3C/svg%3E)
Nathon
0
Reputation points Microsoft Employee
commented 2025-03-07T18:15:32.05+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(304, 6%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKD%3C/text%3E%3C/svg%3E)
Keshavulu Dasari
4,110
Reputation points Microsoft External Staff
1 answer
How to copy data from external volume to Oracle
I am trying to copy the parquet files from external volume to Oracle db, but I am facing this issue : Failure happened on 'Sink' side. ErrorCode=UserErrorOdbcOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=ERROR…
asked 2025-03-06T08:55:43.4666667+00:00
Dheepa Sree
0
Reputation points
commented 2025-03-07T18:13:52.3733333+00:00
Chandra Boorla
9,685
Reputation points Microsoft External Staff
1 answer
How to dynamically switch database names in Mongo DB Linked Service?
How do i call DATABASES dynamically from a MONGO Database in Azure data factory. When trying to create a linked service for mongo DB it is asking to Hardcode the database name without making it a dynamic content. But we have more than 90 databases…
asked 2025-03-05T11:29:05.9566667+00:00
ADFUser
1
Reputation point
commented 2025-03-07T18:09:01.9566667+00:00
Chandra Boorla
9,685
Reputation points Microsoft External Staff
2 answers
One of the answers was accepted by the question author.
ADF Dataflow Derive Column transformation toTimestamp is working for constant timezone but not from column
I am finding Azure Data Factory Dataflow Dataflow Derive Column transformation toTimestamp is working for constant timezone but not from column. I want each column to output the date with time adjusted based on the timezone UTC offset. Any suggestions…
asked 2025-03-06T20:47:30.6366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(35.2, 1%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDB%3C/text%3E%3C/svg%3E)
David Buterbaugh
21
Reputation points
accepted 2025-03-07T18:05:56.8633333+00:00
David Buterbaugh
21
Reputation points
1 answer
Azure DataFactory Hubspot linked service
Dear, We use Azure DataFactory for moving Hubspot data, using the built in Hubspot LinkedService in Azure Data factory. Hubspot announced that on 24th of March 2025 will retire Owners API v2. Link to Hubspot:…
asked 2025-03-05T13:59:45.84+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(44.800000000000004, 66%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESN%3C/text%3E%3C/svg%3E)
Slave Naumoski
0
Reputation points
commented 2025-03-07T16:24:31.04+00:00
J N S S Kasyap
860
Reputation points Microsoft External Staff
1 answer
Azure Data Factory ForEach/If - Force Next Iteration
Our organization receives and loads files into an on-prem SQL Server using SSIS packages written in Visual Studio. As part of the file loads, we run a series of validations on the files to insure that certain things are correct - file name, is the file…
asked 2025-03-06T16:40:32.72+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(92.8, 68%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECL%3C/text%3E%3C/svg%3E)
Corey Livermore
20
Reputation points
commented 2025-03-07T16:04:31.62+00:00
Nandan Hegde
34,351
Reputation points MVP
0 answers
ADF Google Ads Linked Service using Service Authentication as the Authentication Type ERROR pkcs8PrivateKey
Trying to access Google Ads data using a Google Ads Service Account, and the ADF Google Ads Linked Service. The linked service "Authentication type" set to "Service authentication". We generated a private key for this Service…
asked 2025-03-04T21:37:57.02+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 12%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMB%3C/text%3E%3C/svg%3E)
mamadou bhoye sow
5
Reputation points
commented 2025-03-07T14:59:49.7566667+00:00
mamadou bhoye sow
5
Reputation points
1 answer
Using Managed Virtual Network Integration Runtime in Data Flows
I have created a Managed Virtual Network Integrated Runtime (MVNet IR) and successfully connected it to Azure SQL. I can access the database through datasets using this runtime without any issues. However, when creating Data Flows using the same datasets…
asked 2025-03-07T12:20:00.07+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(22.400000000000002, 73%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPS%3C/text%3E%3C/svg%3E)
Paila, S (Sreenu)
0
Reputation points
commented 2025-03-07T14:36:44.46+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
21,160
Reputation points Microsoft External Staff
1 answer
Reduce Cluster startup time in ADF Dataflow
I have create a ADF pipeline which merge data between source and target in parallel fashion for 18 tables. The table name, database name etc are dynamic and assigned at runtime. The pipeline takes more than 7 min to load data and most time is taken by…
asked 2025-02-23T16:50:04.0466667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(76.8, 38%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPR%3C/text%3E%3C/svg%3E)
Patil Rajas
40
Reputation points
commented 2025-03-07T13:54:09.25+00:00
Chandra Boorla
9,685
Reputation points Microsoft External Staff
0 answers
Copy Data Activity with Upsert Operation to MongoDB Not Updating Records
I am trying to use Azure Data Factory's Copy Data Activity to copy data from a CSV file into a MongoDB collection using the upsert operation. However, I am facing an issue where the pipeline run indicates that 4 rows were written, but no records are…
asked 2025-03-06T20:28:21.6766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(48, 4%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERV%3C/text%3E%3C/svg%3E)
Ramakrishna Vaitla
0
Reputation points
commented 2025-03-07T11:16:23.39+00:00
Ramakrishna Vaitla
0
Reputation points
2 answers
User assigned managed identity with credentials ARM deployment of Data Factory using DevOps
I am trying to deploy Credentials associated with User assigned managed identity to higher environment via DevOps and ARM templates. I get the error: User assigned identity does not belong to the resource you are deploying to I want to parametrize…
asked 2023-05-12T07:07:36.9933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(153.6, 48%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
SS23456
5
Reputation points
answered 2025-03-07T10:59:05.1566667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(217.60000000000002, 21%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECA%3C/text%3E%3C/svg%3E)
Ciocan, Andrei-Cosmin
0
Reputation points
2 answers
One of the answers was accepted by the question author.
Migrating SSIS File validation logic to ADF
I work in an environment where inbound files are loaded into an on-prem SQL Server. We currently use SSIS to loop through these files, performing multiple validations to ensure they are correctly formatted. Our validation process follows a sequential…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 64%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EW%3C/text%3E%3C/svg%3E)
1 answer
Unable to use linked service to blob storage account with Test Connection as "To File Path"
I am trying to create a linked service to Azure blob storage account. I need to use System Assigned Managed Identity for Authentication. No restriction on account selection method. I tried using "From Azure Subscription" (which gets modified to…
asked 2025-02-25T14:40:14.99+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(256, 43%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAA%3C/text%3E%3C/svg%3E)
Aqshata Akshay Ajnadkar
5
Reputation points
commented 2025-03-07T08:24:24.7533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(44.800000000000004, 55.00000000000001%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
sr24
0
Reputation points