Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,319 questions
0 answers
Azure Data Factory ForEach/If - Force Next Iteration
Our organization receives and loads files into an on-prem SQL Server using SSIS packages written in Visual Studio. As part of the file loads, we run a series of validations on the files to insure that certain things are correct - file name, is the file…
asked 2025-03-06T16:40:32.72+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(92.8, 68%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECL%3C/text%3E%3C/svg%3E)
Corey Livermore
20
Reputation points
commented 2025-03-07T00:12:32.35+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(252.8, 99%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJN%3C/text%3E%3C/svg%3E)
J N S S Kasyap
775
Reputation points Microsoft External Staff
0 answers
How to Call a ORACLE DATABASE PACKAGE from Azure data factory and receive the output of that package.
Team I need to call a ORACLE DATABASE PACKAGE from azure data factory and use its output in other activities but i am not able to call a PACKAGE. Inside the package there are many stored procedures written so i need to call a specific stored procedure…
asked 2025-03-05T11:24:06.85+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(256, 83%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
ADFUser
1
Reputation point
commented 2025-03-06T23:49:40.7633333+00:00
J N S S Kasyap
775
Reputation points Microsoft External Staff
0 answers
How to fetch the child items within a folder located on azure blob
I am trying to fetch child items of a folder for recursive deletion, so my expectation is to delete the folders within the folder, like if folder A has subfolder B with file b and another subfolder AB has subfolder D, then deleting all items from…
asked 2025-02-26T18:58:40.5766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(144, 0%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAC%3C/text%3E%3C/svg%3E)
Ajay Chowdary Kandula
0
Reputation points
commented 2025-03-06T23:45:20.0366667+00:00
J N S S Kasyap
775
Reputation points Microsoft External Staff
1 answer
I have a case where we need to update some records into Oracke RDS in AWS using ADF
Hi MS Team, I am working on a project where we need to perform Update from Synapse into Oracle RDS using ADF. I have ODBC linked service which I have been using to perform inserts, but no option is available to Update in the ADF pipeline. Has anyone…
asked 2025-03-04T19:31:18.22+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(179.20000000000002, 48%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESK%3C/text%3E%3C/svg%3E)
Samir Kohli
0
Reputation points
commented 2025-03-06T23:05:30.29+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 65%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECB%3C/text%3E%3C/svg%3E)
Chandra Boorla
9,675
Reputation points Microsoft External Staff
0 answers
ADF Dataflow Derive Column transformation toTimestamp is working for constant timezone but not from column
I am finding Azure Data Factory Dataflow Dataflow Derive Column transformation toTimestamp is working for constant timezone but not from column. I want each column to output the date with time adjusted based on the timezone UTC offset. Any suggestions…
asked 2025-03-06T20:47:30.6366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(35.2, 1%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDB%3C/text%3E%3C/svg%3E)
David Buterbaugh
1
Reputation point
edited a comment 2025-03-06T22:56:08.3033333+00:00
Chandra Boorla
9,675
Reputation points Microsoft External Staff
3 answers
I don't get a redirection after typing in 'az login'
I think the problem is in the first line of the error message. The redirection link is login.microsoftonline.DE instead of login.microsoft.COM. Does anyone have an idea how to fix that? I'm glad for anything that could help! PS H:\> az login The…
asked 2023-02-15T08:48:42.23+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(278.4, 83%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETT%3C/text%3E%3C/svg%3E)
Tran, Thomas (ext)
0
Reputation points
answered 2025-03-06T21:01:52.2866667+00:00
Kevin Piatt
80
Reputation points
0 answers
ADF Google Ads Linked Service using Service Authentication as the Authentication Type ERROR pkcs8PrivateKey
Trying to access Google Ads data using a Google Ads Service Account, and the ADF Google Ads Linked Service. The linked service "Authentication type" set to "Service authentication". We generated a private key for this Service…
asked 2025-03-04T21:37:57.02+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 12%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMB%3C/text%3E%3C/svg%3E)
mamadou bhoye sow
5
Reputation points
commented 2025-03-06T20:47:33.51+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(169.60000000000002, 63%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETS%3C/text%3E%3C/svg%3E)
Terry Shelton
0
Reputation points
0 answers
Copy Data Activity with Upsert Operation to MongoDB Not Updating Records
I am trying to use Azure Data Factory's Copy Data Activity to copy data from a CSV file into a MongoDB collection using the upsert operation. However, I am facing an issue where the pipeline run indicates that 4 rows were written, but no records are…
asked 2025-03-06T20:28:21.6766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(48, 4%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERV%3C/text%3E%3C/svg%3E)
Ramakrishna Vaitla
0
Reputation points
edited the question 2025-03-06T20:30:44.4433333+00:00
Ramakrishna Vaitla
0
Reputation points
1 answer
Azure Data Factory and DataBricks end-to-end project
my goal is to copy data from ADLS using ADF(copy activity) and connecting that data to databricks to process and analyse the data and after completion I want save the processed data back to the storage account so, I have my dataset in my local system so…
asked 2025-03-06T12:19:52.7433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(28.799999999999997, 72%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPS%3C/text%3E%3C/svg%3E)
paul sudarshan
0
Reputation points
answered 2025-03-06T18:30:31.5633333+00:00
Chandra Boorla
9,675
Reputation points Microsoft External Staff
1 answer
How to dynamically switch database names in Mongo DB Linked Service?
How do i call DATABASES dynamically from a MONGO Database in Azure data factory. When trying to create a linked service for mongo DB it is asking to Hardcode the database name without making it a dynamic content. But we have more than 90 databases…
asked 2025-03-05T11:29:05.9566667+00:00
ADFUser
1
Reputation point
commented 2025-03-06T18:02:40.1533333+00:00
Chandra Boorla
9,675
Reputation points Microsoft External Staff
2 answers
One of the answers was accepted by the question author.
Azure Data Factory: I'm trying to understand how to get MetaData information from CSV Files used in a Data Flow to get only the latest file created?
I am using the Filter by last modified criteria in the Copy Data task with Azure Data Factory to determine the latest data file (one file created per day). The challenge I'm facing is that the file in Azure are calculated in UTC time but the files…
asked 2025-03-05T18:20:17.9666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(83.2, 82%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMS%3C/text%3E%3C/svg%3E)
Marty Scherr
0
Reputation points
accepted 2025-03-06T16:17:50.7533333+00:00
Marty Scherr
0
Reputation points
1 answer
Azure DataFactory Hubspot linked service
Dear, We use Azure DataFactory for moving Hubspot data, using the built in Hubspot LinkedService in Azure Data factory. Hubspot announced that on 24th of March 2025 will retire Owners API v2. Link to Hubspot:…
asked 2025-03-05T13:59:45.84+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(44.800000000000004, 66%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESN%3C/text%3E%3C/svg%3E)
Slave Naumoski
0
Reputation points
commented 2025-03-06T16:17:50.4966667+00:00
J N S S Kasyap
775
Reputation points Microsoft External Staff
1 answer
Unable to Create Ondemand Hdinsight Hadoop Clusters in Data factory.
I am trying to create a linked service of HDInsight hadoop Clusters. With Cluster size 4, 1 hour as time to live, Provided service principal Id, Azure Key vault Secret Key, Azure Storage account and a SQL Server in Hcatalog. But I am unable to move…
asked 2025-03-05T07:23:18.68+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(169.60000000000002, 46%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVA%3C/text%3E%3C/svg%3E)
Vishesh Agarwal
0
Reputation points
edited a comment 2025-03-06T14:56:43.16+00:00
Vishesh Agarwal
0
Reputation points
1 answer
One of the answers was accepted by the question author.
ADF Switch Expression Adding Unexpected Characters to FileName
The pipeline involves a file landing in an SFTP Blob container, which is decrypted by a .Net application. After the decryption and movement to a designated folder, a pipeline is triggered to load the file into a SQL table. A Get Metadata activity…
asked 2025-03-03T01:20:18.6566667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(310.4, 46%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESJ%3C/text%3E%3C/svg%3E)
Steve Jamrose
20
Reputation points
accepted 2025-03-06T14:55:54.6633333+00:00
Steve Jamrose
20
Reputation points
2 answers
Migrating SSIS File validation logic to ADF
I work in an environment where inbound files are loaded into an on-prem SQL Server. We currently use SSIS to loop through these files, performing multiple validations to ensure they are correctly formatted. Our validation process follows a sequential…
asked 2025-03-06T14:46:03.2933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 64%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EW%3C/text%3E%3C/svg%3E)
WeirdMan
260
Reputation points
answered 2025-03-06T14:55:52.0966667+00:00
Nandan Hegde
34,346
Reputation points MVP
1 answer
One of the answers was accepted by the question author.
Complex transformations, fault tolerance, and scalability
I have the following situation, when preparing to an interview, I want to get the opinion of professionals : You are working on a data integration project where you need to ingest data from multiple on-premises SQL Server databases into Azure Synapse…
1 answer
Data Factory - Unable to read zip files from Amazon S3. Error: Central Directory corrupt. Unable to read beyond the end of the stream.
I am using Azure Data Factory's Copy activity to read csv containing in zip files in Amazon S3. I am experiencing problems since yesterday about reading only zip files from Amazon S3, other types of files were read all fine (or at least pure csv files).…
asked 2025-03-03T15:41:25.5666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(16, 0%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EQD%3C/text%3E%3C/svg%3E)
Quyen Dang
21
Reputation points
commented 2025-03-06T12:52:13.2333333+00:00
Quyen Dang
21
Reputation points
1 answer
How to send 15000 json data to api 1 by 1 in adf pipeline without using data flow and lookup activity?
I can not use lookup activity as it loops only for 5000 records at a time, so what are the possible ways we can send 15000 JSON data to web API, without using Dataflow and lookup activity?
asked 2025-03-04T07:27:48.8733333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(54.400000000000006, 77%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGP%3C/text%3E%3C/svg%3E)
gurudas puthal
0
Reputation points
commented 2025-03-06T12:24:40.4666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
20,935
Reputation points Microsoft External Staff
1 answer
Reduce Cluster startup time in ADF Dataflow
I have create a ADF pipeline which merge data between source and target in parallel fashion for 18 tables. The table name, database name etc are dynamic and assigned at runtime. The pipeline takes more than 7 min to load data and most time is taken by…
asked 2025-02-23T16:50:04.0466667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(76.8, 38%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPR%3C/text%3E%3C/svg%3E)
Patil Rajas
40
Reputation points
commented 2025-03-06T12:09:29.91+00:00
Chandra Boorla
9,675
Reputation points Microsoft External Staff
1 answer
How to copy data from external volume to Oracle
I am trying to copy the parquet files from external volume to Oracle db, but I am facing this issue : Failure happened on 'Sink' side. ErrorCode=UserErrorOdbcOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=ERROR…
asked 2025-03-06T08:55:43.4666667+00:00
Dheepa Sree
0
Reputation points
answered 2025-03-06T11:50:13.8033333+00:00
Amira Bedhiafi
29,636
Reputation points