11,266 questions with Azure Data Factory tags

Sort by: Updated
1 answer

ADF Copy Activity for JSON filesfails to Locate Blob Despite Successful Metadata Retrieval

I’m currently working on an Azure Data Factory (ADF) pipeline that consolidates files from multiple source containers into a single target container. The goal is to gather one file type (lod2) from various resource-based subfolders and versioned folders…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,266 questions
asked 2025-02-17T23:18:32.66+00:00
Denktas, Gökalp Yigit 0 Reputation points
commented 2025-02-20T01:30:04.7233333+00:00
J N S S Kasyap 0 Reputation points Microsoft Vendor
1 answer

How to Pass JSON Data from Azure Blob to On-Prem SQL Server Stored Procedure and Capture Error Messages?

There is a requirement, where there is an Stored Procedure in On Prem SQL Server which return a table with errors/successfully processed records. The SP is inserting the records provided to it via table type parameter. I am having the data in Azure blob…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,266 questions
asked 2025-02-16T08:12:44.86+00:00
Rishikesh S Darwade 0 Reputation points
commented 2025-02-20T01:22:18.7766667+00:00
J N S S Kasyap 0 Reputation points Microsoft Vendor
1 answer

SHIR not able to register to adf via private link

Hi Team, I have an adf with no public access and one private link with private dns configured, I have a vm within the same subnet as the private link of adf Also from the VM I am able to nslookup the private dns of the azure data factory But when i…

Azure Private Link
Azure Private Link
An Azure service that provides private connectivity from a virtual network to Azure platform as a service, customer-owned, or Microsoft partner services.
536 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,266 questions
asked 2023-02-10T07:29:32.49+00:00
Pankaj Dhami 130 Reputation points
commented 2025-02-20T01:09:28.14+00:00
Ben Roberts 0 Reputation points
2 answers

Unable to perform - Exercise: Different types of KQL queries

Hello Support, Recently I have enrolled myself for the Analyze monitoring data with Kusto Query Language Learning Path, where on the first module I have been experiencing an issue with openin the query getting error: Selected user account does not…

Azure Monitor
Azure Monitor
An Azure service that is used to collect, analyze, and act on telemetry data from Azure and on-premises environments.
3,478 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,266 questions
Azure Training
Azure Training
Azure: A cloud computing platform and infrastructure for building, deploying and managing applications and services through a worldwide network of Microsoft-managed datacenters.Training: Instruction to develop new skills.
2,073 questions
asked 2025-02-15T02:25:08.7833333+00:00
Niko 0 Reputation points
commented 2025-02-20T00:52:36.6566667+00:00
Rahul Podila 1,825 Reputation points Microsoft Vendor
2 answers

Logic app to retrieve the latest file from a blob folder

How can I create a logic app that retrieves the latest file from a blob folder when a Http request is received, where there are multiple files, and sends it as an attachment? Are there any specific steps or configurations required for this process?

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,541 questions
Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
3,090 questions
Azure Logic Apps
Azure Logic Apps
An Azure service that automates the access and use of data across clouds without writing code.
3,354 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,193 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,266 questions
asked 2024-04-01T14:00:18+00:00
Aditya Singh 160 Reputation points
commented 2025-02-19T23:07:49.69+00:00
Michael Brunak 0 Reputation points Microsoft Employee
0 answers

Azure Data Factory - Copy data from AWS Redshift to Azure Synapse - Failed to connect error

Hi, I am trying to copy a table of data from AWS Redshift into Azure Synapse, below is the JSON code from the data factory: { "name": "pipeline1", "properties": { "activities": [ { …

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,193 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,266 questions
asked 2025-02-18T22:50:02.35+00:00
Vinny Prajapati 0 Reputation points
commented 2025-02-19T19:52:33.1+00:00
phemanth 13,810 Reputation points Microsoft Vendor
0 answers

delta file Vaccuum Immediately dataflow

Hi , I am using Azure data factory Data Flow source is parquet file sink is Delta file . I wanted run the vacuum immediately keeping no history . do we have any option for that please . Sink is Delta File in the Dataflow

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,266 questions
asked 2025-02-19T17:04:39.2466667+00:00
Vignesh Rajendran 1 Reputation point
commented 2025-02-19T19:20:29.1733333+00:00
phemanth 13,810 Reputation points Microsoft Vendor
0 answers

Azure Devops error invalid parameters

Hi, Invalid parameters error in ADF azure devops armtemplate deployment what can be issue in armtemplate { "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#", "contentVersion":…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,266 questions
asked 2025-02-19T19:12:18.9666667+00:00
Vineet S 1,370 Reputation points
2 answers One of the answers was accepted by the question author.

What is the maximum throughput speed achievable on the Azure Data factory

Hi All, I am performing the data migration from oracle (on prem) to Azure sql database using the Self hosted integration runtime. The throughput speed -2 MB/s on the Azure data factory. 135 GB takes approx. -10 hr. What is the maximum achievable…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,266 questions
asked 2023-07-04T13:56:47.3266667+00:00
Rahul 251 Reputation points
edited a comment 2025-02-19T18:58:35.7966667+00:00
Lu Zhang 0 Reputation points
1 answer

NAT Gateway for Managed VNet IR

How can I make Azure Managed VNet IR to make requests through a static public I.P? I want to pull data from 'xyz.com/books' api. This API provider doesn't allow requests from everyone. They only allow requests from IPs which are whitelisted. I have…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,266 questions
Azure NAT Gateway
Azure NAT Gateway
NAT Gateway is a fully managed service that securely routes internet traffic from a private virtual network with enterprise-grade performance and low latency.
50 questions
asked 2025-02-10T22:46:31.83+00:00
Amrale, Siddhesh 0 Reputation points
commented 2025-02-19T18:51:00.9366667+00:00
AnnuKumari-MSFT 34,011 Reputation points Microsoft Employee
2 answers

Azure Data Factory pipeline to extract data from Snowflake and load it into Azure Blob Storage

Hi Team, Azure Data Factory pipeline to extract data from Snowflake and load it into Azure Blob Storage. However, I’m facing some challenges with the data copy process and could use some assistance. I have several Tables in my snowflake schema. Need…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,266 questions
asked 2025-02-19T16:44:10.35+00:00
Nalini Bhavaraju 20 Reputation points
answered 2025-02-19T18:06:08.06+00:00
Nalini Bhavaraju 20 Reputation points
0 answers

Azure Data Factory Web activity calling child pipeline across ADF instance

I have child pipeline in ADF Instance 1 which invoke rest api POST method through web activity Parent pipeline is in ADF instance 2 which call above child pipeline also through web activity. I am able to successfully call child pipeline from parent…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,266 questions
asked 2025-02-19T15:19:04.2833333+00:00
Pankaj Joshi 371 Reputation points
commented 2025-02-19T17:38:35.5933333+00:00
Pankaj Joshi 371 Reputation points
0 answers

error with ssis package in azure Failed get parameter info of parameter sql password

There is an SSIS package running on a schedule in Azure. The SQL database is also in Azure. The SSIS package moves data from SQL to D365. I am getting this error when the package starts to run: error with ssis package in azure Failed get parameter…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,266 questions
asked 2025-02-19T17:33:28.59+00:00
Al Levy 0 Reputation points
2 answers One of the answers was accepted by the question author.

Need help to copy Data from Snowflake to Azure Blob Storage via ADF Pipeline

I want to create an Azure Data Factory pipeline to perform a bulk data copy from Snowflake tables to Azure Blob Storage. Since I have large tables to handle, I aim to organize the data into folders named after each table. Inside each table’s folder, the…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,266 questions
asked 2025-02-14T17:26:39.2166667+00:00
Nalini Bhavaraju 20 Reputation points
answered 2025-02-19T17:06:56.1766667+00:00
Nalini Bhavaraju 20 Reputation points
1 answer

Why does ADF copy activity to Oracle RDBMS configured with concurrent connections and degree of parallelism always fail?

Configuring ADF copy activity to read from Azure Blob Storage and concurrently write to Oracle database (to speed up the copy activity) fails always with ORA-00054 "Resource busy and acquire with no wait error.....". DBA says ADF had X active…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,266 questions
asked 2025-02-18T16:49:47.3166667+00:00
Ajay Devulapalli 0 Reputation points
commented 2025-02-19T17:06:08.8466667+00:00
Ganesh Gurram 4,170 Reputation points Microsoft Vendor
2 answers One of the answers was accepted by the question author.

check for pipe characters in a field and use if condition on this field

In the dataflow, I am checking for data in a field that could have Pipe (|) characters or alphanumeric characters with Pipe(|) between them. If there is only pipe (|)characters, then use iif condition and mark it as Y . If there are alphanumeric…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,193 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,266 questions
asked 2025-02-18T17:14:18.44+00:00
Peruka, Sikander reddy 90 Reputation points
accepted 2025-02-19T16:47:58.78+00:00
Peruka, Sikander reddy 90 Reputation points
1 answer

Within Azure Data Factory Studio, I get a Failed Connection to my SQL Database

Below is the error I keep getting, I've read all documentation and tried all steps to ensure identification is all there. I'm stuck and have a client deliverable. I would happily pay extra to get over this hurdle as I need to use the data warehouse…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,266 questions
Microsoft Entra ID
Microsoft Entra ID
A Microsoft Entra identity service that provides identity management and access control capabilities. Replaces Azure Active Directory.
23,296 questions
asked 2025-02-14T00:58:25.4166667+00:00
Derek Lilley 20 Reputation points
commented 2025-02-19T16:00:43.1733333+00:00
Chandra Boorla 8,470 Reputation points Microsoft Vendor
0 answers

Removed trigger still triggers

Hi! I had a trigger in Azure Data Factory with a related pipeline. The trigger has been deleted (and the change published) but it is still triggering over 12 hours later. The trigger is not in the GUI, not connected to the pipeline and not in Git.…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,266 questions
asked 2025-02-19T09:54:08.86+00:00
JE 0 Reputation points
commented 2025-02-19T15:59:46.3033333+00:00
Smaran Thoomu 20,295 Reputation points Microsoft Vendor
1 answer

Running MongoDB Aggregation Operation in Azure Data Factory.

Hi, I want to query my MongoDB Atlas (Not Cosmos DB) to fetch data and process it in Azure Data Factory. I have successfully created a connection to MongoDB and can run filter operations. I want to perform aggregation operations like $lookup with the…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,266 questions
asked 2025-02-18T05:05:42.0433333+00:00
Anish Sapkota 5 Reputation points
commented 2025-02-19T15:14:38.93+00:00
J N S S Kasyap 0 Reputation points Microsoft Vendor
2 answers One of the answers was accepted by the question author.

Azure data factory calling restapi hosted on app service

I am calling restapi POST method hosted on app service from azure data factory(web activity) using VNet integration runtime, I am using authentication type as NONE. but it is giving error as Error "403 Forbidden 'the web app you have attempted to…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,266 questions
asked 2025-02-08T05:15:33.36+00:00
Pankaj Joshi 371 Reputation points
accepted 2025-02-19T15:11:24.02+00:00
Pankaj Joshi 371 Reputation points