11,044 questions with Azure Data Factory tags

Sort by: Updated
1 answer

Moving Data From Oracle SQL to Azure SQL Dynamically

Hi All, @{activity('GetTableColumns').output.value[0].ColumnNames} - This line reading my columns from the lookup activity successfully when I try move data from MS SQL(Source) to Azure SQL Databases(Destination). But it's reading the columns from the…

Azure SQL Database
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,044 questions
asked 2024-12-19T19:02:46.3766667+00:00
Ramesh, Harish 0 Reputation points
answered 2024-12-22T16:13:24.1133333+00:00
Sina Salam 14,161 Reputation points
2 answers One of the answers was accepted by the question author.

ADLS Gen2 operation failed for: Operation returned an invalid status code 'Conflict'. Account: 'abc'. ErrorCode: 'EndpointUnsupportedAccountFeatures'. Message: 'This endpoint does not support BlobStorageEvents or SoftDelete.

I am new to ADF. while creating linked service for data lake storage Gen2 I am getting a error ADLS Gen2 operation failed. ADLS Gen2 operation failed for: Operation returned an invalid status code 'Conflict'. Account: 'abc'. ErrorCode:…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,510 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,044 questions
Azure App Service
Azure App Service
Azure App Service is a service used to create and deploy scalable, mission-critical web apps.
8,084 questions
asked 2024-12-21T17:12:26.5366667+00:00
manoj reddy 0 Reputation points
commented 2024-12-22T14:50:01.7933333+00:00
manoj reddy 0 Reputation points
1 answer

Servicenow linked service used connectivity vendor

I am interested in using Servicenow Linked Service to extract data from Servicenow. There are two linked services available; the legacy version and version 2 that uses table API. The documentation mentions a built-in driver for enabling connectivity. Can…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,089 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,044 questions
asked 2024-06-27T10:50:37.5233333+00:00
Amir Bagheripour 0 Reputation points
edited the question 2024-12-22T13:29:20.1733333+00:00
TP 100.9K Reputation points
0 answers

How to create a link service in Azure Data Factory for Rest API which has OAuth grant type as password type of authentication

We need to call Rest API which has OAuth grant type as password type of authentication in a loop to store response in CSV. I know we could use Copy Activity but we are unable to create link service with mentioned authentication. We tried to do simple…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,044 questions
asked 2024-12-20T12:27:29.0933333+00:00
Kukade, Atharva 0 Reputation points
commented 2024-12-20T19:22:29.9166667+00:00
phemanth 12,565 Reputation points Microsoft Vendor
1 answer

Not Able to create connection for Intranet URL to Fabric Data factory

I am working on a project where the requirement is to get content from an Intranet URL through Fabric Data Factory. I have used web Activity and tried to create the connection but getting below error- " request didn't reach the server from the…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,044 questions
Microsoft Fabric Training
Microsoft Fabric Training
Microsoft Fabric: A Microsoft unified data platform.Training: Instruction to develop new skills.
38 questions
asked 2024-12-20T12:59:01.1933333+00:00
Pragya Awasthi 0 Reputation points
answered 2024-12-20T16:51:28.8733333+00:00
Ganesh Gurram 2,380 Reputation points Microsoft Vendor
0 answers

User configuration issue: Failure happened on 'Source' side.

Hi I am trying to copy Azure SQL database Prod Environment to Azure SQL database Sandbox Environment through ADF pipeline and getting below error during execution of copy data activity inside Foreach loop. ADF Pipeline is failing during Copy data…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,044 questions
asked 2024-12-17T17:28:35.7733333+00:00
Kandan-Azure 0 Reputation points
commented 2024-12-20T16:03:19.62+00:00
Ganesh Gurram 2,380 Reputation points Microsoft Vendor
1 answer One of the answers was accepted by the question author.

not being able to create azure free account or even to upgrade to pay as u go

how am i suppose to learn the certification if am not being able to create account. i am really frustrated that am not being able to create account. provide me an efficient way to create account to practice for my certification

Azure Functions
Azure Functions
An Azure service that provides an event-driven serverless compute platform.
5,248 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,089 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,283 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,044 questions
asked 2024-12-18T20:10:24.3266667+00:00
ashfa 25 Reputation points
edited the question 2024-12-20T13:40:32.9233333+00:00
simo-k 1,270 Reputation points
1 answer One of the answers was accepted by the question author.

Azure Data Factory – ForEach overwriting records instead of inserting

I have followed this guide to create a for each loop retrieving the data I need and it is working, however I would like all my JSON records in a single file, but the copy task is overwriting the previously imported record and I cannot see how to make…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,044 questions
asked 2022-12-05T07:55:37.23+00:00
Mia Lancaster 21 Reputation points
commented 2024-12-20T12:21:36.5066667+00:00
Kukade, Atharva 0 Reputation points
2 answers

Delay in Azure data factory pipeline. Why it is taking so long like 15 minutes to complete execution?

Azure data factory pipeline is calling several sub pipelines depending on the use case of the request. It is observed that when using Script activity it is taking minutes to execute and slowing down the pipeline performance. For other activities it is…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,044 questions
asked 2024-12-15T11:06:03.1033333+00:00
Chatti, Niharika [E] 0 Reputation points
commented 2024-12-20T10:03:39.1466667+00:00
Langlais, Jerome [E] 0 Reputation points
1 answer One of the answers was accepted by the question author.

Product tag for the cost with wrong location

Hi support, I have been looking into our cost for the different resources and I see something strange for our Data factory as seen below. Our subscription is located in EU west but the Product tag mention US East. Is this an error or are we paying to…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,044 questions
asked 2024-12-17T07:57:03.5133333+00:00
Andre Drejer 20 Reputation points
accepted 2024-12-20T08:35:27.8666667+00:00
Andre Drejer 20 Reputation points
1 answer One of the answers was accepted by the question author.

Connect Azure Data factory to Dynamic 365 and extract data best practice help

How to connect Dynamic365 application with Azure Data Factory and extract data on daily basis incremental data loading machanism

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,044 questions
asked 2024-03-03T22:27:20.5266667+00:00
MG 45 Reputation points
commented 2024-12-20T07:42:16.2066667+00:00
Kai Hartmann 0 Reputation points
2 answers

Azure Data Factory - delete row in a csv

I am having some issues with the ADF/Data Flow/Pipeline to delete one row using alterRow. I am filtering by ID=1 and deleting ID=1 row. How to include rest of the rows in my final output. I have only 10 rows. Even if there is a x mark on the row that…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,044 questions
asked 2024-12-14T08:35:49.2766667+00:00
Gunasekar-Desaiyan 30 Reputation points
commented 2024-12-20T07:06:20.4833333+00:00
AnnuKumari-MSFT 33,976 Reputation points Microsoft Employee
1 answer

ADF failing CopyData

I have a copy data action that reads from kusto db with a query and has a cosmos db sink. I have validated that the cosmos and kusto are successfully connected, yet while debugging the pipeline I run into this error: Could this be rbac related? I have…

Azure Cosmos DB
Azure Cosmos DB
An Azure NoSQL database service for app development.
1,710 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,044 questions
asked 2024-12-14T00:47:11.2866667+00:00
Saad Khan 1 Reputation point Microsoft Employee
commented 2024-12-20T07:05:10.79+00:00
AnnuKumari-MSFT 33,976 Reputation points Microsoft Employee
1 answer

Azure data factory copy activity DBNull error oracle to Iceberg conversion self-hosted integration runtime

Hi, When using a copy activity in a foreach to iterate through a list of table in an on-premise Oracle database and output into Iceberg tables in Azure Blob Gen2, only 26 (or 28 on last run) tables out of 619 are successful in copying. Each failure gives…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,044 questions
asked 2024-12-11T15:21:20.22+00:00
MarkDG 0 Reputation points
commented 2024-12-20T07:00:25.4666667+00:00
AnnuKumari-MSFT 33,976 Reputation points Microsoft Employee
2 answers

Creating sql database table from csv schema file using Azure data factory

Hi, i have requirement, we will get schema files in csv(see below highlighted), there were around 100s of files. here is sample schema csv…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,044 questions
asked 2024-12-11T00:58:02.5066667+00:00
Anil 20 Reputation points
commented 2024-12-20T06:59:10.75+00:00
AnnuKumari-MSFT 33,976 Reputation points Microsoft Employee
2 answers

Import schema csv file into sqlserver

Hi, i have scenario , where i will get schema files in csv, there were around 100s of files. here is sample schema csv…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,044 questions
asked 2024-12-10T19:14:17.3833333+00:00
Anil 20 Reputation points
commented 2024-12-20T06:51:21.1166667+00:00
AnnuKumari-MSFT 33,976 Reputation points Microsoft Employee
2 answers

How to use Dynamic add content in Compression type in copy activity

Hi ,  I created the dynamic dataset and linker service. In copy activity i used that dynamic dataset and want to pass parameter in copy activity or use Compression type option as parameter/Dynamic when i click on Compression type there is drop down menu…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,089 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,044 questions
asked 2024-11-20T20:55:07.4966667+00:00
Junaid Bilal 0 Reputation points
commented 2024-12-20T06:47:40.2333333+00:00
AnnuKumari-MSFT 33,976 Reputation points Microsoft Employee
0 answers

Select transformation output schema is empty after renaming columns

I've followed the example from the link: https://learn.microsoft.com/en-us/azure/data-factory/tutorial-data-flow-dynamic-columns It's exactly what I need for my project, but unfortunately, no matter what I do, the output schema of the select…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,089 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,044 questions
asked 2024-11-20T16:26:53.2266667+00:00
AdnanT 0 Reputation points
commented 2024-12-20T06:44:27.0466667+00:00
AnnuKumari-MSFT 33,976 Reputation points Microsoft Employee
1 answer

User could debug the copy data activity in pipeline successfully but sometimes fail

Problem Description: Recently, when users run the Copy Data Activity in Azure Data Factory, the debug operations are successful, but sometimes they fail. There have been no changes to the pipeline configuration. Both the source and sink used by the users…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,044 questions
asked 2024-12-20T01:56:41.6+00:00
Yujin Zhu (Shanghai Wicresoft Co Ltd) 0 Reputation points Microsoft Vendor
edited an answer 2024-12-20T03:08:12.2233333+00:00
Chandra Boorla 5,755 Reputation points Microsoft Vendor
2 answers One of the answers was accepted by the question author.

Role required to read/write data from cosmos DB (SQL API) from ADF?

We are trying to read/write data from Azure data factory, since local authentication is disabled in cosmos we are trying to access cosmos DB using managed identity. However even with Cosmos DB Account Contributor role assigned to managed identity of…

Azure Cosmos DB
Azure Cosmos DB
An Azure NoSQL database service for app development.
1,710 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,044 questions
Microsoft Entra ID
Microsoft Entra ID
A Microsoft Entra identity service that provides identity management and access control capabilities. Replaces Azure Active Directory.
22,620 questions
asked 2022-04-06T15:47:26.763+00:00
Ashutosh Saini 36 Reputation points
commented 2024-12-20T01:44:53.0133333+00:00
Thuy Nguyen 0 Reputation points Microsoft Employee