11,066 questions with Azure Data Factory tags

Sort by: Updated
1 answer

I am not able to signup for azure portal because it says my mobile number is not verified

Hi team, Since last 2 days I am trying to signup for the Microsoft azure portal but every time I am getting this error that my mobile number is not getting verified, I even tried with other mobile number and it had the same error please help

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,066 questions
asked 2025-01-04T11:23:39.7033333+00:00
Naman 0 Reputation points
answered 2025-01-04T19:57:05.7933333+00:00
Andreas Baumgarten 114.9K Reputation points MVP
0 answers

Data Factory Copy Activity - SYNC - Stored Proc - Has anyone used generic parameter to use same proc on multiple tables?

I am trying to use the copy activity and in the sync side, use the stored procedure option. It requires two parameters. One of them is a custom data type for the table. Has anyone figured out a way to use a generic parameter so that this same copy could…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,066 questions
asked 2025-01-03T16:33:55.7+00:00
Blasko, Dan W 20 Reputation points
commented 2025-01-03T20:31:16.27+00:00
Blasko, Dan W 20 Reputation points
0 answers

ADF - FullandIncremental load with selections, sometimes working but not all the time

Hello ADF Forum, We are using ADF to get the data from SAP ECC extractors (CDC) to snowflake, we have built the data flow with conditions and without conditions. ADF run mode -> "Full and Incremental", when I choose with selections…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,066 questions
asked 2024-12-30T21:07:37+00:00
Naveen Nunna 0 Reputation points
commented 2025-01-03T18:17:12.2066667+00:00
Ganesh Gurram 2,550 Reputation points Microsoft Vendor
1 answer

access to my organization

I can not access to my organization on Azure DevOps portal. It was CICD pipeline for ADF. this error "401 - Uh-oh, you do not have access. Your account, Mahdi@nematshahimahdigmail.onmicrosoft.com is not authorized to view this page. Make sure the…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,066 questions
asked 2025-01-02T17:10:50.4966667+00:00
Mahdi Nematshahi 0 Reputation points
commented 2025-01-03T16:05:37.18+00:00
Ganesh Gurram 2,550 Reputation points Microsoft Vendor
0 answers

ADF Pipeline created using Python fails its first run but succeeds later

Hi, I am using python script to create pipelines in ADF. The issue I am facing is the pipeline gets created but fails its first run with schema not matching or column mapping issues. But Once i click on some of the components but do not make any…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,066 questions
asked 2025-01-02T07:08:31.6333333+00:00
vijayshankar 0 Reputation points
commented 2025-01-03T10:19:47.6+00:00
Smaran Thoomu 18,800 Reputation points Microsoft Vendor
1 answer

Azure Data Factory pipeline fails randomly with error: "Specified argument was out of the range of valid values"

Hey, Currently I'm struggling with ADF pipeline that fails pretty randomly. It loads multiple tables from sql database to data lake in parquet format. There are 3 retries for copy data activity, sometimes it succeeds on one of the retries, sometimes…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,066 questions
asked 2025-01-02T08:13:34.18+00:00
Sylwester Jedrys 0 Reputation points
commented 2025-01-03T08:07:11.0966667+00:00
Sylwester Jedrys 0 Reputation points
1 answer

I am getting the below error when I try to insert into a azure data lake table. "Type=System.IO.FileNotFoundException,Message=Could not load file or assembly 'System.Text.Json"

I am getting the below error when I try to insert into a azure data lake table from adf pipeline . "The type initializer for 'Azure.Core.Serialization.JsonObjectSerializer' threw an…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,516 questions
Azure Functions
Azure Functions
An Azure service that provides an event-driven serverless compute platform.
5,271 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,066 questions
asked 2024-12-27T08:56:47.04+00:00
Beebi Thasleem M 20 Reputation points
commented 2025-01-03T07:24:46.0566667+00:00
AnnuKumari-MSFT 33,981 Reputation points Microsoft Employee
1 answer

We want to loop all folder and file recursively of BLOB Storage for given container using python in azure data factory custom activity and write data in in blob in .CSV file

Hi All, We want to loop all folder and file recursively of BLOB Storage for given container using python in azure data factory custom activity and write data in in blob in .CSV file. also when we connect blob storage using ADF custom activity using SAS…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,516 questions
Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
3,008 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,066 questions
asked 2025-01-02T07:49:43.5366667+00:00
manish verma 481 Reputation points
commented 2025-01-03T06:30:47.54+00:00
AnnuKumari-MSFT 33,981 Reputation points Microsoft Employee
0 answers

Ensuring Private Connectivity for Cross-Cloud Data Transfers Using STS and Azure Data Factory

Hello, I am working on a project where Google Cloud and Azure are connected via a private connection (Cloud Interconnect/ExpressRoute). I need to ensure that data transfers between Google Cloud Storage and Azure Storage happen over this private…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,066 questions
asked 2024-12-25T21:40:27.41+00:00
Nikunj Patel 0 Reputation points
edited the question 2025-01-03T05:08:22.38+00:00
KapilAnanth-MSFT 48,091 Reputation points Microsoft Employee
1 answer

ADF Managed Virtual Network unexpectedly incurring cost in other region

Recently I have implemented denied public access, virtual networking and private endpoints for my SQL Servers, as well as Managed Virtual Networks in my ADF. Everything works as intended, but there was a quick spike in the cost for my ADF, which I…

Azure Cost Management
Azure Cost Management
A Microsoft offering that enables tracking of cloud usage and expenditures for Azure and other cloud providers.
2,858 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,066 questions
asked 2024-12-04T13:31:46.2433333+00:00
Thijs 0 Reputation points
edited the question 2025-01-03T05:03:34.3466667+00:00
KapilAnanth-MSFT 48,091 Reputation points Microsoft Employee
1 answer

Data flow failed due to transient error

The ADF data flow failed due to a transient error, and during the retry, some data was already loaded into the sink, leading to duplicate records in the insert scenario. Since I didn't implement CDC at the source level and there are no sink checkpoint…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,101 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,066 questions
asked 2024-12-31T02:11:17.29+00:00
Ananya patil 20 Reputation points
commented 2025-01-03T03:29:26.3866667+00:00
Ganesh Gurram 2,550 Reputation points Microsoft Vendor
1 answer One of the answers was accepted by the question author.

hi, how can i have the discounts for students for certifications?

I went to https://learn.microsoft.com/es-mx/credentials/certifications/student-discounts to get the discount for some certifications but I already completed the survey but I don't have the discount for the exams

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,066 questions
asked 2025-01-02T06:56:55.4333333+00:00
manuel rondon 20 Reputation points
accepted 2025-01-03T03:00:19.9266667+00:00
manuel rondon 20 Reputation points
1 answer One of the answers was accepted by the question author.

ADF Pipeline "Bad Request" Error when Debug run

Hi all, I have a pipeline in ADF that is incrementally loading objects from Salesforce whose LastModifiedDate is between the last date in my sink (Fabric Warehouse) and the current runtime. I want to do that in a single pipeline now with parameters. The…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,066 questions
asked 2025-01-01T01:13:31.68+00:00
Mathias Denzin 20 Reputation points
accepted 2025-01-03T02:46:12.7866667+00:00
Mathias Denzin 20 Reputation points
1 answer

Getting this error - "at Source 'source1': com.jcraft.jsch.JSchException: SSH_MSG_DISCONNECT: 11" when previewing data on azure data factory dataflow

I have a dataset that is a csv file existing on azure sftp. I am able to connect to sftp from linked service and able to preview data on the dataset. I have a dataflow where I want to select a few columns from this csv and copy them onto another…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,066 questions
asked 2025-01-02T04:47:03.9966667+00:00
Beebi Thasleem M 20 Reputation points
answered 2025-01-03T02:40:54.8333333+00:00
Nandan Hegde 33,081 Reputation points MVP
1 answer One of the answers was accepted by the question author.

Add a public i.p. to data factory

I'd like to set up a public static i.p. to data factory, so my client could whitelist me in his firewall. I went through all of the documentation, created virtual network(s) & public i.p.(s). When I try to create a new Integration runtime, in the…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,066 questions
asked 2025-01-02T15:07:19.3233333+00:00
‪Shacham Zaguri‬‏ 25 Reputation points
commented 2025-01-02T17:24:18.95+00:00
‪Shacham Zaguri‬‏ 25 Reputation points
2 answers One of the answers was accepted by the question author.

DATA FACTORY COPY DATA TO REST API - Failure happened on 'Sink' side. ErrorCode=UserErrorPropertyInvalidOrMissing,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The property 'additionalHeaders' is invalid or missing.,Sour

DATA FACTORY COPY DATA TO REST API returns Failure happened on 'Sink' side. ErrorCode=UserErrorPropertyInvalidOrMissing,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The property 'additionalHeaders' is invalid or…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,066 questions
asked 2024-12-30T07:02:15.7366667+00:00
Kafka Hava 20 Reputation points
answered 2025-01-02T14:38:52.45+00:00
Kafka Hava 20 Reputation points
1 answer

Connect Azure Data Factory to existing VM and run the R script

Need to execute the R script in existing Azure virtual machine on ADF. is this feasible ?

Azure Virtual Machines
Azure Virtual Machines
An Azure service that is used to provision Windows and Linux virtual machines.
8,194 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,066 questions
asked 2022-08-30T14:11:05.92+00:00
Prakash14 121 Reputation points
commented 2025-01-02T12:35:05.0566667+00:00
Mohan Gangadhar 0 Reputation points
1 answer

Pipeline throws exception as InvalidParameter for the parameter which is not specified

collectionName is not configured in pipeline ErrorCode=InvalidParameter, Message=The value of the property 'collectionName' is invalid: 'The required property is not specified.\r\nParameter name: collectionName'

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,066 questions
asked 2021-09-06T09:15:00.087+00:00
Kamal Sharma 1 Reputation point Microsoft Employee
commented 2025-01-02T09:33:07.0533333+00:00
suganya mahadevan 0 Reputation points
1 answer One of the answers was accepted by the question author.

Unexpected Activation of Tumbling Trigger Outside Scheduled Time

A tumbling trigger was set to start at 12:00 AM on February 6, 2022, and end at 12:00 AM on September 11, 2022, with a recurrence of every 1440 minutes (once every 24 hours). However, it has activated unexpectedly without manual initiation. What could be…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,066 questions
asked 2024-12-27T08:22:06.7033333+00:00
Yujin Zhu (Shanghai Wicresoft Co Ltd) 60 Reputation points Microsoft Vendor
accepted 2025-01-02T06:40:36.9366667+00:00
Yujin Zhu (Shanghai Wicresoft Co Ltd) 60 Reputation points Microsoft Vendor
2 answers

Changing ADF Trigger changes Advanced Filters in Container Blob EventGrid

Hi, I am using SFTP in a Storage Container set up with an EventGrid that triggers an ADF pipeline when a new file is uploaded to the container. My event grid subscription is set to the filters values you can see in the first attached image. With the…

Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
3,008 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,066 questions
Azure Event Grid
Azure Event Grid
An Azure event routing service designed for high availability, consistent performance, and dynamic scale.
409 questions
asked 2024-12-04T15:07:23.4833333+00:00
Luis Chaves Rodriguez 0 Reputation points
commented 2024-12-31T18:27:26.4666667+00:00
phemanth 12,740 Reputation points Microsoft Vendor