11,192 questions with Azure Data Factory tags

Sort by: Updated
1 answer

how to connect azure cache redis to azure databricks through service principal

I need to connect azure redis to databricks using service principal . but I am getting this error Failed to connect to Redis: invalid username-password pair

Azure Cache for Redis
Azure Cache for Redis
An Azure service that provides access to a secure, dedicated Redis cache, managed by Microsoft.
270 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,323 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,192 questions
asked 2025-02-04T09:34:56.25+00:00
Nageswara.Podapati 0 Reputation points
commented 2025-02-04T20:37:07.6833333+00:00
Oury Ba-MSFT 20,106 Reputation points Microsoft Employee
0 answers

copy data gives error NullReferenceException: Object reference not set to an instance of an object

I'm copying from Azure Blob Storage .csv files into Snowflake, and receiving the error ErrorCode=SessionLogFailToWriteLog,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Write log is failed with error message:…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,192 questions
asked 2025-02-03T23:51:04.0566667+00:00
MarkDG 0 Reputation points
edited a comment 2025-02-04T19:58:29.9733333+00:00
MarkDG 0 Reputation points
1 answer

Migrating ADFS to Entra ID question

Dear All, i'm new in Azure, i have a hybird environment with ADFS, i was planning to migrate ADFS to Entra ID, and i have already downloaded entra health connect agent and install it on ADFS and domain controller, having waited for 1 day i go…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,192 questions
Microsoft Entra ID
Microsoft Entra ID
A Microsoft Entra identity service that provides identity management and access control capabilities. Replaces Azure Active Directory.
23,099 questions
asked 2025-01-25T17:53:10.1933333+00:00
keith li 51 Reputation points
commented 2025-02-04T16:43:05.3566667+00:00
Venkata Jagadeep 80 Reputation points Microsoft Vendor
1 answer One of the answers was accepted by the question author.

Azure Data Factory Pipeline

Hi there, This pipeline runs Azure SQL SPs. And wanted to check with you if my connections achieve the below. SP3,4,5,6 should run only after SP1 and 2 gets completed. Is there a container so its an AND operation? SP 7 should run only when SP3,4,5,6 have…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,192 questions
asked 2025-02-03T22:29:24.8233333+00:00
RJ 286 Reputation points
accepted 2025-02-04T15:32:20.6133333+00:00
RJ 286 Reputation points
0 answers

I'm using data factory and my explorer seems to be frozen. Behind the scenes I am able to create tables but when I request to drop a table it says that I have two columns with similar names and that I need to use SQL commands to rename these columns.

I have a table InstallBase which I have created successfully using a copy data task in Data Factory. When I go to Explorer I can't see the table. When I then do to drop the table using a drop table SQL command it drops the table but reports that I have…

Azure Data Explorer
Azure Data Explorer
An Azure data analytics service for real-time analysis on large volumes of data streaming from sources including applications, websites, and internet of things devices.
545 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,192 questions
asked 2025-02-04T11:57:09.39+00:00
Peter Carruthers 0 Reputation points
commented 2025-02-04T15:03:21.4266667+00:00
phemanth 13,620 Reputation points Microsoft Vendor
0 answers

Having trouble in copy data between cosmos mongodb

Operation on target ForEach_q2g failed: Activity failed because an inner activity failed; Inner activity name: Copy_q2g, Error: Failure happened on 'Sink' side.…

Azure Cosmos DB
Azure Cosmos DB
An Azure NoSQL database service for app development.
1,745 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,192 questions
asked 2024-01-29T14:48:52.7+00:00
Rajagopal Venkataramani 0 Reputation points
edited a comment 2025-02-04T14:29:32.1533333+00:00
Mamidi Vasavi 0 Reputation points
1 answer

Regarding not being able to deploy the project in Azure which is connected to Github

Hello, I have a Django project which I uploaded to GitHub first and then tried to connect to Azure. The build process has been successful as per the Action tab in GitHub.But the process gets to the Deploy part, it takes around 30 minutes and then provide…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,192 questions
Azure App Service
Azure App Service
Azure App Service is a service used to create and deploy scalable, mission-critical web apps.
8,243 questions
asked 2025-01-23T07:53:31.06+00:00
Sakar Dahal 0 Reputation points
edited the question 2025-02-04T13:08:45.9+00:00
Mounika Reddy Anumandla 2,070 Reputation points Microsoft Vendor
1 answer

accidentally deleted during a deployment to my Azure App Service, Unfortunately, I do not have a backup enabled for this app

I am reaching out to request your assistance in recovering files that were accidentally deleted during a deployment to my Azure App Service. Unfortunately, I do not have a backup enabled for this app, and the deployment process seems to have overwritten…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,192 questions
Microsoft Deployment Toolkit
Microsoft Deployment Toolkit
A collection of Microsoft tools and documentation for automating desktop and server deployment. Previously known as Microsoft Solution Accelerator for Business Desktop Deployment (BDD).
932 questions
Azure App Service
Azure App Service
Azure App Service is a service used to create and deploy scalable, mission-critical web apps.
8,243 questions
asked 2025-01-20T07:27:08.55+00:00
Vijan Bandarathilake 0 Reputation points
commented 2025-02-04T11:37:06.27+00:00
Harshitha Veeramalla 1 Reputation point Microsoft Vendor
0 answers

Error during batchpoint invoke: Access to this resource is denied. Please check your ACL rules on the resource

Hello, I am getting a "Access to this resource is denied. Please check your ACL rules on the resource." when attempting to invoke a batch endpoint. I have tried both with Azure CLI and Azure Data Factory to invoke a batch endpoint, with both…

Azure Machine Learning
Azure Machine Learning
An Azure machine learning service for building and deploying models.
3,097 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,192 questions
asked 2024-07-22T11:59:17.8133333+00:00
Khai-Tam Nguyen 20 Reputation points
commented 2025-02-04T11:05:56.1866667+00:00
Sajko Miha 0 Reputation points
2 answers

Unable to write rows into sink from SAP CDC Connector

I am working on extracting data from SAP BW and writing it to Azure Blob Storage using an ETL pipeline. Initially, I experimented with delta/incremental loads, but now I need to perform a full load. However, after switching back, I am unable to get any…

Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
3,054 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,192 questions
asked 2025-01-29T21:43:41.8166667+00:00
Dennis Tran 0 Reputation points
commented 2025-02-04T10:52:48.5233333+00:00
phemanth 13,620 Reputation points Microsoft Vendor
2 answers

Self Hosted Integration Studio Out of Memory errors but system has plenty of memory

Greetings - I am fairly new to supporting a portion of a Azure Data Factory. One of our servers that is running the Integration Runtime has a current version of runtime and plenty of CPU and memory available but it is throwing out this error a couple of…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,192 questions
asked 2025-02-01T23:06:10.99+00:00
mpls 125 Reputation points
commented 2025-02-04T09:16:11.6866667+00:00
Chandra Boorla 7,490 Reputation points Microsoft Vendor
0 answers

How to call AzureML Batch Model endpoint from Azure Data Factory (ADF Pipeline).

Uzing Azure ML I am able to create and deploy models to a batch endpoint and run them through the UI. When I try and execute them through the Machine Learning Execute Pipeline activity in ADF I get the error "Data set node xxx references parameter…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,192 questions
asked 2025-02-03T22:25:35.05+00:00
Kirby Jackson 0 Reputation points
commented 2025-02-04T06:25:37.13+00:00
Ganesh Gurram 3,690 Reputation points Microsoft Vendor
1 answer

How to make sure delete rows are also synced with Incremental column as watermark column

Hi Team We have setup sync using ADF CDC between 2 azure SQL database. We have one table "X" having a watermark column "UpdatedAt". We have created CDC for table X with incremental column "UpdatedAt" and Keys" as…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,192 questions
asked 2025-01-27T11:59:40.0966667+00:00
Tushar Kumar 0 Reputation points Microsoft Employee
commented 2025-02-04T05:52:12.1666667+00:00
Tushar Kumar 0 Reputation points Microsoft Employee
1 answer

Zoho Authentication: Successful in Fabric Data Factory, Fails in Azure Data Factory"

I am trying to access https://accounts.zoho.com/oauth/v2/token to get the token for my zoho endpoint.. When using Web activity on fabric it is a cakewalk and I am getting token but when I am trying same in azure data factory I am getting an error as…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,192 questions
asked 2025-01-31T08:55:06.53+00:00
Amit Mohanty 0 Reputation points
commented 2025-02-04T05:35:32.7933333+00:00
Ganesh Gurram 3,690 Reputation points Microsoft Vendor
1 answer

Azure Data Factory random disconnections and pipeline experienced a "ghost run" with no trigger debug/history present.

Hello, I'm experiencing a few issues with Azure Data Factory. Our team has Azure Data Factory integrated with GIT, but at random times it disconnects. It may show as connected for one team member but appear disconnected for another. This also…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,192 questions
asked 2025-01-30T01:50:23.41+00:00
Kandice 0 Reputation points Microsoft Employee
commented 2025-02-04T05:30:38.0333333+00:00
Ganesh Gurram 3,690 Reputation points Microsoft Vendor
2 answers

How to stop ForEach activity if previous itertion fails in ADF/Synapse pipeline

I am trying to pull multiple tables from a Azure SQL Database . When data is loaded to its destination I am calling an SP that will update the status of a table to Success. Using ForEach activity multiple tables are puling. But, While pulling one table…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,167 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,192 questions
asked 2022-06-13T19:20:26.183+00:00
heta desai 357 Reputation points
commented 2025-02-03T23:41:52.31+00:00
Sasha Whittaker (CSPS-EFPC) 0 Reputation points
1 answer

Lookup activity issue from Snowflake connector in Azure data factory

Hello Team, I am facing an issue from lookup activity, I have a pipeline which is having a snowflake connector in the lookup activity and trying to fetch the records from the table. but lookup activity not getting any output when we check in the…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,192 questions
asked 2025-01-29T07:03:33.5+00:00
Suresh Boya 0 Reputation points
commented 2025-02-03T19:38:24.5866667+00:00
Ajay Hirapara 0 Reputation points
0 answers

Data factory copy operation in immutable storage fails

I have data factory with one pipeline which do copy operation from one container to another within same storage account. I have two containers "daily" and "weekly" and my pipeline copy files from daily to weekly. When new files are…

Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
3,352 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,192 questions
asked 2025-01-29T13:02:41.8833333+00:00
Akshay Vithalkar 1 Reputation point
commented 2025-02-03T16:55:55.2333333+00:00
Syed Aaqid Ali 5 Reputation points Microsoft Vendor
1 answer

Need to extract zreports from SAP HANA

Hi, I have a use-case where I want to extract data from SAP Hana. The use-case is as follows: I have SAP Hana deployment from where I need to extract data. The data is stored in zreports which are extracted using T-Codes. Now, I want to extract the data…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,537 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,192 questions
asked 2025-01-30T05:24:40.5533333+00:00
Om Bhimani 0 Reputation points
commented 2025-02-03T13:07:16.4666667+00:00
Smaran Thoomu 19,630 Reputation points Microsoft Vendor
2 answers

Changing ADF Trigger changes Advanced Filters in Container Blob EventGrid

Hi, I am using SFTP in a Storage Container set up with an EventGrid that triggers an ADF pipeline when a new file is uploaded to the container. My event grid subscription is set to the filters values you can see in the first attached image. With the…

Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
3,054 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,192 questions
Azure Event Grid
Azure Event Grid
An Azure event routing service designed for high availability, consistent performance, and dynamic scale.
418 questions
asked 2024-12-04T15:07:23.4833333+00:00
Luis Chaves Rodriguez 10 Reputation points
commented 2025-02-03T12:45:23.5033333+00:00
phemanth 13,620 Reputation points Microsoft Vendor