Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,884 questions
0 answers
How to remove escape character '//' in the output file of the JSON sink dataset
Good day community, I'm seeking your help in removing the escape character \ from the output of the dataflow with JSON as the sink dataset. The objective is to produce a JSON output file with a row containing the value: ["all.be"] In the…
asked 2024-11-19T10:23:15.6066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(252.8, 86%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKN%3C/text%3E%3C/svg%3E)
Kien Nguyen Mai Tuan
0
Reputation points
asked 2024-11-19T10:23:15.6066667+00:00
Kien Nguyen Mai Tuan
0
Reputation points
0 answers
Clarification on ACID Compliance of "Truncate Table" Action in Snowflake Sink for ADF Data
Dear Microsoft Support Team, I am working with Azure Data Factory (ADF) Data Flows and utilizing the Snowflake sink for data loading. In the Settings tab, there is an option to select "Truncate table" under the Table action dropdown. I have…
asked 2024-11-19T09:47:38.3066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(236.8, 74%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKS%3C/text%3E%3C/svg%3E)
Kudras, Stefan
20
Reputation points
asked 2024-11-19T09:47:38.3066667+00:00
Kudras, Stefan
20
Reputation points
2 answers
Help me on DP900
I was preparing for DP 900 in the official website of Microsoft Account. I came up with such question and have confusion in this. " Q1. Q2. Therefore, Which data service allows you to use every feature of Microsoft SQL Server? Please…
asked 2024-11-18T16:09:19.9533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(233.6, 17%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESM%3C/text%3E%3C/svg%3E)
SUBHAJIT MUKHERJEE
0
Reputation points
commented 2024-11-19T08:11:01.92+00:00
Nandan Hegde
32,421
Reputation points MVP
0 answers
Unable to create HDInsight On Demand Cluster in Azure Data Factory
I am getting the issue to creating the On Demand HDInsights in Azure Data Factory. The status is stuck at "PreparingCluster". Here's my LinkedService configuration. OS type : Linux
asked 2024-11-13T03:52:15.9866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 19%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETC%3C/text%3E%3C/svg%3E)
Tan Check Leam
10
Reputation points
commented 2024-11-19T06:35:39.65+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 65%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECB%3C/text%3E%3C/svg%3E)
Chandra Boorla
3,380
Reputation points Microsoft Vendor
0 answers
Azure Data Factory Linked Service Connection Timeout Error with MongoDB
Experiencing an error while trying to connect to MongoDB from a linked service in Azure Data Factory. Connection to MongoDB server is timeout. A timeout occurred after 30000ms selecting a server using CompositeServerSelector{ Selectors =…
asked 2024-11-19T03:53:52.4433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(310.4, 42%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHR%3C/text%3E%3C/svg%3E)
Hari Ramamurthy
0
Reputation points
asked 2024-11-19T03:53:52.4433333+00:00
Hari Ramamurthy
0
Reputation points
1 answer
nested loop in adf
Hi, what happened in ADF engineer when nested loop is applied which are the cases its required apart getmetadata and count of files and all
asked 2024-11-18T21:28:29.8133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 13%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVS%3C/text%3E%3C/svg%3E)
Vineet S
1,090
Reputation points
answered 2024-11-19T03:43:17.8166667+00:00
Nandan Hegde
32,421
Reputation points MVP
1 answer
How to fix UserErrorPermissionDeniedOnCloudIR
Hi, I'm trying to copy files from my sharepoint online document library into azure blob storage, but I am getting this error: ErrorCode=UserErrorPermissionDeniedOnCloudIR,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Access…
asked 2024-11-17T11:35:12.6033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 30%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGY%3C/text%3E%3C/svg%3E)
Gal Yaniv
65
Reputation points
commented 2024-11-19T01:15:00.76+00:00
Chandra Boorla
3,380
Reputation points Microsoft Vendor
1 answer
Publishing to event hub via rest API is slow from ADF web activity
Publishing to message via rest api event hub using managed identity from dedicated integration runtime. Event Hub is premium with auto scale and there is enough TU. Messages are being sent one at a time. Each call takes 3 to 15 seconds to publish…
asked 2024-11-16T04:06:04.1233333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(252.8, 47%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHZ%3C/text%3E%3C/svg%3E)
Himanshu Zinzuwadia
45
Reputation points
commented 2024-11-19T01:09:38.8566667+00:00
Chandra Boorla
3,380
Reputation points Microsoft Vendor
2 answers
How can I write to a Dataverse lookup using an alternate key?
Hi I want to write data to Dataverse via a Synapse Copy Data task in a pipeline. The destination tables all have alternate keys set up. So for example, I could have a Country entity, with CountryAlternateKey (e.g. ISO code) and Name, and a City with…
asked 2024-11-13T10:32:20.5433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(278.4, 86%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDP%3C/text%3E%3C/svg%3E)
Duncan Pryde
1
Reputation point
commented 2024-11-19T01:06:20.76+00:00
Chandra Boorla
3,380
Reputation points Microsoft Vendor
1 answer
One of the answers was accepted by the question author.
Redshift Connector in Azure Data Factory does not accept sslmode=verify-ca
From Azure Data Factory, I am trying to connect to a Redshift database as the source. I have created a Redshift Connector, but it fails to connect, with: ERROR [08S01] [Microsoft][Amazon Redshift] (10) Error occurred while trying to connect: [SQLState…
asked 2024-11-17T21:53:02.3866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(249.60000000000002, 69%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPM%3C/text%3E%3C/svg%3E)
Peter McNamara
20
Reputation points
accepted 2024-11-18T23:39:13.4533333+00:00
Peter McNamara
20
Reputation points
1 answer
How to copy a csv file from blob storage to snowflake table?
Trying to copy a csv file from blob storage to snowflake table. Been trying various ways but keep getting all sorts of errors. Getting various errors…
asked 2024-11-18T20:01:42.06+00:00
LIU, PETER
0
Reputation points
commented 2024-11-18T22:22:41.28+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 18%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGG%3C/text%3E%3C/svg%3E)
Ganesh Gurram
1,430
Reputation points Microsoft Vendor
2 answers
Uanble to read excel file from SFTP server in Azure ADF
Error details Error code 21359 Details Encrypted excel file 'SAMPLE.XLSX' is not supported, please remove its password. Wrong Local header signature: 0x41D Activity ID: 70462da3-07e6-4d98-8373-85bdbb0fe9bc
asked 2022-10-01T22:18:08.103+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(166.4, 40%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
siya1234
1
Reputation point
commented 2024-11-18T21:24:01.1833333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(304, 93%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ER8%3C/text%3E%3C/svg%3E)
Rajeshk-8659
0
Reputation points
1 answer
ErrorCode=EncryptedExcelIsNotSupported,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Encrypted excel file 'EYG Daily AG Knowledge Article Usage.xlsx' is not supported, please remove its password
We are extracting data from ServiceNow tool and loading into the database via the ADF pipeline and since 3rd Jun, this error has been coming and pipeline fails. If we copy the content of the said file into a new .xlsx file with same name and load it…
asked 2023-06-09T12:56:35.12+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 68%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJM%3C/text%3E%3C/svg%3E)
Jayadev M
0
Reputation points
commented 2024-11-18T21:19:24.2866667+00:00
Rajeshk-8659
0
Reputation points
1 answer
How to pass outputs to another activity from a Custom Activity in an Azure Data Factory pipeline?
Hello, I need to capture output values from a "Custom Activity" in an Azure Data Factory pipeline to be used, afterwards, by another activity. I have followed the guidelines in…
asked 2023-11-06T09:29:03.87+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 6%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECM%3C/text%3E%3C/svg%3E)
Calabria Montero, Salvador (SGRE SE D FP&DC WEF)
81
Reputation points
answered 2024-11-18T20:40:59.8733333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(112.00000000000001, 86%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
Ashoke Sati
0
Reputation points
1 answer
Wildecard in ADF copy activity
Hello , Is there a way in wildcard pattern in ADF copy activity to pick files from folders with certain naming convection. For example I have a following folder directory under bucket Myfolder **Main Folder** …
asked 2024-11-13T22:05:01.7466667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(131.20000000000002, 0%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESC%3C/text%3E%3C/svg%3E)
Shweta Chavan
40
Reputation points
commented 2024-11-18T19:20:16.45+00:00
Shweta Chavan
40
Reputation points
1 answer
ADF PII Detection and Masking - Schema Drift Issue
I am following the tutorial for setting up a pipeline for PII detection and masking (linked below). All steps have been followed exactly as described, without any changes to the pipeline settings or configurations. However, I am encountering an issue…
asked 2024-11-15T19:42:49.95+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(121.6, 7.000000000000001%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMA%3C/text%3E%3C/svg%3E)
Muhammad Abdulbaqi
20
Reputation points
commented 2024-11-18T17:33:42.6366667+00:00
Muhammad Abdulbaqi
20
Reputation points
1 answer
Unable to get IP address of nodes using the same Integration Runtime (IR) across multiple ADFs
Hello all, I have an issue with nodes using the same Integration Runtime (IR) across multiple ADFs. I am currently unable to fetch the IP of these nodes. Please see the image below: This does not happen with nodes that are being used for just 1…
asked 2024-11-18T13:43:52.24+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(96, 54%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EFN%3C/text%3E%3C/svg%3E)
Fabián Avilés
55
Reputation points
answered 2024-11-18T16:29:39.8166667+00:00
Amira Bedhiafi
26,656
Reputation points
1 answer
Evaluating a bit/boolean in ADF If Condition Activity
You would think this would be incredibly simple, but apparently not. A SQL statement used in a lookup returns either CAST(1 as bit) or CAST(0 as bit) to a pipeline as column is_enabled. An If condition activity (inside a For Each) uses this expression to…
asked 2024-11-13T20:35:55.0633333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(28.799999999999997, 19%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPD%3C/text%3E%3C/svg%3E)
Pendleton, David
20
Reputation points
answered 2024-11-18T16:08:30.5+00:00
Ganesh Gurram
1,430
Reputation points Microsoft Vendor
1 answer
One of the answers was accepted by the question author.
Azure Data Factory SnowflakeV2 linked service supports only Basic authentication in data flow, will this changed to key pair as well?
I am using Azure Data Factory and Snowflake for data processing. Currently I am migration the Snowflake connection from the Snowflake legacy linked service to the new SnowflakeV2 linked service because the new linked Service supports key pair…
asked 2024-06-26T08:40:22.59+00:00
Kudras, Stefan
20
Reputation points
commented 2024-11-18T15:18:26.6566667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 20%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
Marcin Molak
0
Reputation points
1 answer
One of the answers was accepted by the question author.
Azure-ADF pipeline failure while coping data to parquet file as Sink.
We are facing below error while copying data from RDBMS source to ADLS(Parquet File) as Sink. Error Message: ParquetJavaInvocationException,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=An error occurred when invoking java,…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 72%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)