Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,911 questions
1 answer
Can anyone tell me why there are temp tables in lakehouse after successfull run of notebook activity.
i am creating temp tables using copy activity. then after successful run of copy activity, i am running notebook activity to process those temp table. if temp table exists then i rename them by remove underscore _temp and delete previous table. below is…
asked 2024-11-19T12:37:51.25+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(28.799999999999997, 15%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKB%3C/text%3E%3C/svg%3E)
Ketan Bharat | MAQ Software
40
Reputation points
commented 2024-11-22T01:33:41.92+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 65%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECB%3C/text%3E%3C/svg%3E)
Chandra Boorla
3,460
Reputation points Microsoft Vendor
1 answer
How to remove escape character '//' in the output file of the JSON sink dataset
Good day community, I'm seeking your help in removing the escape character \ from the output of the dataflow with JSON as the sink dataset. The objective is to produce a JSON output file with a row containing the value: ["all.be"] In the…
asked 2024-11-19T10:23:15.6066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(252.8, 86%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKN%3C/text%3E%3C/svg%3E)
Kien Nguyen Mai Tuan
0
Reputation points
commented 2024-11-22T01:31:18.85+00:00
Chandra Boorla
3,460
Reputation points Microsoft Vendor
1 answer
Wildecard in ADF copy activity
Hello , Is there a way in wildcard pattern in ADF copy activity to pick files from folders with certain naming convection. For example I have a following folder directory under bucket Myfolder **Main Folder** …
asked 2024-11-13T22:05:01.7466667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(131.20000000000002, 0%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESC%3C/text%3E%3C/svg%3E)
Shweta Chavan
40
Reputation points
commented 2024-11-22T01:28:58.93+00:00
Chandra Boorla
3,460
Reputation points Microsoft Vendor
0 answers
Unable to create HDInsight On Demand Cluster in Azure Data Factory
I am getting the issue to creating the On Demand HDInsights in Azure Data Factory. The status is stuck at "PreparingCluster". Here's my LinkedService configuration. OS type : Linux
asked 2024-11-13T03:52:15.9866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 19%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETC%3C/text%3E%3C/svg%3E)
Tan Check Leam
10
Reputation points
commented 2024-11-22T01:25:18.93+00:00
Chandra Boorla
3,460
Reputation points Microsoft Vendor
0 answers
How do I dynamically read multiple file names?
I am trying to read name of about 20+ csv files, and ingest data dynamically. Below is diagram of how I ingest data from each csv file individually. So, I have at least 20+ pipelines for each csv files. Below is how each csv files' name look like: If I…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 39%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKJ%3C/text%3E%3C/svg%3E)
1 answer
How to get return value from Oracle stored procedure?
Hi, I'm trying to use a Script activity to call a Stored Procedure which exists in my on-prem Oracle database. I used script parameters to pass the inputs which works fine. But when I add my output parameter to the script activity, it is returning a…
asked 2024-11-21T12:58:14.3333333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(179.20000000000002, 66%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBN%3C/text%3E%3C/svg%3E)
Bansal, Nimish
40
Reputation points
answered 2024-11-21T23:35:51.6566667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(246.4, 4%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Sina Salam
12,816
Reputation points
1 answer
Azure Data Factory Linked Service Connection Timeout Error with MongoDB
Experiencing an error while trying to connect to MongoDB from a linked service in Azure Data Factory. Connection to MongoDB server is timeout. A timeout occurred after 30000ms selecting a server using CompositeServerSelector{ Selectors =…
asked 2024-11-19T03:53:52.4433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(310.4, 42%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHR%3C/text%3E%3C/svg%3E)
Hari Ramamurthy
0
Reputation points
commented 2024-11-21T19:55:18.58+00:00
Hari Ramamurthy
0
Reputation points
1 answer
Best way to read from MongoDB change streams from within Azure
Hello! We have an application that needs events from our MongoDB change stream and insert the change event into Data Lake storage. We cannot use MongoDB Triggers to call an Azure Function because we have no way of authenticating the Trigger (we have been…
asked 2024-11-21T18:06:15.1766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(176, 23%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJB%3C/text%3E%3C/svg%3E)
Jaime Becker
0
Reputation points
answered 2024-11-21T18:59:32.81+00:00
hossein jalilian
8,230
Reputation points
1 answer
Extraction and Loading of OData to Postgresql issue with ADF pipeline with SOOM
our requirement to perform Extraction and Loading of OData to Postgresql using ADF pipeline is having issues related to System Out Of Memory exception randomly and sometimes with the max buffer issues. For the same pipeline run which worked and completed…
asked 2024-11-20T17:41:10.1633333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(291.2, 71%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETV%3C/text%3E%3C/svg%3E)
Tunikuntla venkata naga, Aswanth [JJCUS NON-J&J]
0
Reputation points
commented 2024-11-21T18:50:42.6733333+00:00
Tunikuntla venkata naga, Aswanth [JJCUS NON-J&J]
0
Reputation points
0 answers
We are looking for a spark structured based streaming to consume data from activeMQ
We are looking for a spark structured based streaming to consume data from activeMQ. However, we dont see any default connector or support as such available for the same. Can you please suggest, is there any roadmap in place to support this ?
asked 2024-11-21T15:47:57.1766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(121.6, 83%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPC%3C/text%3E%3C/svg%3E)
Prashant Chouhan
0
Reputation points
commented 2024-11-21T18:50:02.8666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(118.4, 98%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
KarishmaTiwari-MSFT
20,212
Reputation points Microsoft Employee
1 answer
Unable to copy payload from snowflake sql rest api to Azure blob
Hi , I am using web activity to get oauth token and passing subscription key and token to get method source dataset in a copy activity. postman is giving correct result but getting below error:- "Code": 22755, "Message":…
asked 2024-11-10T23:09:03.8133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(272, 5%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ%3C/text%3E%3C/svg%3E)
John
190
Reputation points
commented 2024-11-21T17:49:56.32+00:00
Chandra Boorla
3,460
Reputation points Microsoft Vendor
1 answer
How to use Dynamic add content in Compression type in copy activity
Hi , I created the dynamic dataset and linker service. In copy activity i used that dynamic dataset and want to pass parameter in copy activity or use Compression type option as parameter/Dynamic when i click on Compression type there is drop down menu…
asked 2024-11-20T20:55:07.4966667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(140.8, 99%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJB%3C/text%3E%3C/svg%3E)
Junaid Bilal
0
Reputation points
commented 2024-11-21T17:39:25.7733333+00:00
AnnuKumari-MSFT
33,556
Reputation points Microsoft Employee
2 answers
Auto update on IR onpremise hosting
I'm using the self-hosted IR for on-premise servers. I found that there is an auto update setting in IR, but the comments is weird. Can anyone tell me if it is right for auto updating?
asked 2024-02-06T01:33:31.1733333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(316.8, 48%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EEK%3C/text%3E%3C/svg%3E)
EUNSIL kim
50
Reputation points
commented 2024-11-21T16:38:36.49+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(96, 41%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETM%3C/text%3E%3C/svg%3E)
Toby MacAlister
0
Reputation points
0 answers
Is there a way to deploy Azure Data Factory pipeline using DevOps CI/CD without using ARM template?
I want to deploy Azure Data Factory pipelines using DevOps release pipeline without using ARM template.
asked 2024-11-21T15:00:21.11+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(227.2, 55.00000000000001%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKK%3C/text%3E%3C/svg%3E)
Kun Kuang
0
Reputation points
commented 2024-11-21T16:06:22.32+00:00
Nandan Hegde
32,651
Reputation points MVP
1 answer
ADF for OnPrem Target
What's is recommended ETL approach from OnPrem Source to OnPrem target/sink, inside Enterprise network, using ADF, without data movement to azure/cloud.
asked 2024-11-19T13:42:15.5633333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 72%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESD%3C/text%3E%3C/svg%3E)
Swapan Das
0
Reputation points
commented 2024-11-21T15:47:44.5366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
17,520
Reputation points Microsoft Vendor
2 answers
In ADF cannot skip columns in Copy activity Upsert
I have generic pipeline to upsert data in target synapse tables. Column mapping is also dynamic(using json). I want to skip few column in mapping but I am getting error of column count mismatch.
asked 2022-11-02T05:04:44.133+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(163.2, 39%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EYN%3C/text%3E%3C/svg%3E)
Yogesh Nikam
1
Reputation point
commented 2024-11-21T13:39:26.95+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(48, 85%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EYN%3C/text%3E%3C/svg%3E)
YogeshL Nikam
0
Reputation points
0 answers
I am getting Negsignal.SIGKILL Airflow
I have multiple integration running but for one integration when I extract zip file size of zip is 2-3 gbs, extract and upload each file to our storage.It shows Negsignal.SIGKILL. It was running fine a month ago but now its this error. The file is same…
asked 2024-11-21T11:16:44.95+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(32, 73%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECL%3C/text%3E%3C/svg%3E)
Chirag Lalwani
0
Reputation points
asked 2024-11-21T11:16:44.95+00:00
Chirag Lalwani
0
Reputation points
2 answers
Copy activity in ADF errors out while using dynamic range partitioning
Hey, I've been trying to copy some data using the Copy Activity from SQL Server 16 to Snowflake through a self hosted IR that has direct Line of Sight to the Server. I'm trying to enable dynamic range partitioning on one of the column so that the…
asked 2024-11-19T21:44:46.29+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(294.40000000000003, 81%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPR%3C/text%3E%3C/svg%3E)
Putatunda, Rishav
0
Reputation points
commented 2024-11-21T10:37:21.3666667+00:00
Putatunda, Rishav
0
Reputation points
1 answer
How to parametrize Email ID while sending notifications using ADF
Hi, We are sending email notifications while executing the ADF pipelines for success and failures. At multiple activities, we are using these Emails. How to parametrize Email ID while sending notifications using ADF? Thanks.
asked 2024-11-19T11:53:58.6433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(156.8, 77%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELN%3C/text%3E%3C/svg%3E)
Lakshmi Narayana Sarma Bhamidipati
110
Reputation points
commented 2024-11-21T09:53:10.35+00:00
Smaran Thoomu
17,520
Reputation points Microsoft Vendor
1 answer
audit table
Hi, how to set a up a audit log table in in adf pipeline.... i know single copy activity can be logged using sp for failure and success but how about multiple copy activity or any other activity failed have tried below for single copy…
asked 2024-11-20T15:28:49.49+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 13%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVS%3C/text%3E%3C/svg%3E)
Vineet S
1,230
Reputation points
answered 2024-11-21T08:57:16.6966667+00:00
Nandan Hegde
32,651
Reputation points MVP