Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,593 questions
1 answer
Salesforce Connector deprecation details
As part of Sales force legacy connector, can we get confirmation for below ask please: "You can continue to use existing Linked services and datasets of the legacy connector and make changes to them. You cannot create a new Linked service for the…
asked 2024-09-19T14:03:55.46+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(259.20000000000005, 89%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENK%3C/text%3E%3C/svg%3E)
Narendra Kandula
0
Reputation points
edited an answer 2024-09-19T22:13:09.03+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
15,040
Reputation points Microsoft Vendor
0 answers
Error on sink data to MySQL - communications link failure
I've got a Pipeline with some Data Flow activities that runs in parallel. I've got the error on last night's execution and I'd like some help to understand why and how to prevent it from happenning. Job failed due to reason: at Sink 'EnviaFerias':…
asked 2024-09-19T11:24:37.3633333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(291.2, 25%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJL%3C/text%3E%3C/svg%3E)
Joffre L. A. Mota (LG lugar de gente)
0
Reputation points
commented 2024-09-19T21:48:44.3966667+00:00
Smaran Thoomu
15,040
Reputation points Microsoft Vendor
2 answers
Pipeline run stucked on queue
I have some pipelines that ingest data from Oracle and write directly to Delta in a lakehouse on Fabric. I have other pipelines that ingest data from Oracle and write to a .parquet file in a lakehouse on Fabric. The pipelines that write directly to Delta…
asked 2024-09-16T14:11:07.89+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(208, 25%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGS%3C/text%3E%3C/svg%3E)
Gabriel Santos
0
Reputation points
commented 2024-09-19T20:55:01.3266667+00:00
Gabriel Santos
0
Reputation points
1 answer
One of the answers was accepted by the question author.
How to convert JSON-type columns from a CSV into JSON and apply a flatten operation
Hi all, A file is downloaded via a web service in CSV format, containing several columns. However, these columns are of JSON type. Example…
asked 2024-09-16T15:03:13.74+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 57.99999999999999%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDA%3C/text%3E%3C/svg%3E)
Diego Andres Vasquez Ruiz
20
Reputation points
accepted 2024-09-19T19:45:38.9833333+00:00
Diego Andres Vasquez Ruiz
20
Reputation points
0 answers
Failing to copy large dataset from Salesforce using new salesforce V2 connector in Azure data factory
I am trying to migrate the salesforce legacy linked service to new salesforce V2 linked service in Azure data factory. The test connection is successful and I am able to copy the smaller datasets using new salesforce V2 linked service. But when i am…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 2%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EL%3C/text%3E%3C/svg%3E)
1 answer
adf/synapse - how to copy all files based on given lastmodified date from source container and save in sink container in respective folder based on extension
Req: Copy files from one container and store in another container- I am specifying the last modified date , i want to copy files only before that last last modified date. The files are of any type. csv,xml,json,txt etc. Also, in source container my files…
asked 2024-09-17T20:33:13.42+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(313.6, 57.00000000000001%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EW%3C/text%3E%3C/svg%3E)
Wasim
0
Reputation points
commented 2024-09-19T18:01:28.8033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(227.2, 46%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENN%3C/text%3E%3C/svg%3E)
NIKHILA NETHIKUNTA
1,920
Reputation points Microsoft Vendor
1 answer
Configuration of Azure data factory with Azure Dev ops
Hello Team, I am looking to configure Azure Data Factory with Azure DevOps for version control and to set up pipelines/flows that enable code check-ins to the next environment with an approval process in place. I would appreciate it if you could provide…
asked 2024-09-17T17:02:21.8166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 13%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERK%3C/text%3E%3C/svg%3E)
Rohit Kulkarni
691
Reputation points
commented 2024-09-19T17:59:54.7533333+00:00
NIKHILA NETHIKUNTA
1,920
Reputation points Microsoft Vendor
0 answers
Json response could not parse successfully in Copy activity (source)
For the same HTTP request, Web activity gives us the correct response all correctly parse, but Copy activity couldnt. This has worked fine before. Screenshots are cut from Preview data. When I actually run the copy activity there is no data coming from…
asked 2024-09-19T14:55:37.5466667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(16, 0%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EQD%3C/text%3E%3C/svg%3E)
Quyen Dang
21
Reputation points
commented 2024-09-19T17:44:27.9333333+00:00
Quyen Dang
21
Reputation points
2 answers
One of the answers was accepted by the question author.
Cosmos db in synapse
Hi, How to get cosmos db data load in azure synapse using container or table levels.. Pls share screenshot
asked 2024-09-18T16:00:18.14+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 13%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVS%3C/text%3E%3C/svg%3E)
Vineet S
770
Reputation points
commented 2024-09-19T17:17:41.5233333+00:00
Smaran Thoomu
15,040
Reputation points Microsoft Vendor
0 answers
Azure Data Factory Pipelines stuck in queue again
Azure Data Factory pipelines stuck currently in queue / in-progress indefinitely. We can however preview the data, and this seems to be happening only to few pipelines, not impacting for others... Sharing pipeline ID of one such for reference...…
asked 2024-09-17T08:36:58.8833333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(224.00000000000003, 84%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBG%3C/text%3E%3C/svg%3E)
Bhargav Guirish
0
Reputation points
commented 2024-09-19T16:05:07.8466667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
phemanth
10,250
Reputation points Microsoft Vendor
0 answers
Migrating ADF to Salesforce v2 Dataset, how to get MAX() value of Object's LastModifiedDate
Hello, I'm migrating to the new Salesforce v2 linked service and dataset. We have a dynamic set up to load the delta from Salesforce into our SQL Server DWH every night. We loop over all Objects that we sync, and start off with executing this statement…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(224.00000000000003, 17%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EM%3C/text%3E%3C/svg%3E)
1 answer
Increased expenses 10 times
My expenses increased 10 times from 3e per day to 35e per day. How to check which pipeline cost how much? The invoice only gives the total price. The pipeline itself uses Azure IR and uses a script, but it runs for 2 minutes. The logic is as follows in…
asked 2024-09-13T07:45:55.1+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(131.20000000000002, 52%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDV%3C/text%3E%3C/svg%3E)
Darjuš Vasiukevič
20
Reputation points
commented 2024-09-19T15:01:18.7666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 36%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
Bhargava-MSFT
30,891
Reputation points Microsoft Employee
3 answers
There are substantial concurrent MappingDataflow executions which is causing failures due to throttling under Integration Runtime 'AutoResolveIntegrationRuntime'
Hi, I'm having the error above: There are substantial concurrent MappingDataflow executions which is causing failures due to throttling under Integration Runtime 'AutoResolveIntegrationRuntime' I have checked other questions asking the…
asked 2022-09-21T06:42:06.567+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(224.00000000000003, 50%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EFD%3C/text%3E%3C/svg%3E)
Francisco Dominguez
386
Reputation points
answered 2024-09-19T14:57:09.6366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(9.6, 95%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAK%3C/text%3E%3C/svg%3E)
Ashir Khan
0
Reputation points
1 answer
Verify the existence of a folder in the metadata ChildItem list; if it does not exist, create the folder.
I have a pipeline that searches for a folder formatted as "yyyymm". I need assistance in verifying whether the folder name is present in the Get Metadata ChildItem name list. If the folder name does not exist, I need guidance on creating it in…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 10%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EL%3C/text%3E%3C/svg%3E)
0 answers
Azure Data Factory, Trigger is not working for weekly and monthly schedule but works for Daily Schedule
Create a test azure pipeline, Add a trigger to it with Monthly/Weekly frequency Let the time come for trigger, the trigger won't start the Pipeline. Now shift some time in trigger Manually, the trigger will start working.
asked 2024-09-19T14:07:32.1233333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(316.8, 17%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGD%3C/text%3E%3C/svg%3E)
Gupta, Dheeraj
0
Reputation points
asked 2024-09-19T14:07:32.1233333+00:00
Gupta, Dheeraj
0
Reputation points
1 answer
Server Configuration for Self Hosted IR
Helllo Team, I need to install and configure the Self Hosted IR in one of the server. Please let me know the configuration for the server. Regards Rohit
asked 2024-09-17T12:05:37.17+00:00
Rohit Kulkarni
691
Reputation points
commented 2024-09-19T11:46:19.4533333+00:00
phemanth
10,250
Reputation points Microsoft Vendor
0 answers
Azure data lake folder structure
Hi, the link provided in the below thread is not working, any way to find the information or a new URL/link: FAQs About Organizing a Data…
asked 2024-09-19T08:43:49.7+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(243.2, 88%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
azure_learner
240
Reputation points
commented 2024-09-19T09:49:49.6333333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(150.4, 64%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENR%3C/text%3E%3C/svg%3E)
Nehruji R
7,556
Reputation points Microsoft Vendor
1 answer
One of the answers was accepted by the question author.
UserErrorInvalidColumnMappingColumnNotFound
using ADF for below activity Copying data from source table to destination table with same column name both side and column present in both source and destination table, still receiving the error saying Failure happened on 'Sink' side.…
asked 2024-08-05T03:36:01.2666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(118.4, 1%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Sandip Shelke
25
Reputation points
accepted 2024-09-19T09:21:47.25+00:00
Sandip Shelke
25
Reputation points
1 answer
Where did the 'Script Parameters' section go in the new Snowflake ADF connector?
Where did the 'Script Parameters' section go in the new Snowflake ADF connector? Old: New:
asked 2024-08-29T13:49:05.09+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(89.60000000000001, 7.000000000000001%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERB%3C/text%3E%3C/svg%3E)
Robert Baber
0
Reputation points
commented 2024-09-19T08:39:34.58+00:00
AnnuKumari-MSFT
32,821
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
How to get weekly and monthly schedule triggers to work in Azure Data Factory V2?
Scheduling weekly or monthly triggers does not appear to work in Azure Data Factory V2, although hourly or daily works fine on the same simple test pipeline. All components have of course been Saved and Published.
asked 2023-12-19T14:49:25.71+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(51.2, 10%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ1%3C/text%3E%3C/svg%3E)
JonathanQuick-1666
30
Reputation points
commented 2024-09-19T08:23:20.2266667+00:00
Gupta, Dheeraj
0
Reputation points