Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,159 questions
0 answers
execute few tables in parallel and few tables in sequential based on the exec priority
I have a ADF pipeline where it reads the control table to get the list of source and target tables that needs to copy from source to target. and pass the list to foreach where it executes copy activity. Enhancement needed: I have the exec_pprioroty in…
asked 2025-01-26T20:52:11.75+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(156.8, 90%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EYR%3C/text%3E%3C/svg%3E)
yogeswara Reddy Chegireddy
0
Reputation points
asked 2025-01-26T20:52:11.75+00:00
yogeswara Reddy Chegireddy
0
Reputation points
6 answers
One of the answers was accepted by the question author.
Getting error "{"code":"BadRequest","message":null,"target":"pipeline..." when trying to debug pipeline
Have confirmed through reproducing a couple of times. I have 3 data factories, dev/uat/prod. I develop something on DEV, then use Azure devops to run through and release the ARM templates onto UAT and prod. This all works OK. However I then change…
asked 2021-03-08T02:37:14.197+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(128, 8%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
Anonymous
commented 2025-01-26T19:18:09.4633333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(28.799999999999997, 68%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
Kieran McKewen (Parallel42)
0
Reputation points
2 answers
How to fix nested split issue in Azure data factory Pipeline expression builder?
Hi there, I'm trying to make use of hierarchical namespaces for Azure data lake storage gen2. When I'm dynamically setting the file structure in the Pipeline expression builder in the ADF, the following error is returned: Cannot fit unknown into the…
asked 2025-01-07T23:56:02.7266667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(112.00000000000001, 30%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EZS%3C/text%3E%3C/svg%3E)
Zohaib Sheraz Mughal
15
Reputation points
edited an answer 2025-01-26T13:20:13.0033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(304, 6%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKD%3C/text%3E%3C/svg%3E)
Keshavulu Dasari
3,090
Reputation points Microsoft Vendor
1 answer
ADF Dataflow Preview SQUISHED
Currently the ADF UI is not rendering the Dataflow preview properly -its SQUISHED. when exported to csv the data is there. I have tried all possible debugging steps for the Dataflow data preview, including: Clearing cache Switching browsers Adjusting…
asked 2025-01-23T17:17:51.34+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 40%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAL%3C/text%3E%3C/svg%3E)
Anatoly La Pere (London)
0
Reputation points
answered 2025-01-26T07:30:36.64+00:00
Vinodh247
27,281
Reputation points MVP
1 answer
ErrorCode=AdlsGen2OperationFailed...Failure happened on 'Sink' side.
Getting the below error while copying the data to ADLS from SAP Hana, however this error is random to tables failing daily. When I try load each of the failed tables from the same source and with the same file name then there is no issue and it works as…
asked 2025-01-25T11:59:38.9966667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(131.20000000000002, 82%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETA%3C/text%3E%3C/svg%3E)
Tyagi, Amit
5
Reputation points
answered 2025-01-26T07:28:24.8066667+00:00
Vinodh247
27,281
Reputation points MVP
2 answers
Issue with Azure SQL Database Table Detection in CDC
I have a table in azure sql database with schema, TABLE NAME: EMPLOYEE COLUMNS: PersonID int NOT NULL, Name varchar(255), Age int But this table is not detected in the pane list of source table. As per MS doc, The pane lists only…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(310.4, 30%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
1 answer
Optimize Nested Loop Processing in ADF Pipeline for Dynamic JSON Transformation
I have two JSON responses from Lookup activities in my Azure Data Factory pipeline, as shown below: Lookup Activity 1 Response: [ { "cra5a_columnname": "SALES_CHANNEL", "cra5a_datatype": "OptionSet", …
asked 2025-01-24T16:27:16.6033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(163.2, 13%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENA%3C/text%3E%3C/svg%3E)
Naresh Arvapalli
0
Reputation points
commented 2025-01-25T12:30:03.2633333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(246.4, 4%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Sina Salam
16,446
Reputation points
1 answer
ADF - flatten activity
Hi team, I am facing the issue while reading the xml using ADF copy activity. string value “6E12” is getting converted into 6000000000000 while reading the XML file with flatten hierarchy. Please help in resolving the issue. thanks, shraddha
asked 2025-01-23T10:41:18.9933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(51.2, 93%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESC%3C/text%3E%3C/svg%3E)
Shraddha Choudhary
0
Reputation points
commented 2025-01-25T11:18:23.8766667+00:00
Shraddha Choudhary
0
Reputation points
1 answer
ADF Dataflow Preview not working
The ADF UI for Dataflow preview is not working - it appears hidden or squished. I have tried all possible debugging steps for the Dataflow data preview, including: Clearing cache Switching browsers Adjusting sample size Increasing CPU runtime …
asked 2025-01-23T17:19:46.62+00:00
Anatoly La Pere (London)
0
Reputation points
answered 2025-01-25T11:13:04.7666667+00:00
Vinodh247
27,281
Reputation points MVP
1 answer
Monitor AutoResolve Integration Runtime in Azure Data Factory
Hi all I need to monitor the Integration Runtime performance for a pipeline that uses stored procedure activity to load data from ADLS to Snowflake. The stored procedure in this case resides in snowflake itself and it’s called from Snowflake Tenant to…
asked 2025-01-24T06:12:53.87+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(188.79999999999998, 64%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA6%3C/text%3E%3C/svg%3E)
AyusmanBasu-6332
6
Reputation points
answered 2025-01-25T11:11:19.7833333+00:00
Vinodh247
27,281
Reputation points MVP
0 answers
Azure Synapse Notebook does't fail immediately in case of error in code
Hello, I am struggling with really weird behavior when executing a notebook within Azure Synapse. I am utilizing pyspark and sql within this notebook and noticed that the notebook doesn't fail immediately in case of a bug or let's say typo in a code. For…
asked 2025-01-23T18:26:28.9+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(182.40000000000003, 38%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPP%3C/text%3E%3C/svg%3E)
Peter Polacek
20
Reputation points
commented 2025-01-24T21:49:13.04+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 65%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECB%3C/text%3E%3C/svg%3E)
Chandra Boorla
7,260
Reputation points Microsoft Vendor
1 answer
In ADF Dataflow Preview i am not able to see the preview properly how to set this can anyone help
This preview is not good i can't do manually for 100 of tables is their any way for getting better preview on this?
asked 2025-01-23T10:55:29.5466667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(73.60000000000001, 13%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMW%3C/text%3E%3C/svg%3E)
Makrand Waghmare
5
Reputation points
commented 2025-01-24T21:09:17.5833333+00:00
Chandra Boorla
7,260
Reputation points Microsoft Vendor
1 answer
I am using ADF ,copying data from on premises Data to azure SQL.when it trying to copying data from SQL(On Premises) to Azure SQL it giving error
ErrorCode=SqlFailedToConnect,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Cannot connect to SQL Database. Please contact SQL server team for further support. Server: 'd-puma-eastus-sql.database.windows.net', Database:…
asked 2025-01-14T13:24:35.06+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(118.4, 56.00000000000001%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESP%3C/text%3E%3C/svg%3E)
Soubhagya Patra
0
Reputation points
commented 2025-01-24T21:07:52.27+00:00
Chandra Boorla
7,260
Reputation points Microsoft Vendor
1 answer
With ADF Pagination Rules, how do I add query parameters properly to subsequent calls?
I am running calls on an API, where my base URL specifies a query parameter "limit=500" to return a max of 500 records in the response. I have an AbsoluteURL pagination rule set up to follow a next-page URL provided in the response. When the…
asked 2025-01-24T18:11:27.0333333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 52%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAM%3C/text%3E%3C/svg%3E)
Angelo Mariano
25
Reputation points
answered 2025-01-24T20:55:21.5566667+00:00
Chandra Boorla
7,260
Reputation points Microsoft Vendor
0 answers
What is the issue with dataflow yesterday
What was the issue with dataflows yesterday as a result where all of our pipelines failed
asked 2025-01-10T05:40:05.54+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(147.20000000000002, 98%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETA%3C/text%3E%3C/svg%3E)
T, Athaullakhan
0
Reputation points
commented 2025-01-24T16:57:12.6266667+00:00
AnnuKumari-MSFT
33,986
Reputation points Microsoft Employee
1 answer
How to Reduce Queue Time in Azure Data Factory for Self-hosted Integration Runtime?
Hello everyone, We have been experiencing significant queue times in Azure Data Factory for our self-hosted integration runtime (SHIR). Despite reaching out to Microsoft support for several months, we have not been able to resolve this issue. Our SHIR…
asked 2025-01-20T06:49:04.9366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(128, 12%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECR%3C/text%3E%3C/svg%3E)
Contreras,Rafael,MX-Ciudad de México
0
Reputation points
commented 2025-01-24T16:49:12.99+00:00
Contreras,Rafael,MX-Ciudad de México
0
Reputation points
1 answer
ADF Power Query Sink Settings UI Missing Options
Why would my data factory not have the same UI as I see in training videos and QA answers on the Power Query data wrangling part of ADF? I can only see the delete button next to the dataset in the Power Query activity Sink…
asked 2025-01-23T15:50:49.8533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(272, 24%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EES%3C/text%3E%3C/svg%3E)
Elisa Schrader
20
Reputation points
answered 2025-01-24T16:46:32.2466667+00:00
AnnuKumari-MSFT
33,986
Reputation points Microsoft Employee
0 answers
GetMetadat Activity returns False even if folder exists
I have enabled GetMetadata Activity to check any folder/files existing in a Blob Storage. The Exists property which was working earlier isn't functioning as expected as observed this week and returns False even if a folder exists in the path.
asked 2025-01-22T12:01:59.2466667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(19.2, 83%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Sambit Sourav
0
Reputation points
commented 2025-01-24T16:33:58.68+00:00
Chandra Boorla
7,260
Reputation points Microsoft Vendor
2 answers
How to connect synapse external table using ADF.
I have Azure Synapse external tables and wanted to bring those table data into another adls account using adf. Created Link service in ADF using authentication type as sql authentication. used admin username and password in the same. Test connection…
asked 2025-01-24T11:28:31.66+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(25.6, 48%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHS%3C/text%3E%3C/svg%3E)
Harshraj Sawant
0
Reputation points
answered 2025-01-24T13:51:30.9133333+00:00
Nandan Hegde
33,561
Reputation points MVP
1 answer
Data Preview in ADF Debug Mode is not as expected
Hi Team, When i am trying preview data in azure data factory dataflow (debug mode). The visibility of data preview is not clear. Attached Screen shot. Is this a bug? Could you please help me to get this Thanks, Santhosh
asked 2025-01-24T02:04:24.9166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(57.599999999999994, 51%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESJ%3C/text%3E%3C/svg%3E)
Santosh J
30
Reputation points
answered 2025-01-24T13:35:38.4266667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
19,310
Reputation points Microsoft Vendor