ADF Pipelines to Synapse or Fabric

RJ 326

Hi there,

I would like to know if the actively running ADF pipelines could be copied and restored into Synapse or Fabric? (lift and shift). If possible Across accounts (to use as templates across accounts and customize/tune ADF pipelines little bit in new environment).

ADF Pipeline mostly contains copy activity and SQL SP.

option 1: Looking for steps to Download ADF pipelines & keyvaults to my local computer and restore it on separate new account synapse or fabric accounts.

Option 2: Is there any steps to follow to directly copy / restore ADF into different workspace within same subscription account? ( i prefer to copy to a spearate workspace so if i need to run in parallel and shut the new fabric down, it should not impact existing ADF set up running smoothly)

Same way can if SQL Azure DB can be lift and shift into Synapse or Fabric?

Thanks

Accepted answer

Nandan Hegde 34,171 Reputation points MVP

2025-02-13T03:20:10.9+00:00

Hey,
ADF pipeline templates can be extracted and imported into Synapse pipelines directly.

But ADF pipeline to fabric integration is not direct aspect.

Below link provides details:

https://learn.microsoft.com/en-us/fabric/data-factory/migrate-planning-azure-data-factory
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

2 additional answers

RJ 326 Reputation points

2025-02-12T23:35:47.4633333+00:00
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.
Chiugo Okpala 160 Reputation points MVP

2025-02-11T20:04:40.8366667+00:00
@RJ

There are different options to this.

Option 1: Download ADF Pipelines & Key Vaults to Local Computer and Restore

Export ADF Pipelines:

You can export your ADF pipelines as JSON files. In the ADF portal, navigate to the pipeline you want to export, click on the "Export ARM Template" option, and save the JSON file to your local computer.

For detailed steps, you can refer to this guide https://learn.microsoft.com/en-us/answers/questions/519416/migrate-and-move-pipelines-created-in-azure-data-f

Export Key Vault Secrets:

To export Key Vault secrets, you can use Azure PowerShell or Azure CLI to retrieve the secrets and save them locally. For example, using Azure CLI:
az keyvault secret download --vault-name <YourKeyVaultName> --name <SecretName> --file <FilePath>

For more details, check out this article https://learn.microsoft.com/en-us/azure/data-factory/how-to-use-azure-key-vault-secrets-pipeline-activities

Restore to Synapse or Fabric:

Use Azure PowerShell cmdlets for Azure Synapse Analytics to create the same artifacts in the Synapse workspace. You can import the JSON files you exported earlier.

For Microsoft Fabric, you might need to manually recreate the pipelines as the JSON code is read-only for now.

Option 2: Copy/Restore ADF into Different Workspace within Same Subscription

1. Export and Import Pipelines:

Similar to Option 1, export the pipelines as JSON files.

In the new workspace, use the "Import from pipeline template" option to import the JSON files. You will need to recreate the linked services during the import process.

2.Move Data Factory to Another Subscription:

You can move your Data Factory to another subscription by exporting the ARM template and importing it into the new subscription. Ensure that all linked services and datasets are correctly configured in the new environment

SQL Azure DB Lift and Shift into Synapse or Fabric

1.Lift and Shift to Synapse:

You can use Azure Data Factory (ADF) to migrate your SQL Azure DB to Synapse. ADF supports parallelism and can handle large datasets efficiently.

2.Lift and Shift to Fabric:

For Microsoft Fabric, you can use the mirroring feature to continuously replicate your existing Azure SQL Databases directly into Fabric's OneLake.

See:

https://community.fabric.microsoft.com/t5/Data-Pipeline/How-to-migrate-the-existing-pipelines-from-ADF-or-Azure-Synapse/m-p/3678244

https://learn.microsoft.com/en-us/answers/questions/1079151/adf-how-to-copy-and-customize-same-pipeline-from-o

https://learn.microsoft.com/en-us/answers/questions/809733/how-can-i-move-my-data-factory-to-another-subscrip

https://learn.microsoft.com/en-us/answers/questions/2082405/how-to-lift-and-shift-huge-on-premise-data-warehou

https://learn.microsoft.com/en-us/fabric/database/mirrored-database/azure-sql-database

I hope this helps, if so accept the answer or let me know if you need further help
Please sign in to rate this answer.
RJ 326 Reputation points

2025-02-11T23:53:39.93+00:00

Thanks @Chiugo Okpala . Does it mean I can not use import pipeline template in synapse or fabric? and only use PowerShell commands?

Chiugo Okpala 160 Reputation points MVP

2025-02-12T05:08:19.3933333+00:00

Hi @RJ,

You can indeed use import pipeline templates in both Synapse and Fabric. You don't have to rely solely on PowerShell commands. Here's how you can do it:

Import Pipeline Templates in Synapse

Export the Pipeline:

In Azure Data Factory, export your pipeline as a JSON file.

Import the Pipeline:

In Synapse Studio, navigate to the "Integrate" hub.

Click on "Import from pipeline template" and select the JSON file you exported.

Import Pipeline Templates in Fabric

Export the Pipeline:

Similar to Synapse, export your pipeline as a JSON file from Azure Data Factory.

Import the Pipeline:

In Microsoft Fabric, go to the Data Factory section.

Use the "Import from pipeline template" option to import the JSON file. You can also use the Pipeline template gallery to get started with predefined templates

See: https://learn.microsoft.com/en-us/fabric/data-factory/templates

Ganesh Gurram 3,930 Reputation points Microsoft Vendor

2025-02-12T15:57:06.8733333+00:00

@RJ - Just checking in to see if the above answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

RJ 326 Reputation points

2025-02-12T23:01:49.51+00:00

@Chiugo Okpala Thanks again. I exported from ADF by Download pipeline aritifacts and it got downloaded as compressed format.

I went into to fabric import template and it errored out as invalid template.

Wondering if im missing anything.

Chiugo Okpala 160 Reputation points MVP

2025-02-13T05:55:33.25+00:00

@RJ

Try this two options below to solve the problem:

Option 1:

Exporting Pipelines Correctly from Azure Data Factory

Use "Save as Template" Instead of "Download Pipeline Artifacts"

Navigate to Your Pipeline:

Open your ADF portal and locate the pipeline you want to export.

Save as Template:

Click on the ellipsis (`...`) next to the pipeline name. Select **"Save as Template"** from the dropdown menu. Follow the prompts to save the pipeline as a JSON template file.

Export the ARM Template

Export Entire Data Factory:

In the ADF portal, click on the "ARM Template" option at the top.

Choose "Export ARM Template".

This will download a set of JSON files representing your Data Factory's configuration.

Importing into Microsoft Fabric

1. Prepare the JSON Template

Ensure Compatibility:

Open the JSON template file in a text editor.

Verify that the activities and linked services used in your pipeline are supported in Microsoft Fabric. Fabric might have different supported services or parameters, so minor adjustments might be necessary.

2. Import the Template into Fabric

Navigate to Fabric's Data Factory Experience:

Open Microsoft Fabric and go to the Data Factory section.

Import Template:

Click on "Import" and select "Pipeline Template".

Choose the JSON template file you exported from ADF.

Resolve Any Errors:

If you encounter errors, note the specific issues.

Common problems include unsupported activities or missing linked services.

Adjust the JSON file accordingly and retry the import.

Common Issues and Solutions

Issue: Invalid Template Error

Cause: Fabric doesn't recognize certain elements in the ADF template.

Solution:

Extract and Use Individual JSON Files:

If you downloaded a compressed file, extract it to access individual JSON files.

Modify the Template:

Remove or replace unsupported activities.

Ensure all parameters and namespaces align with Fabric's expectations.

Validate the JSON:

Use a JSON validator to check for syntax errors.

Issue: Compressed File Not Accepted

Cause: Fabric's import function doesn't accept compressed (zip) files.

Solution:

Extract the Zip File:

Unzip the downloaded artifacts to access the JSON templates inside.

Import the JSON Template:

Use the extracted JSON file for the import process.

Alternative Approach: Manual Recreation

Given that your pipelines mainly consist of copy activities and SQL stored procedures, manually recreating them in Fabric could be straightforward.

Steps to Recreate Pipelines in Fabric

Set Up Linked Services:

Define connections to your data sources and destinations within Fabric.

Recreate the Copy Activities:

Use the Copy Data tool in Fabric's Data Factory to set up copy operations.

Configure source and sink datasets as per your requirements.

Implement SQL Stored Procedures:

If you're calling SQL stored procedures, use the Stored Procedure activity in Fabric.

Configure the activity with your database details and procedure parameters.

Configure Parameters and Variables:

Recreate any parameters or variables used in your ADF pipelines to maintain dynamic behavior.

Test Your Pipeline:

Run the pipeline in debug mode to ensure it functions as expected.

Monitor the execution and make necessary adjustments.

Option 2:

Steps to Ensure Successful Import

1.Extract the Compressed File:

Make sure to extract the contents of the compressed (zipped) folder you downloaded from ADF. The import process in Fabric might not recognize the compressed format directly.

2. Check the JSON Structure:

Ensure that the JSON files inside the extracted folder are correctly formatted. Sometimes, minor issues in the JSON structure can cause import errors.

3. Validate the Template:

Use a JSON validator tool to check for any syntax errors in the JSON files. This can help identify any issues that might be causing the import to fail.

4.Import the Correct File:

When importing the template into Fabric, make sure you are selecting the correct JSON file. Sometimes, there might be multiple JSON files, and selecting the wrong one can cause errors.

Additional Resources

You can refer to this guide for more details on publishing and downloading pipeline artifacts.

For more information on supported file formats and compression codecs, check out this article.

If you've tried all these steps and are still facing issues, let me know, and we can troubleshoot further.

RJ 326 Reputation points

2025-02-13T17:14:32.8266667+00:00

tried exporting ADF as template

Chiugo Okpala 160 Reputation points MVP

2025-02-14T07:53:24.98+00:00

@R J Just checking to see if it works?
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

ADF Pipelines to Synapse or Fabric

2 additional answers

Your answer