ADF Pipelines to Synapse or Fabric

Question

Hi there,

I would like to know if the actively running ADF pipelines could be copied and restored into Synapse or Fabric? (lift and shift). If possible Across accounts (to use as templates across accounts and customize/tune ADF pipelines little bit in new environment).

ADF Pipeline mostly contains copy activity and SQL SP.

option 1: Looking for steps to Download ADF pipelines & keyvaults to my local computer and restore it on separate new account synapse or fabric accounts.

Option 2: Is there any steps to follow to directly copy / restore ADF into different workspace within same subscription account? ( i prefer to copy to a spearate workspace so if i need to run in parallel and shut the new fabric down, it should not impact existing ADF set up running smoothly)

Same way can if SQL Azure DB can be lift and shift into Synapse or Fabric?

Thanks

Answer 1

@RJ

There are different options to this.

Option 1: Download ADF Pipelines & Key Vaults to Local Computer and Restore

1. Export ADF Pipelines:
   • You can export your ADF pipelines as JSON files. In the ADF portal, navigate to the pipeline you want to export, click on the "Export ARM Template" option, and save the JSON file to your local computer.
   • For detailed steps, you can refer to this guide https://learn.microsoft.com/en-us/answers/questions/519416/migrate-and-move-pipelines-created-in-azure-data-f
2. Export Key Vault Secrets:
   • To export Key Vault secrets, you can use Azure PowerShell or Azure CLI to retrieve the secrets and save them locally. For example, using Azure CLI:

          az keyvault secret download --vault-name <YourKeyVaultName> --name <SecretName> --file <FilePath>
     

   • For more details, check out this article https://learn.microsoft.com/en-us/azure/data-factory/how-to-use-azure-key-vault-secrets-pipeline-activities
3. Restore to Synapse or Fabric:
   • Use Azure PowerShell cmdlets for Azure Synapse Analytics to create the same artifacts in the Synapse workspace. You can import the JSON files you exported earlier.

For Microsoft Fabric, you might need to manually recreate the pipelines as the JSON code is read-only for now.

Option 2: Copy/Restore ADF into Different Workspace within Same Subscription

1. Export and Import Pipelines:

• Similar to Option 1, export the pipelines as JSON files.
• In the new workspace, use the "Import from pipeline template" option to import the JSON files. You will need to recreate the linked services during the import process.

2.Move Data Factory to Another Subscription:

• You can move your Data Factory to another subscription by exporting the ARM template and importing it into the new subscription. Ensure that all linked services and datasets are correctly configured in the new environment

SQL Azure DB Lift and Shift into Synapse or Fabric

1.Lift and Shift to Synapse:

• You can use Azure Data Factory (ADF) to migrate your SQL Azure DB to Synapse. ADF supports parallelism and can handle large datasets efficiently.

2.Lift and Shift to Fabric:

• For Microsoft Fabric, you can use the mirroring feature to continuously replicate your existing Azure SQL Databases directly into Fabric's OneLake.

See:

• https://community.fabric.microsoft.com/t5/Data-Pipeline/How-to-migrate-the-existing-pipelines-from-ADF-or-Azure-Synapse/m-p/3678244
• https://learn.microsoft.com/en-us/answers/questions/1079151/adf-how-to-copy-and-customize-same-pipeline-from-o
• https://learn.microsoft.com/en-us/answers/questions/809733/how-can-i-move-my-data-factory-to-another-subscrip
• https://learn.microsoft.com/en-us/answers/questions/2082405/how-to-lift-and-shift-huge-on-premise-data-warehou
• https://learn.microsoft.com/en-us/fabric/database/mirrored-database/azure-sql-database

I hope this helps, if so accept the answer or let me know if you need further help