Configuration of Azure data factory with Azure Dev ops

Question

Hello Team,

I am looking to configure Azure Data Factory with Azure DevOps for version control and to set up pipelines/flows that enable code check-ins to the next environment with an approval process in place. I would appreciate it if you could provide relevant documentation or links, so I can refer to them and successfully configure Azure Data Factory with Azure DevOps for this purpose.

Regards

Rohit

Answer 1

Follow these key steps to integrate ADF with Azure DevOps and establish an approval-based deployment pipeline:

1. Connect Azure Data Factory to Azure DevOps for Source Control

ADF supports Git repositories, including Azure DevOps Git and GitHub, for source control. Here's how to link your ADF to an Azure DevOps Git repository:

• Open Azure Data Factory Studio: In the ADF portal, go to the Author & Monitor section.
• Set Up Git Integration:
  1. In the top menu, click on the Manage hub (the gear icon).
  2. Under Git Configuration, select Azure DevOps Git as the repository type.
  3. Enter details such as Azure DevOps organization, project, repository name, collaboration branch (usually main or master), and root folder.
• Publish from Collaboration Branch: Once integrated, any changes made in the ADF studio are automatically committed to the collaboration branch in Azure DevOps. You can publish changes from the Collaboration branch to the Data Factory’s service instance by clicking Publish All.
• Branching: You can use feature branches for development and then merge changes into the collaboration branch via pull requests.

You can follow this Microsoft guide for version control setup: Set up source control in Azure Data Factory

2. Set Up Azure Pipelines for CI/CD

To move code from development to other environments (such as QA, production), set up a Continuous Integration (CI) and Continuous Delivery (CD) pipeline in Azure DevOps.

• Create a Pipeline:
  1. In Azure DevOps, navigate to Pipelines.
  2. Click New Pipeline and choose the source repository where ADF code is stored.
  3. Define a pipeline using the YAML pipeline editor or use the Classic Editor for a GUI-based approach.
  4. You can use Azure Resource Manager (ARM) templates to deploy Data Factory objects (datasets, pipelines, etc.) across environments.
• Pipeline Configuration:
  • The ADF service automatically generates ARM templates when you publish changes. These templates are stored in the repository's adf_publish branch.
  • Use these templates in your deployment pipeline to deploy changes to higher environments (QA/Prod).
  • Example of YAML pipeline:

    
           trigger:
    
             branches:
    
               include:
    
                 - master
    
           pool:
    
             vmImage: 'windows-latest'
    
           
    
           steps:
    
           - task: AzureResourceManagerTemplateDeployment@3
    
             inputs:
    
               deploymentScope: 'Resource Group'
    
               azureResourceManagerConnection: 'AzureRMServiceConnection'
    
               action: 'Create Or Update Resource Group'
    
               resourceGroupName: 'YourResourceGroup'
    
               location: 'YourLocation'
    
               templateLocation: 'Linked artifact'
    
               csmFile: '$(System.DefaultWorkingDirectory)/adf_publish/ARMTemplateForFactory.json'
    
               csmParametersFile: '$(System.DefaultWorkingDirectory)/adf_publish/ARMTemplateParametersForFactory.json'
    
    

3. Implement an Approval Process

Azure DevOps allows you to add manual intervention or approval steps before deploying to specific environments.

• Set up Environments with Approvals:

1. Go to the Pipelines > Environments section in Azure DevOps.
   2. Create a new environment (e.g., QA, Production).
   3. Configure approvers for that environment, who will be notified to approve deployments before they proceed.
   4. In your pipeline, use a stage for each environment and specify the environment configured for approvals.
   Example:

  
  stages:
  
  - stage: QA
  
  jobs:
  
  - deployment: QA_Deploy
  
    environment: 'QA'
  
    strategy:
  
      runOnce:
  
        deploy:
  
          steps:
  
          - script: echo Deploying to QA
  

More links :

Configure Git integration with Azure Data Factory

Continuous integration and delivery in Azure Data Factory

Azure Pipelines Environments and Approvals

These steps will enable you to set up version control, configure CI/CD pipelines, and implement an approval process for deploying ADF assets across different environments.