Updating an existing ADF pipeline - ( Importing new code )

Question

We use different adf instances for dev and production and I was wondering how to update an existing pipeline in prod instance. The updated pipeline sitting in the dev instance and if we try to import again it creates a duplicate pipelines and datasets. Is there a way to update without duplicating ? Would just updating json work?

Answer 1

@Jiby Jacob

Greetings & Welcome to Microsoft Q&A forum! Thanks for posting your query!

As I understand that you are attempting to update Azure Data Factory pipelines in a production environment, with the updated versions currently in a development environment. You aim to avoid creating duplicate pipelines and datasets during this update process. You are seeking the best method to deploy changes from development to production without duplication and are specifically inquiring if directly updating the JSON definitions of the pipelines would be effective.

Why simply importing creates duplicates

ADF identifies pipelines and datasets by their names. When you import a pipeline with the same name as an existing one, ADF treats it as a new entity, hence creating a duplicate. It doesn't automatically perform an "update" or "merge" operation.

Would just updating json work? Why directly updating JSON is generally not recommended

While technically possible to directly modify the JSON behind ADF resources, it's strongly discouraged for several reasons:

Risk of errors - Manually editing JSON is prone to syntax errors, which can corrupt your ADF environment.

Lack of version control - Direct JSON manipulation bypasses ADF's built-in version control and deployment mechanisms, making it difficult to track changes and rollback if necessary.

Potential for inconsistencies - Manually changing JSON can lead to inconsistencies between the ADF portal's view and the actual underlying configuration.

Complexity - ADF resources can have complex JSON structures, making manual editing tedious and error-prone.

Is there a way to update without duplicating ? Recommended Approaches for Deploying ADF Changes

The standard and recommended approach for deploying ADF changes from one environment to another is using Azure DevOps or other CI/CD (Continuous Integration/Continuous Deployment) pipelines along with ARM (Azure Resource Manager) templates.

Declarative approach - ARM templates use a declarative approach, where you define the desired state of your resources. The deployment engine then figures out how to achieve that state, whether it's creating new resources or updating existing ones.

Idempotency - ARM template deployments are idempotent, meaning you can deploy the same template multiple times, and it will have the same result. This is crucial for CI/CD, as it ensures consistent deployments.

Dependency management - ARM templates handle dependencies between resources automatically, ensuring that they are deployed in the correct order.

Validation - The Azure deployment engine validates ARM templates before deployment, catching errors early on.

Integration with CI/CD - ARM templates integrate seamlessly with CI/CD tools like Azure DevOps and GitHub Actions, enabling automated deployments and a smooth development workflow.

For further details, please consult the following Microsoft documentation, which may provide valuable insights:

• Applying DataOps to Azure Data Factory
• Continuous integration and delivery in Azure Data Factory

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

Thank you.