I am integrating the dataflow activity which removes duplicates from the csv file to the existing pipeline i have. I need a help to configure the parameters.

Question

I created a dataflow activity which removes duplicates from the source files( csv formatted files). So now i integrated this dataflow activity to existing pipeline i have. but i am not able to parametrize the source files.

It would be helpful if i get some options on this to solve my issue.

without attaching to the pipeline it's working completely fine i am getting the proper count without duplicates. But after integrating the datflow activity i am not able to provide the source options and the particular files i want.

Answer 1

@Lakshmi Moulya Nerella

Thanks for using Microsoft Q&A forum and posting your query.

you're on the right track with your dataflow activity! To help you parameterize the source files in your existing pipeline

I have reproduced from me to parametrize the data flow to an existing pipeline please follow the below steps and let me know if it works

Create Parameters in Your Data Flow:

• Open your data flow and click on the blank portion of the canvas to access the general properties.
• In the settings pane, go to the Parameters tab and click New to create a new parameter.
• Assign a name, select a type (e.g., string), and optionally set a default value.

Use Parameters in Your Data Flow:

• You can reference these parameters in any data flow expression. Parameters start with a $ sign.
• For example, if you have a parameter named sourceFilePath, you can use it in your source transformation settings.

Assign Parameter Values from the Pipeline:

• In your pipeline, add an Execute Data Flow activity.
• In the activity's Parameters tab, you will see the parameters you created in your data flow.

• Assign values to these parameters using either the pipeline expression language or the data flow expression language.

For more detailed guidance, you can refer to the official documentation on parameterizing mapping data flows

I hope the above steps will resolve the issue, please do let us know if issue persists. Thank you

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.