ADF Pipeline "Bad Request" Error when Debug run

Mathias Denzin 20

Hi all,

I have a pipeline in ADF that is incrementally loading objects from Salesforce whose LastModifiedDate is between the last date in my sink (Fabric Warehouse) and the current runtime.

I want to do that in a single pipeline now with parameters. The pipeline will be called from another pipeline with different parameters for each object.

User's image

As soon as I debug run this pipeline, I get this error:

User's image

The other activities run successfully when I set a stop point before the "Copy data" activity. As soon as I remove the stop point, the whole pipeline does not start.

Does anyone have an idea or similar problems?

Chandra Boorla 6,130 Reputation points Microsoft Vendor

2025-01-02T01:49:26.6866667+00:00

Hi @Mathias Denzin

Greetings & Welcome to Microsoft Q&A forum! Thanks for posting your query!

The "Bad Request" error in Azure Data Factory (ADF) can often be caused by issues related to the parameters being passed to the pipeline or the configuration of the activities within the pipeline. Since you mentioned that the other activities run successfully when a stop point is set before the "Copy data" activity, it suggests that there may be a problem with the data being processed or the parameters being passed when the pipeline is executed without the stop point.

"Bad Request" error might be caused by several factors, and it's important to check the pipeline configuration and input parameters closely.

Here are a few things you can check to troubleshoot the issue:

Parameter Configuration - Ensure that the parameters being passed from the calling pipeline are correctly formatted and match the expected types in the child pipeline. If you're passing an array, make sure it is not being treated as a string.

Incorrect Query/Mapping - The query used to extract data from Salesforce or the mapping between Salesforce fields and warehouse columns might be incorrect. This is particularly relevant given the incremental loading requirement, as the query likely includes a WHERE clause filtering by LastModifiedDate.

Data Types - Verify that the data types of the parameters match what the "Copy data" activity expects. Mismatched data types can lead to a "Bad Request" error.

Payload Size - Check if the payload being sent to the "Copy data" activity exceeds the size limit. The payload for each activity run should not exceed 896 KB.

Debugging with Stop Points - Since the pipeline runs successfully with a stop point, consider using this to isolate the issue further. You can run the pipeline step-by-step to see which specific configuration or data is causing the failure.

If none of these step's work, you can try recreating the entire ADF pipeline from scratch. Sometimes, recreating the activity or pipeline can resolve any underlying issues.

In case if you are still blocked and need assistance, please do share the JSON payload of your pipeline which would help us to troubleshoot the issue.

For further information, please refer to the links below, which may provide valuable insights.

Troubleshoot pipeline orchestration and triggers in Azure Data Factory

Troubleshoot Azure Data Factory and Synapse pipelines

For common bad request errors watch this video - Bad Request error in ADF | Message Null | Common errors in ADF for more information.

I hope this information helps. Please do let us know if you have any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Mathias Denzin 20 Reputation points

2025-01-02T16:12:43.19+00:00

I think I found a bug here in ADF. The pipeline was copied and adjusted. I created parameters to call the pipeline with parameters. One of them was the objectname. When I replace all occurrences of the variable "objectname" with the new parameter "ObjectName", all works fine. As soon as I delete the unused variable "objectname", the pipeline crashes. I recreated the variable to confirm, and the pipeline was running well again. I also confirmed in the pipeline JSON that the variable "objectname" is not used anywhere any more. And that is a fact. The only place it is visible in the JSON is the variables declaration block. So, I will manually recreate the pipeline and not work from a copy.
Mathias Denzin 20 Reputation points

2025-01-02T16:28:00.6+00:00

Addition: My sink dataset had a default value that included the variable. Even if I overwrite the default value in my pipeline, it still crashes as long as the data set has the default value. I removed the default value from the dataset parameter, and now everything works fine.
Chandra Boorla 6,130 Reputation points Microsoft Vendor

2025-01-03T01:15:16.38+00:00

@Mathias Denzin

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer.

Accepted answer

Chandra Boorla 6,130 Reputation points Microsoft Vendor

2025-01-03T01:14:49.5366667+00:00

@Mathias Denzin

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer.

Issue:

I have a pipeline in ADF that is incrementally loading objects from Salesforce whose LastModifiedDate is between the last date in my sink (Fabric Warehouse) and the current runtime.

I want to do that in a single pipeline now with parameters. The pipeline will be called from another pipeline with different parameters for each object.

As soon as I debug run this pipeline, I get the error:

The other activities run successfully when I set a stop point before the "Copy data" activity. As soon as I remove the stop point, the whole pipeline does not start.

Solution:

I encountered an issue with my sink dataset due to a default value that included a variable. Despite overwriting the default value in my pipeline, the problem persisted as long as the dataset retained the default value. After removing the default value from the dataset parameter, the issue was resolved, and everything is now functioning correctly.

If I missed anything please let me know and I'd be happy to add it to my answer, or feel free to comment below with any additional information.

Hope this helps. Do let us know if you have any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

ADF Pipeline "Bad Request" Error when Debug run

0 additional answers

Your answer