Error 1502

Siphe 25

Im new to Azure....We have spark job thats has been running just fine for a long time we've had some issues with the data format lately but the pipeline was breaking in a different place than now. Now Im getting this error

"errorCode": "1502", "message": "Synapse ADF RP call failed with response = Synapse ADF RP call failed with response =

"failureType": "SystemError", "target": "claims_staging_job", "details": [] }

Could the issue be the spark job or is it something else

phemanth 13,905 Reputation points Microsoft Vendor

2025-02-14T20:24:27.5866667+00:00
@Siphe

Welcome to Microsoft Q&A platform and thanks for posting your query here

It looks like you're encountering a SystemError with error code 1502 in your Azure Synapse pipeline. This error typically indicates a failure in the Azure Data Factory (ADF) or Synapse Resource Provider (RP) call.

Given that your Spark job was running fine previously and you've had recent issues with the data format, it's possible that the error could be related to the data format changes. However, it could also be due to other factors such as:

Configuration Issues: Ensure that all configurations for your Spark job and pipeline are correct and haven't been altered inadvertently.

Resource Limits: Check if there are any resource constraints or limits being hit, such as memory or compute capacity.

Network Issues: Verify that there are no network connectivity issues between your data sources and the Synapse workspace.

For more detailed troubleshooting, please refer to the Azure Data Factory and Synapse troubleshooting guide

I hope the above steps will resolve the issue, please do let us know if issue persists. Thank you
Siphe 25 Reputation points

2025-02-17T07:55:51.23+00:00

Hi @phemanth thank you.
I've used an old file that didn't have any issues before and i still get the same error now
1.Configuration Issues - Nothing has changed or has been altered.
2.Resource Limits - No resource limits are being hit
3.Network Issues - No network issues, the pipeline fetches the file from the SFTP server just fine but fails on the next step of staging
phemanth 13,905 Reputation points Microsoft Vendor

2025-02-17T17:59:31.5233333+00:00
@Siphe

Thanks for the additional details! Since you've ruled out configuration, resource limits, and network issues, please consider few more possibilities:

Data Corruption: Even though the file worked before, it might have become corrupted. Try validating the file's integrity.

Service Updates: Sometimes, updates to Azure services can introduce changes that affect existing pipelines.

Dependency Issues: Ensure that all dependencies and linked services are functioning correctly. Sometimes, issues in dependent services can cause failures in your pipeline.

You might also want to look into the Azure Synapse known issues and general troubleshooting guide for more insights
phemanth 13,905 Reputation points Microsoft Vendor

2025-02-18T17:49:33.25+00:00

@Siphe We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Accepted answer

Dizzy 75 Reputation points

2025-02-19T06:45:07.39+00:00

We recently experienced the same issue.

The spark job was running fine before and no changes have been made to it before the error suddenly occurred. While we were unable to find the cause of the issue, what did solve it for us was to use a different spark pool. The sole difference being their respective spark version (old: 3.2, new: 3.4).
Please sign in to rate this answer.
Siphe 25 Reputation points

2025-02-19T08:16:21.2066667+00:00

Yes, I did the exact same thing and it worked
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Error 1502

0 additional answers

Your answer