ErrorCode=AdlsGen2OperationFailed...Failure happened on 'Sink' side.

Question

Getting the below error while copying the data to ADLS from SAP Hana, however this error is random to tables failing daily.

When I try load each of the failed tables from the same source and with the same file name then there is no issue and it works as expected, but it fails while the data pipelines are running as per the schedule in morning.

Failure happened on 'Sink' side. ErrorCode=AdlsGen2OperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=ADLS Gen2 operation failed for: Error while copying content to a stream.. Account: 'stedhenis01landrun'. FileSystem: 'harnes-oracle'. Path: 'abc/2025/01/25/ABC_XYZ.parquet'..,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Net.Http.HttpRequestException,Message=Error while copying content to a stream.,Source=mscorlib,''Type=System.IO.IOException,Message=Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.,Source=System,''Type=System.Net.Sockets.SocketException,Message=An existing connection was forcibly closed by the remote host,Source=System,'

Any quick response must be appreciated as we are not able to provide the latest data to the end users.

Answer 1

Hi ,

Thanks for reaching out to Microsoft Q&A.

The error "AdlsGen2OperationFailed... Failure happened on 'Sink' side" typically occurs due to connectivity or throttling issues between the source (SAP HANA) and the sink (adls gen2).

Possible Causes:

1. The connection to ADLS Gen2 might be temporarily interrupted or throttled due to heavy traffic or resource contention during peak hours. Scheduled pipeline runs during high usage periods may cause resource strain.
2. The pipelines may have high concurrency settings or are processing too many tables simultaneously, leading to contention for resources.
3. The SAP HANA source or ADLS Gen2 sink might experience intermittent failures or maintenance-related issues.
4. Errors like "An existing connection was forcibly closed by the remote host" indicate potential issues with the underlying network or socket connection.

Recommended Actions:

Enable Retry Logic:

• Configure retry settings in the pipeline activities to handle transient failures automatically.
  • Increase the retry count and set a reasonable retry interval in the activity settings.
  Adjust Pipeline Concurrency:

  - Reduce the concurrency or parallelism of the pipeline to minimize resource contention during execution.
  

1. Optimize Resource Utilization:
   • If using Integration Runtime (IR), ensure it has sufficient capacity and is not overloaded during peak hours. Scale up the IR temporarily to handle morning loads more effectively.
   Review ADLS Gen2 Throttling Limits:
   • Check Azure Monitor or ADLS Gen2 metrics for throttling events.
     • If throttling is observed, consider increasing ADLS Gen2's performance tier or optimizing the data volume and frequency.
2. Validate Network Connectivity:
   • Run a network diagnostic to ensure there are no packet losses or connection drops between SAP HANA and ADLS Gen2. Consider setting up a SHIR close to SAP HANA for better connectivity.
3. Monitor and Debug Specific Failures:
   • Capture logs of failed tables and analyze if there are patterns (file size, specific tables).
   • Test failed table loads outside of peak hours to see if the issue persists.

Check File Naming or Overwrite Conflicts:

• Ensure there are no file overwrite conflicts caused by concurrent table writes. If necessary, implement unique file naming conventions or transactional handling for writes.

Quick Fix for Immediate Resolution:

• Since manual reruns are succeeding, schedule a retry or reprocess the failed tables immediately.
• Shift pipeline schedules to less busy periods, if feasible, to avoid high contention during peak hours.

Please feel free to click the 'Upvote' (Thumbs-up) button and 'Accept as Answer'. This helps the community by allowing others with similar queries to easily find the solution.