Receiving Duplicate Files

Question

We have been struggling with the issue of receiving duplicate files for several months now.It's not a constant problem. We encounter it once or twice a week.

 

We have around 4000 SFTP Receive Locations. All connect to the same SFTP server but different directories.

 Our 4000 customers upload their payment files to their assigned folders. As SFTP adapter we use NSoftware Biztalk SFTP.

 

It seems there is a  10 minutes gap between downloading the file and Biztalk approves the batch. As far as I understood, Nsoftware adapter downloads the file and conveys it to Biztalk. While waiting for confirmation, the SFTP server closes the connection.At this point , SFTP assumes the batch is failed and tries to download it again. We have a custom pipeline component at the beginning that writes a startup log. There is a  10 minutes GAP between downloading the file and the execution of the pipeline component.

 

A comment from the NSoftware support.

The delay in this case happens between submitting the messages to BizTalk and starting cleanup: there are log entries saying we’ve submitting the messages to BizTalk, then a minute later a log entry saying the server has disconnected, then 9 minutes later a message saying the adapter is starting the cleanup process. >During the gap, we are simply waiting for BizTalk to notify us that the message batch is complete, and have no information on what it’s doing: It could be that the receive pipeline is taking 10 minutes to run, or it could be somehow introduced by BizTalk itself.