How to download multiple files concurrently using WebClient and the DownloadFileAsync method.

 

I had a case the other day where customer was using WebClient and the DownloadFileAsync method.

Customer then issued 10 concurrent calls to the DownloadFileAsync method and expected all the files to be downloaded at the same time.

However, only 2 files were concurrently downloading. 10 files were created on disk but only 2 were downloading.

When one file download was complete. The next would start and so on until all 10 files were downloaded.

 

I did some research but found no good explanation or documentation on why this happened or how to fix this.

So, here is the why and the how. I will do this with an example.

 

First place 10 files that will be downloaded on web server.

In my case I’ve simply called them File1.txt, File2.txt, etc. up to File10.txt and put them in a “Files” directory.

 

Create a new Windows Form Application and drop a button on it.

In the code behind, use the following code:

 

    publicpartialclassForm1 : Form

    {

        public Form1()

        {

            InitializeComponent();

        }

 

        privatevoid button1_Click(object sender, EventArgs e)

        {

            // Setting up source and destination

            // and creating a list of the file source and destination.

            string src = "https://<your web server>/Files/File";

            string dst = @"C:\Temp\Files\File";

            string extension = ".txt";

 

  List<FileItem> files = newList<FileItem>();

            for (int i = 1; i <= 10; i++)

            {

                files.Add(newFileItem { Source = newUri(src + i.ToString() + extension), Destination = dst + i.ToString() + extension });

     }

 

            // Just iterated through the list to issue 10 concurrent async file downloads

            foreach (FileItem fi in files)

            {

                WebClient client = newWebClient();

                client.DownloadFileAsync(fi.Source, fi.Destination);

            }

        }

 

        classFileItem

        {

            publicUri Source { get; set; }

            publicString Destination { get; set; }

        }

    }

 

Now create the destination directory (C:\Temp\Files\ in this case) if you haven’t do so yet.

Run the Win Form application but do not hit the button yet.

Open the destination directory in windows explorer (should be empty) and hit the button.

It should now looks as follows. All files are created but only 2 are downloading at the same time. The rest of the files have a size of 0.

Once one file is downloaded, another one will be downloading until all 10 are downloaded.

 

 

So, what we want is to have all 10 files to download concurrently. Not only 2.

The reason for only 2 files being downloaded concurrently is that this is the default number for outgoing connections in the System.Net / <connectionManagement> / maxconnection setting.

Since you only have 2 outgoing connections. There will be only 2 file downloads allowed at the same time. Async or not.

See more here:

 

<connectionManagement>

https://msdn.microsoft.com/en-us/library/fb6y0fyc.aspx

 

The solution is simply to up this number to, in this example, 10.

This is done like this in the example above.

Open the App.config file for the project and add the maxconnection setting like so:

 

<?xmlversion="1.0"encoding="utf-8" ?>

<configuration>

 

    <system.net>

      <connectionManagement>

        <addaddress = "*"maxconnection = "10" />

      </connectionManagement>

    </system.net>

 

</configuration>

 

In this case we allow 10 outgoing connections to any address (*). The Maxconnection value can be different for different endpoints. See the example in the documentation.

Now empty the destination directory and rerun the application. You should now have all 10 files downloading at the same time.

 

 

 

Hope this helps.

Comments

• Anonymous
  August 18, 2015
  WebClient is disposable and you make a loop of it without disposing!!!