Compartir a través de

Are you getting OutOfMemoryExceptions when uploading large files?

  • Artículo

Problem:

Using the WebClient.Upload method for posting large files will eventually leave you stranded with OutOfMemoryExceptions.

Cause:

WebClient.Upload reads the entire file to memory by default.

Resolution:

Build your own uploader.

Scenario:

One of my customers was using WebClient.Upload in a Winforms application to transfer files to a webserver. The idea in itself was fine, but when they transferred a couple of large files they'd get OutOfMemoryExceptions. When uploading a 500 MB file the application would need approximately 520 MB of memory and if you uploaded a few large files after each other you quickly hit the roof. Running GC.Collect(); after each transfer didn't help. Judging from the number of hits on the internet for this scenario they weren't the only ones with this problem.

This is the code they were using:

WebClient oWeb = new WebClient();
oWeb.UploadFile("https://localhost/test.aspx", "c:\\bigfile.cab");

Okay, so why was this happening?
Well, first of all I wouldn't recommend running GC.Collect(); in any application. A lot has been written on this allready, but if you're interested in why I suggest you look at Rico Mariani's post on the subject. Anyway, for testing purposes we ran the following instead:

GC.Collect(3);

GC.WaitForPendingFinalizers();

GC.Collect(3);

And this cleared the memory. So why isn't this a valid solution? Well, like I said, I wouldn't recommend using GC.Collect(); in any application, and why read the entire file to memory when you can stream it? I looked up the UploadFile-method and it seems like it does read the entire file to a byte array before posting. This is great for smaller files, but in this particular scenario it wasn't too good. So what I did was to write my own uploader:

public static string MyUploader(string strFileToUpload, string strUrl)

{

    string strFileFormName = "file";

    Uri oUri = new Uri(strUrl);

    string strBoundary = "----------" + DateTime.Now.Ticks.ToString("x");

    // The trailing boundary string

    byte[] boundaryBytes = Encoding.ASCII.GetBytes("\r\n--" + strBoundary + "\r\n");

    // The post message header

    StringBuilder sb = new StringBuilder();

    sb.Append("--");

    sb.Append(strBoundary);

    sb.Append("\r\n");

    sb.Append("Content-Disposition: form-data; name=\"");

    sb.Append(strFileFormName);

    sb.Append("\"; filename=\"");

    sb.Append(Path.GetFileName(strFileToUpload));

    sb.Append("\"");

    sb.Append("\r\n");

    sb.Append("Content-Type: ");

    sb.Append("application/octet-stream");

    sb.Append("\r\n");

    sb.Append("\r\n");

    string strPostHeader = sb.ToString();

    byte[] postHeaderBytes = Encoding.UTF8.GetBytes(strPostHeader);

    // The WebRequest

    HttpWebRequest oWebrequest = (HttpWebRequest)WebRequest.Create(oUri);

    oWebrequest.ContentType = "multipart/form-data; boundary=" + strBoundary;

    oWebrequest.Method = "POST";

    // This is important, otherwise the whole file will be read to memory anyway...

    oWebrequest.AllowWriteStreamBuffering = false;

    // Get a FileStream and set the final properties of the WebRequest

    FileStream oFileStream = new FileStream(strFileToUpload, FileMode.Open, FileAccess.Read);

    long length = postHeaderBytes.Length + oFileStream.Length + boundaryBytes.Length;

    oWebrequest.ContentLength = length;

    Stream oRequestStream = oWebrequest.GetRequestStream();

    // Write the post header

    oRequestStream.Write(postHeaderBytes, 0, postHeaderBytes.Length);

    // Stream the file contents in small pieces (4096 bytes, max).

    byte[] buffer = new Byte[checked((uint)Math.Min(4096, (int)oFileStream.Length))];

    int bytesRead = 0;

    while ((bytesRead = oFileStream.Read(buffer, 0, buffer.Length)) != 0)

        oRequestStream.Write(buffer, 0, bytesRead);

   oFileStream.Close();

 

// Add the trailing boundary

    oRequestStream.Write(boundaryBytes, 0, boundaryBytes.Length);

    WebResponse oWResponse = oWebrequest.GetResponse();

    Stream s = oWResponse.GetResponseStream();

    StreamReader sr = new StreamReader(s);

String sReturnString = sr.ReadToEnd();

 

// Clean up

oFileStream.Close();

oRequestStream.Close();

s.Close();

sr.Close();

return sReturnString;

}

 

One of the things worth noting is that you need to set oWebrequest.AllowWriteStreamBuffering = false; Otherwise you will read the entire file to memory anyway. This is because the default behavior of the WebRequest is to buffer the entire request in case it needs to re-send it due to authentication, connectivity problems, etc. Again, this is a default behavior that normally is a performance boost, but in this case is a performance killer.

 

So what was the end result?

During my first test runs the application needed as much memory as the file I was trying to upload, and then some. So in order to upload a 500 MB .cab-file the application needed at least 520 MB. The application using the custom uploader never went above 23 MB.

End of transmission

/ Johan

Comments

  • Anonymous
    December 23, 2006
    Thanks bunch! Hungry for more!!!! Please keep writing :) Rahul

  • Anonymous
    December 23, 2006
    Getting OutOfMemoryExceptions when uploading large files is a very common problem and we troubleshoot

  • Anonymous
    January 12, 2007
    Hey Johan, nice post, but you should complete your code with Closing and Disposing calls.   Is a common mistake in many developers that they dont call those methods when a type has them, specially those related with file and streams. bye

  • Anonymous
    January 14, 2007
    Hi Patrick, You're absolutely, 100% right. I can't believe I missed this. The code should be fine now. I keep telling my customers the importance of calling close and dispose and fail to do this myself in my sample code. - Well, I think I have the topic for my next post. :) Thanks for the heads up! / Johan

  • Anonymous
    March 21, 2007
    Can you also list the code for the .aspx page that this uploader would post to and any server-side configurations that should be considered? for example, the maxRequestLength or anything else... Thanks! Already a great help!

  • Anonymous
    March 29, 2007
    Hi Mike, The file is uploaded "the normal way", so you deal with it as you would with any file uploaded through <input type="file> For more info on how to do that, as well as what to take into consideration regarding file size, please take a look at http://support.microsoft.com/kb/323245 / Johan

  • Anonymous
    May 29, 2007
    Hi Johan I tried your "MyUploader" code to upload files. Its going thru all the procedure. There is no error, but it is also not uploading the file!. Am i missing something? Do i need to configure something?

  • Anonymous
    May 29, 2007
    Hi Gagan, The name of the file is simply "file", so if you just change that, you should be able to use the code from http://support.microsoft.com/kb/323245 Using Request.Files(0) should work fine as well. Additional troubleshooting tips: Try setting a breakpoint in the code you're using to receive the file. Add a watch on "Request.Files". Take a look at the values under the "AllKeys"-property. You should see a key named "file". It would also be interesting to add a watch on Request.Files(0). It's length should definitively be > 0. Let me know if this works out or if you need additional help. / Johan

  • Anonymous
    October 11, 2007
    Thanks! You're a savior. I had already written code to upload my file piecemeal, but I missed the mysterious AllowWriteStreamBuffering property (which sounds like a good thing). Easiest bug fix ever! Thanks again.

  • Anonymous
    October 24, 2007
    The comment has been removed

  • Anonymous
    October 24, 2007
    well sorry for the double post i found the solution: changing the oWebrequest.Timeout value :)

  • Anonymous
    November 12, 2007
    The comment has been removed

  • Anonymous
    April 03, 2008
    I'm getting an error when trying to implement this code:     This request requires buffering data to succeed Google search says that this is an anonymous authentication problem with IIS... but I am using a secured page on an Apache server. Any idea?

  • Anonymous
    April 03, 2008
    Hi Joe, I'm afraid I don't know much about Apache. I'd try the standard troubleshooting stuff, such as:

  • Attempt to contact a page using anonymous access

  • Try a different web server, IIS, etc. with both anonymous and authenticated requests

  • Run Netmon / Fiddler2 in each scenario and compare the logs / Johan

  • Anonymous
    June 05, 2008
    Hi I tried to use your method , but why i get back on the string is the webpage i put in the uri, not textfile i am trying to upload, am i doing anything wrong? John

  • Anonymous
    June 08, 2008
    Hi John, sReturnString will contain whatever the webpage you're uploading to decides to return in response. / Johan

  • Anonymous
    June 11, 2008
    The comment has been removed

  • Anonymous
    June 12, 2008
    OK, I fixed the issues today, now I can upload a large file (up to 1G), here is my update code:  // keep session for a long time, without timeout   req.KeepAlive = false;   req.Timeout = Timeout.Infinite;   //use protocol version 10 instead of 11 by default   req.ProtocolVersion = HttpVersion.Version10; Hope this help for anyone else have same problem with me Thnks, Tony

  • Anonymous
    August 14, 2008
    hi, great article, but what if want to upload more than one file??

  • Anonymous
    August 15, 2008
    Hi Roey, I'm not sure I follow? What would prevent you from calling the function a second time? / Johan

  • Anonymous
    February 16, 2009
    Thank you for blogging about the AllowWriteStreamBuffering property.

  • Anonymous
    March 09, 2009
    According to RFC 1867 the trailing boundary MUST end with -- . Otherwise the upload will fail on certain web servers. Replace: byte[] boundaryBytes = Encoding.ASCII.GetBytes("rn--" + strBoundary + "rn"); by: byte[] boundaryBytes = Encoding.ASCII.GetBytes("rn--" + strBoundary + "--rn"); Good job otherwise.

  • Anonymous
    March 26, 2009
    Everyone seems to post only one file at a time. The challenge I have is streaming multiple files...  can this be done?

  • Anonymous
    March 29, 2009
    @Jose: Well, like I've said before: What prevents you from calling the function a second time? / Johan

  • Anonymous
    April 22, 2009
    Hi, Nice Code ! What can be done in case your internet is gone while oRequestStream.Write(buffer, 0, bytesRead); i have a try catch, so it goes to catch with the exception "Unable to read data from the transport connection". Then Let's suppose that i have a Thread.Sleep for 5 minutes, during that time the internet is back and i want that continues writing where it was, but i continue getting the exception. Thanks, Fernando

  • Anonymous
    April 22, 2009
    @Fernando: In that case the connection has been closed, so you'd need to restart. To avoid having to restart completely you'd have to write your own method for dividing the transfer into suitable packets which you would then assemble into the final file. / Johan

  • Anonymous
    April 23, 2009
    @Johan: Thanks for the answer, i was looking for a way to avoid that solution but looks like it can't. I really appreciate your help ! Thanks, Fernando.

  • Anonymous
    June 13, 2009
    話題の小向美奈子ストリップを隠し撮り！入念なボディチェックをすり抜けて超小型カメラで撮影した神動画がアップ中！期間限定配信の衝撃的映像を見逃すな

  • Anonymous
    August 06, 2009
    Thanks for your findings. This helped my team.

  • Anonymous
    December 08, 2009
    Thank you! Because of System.OverflowException uploading of large files I replaced byte[] buffer = new Byte[checked((uint)Math.Min(4096, (int)oFileStream.Length))]; by: byte[] buffer = new Byte[checked((ulong)Math.Min(4096, (long)oFileStream.Length))];

  • Anonymous
    December 30, 2010
    Hi there Johan: I tried your code but I'm getting this error on IIS Windows XP SP2: "This request requires buffering data to succeed" Any ideas? When I set AllowWriteStreamBuffering to true it works, but that's not the idea right?

  • Anonymous
    December 30, 2010
    I found the problem, Authentication was required, now it's ok.

  • Anonymous
    March 20, 2012
    The comment has been removed

  • Anonymous
    January 08, 2013
    Very nice post, thank you. I have a question about the server side: Can you list a code example for the .aspx page that this uploader would post to? (I don't know how to handle this, the link support.microsoft.com/.../323245 does not work anymore) Thank you!

  • Anonymous
    January 08, 2013
    On the server side, I have an aspx file but this doesn't work in combination with your method. Do you have a piece of server code illustrating the way to do it. Protected Sub Page_Load(ByVal sender As Object, ByVal e As System.EventArgs) Handles Me.Load        Dim f As String        Dim file        Dim SaveAsPath As String        For Each f In Request.Files.AllKeys            file = Request.Files(f)            SaveAsPath = Server.MapPath("uploadedfiles/") & file.FileName            'file.SaveAs("c:inetpubtestUploadedFiles" & file.FileName)            file.SaveAs(SaveAsPath)        Next f    End Sub

  • Anonymous
    May 01, 2013
    Hi, Can I get a little help here, I am getting the the below exception {System.Net.WebException: The remote server returned an error: (405) Method Not Allowed.   at System.Net.HttpWebRequest.GetResponse() It happens when i try to call the below method System.Net.WebResponse oWResponse = oWebrequest.GetResponse();

  • Anonymous
    October 17, 2013
    While using below code byte[] bytes= new Byte[checked((uint)Math.Min(4096, (int)fls.Length))]; Is there any chance of loosing the data.?  because if we upload file of 5 mb it will only save the 4096 bytes.

  • Anonymous
    October 27, 2013
    I get the following exception while uploading a 2 MB file to a SharePoint site "Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host." Please help me.....