Streaming over HTTP with WCF

Recently I had a customer email me looking for information on how to send and receive large files with a WCF HTTP service. WCF supports streaming specifically for these types of scenarios.  Basically with streaming support you can create a service operation which receives a stream as it’s incoming parameter and returns a stream as it’s return value (a few other types like Message and anything implementing IXmlSerializable are also supported). MSDN describes how streaming in WCF works here, and how to implement it here. There’s a few gotchas however if you are dealing with sending large content with a service that is hosted in ASP.NET. If you scour the web you can find the answers, such as in the comments here.

In this post I’ll bring everything together and walk you through building a service exposed over HTTP and which uses WCF streaming. I’ll also touch on supporting file uploads with ASP.NET MVC, something I am sure many are familiar with. The sample which we will discuss requires .NET 4.0 and ASP.NET MVC 3 RC. If you don’t have MVC you can skip right to the section “Enabling streaming in WCF”. Also it’s very easy to adopt the code to work for web forms.

The scenario

For the sample we’re going to use a document store. To keep things simple and stay focused on the streaming, the store allows you to do two things, post documents and retrieve them through over HTTP. Exposing over HTTP means I can use multiple clients/devices to talk to the repo.  Here are more detailed requirements.

1. A user can POST documents to the repository with the uri indicating the location where the document will be stored using the uri format “https://localhost:8000/documents/{file}”. File in this case can include folder information, for example the following is a valid uri, “https://localhost:8000/documents/pictures/lighthouse.jpg”.

Below is what the full request looks like in Fiddler. Note: You’ll notice that the uri (and several below) has a “.” in it after localhost, this is a trick to get Fiddler to pick up the request as I am running in Cassini (Visual Studio’s web server).

And the response

2. On a POST, the server creates the necessary folder structure to support the POST. This means if the Pictures sub-folder does not exist it will get created.

3. A user can GET a document from the repository using the same uri format for the POST. Below is the request for retrieving the document we just posted.

And the response: (To keep thing simple I am not wrestling with setting the appropriate content type thus application/octet-stream is the default).

4. The last requirement was I needed a simple front end for uploading the file.  For this I decided to use ASP.NET MVC3 to create a really simple front end. That also gave me a chance to dip my feet in the new Razor syntax.

Creating the application

The first thing I did was create a new MVC 3 application in Visual Studio. I called the application “DocumentStore” and selected to create an empty application when prompted.

Next thing I did was was add a HomeController and a new view Index.cshtml which I put in the Views/Home folder. The view which is below allows me to upload a document and specify the folder. I did tell you it was simple right?

Next I added an Upload view also in the View/Home folder. That view show she result of the upload.

Then in the HomeController I implemented the Upload action which contains the logic for POSTing to the service. I used HttpClient form the Rest Starter Kit to do the actual post.

Here’s the flow

• Grab the uploaded file and the path.
• Create the uri for the resource I am going to post.
• Post the resource using HttpClient.
• Set the filename info to return in the view.

Up until this point I’ve simply created the front end for uploading the document. Now we can get to the streaming part.

Enabling streaming over HTTP

Streaming is enabled in WCF through the transferMode property on the binding. In this case I set transferMode to “Streamed” as I want to support it bi-directionally. The setting accepts other values which can be used to enable streaming on the request OR the response. I also set the maxReceivedMessageSize to the maximum size that I want to allow for transfers. Finally I also also set the maxBufferSize, though this is not required. WCF 4.0 introduced default bindings which greatly simplifies this configuration. In this case because I’m going to use WCF HTTP, I can use the standard webHttpBinding as is shown.

If you read the MSDN articles I cited above, you might think I’m done, however because I’m hosting our service on ASP.NET, there’s one more critical setting I need to mess with. This one definitely threw me for a loop.

ASP.NET doesn’t know about WCF and it has it’s own limits for the request size which I have now increased to match my limit. Without setting this the request will be killed if it exceeds the limits. This setting and the previous settings have to kept in sync. If not, either ASP.NET will kill the request, or WCF will return a Status 400: Invalid Request. Service tracing is definitely your friend in figuring out any issues on the WCF site.

Building the DocumentsResource

Now that streaming is enabled, I can build the resource. First I created a DocumentsResource class which I marked as a ServiceContract and configured to allow hosting in ASP.NET.

 

Registering the resource through routes (.NET 4.0 Goodness!)

Because I am using .NET 4.0, I can take advantage of the new ASP.NET routing integration to register my new resource without needing an .SVC file. To do this, I create a ServiceRoute in the Global.asax specifying the base uri and the service type (resource).

Document retrieval (GET)

Our document store has to allow retrieval of documents. Below is the implementation.

As I mentioned earlier, when you enable streaming your operations only work with a limited set of input / return values. In this case my GET method returns a stream which represents the document. Notice the method is has no parameters. When building a streamed service, the method can only have a single parameter which is either of type stream, message or which implements IXmlSerializable. This forces me to jump through a small hoop to get access to the path info for the document, but it’s manageable.

Here’s the logic in the method.

• The method is annotated with a WebGetAttribute to indicate that it is a GET. The UriTemplate is set to “*” in order to match against any uri that matches the base uri for this service.
• Next I grab all the uri segments and concatenate in order to create the relative file location. For example for the uri “https://localhost:8080/documents/pictures/lighthouse.jpg” this will result in “pictures\lighthouse.jpg” assuming the base address is “documents'”.
• If a file path was actually passed in, I then create a the relative file path by concatenating the relative location with the server root and the relative path for the doc store.
• Finally I create a stream pointing to the file on disk and I return it. The resulting stream is then returned back to the client.

Document upload (POST)

Whereas in the Get method we return a stream, in the POST case we simply accept a stream. In this case we need to create the directory if it does not exist and write the document to disk so there’s a bit more work.

Here’s the logic:

• The method is annotated with WebInvoke with the method set to POST. Similar to the GET, the UriTemplate is set to “*” to match everything.
• Next I grab on to the uri of the request so that I can return it later as part of the Location header in the response. This is appropriate according to the HTTP spec. It’s important that I grab it before I actually stream the file as I found if I wait till after, the request is disposed and I get an exception.
• Similar to the GET I also grab the path segments to create the file name and create the local path.
• Next I check to see if the directory that was requested actually exists. If it does not, I create it.
• I create a stream to write the file to disc and use the convenient CopyTo method to copy the incoming stream to it.
• Finally I set the status code to Created and set the location header.

Testing out the app

That’s it, the app is done. Now all there is to do is launch it and see the streaming in action as well.

Like I said, simple UI. . Go press “Browse” and then grab an image or a video. In this case I’ll grab the sample wildlife.wmv included in win7.

Next enter “Videos” for the folder. You can also put a deep hierarchy if you want like Foo/Bar/Baz.

Select Upload and you will see the following once it completes.

Now that the file is uploaded, it can be retrieved from the browser.

Which in this case will cause Windows media player to launch and show our video,….w00t!

Now that our document store is in place, we can use different clients to access it. In this example we used a browser / MVC application to keep things simple. The same store can also be accessed from a mobile device line an IPad,, a Silverlight application, or from a desktop application. All it needs is a way to talk HTTP.

Why not just MVC?

I know some folks are going to see this and say “why didn’t you just use MVC?'”.  I could have accomplished the upload / retrieval completely through MVC. The main point of this post however was to show how you can use streaming with WCF. Let’s move on

Get the code here: https://cid-f8b2fd72406fb218.office.live.com/self.aspx/blog/DocumentStore.zip

Comments

• Anonymous
  November 24, 2010
  Hi, all in all a nice post. Always nice to learn something at the start of your day. I have only one suggestion, and that is to put a little more security thought into the sample code. I'm shocked at the path traversal vulnerabilities in the post/get methods. Yes, I understand its sample code. but still, this code will probably be used verbatim by someone, somewhere. A little bit more care will make the web a safer place. Right?

• Anonymous
  November 25, 2010
  Hi Rjvdboon Thanks for the feedback. Yes the code is insecure, as the focus of the sample is on streaming, not on how to build a secure application. I would hope that anyone who looks at ANY code on the internet takes the time to evaluate the appropriate way to apply the concepts being illustrated to their environment. For example if this service was used behind the firewall, the security concerns might be far less than if it was exposed across the internet. I will add a disclaimer to the post though emphasizing that this is not secure code. Glenn

• Anonymous
  November 28, 2010
  Can we use MTOM message encoder along with specified config section. I believe it helps to boost the file transfer...

• Anonymous
  November 30, 2010
  Nice writeup. I built something similar a few years ago, but had to abandon the WCF download part due to a shortcoming/bug in HttpTransportBinding's handling of Content-Length with streamed responses. It forces all streamed downloads to use Transfer-Coding: Chunked even when the content length is known (as in this case), causing a truncated download to "poison" any caches and preventing definite progress monitoring of large downloads. Connect issue is at connect.microsoft.com/.../support-non-chunked-streamed-responses-on-wcf-httptransport. Nobody inside seems to think this is worth fixing, but I'd argue that it makes streamed binary responses over HTTP nearly unusable (safely, anyway).

• Anonymous
  December 03, 2010
  Matt, thanks for the feedback. I will follow up on this.

• Anonymous
  December 20, 2010
  Updates - something must have changed with the MVC 3 RC2. Note, the View and ViewModel (dynamic) are not availabe to me.  I had to change things to use the ViewBag, and created a ViewModel. No biggie, but someone else might have an issue. Also, can you point to the REST toolkit you used. I found two different ones and had some issues there when trying to recreatet your project from scratch. Thanks !

• Anonymous
  December 21, 2010
  Yes, the MVC3 bits were recently released with changes to the api that you noted. Which issues with RSK? You can get the binaries in our new codeplex site at wcf.codeplex.com here: wcf.codeplex.com/.../57702. The RSK binaries are in the Lib folder.