Async Multi File Upload Control with Metadata support for Silverlight 2 (Beta 1)

By now a lot of you must be having a lot of fun of Silverlight 2 Beta 1. If you have not downloaded, installed and played with SL 2 Beta 1, I urge you do it right away - you are missing out!!! You can download SL2 here.

I have been working on a number of projects that are using the Silverlight 2 beta bits, and several of them needed file upload functionality. I looked around, and found a few controls, but while all of them did certain things admirably well, there were a few things missing.

• A few of the controls I found allowed you to select multiple files, and upload them, but all of them did so sequentially i.e. one file at a time. Parallel uploads were a necessity for my projects, with the extent of parallelization as a controllable parameter.
• None of them really allowed you to control how a large file gets chunked i.e. at what size boundaries. This can be an important parameterization to improve efficiency.
• Since most of my projects are media related, it is common for me to work on applications where a user is allowed to attach metadata to the file being uploaded. And metadata could vary from application to application. I needed a control that would allow some sort of metadata architecture, that is not hard wired to the control and pluggable by design. I wanted to reuse the same control in many apps, with the consuming apps defining the metadata UI and serialization process, and wiring it to the control through a standard interface.
• And lastly I also needed a completely templatable control.

Needless to say I could not find a combination of all of the above in one control - so I set out to do the next best thing - write one for my own. And the point of this entry is to share it with you. The code is zipped and attached.

First a little about the solution structure. You will find several projects in the solution:

• MFUControl - This the main control project containing most of the functionality behind the control. It also contains all of the control's UI.For those of you who are just starting out with SL 2, or have not played with WPF custom controls before, the default UI for a custom control is defined as a control template that gets applied to the control through a style defined in a file called generic.xaml - this file has to be named exactly this way for the runtime to recognize it(this is similar to WPF). Also, applying a UI to a custom control by way of a control template allows the consumer of the control to later apply other templates to change the L&F.
• ProgressBar - Since SL 2 Beta 1 does not have a ProgressBar control yet, and I needed one, I created one by extending the RangeBase control. Again a default control template is applied and can be found in the generic.xaml file in the project.
• MFUTestApp - This the test harness for the app - standard SL 2 app that has one page with the control on it.
• TestWeb - An ASP.Net web app that can be used to run MFUTestApp from IIS
• UploadService - This the HTTP backend to which the files are uploaded. I have chosen to use a WCF service, using a RESTful interface (implemented using the WCF Web Programming layer), with a single operation that gets invoked using an HTTP POST accepting a file stream.

I will attempt to sparingly explain some of the key pieces of the code here. I have commented the code pretty well - so hopefully in reading through the downloaded code most of it will be obvious to you.

But first things first - below is a screen shot of the control in action, uploading some videos. By the way, if some of you are designers and other kind of creative professionals, and are snickering at the UI, you will have to forgive the paucity of my artistic skills. I can code, but let it be known - I cannot design nice UI's :-).

So the key pieces of the code as I said :

• The actual upload functionality - The upload piece is pretty simple. I use the HttpWebRequest type to do the upload. Initialize it properly, plug in the file name in the HTTP headers, get the request stream, write the file content to it, and POST - that's about it. On the service side, it is even simpler - open the stream, get the file name from the HTTP headers, create a new file at a known location and write it out. This goes on for every file that is being uploaded.

• The parallelization/multi threaded upload - Unfortunately while SL 2 allows you to spin up your own threads, it does not allow making network calls from anywhere other than the UI thread(at least not yet). So I had to look for a way around. The good thing is that the network API's in SL 2 are all async, and use the standard Begin...()/End...() async invocation pattern established in .Net. In this pattern as soon as you make the egin..() call, control returns to your code and the actual unit of work initiated with the call gets automatically shifted to a background thread from the thread pool.

In my code I break up the entire list of files of chosen by the user into sublists based on the number of background threads the user desires to use (this is specified by the MaximumUploadThreads dependency property on the control) - one sublist for each thread. As an example if you had selected 14 files and specified 4 threads, the first 3 sublists each get 4 files, and the last one gets 2.

Once these lists are broken up, I loop through them, and call BeginGetRequestStream() once for the first file in each list. Control returns and only as many threads get employed as there are sublists, and upload starts. To ensure that at any point in time, no more threads get employed, I start uploading the next file in any sublist, once the previous upload for that sublist is completed i.e. initiate the next upload in the callback handler specified in the BeginGetResponseStream() method for the previous upload, which gets invoked once a response comes back from the service after the previous upload completes (or fails).

Note that upping the number of threads will allow you to achieve more parallelization, but at the cost of more background threads on the client.

• The chunking of large files - The control has another dependency property called FileChunkBoundaryInKiloBytes. What you specify is used as the chunking size boundary. For files that are smaller in size than the size specified here are uploaded in one fell swoop, and the progress bar goes straight from 0 to 100%. This is also because the HttpWebRequest has no mechanism for reporting true progress of the network activity :-(. However for files that are larger than the chunking size, are uploaded in chunk sizes as specified. the code follows a different route for those files, and all chunks of a file get uploaded in the same background thread as being used for the sublist to which the file belonged. An internal data structure named FileChunk is used to maintain stream positions, so that successive chunk uploads can be done accurately.

Progress is reported for each chunk, so you will see the progress bar report things somewhat more realistically here. Also additional HTTP headers are passed in to the backend, for each chunk, including the chunk #, total chunks and the chunk size, to facilitate the proper appending of chunks to the file as it gets built on the server with each POST of a chunk.

Note that upping the boundary may be a lucrative option for very large file uploads to achieve faster uploads, but this is client memory, and you can soon get OutOfMemory exceptions if you are not careful.

• The metadata piece - The assumption here is that the client (the app using the control) and the server has some well known contract for the metadata passed and expected respectively, and I needed a way for the control to facilitate the collection, packaging and uploading of that metadata, without itself being aware of any specifics of the metadata - that is the only way it would be reusable across all possible metadata requirements.

The metadata UI is supplied to the control by you defining a Data Template in your app code with the appropriate UI, and supplying it to the control through the FileMetadataTemplate dependency property.

You also need to implement a custom type to act as the data source for this data template. This type needs to implement an interface named IFileMetadataBase that I defined, and that has a single method named SerializeFileMetadata() returning a string. This method is where you supply your serialization logic (in the provided sample I use Json - but you could as easily use XML) , and as long as the serialized form is a string - we are good. This type also will typically implement the metadata elements as properties that you will bind to the appropriate portions of your metadata Data Template.

So how does all this get wired up ? At runtime if the control senses that a Data Template has been specified for metadata, it raises an event called ProvideMetadataInstance and you will need to handle this. The event arguments to this event has a single property of type IFileMetadataBase (the interface I mentioned above), and you set it to a new instance of the implementing type that you created - and voila!!. The control wires up the instance to the data template. When files are uploaded, it calls the function to serialize the metadata to a string, and passes it on as another additional, well named HTTP header, so that the backend can do whatever it wants with it.

I provide a sample of this in the code, with two metadata fields called "Comments" and "Description". Clicking the little button on each file entry, displays the data template which you can fill out.

• The backend - I implement this as a WCF service. But since the interface is simple (HTTP POST + Headers) - I am certain that you ASP.Net (or even other web platform) experts out there can reimplement it using other patterns.

Also the service now puts all uploaded files in a subfolder called Assets under the service directory. This is hardwired in the service code. Change it as you see fit, but do remember to change service code accordingly as well.

• There is also remove functionality in the control - you can check the little checkboxes, a Remove button appears, and you can then remove files from the list

There are a lot of things incomplete in the control:

• Slick UI
• There should be a ceiling enforced on both the chunking size and max thread count to prevent inadvertent memory outage/resource outage on the client
• The button (or whatever else) that shows the metadata, needs to be hidden when no data template is attached
• In future revs of SL 2, as network calls and background threads become more friendly, you can take better advantage of that
• Any other bugs/incomplete features you can find (of which there may be a ton)
• Slick UI (Have I said that already ??)

In any case my goal was to give you all something to play with and certainly improve upon. Let me know if this helped.

Remember to uninstall any older versions of SL on your machine, download and install the runtime , the SDK and the tools, and enjoy!!!

 

Download Link : https://cid-60bb739cbf5d117b.skydrive.live.com/self.aspx/Public/MultiFileUpload/MultiFileUpload.zip

Comments

• Anonymous
  March 29, 2008
  hi Jitghosh, very informative post! however your link to attachment doesn't work...

• Anonymous
  March 29, 2008
  Sorry for the broken link to attachment. It has now been replaced with a link to the code on my Windows Live SkyDrive

• Anonymous
  May 15, 2008
  Hi: I gave write permission on the asset folder and whne I click on upload button nothing happened and the file icon disappear.

• Anonymous
  May 15, 2008
  I search around and there seems to be a file missing called scratch.xaml, will that cause the app not working.

• Anonymous
  May 15, 2008
  No - that was just a temp file used to hold some work in progress XAML. Can you please debug the code and let me know if you see any exceptions ? And send me the exception stack ?

• Anonymous
  May 15, 2008
  Hi: Cannot even go into debug at all. The project file is all fouled up, it points to localhost/slbook/recepefactory which is not on my machine, so I have no clue on how to make it work if the project file is not clean.

• Anonymous
  May 15, 2008
  There are two web projects in the solution - one for the WCF service that acts as the upload backend, and another for testing the SL application. The project files reflect the virtual directory structure that I used on my machine.To package the web projects I would have had to use some structure or the other. If you do not like that structure, just create two web projects in the solution at URL's of your liking and copy all the files from my web projects to yours. Then open Page.XAML in the MFUTestApp and change the UploadUri appropriately to point to the WCF service endpoint. HTH, Jit

• Anonymous
  June 11, 2008
  Do you have plans to update this demo in Beta 2?  I was playing around with this demo a week ago and now I'm updated to Silverlight 2 Beta 2 and a lot of it doesn't port over ... for example the INotifyPropertyChanged doesn't seem to exist in Beta 2?!?!?! BTW great demo and great code! Thanks

• Anonymous
  July 09, 2008
  The comment has been removed

• Anonymous
  August 16, 2008
  The download link doesnt seem to work.

• Anonymous
  August 16, 2008
  Could you please fix the download link as I would really like to see how you do this for something I am having problems with in my project. Thanks and great article.

• Anonymous
  August 19, 2008
  Hi Looks VERY good! :) Could you please put the .zip file up again? The link seems to be dead :( I would very much like to see an beta2 version also :) //Mads

• Anonymous
  September 01, 2008
  Hi. Very good and useful control to use on the web. Gives quite a usability to the website. Though, the link seems to be dead. I tried the link multiple times at various intervals over the past 3 days....just to make sure the server issue...... but the file is not just found. Could you re-upload the zip file / source code please?

• Anonymous
  January 15, 2009
  Great information, thanks.  Please reestablish the link to the download.

• Anonymous
  March 22, 2009
  Do you still have to code available? The link is broken...