Uploading Large Files to SharePoint 2013 from .NET Using CSOM and REST

This is a topic that seems to come up with some frequency and when I needed to do it recently I could not find a good working sample of doing this from server-side code. The scenario here is imagine you want to upload some very large files to SharePoint via CSOM. You have some code running "somewhere" - could be in your own Web API controller or something similar to it that can tolerate a long upload time. The upload time could be long because you want to upload large files, i.e. larger than the 1.5MB support from CSOM alone when uploading files to SharePoint. In that case you need to use the REST interface into SharePoint. In pulling together the code to do this I didn't find a complete sample anywhere, but I managed to cobble together the code I wrote along with random pointers here and there from about four other TechNet articles. Blech! I feel like I should get some kind of finder's fee for figuring this out and pulling it together. But I digress... ;-) Trust me when I say you're probably going to want to bookmark this posting because I think you will find it handy.

So let me start I suppose by hitting up some of the main challenges I hit when doing this and how I tackled each one, and then I'll finish with a fairly complete code sample (i.e. it will work for you when you plug in your own file you want to upload). Here we go:

Challenge #1 - I Need an App Only Token and I'm using ACS

Virtually every sample I've ever seen for developing SharePoint apps using ACS (i.e. low trust) assume that there is some browser and user context present. Well if you are really doing this via some code running "somewhere" then chances are it may not be triggered directly from a browser request. Maybe it's a scheduled job, who knows, but for now let's just say that it was part of my scenario so I needed to solve it. The problem gets more complex when you are doing this from something like a WCF or REST solution because they don't expose an HTTP Context object. That means that you can't use the method in TokenHelper to get an access token because it requires the HTTP Context. The good news is there is another method in TokenHelper that we can use to get the access token that we'll use to talk to SharePoint. It looks like this:

//start out by getting a client context for o365

string o365ClientId = ConfigurationManager.AppSettings["o365ClientId"];

string o365ClientSecret = ConfigurationManager.AppSettings["o365ClientSecret"];

//get the Uri for the personal site

Uri siteUri = new Uri(siteUrl);

//this makes a connection to o365 using an App Only token since the user is not currently connected to o365

//NOTE: The 5 parameter version of this method is something custom I wrote, but not needed for the general

//use case; just use the standard 3 parameter version of this method that comes out of the box. Thanks!

var token = TokenHelper.GetAppOnlyAccessToken(TokenHelper.SharePointPrincipal, siteUri.Authority, TokenHelper.GetRealmFromTargetUrl(siteUri),

o365ClientId, o365ClientSecret).AccessToken;

So what I've done here is I had the ClientId and ClientSecret in my web.config for my SharePoint App anyways. For reasons that aren't important to this scenario, I copied those values into two new settings in web.config - one called o356ClientId and the other called o365ClientSecret. I take those values along with the Url to the site where I'm going to upload the file (which I put the siteUri variable) and let SharePoint and ACS go do their authentication oauth token thing. When I'm done I have an access token I can use for an App Only call into the SharePoint site.

Challenge #2 - I need to get a Form Digest Value

Almost all of the example code you finds assumes that you are in a SharePoint hosted app and you are running client side code. That's great because SharePoint emits a form digest value into every SharePoint page. However in our scenario we're not working with a SharePoint page so how do we get this? Well you can just take the siteUri you created above, and go make a POST to that site in the _api/contextinfo path and you will get a bunch of JSON back that includes a form digest value. You don't include any content when you make your POST request however. Here's an example of what that looks like:

HttpWebRequest restRqst = (HttpWebRequest)HttpWebRequest.Create(siteUrl + "_api/contextinfo");

restRqst.Method = "POST";

restRqst.Accept = "application/json;odata=verbose";

restRqst.Headers.Add("Authorization", "Bearer " + token);

restRqst.ContentLength = 0;

//get the response so we can read in the request digest value

HttpWebResponse restResponse = (HttpWebResponse)restRqst.GetResponse();

Stream postStream = restResponse.GetResponseStream();

StreamReader postReader = new StreamReader(postStream);

string results = postReader.ReadToEnd();

So you can see I use the access token I got in the previous chunk of code and use that in an authorization header to get access to the SharePoint site. I then make my 0 byte POST request and I get back the JSON string, which in my code above has been stuck in the "results" variable.

Challenge #3 - How Do I Dig out the Form Request Digest

This of course is not a huge problem, but I did come across an interesting solution so I'm sharing here. Typically when I work with JSON on the server side I'll design some classes into which I'll serialize the JSON so I have a nice object model that I can use to work with the results. Well some very sharp folks that know ASP.NET much better than me turned me on to a much simpler way of digging out that data than trying to do some string parsing. Here's what that code looks like:

JavaScriptSerializer jss = new JavaScriptSerializer();

var d = jss.Deserialize<dynamic>(results);

string xHeader = d["d"]["GetContextWebInformation"]["FormDigestValue"];

I really haven't deconstructed the "<dynamic>" feature enough to explain it well. For now I will just say "hey, look at my cool code sample!" The way I figured out the hierarchy to get down to FormDigestValue was just to look at the "d" variable in the debugger after it had been populated. It essentially has a series of Dictionary objects that it builds on the fly to map a pseudo object model the JSON that was consumed. So for example it had a Dictionary with a key of "d". The value for that item was another Dictionary that had a key of "GetContextWebInformation". And so on, and so on it goes, until it maps out the JSON that was returned. It's pretty cool.

Challenge #4 - Where In the Blazes Do I Upload by File

This may have actually been the most challenging aspect of all. Pretty much every single upload example I saw for using the REST endpoint tells you to upload it to "/_api/web/GetFolderByServerRelativeUrl('" + serverRelativeUrl + "')/Files/Add...blah... No joy. Never. Ever. To be clear, all I was trying to do was to upload the file into the Documents library in a user's OneDrive site. Wasted LOTS of time trying different tweaks around that Url and got no where. Finally found kind of a random sample in one of the TechNet docs that got me pointed to the correct location, so here is where I POST'ed my file upload: siteUrl + "_api/web/lists/getByTitle('Documents')/RootFolder/Files/Add(url='" + fileName + "', overwrite=false)". Yeah...get the list by title and then go into its RootFolder. Yahooo, order is restored in my world.

So...assuming you have a stream of data from somewhere (like opening a local file, taking a file that's been uploaded, etc.) and you've stuck it in a byte array (I put mine in fileBytes), here is the complete chunk of code to do an upload for completeness; apologies in advance for the absurd formatting this blog site will put on the code:

string uploadUrl = siteUrl + "_api/web/lists/getByTitle('Documents')/RootFolder/Files/Add(url='" + fileName + "', overwrite=false)";

bool fileUploadError = false;

 

//now that we have the Url and byte array, we can make a POST request to push the data to SharePoint

try

{

//need to get the X-RequestDigest first by doing an

//empty post to https://<site url>/_api/contextinfo

HttpWebRequest restRqst = (HttpWebRequest)HttpWebRequest.Create(siteUrl + "_api/contextinfo");

restRqst.Method = "POST";

restRqst.Accept = "application/json;odata=verbose";

restRqst.Headers.Add("Authorization", "Bearer " + token);

restRqst.ContentLength = 0;

 

//get the response so we can read in the request digest value

HttpWebResponse restResponse = (HttpWebResponse)restRqst.GetResponse();

Stream postStream = restResponse.GetResponseStream();

StreamReader postReader = new StreamReader(postStream);

string results = postReader.ReadToEnd();

 

//get the FormDigestValue node

JavaScriptSerializer jss = new JavaScriptSerializer();

var d = jss.Deserialize<dynamic>(results);

string xHeader = d["d"]["GetContextWebInformation"]["FormDigestValue"];

//now create a new request to do the actual upload

restRqst = (HttpWebRequest)HttpWebRequest.Create(uploadUrl);

restRqst.Method = "POST";

restRqst.Accept = "application/json;odata=verbose";

restRqst.Headers.Add("Authorization", "Bearer " + token);

restRqst.Headers.Add("X-RequestDigest", xHeader);

restRqst.ContentLength = fileBytes.Length;

 

//take the document and get it ready to upload

postStream = restRqst.GetRequestStream();

postStream.Write(fileBytes, 0, fileBytes.Length);

postStream.Close();

 

//do the upload

restResponse = (HttpWebResponse)restRqst.GetResponse();

 

//assuming it all works then we'll get a chunk of JSON back

//with a bunch of metadata about the file we just uploaded

postStream = restResponse.GetResponseStream();

postReader = new StreamReader(postStream);

results = postReader.ReadToEnd();

}

There you go, hope you can find a good use for this.

Comments

• Anonymous
  January 01, 2003
  The comment has been removed
• Anonymous
  January 01, 2003
  thanks for sharing.
• Anonymous
  August 02, 2014
  Great explanation. Thanks for sharing!
• Anonymous
  October 17, 2014
  The comment has been removed
• Anonymous
  December 31, 2014
  The comment has been removed
• Anonymous
  January 08, 2015
  m88 : http://m88en.com
  M88.com offer online sports games Asia, Sports Betting Asia, Sports Betting Sites Asia.
  m88asia : http://m88en.net
  Link to M88BET phone: m88en.com. – Register and Open Betting Account and Membership M88BET.
  m88bet : http://www.linkm88vip.com
  MANSION88 the house is one of the largest and most prestigious. Appeared quite early in the Asian market, the so-MANSION88 currently attracts more players.
  link m88 : http://m88wiki.com
  Home the M88 is the official sponsor of the football club in the Premier League
  Wish you happy with the new M88
  m88 casino online : http://m88free.com
• Anonymous
  January 27, 2015
  I absolutely love your blog and find nearly all of your post’s to be precisely what I’m looking for.
  http://www.hungrymind.in">Free Online Training, http://www.hungrymind.in">Online Training Courses, http://www.hungrymind.in">Free Online Training Videos
• Anonymous
  January 27, 2015
  Great article ...Thanks for your great information, the contents are quiet interesting. I will be waiting for your next post.
  http://staygreenacdemy.com">SharePoint 2013 Online Training in Hyderabad
• Anonymous
  January 27, 2015
  The comment has been removed