Making SharePoint Apps Scale with Azure Redis Cache

This post will show how to create custom classes for a SharePoint 2013 app that enable the use of Azure Redis Cache.

Background

Often times there are web application design constraints that require use of session state.  From Scott Guthrie’s book:

It's often not practical in a real-world cloud app to avoid storing some form of state for a user session, but some approaches impact performance and scalability more than others. If you have to store state, the best solution is to keep the amount of state small and store it in cookies. If that isn't feasible, the next best solution is to use ASP.NET session state with a provider for distributed, in-memory cache. (See “ASP.NET session state using a cache provider,” in Chapter 12, “Distributed Caching.”) The worst solution from a performance and scalability standpoint is to use a database-backed session state provider.

[page 56 of Building Cloud Apps with Microsoft Azure: Best practices for DevOps, data storage, high availability, and more , by Scott Guthrie, Mark Simms, Tom Dykstra, Rick Anderson, and Mike Wasson]

Azure Websites make a very compelling target for provider-hosted apps that target Office 365.  The default code generated by Visual Studio for SharePoint apps using the SharePointContext class require ASP.NET session state, and the default is to use InProc mode for session state.  Of course that’s not going to work when we scale our website to more than 1 instance.  To achieve scale, I thought I would use Azure Redis Cache to store session state data (instead of a database as per the above quote).  I followed the instructions here and then received this error:

Type 'KirkeDemoWeb.SharePointContext' in Assembly 'KirkeDemoWeb, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' is not marked as serializable.

OK, fair enough, I’ll just mark is with the SerializableAttribute.  Then I hit this error:

Type 'Microsoft.IdentityModel.S2S.Tokens.JsonWebSecurityToken' in Assembly 'Microsoft.IdentityModel.Extensions, Version=2.0.0.0, Culture=neutral, PublicKeyToken=69c3241e6f0468ca' is not marked as serializable.

Since I don’t have the source code to that class, I can’t mark it as serializable.  The rest of this post shows how to solve the problem by creating a new SharePointContext implementation that is serializable.

The Problem

When you generate a new App for SharePoint project in Visual Studio, the code will store sensitive information in session data, namely the OAuth refresh and access tokens that should not be stored on the client.  For this reason, the SharePoint context token provides a CacheKey claim in the context token that is used to safely store in an HTTP cookie while the corresponding data can be stored in session state, retrieved by using the CacheKey.  The out-of-box code generated by Visual Studio uses a provider model to save the refresh and access tokens to session state, reducing the number of round trips to obtain an OAuth token. 

The use of session state here is appropriate, but as we saw before we hit the error that SharePointContextToken is not marked serializable.  The reason why is highlighted in the class diagram.

The solution is to create a new SharePointContext implementation that does not have the contextTokenObj (of type SharePointContextToken) field. If we go into the source and find all references to the contextTokenObj field, we can get a better sense of how it is being used:

The lines highlighted in the green box are simply checking the ValidTo property to determine when the context token is no longer valid.  That’s easy enough to replace.  The code in the red box obtains an access token.  It turns out that TokenHelper makes replacing this code easy as well.

SharePointAcsSerializableContext

The first part of the solution is to create a new class that derives from SharePointContext.  Instead of using the SharePointContextToken class, we simply capture the required data in fields within our class.  You can see the difference in this class diagram, where we take the data that was stored in the SharePointContextToken class and store it within fields in our class.  All of the fields are native .NET types, meaning they are all serializable.

The implementation is provided in full.

Serializable SharePointContext

1. [Serializable]
2. publicclassSharePointAcsSerializableContext : SharePointContext
3. {
4.     privatereadonlystring _contextToken;
5.     privateDateTime? _contextTokenValidTo;
6.  
7.     privatestring _refreshToken;
8.     privatestring _targetPrincipalName;
9.     privatestring _targetRealm;
10.     privatestring _cacheKey;
11.  
12.  
13.     privateDateTime ContextTokenValidTo
14.     {
15.         get
16.         {
17.             returnthis._contextTokenValidTo == null ? DateTime.MinValue : this._contextTokenValidTo.Value;
18.         }
19.     }
20.  
21.     ///<summary>
22.     /// The context token.
23.     ///</summary>
24.     publicstring ContextToken
25.     {
26.         get { return ContextTokenValidTo > DateTime.UtcNow ? this._contextToken : null; }
27.     }
28.  
29.     ///<summary>
30.     /// The context token's "CacheKey" claim.
31.     ///</summary>
32.     publicstring CacheKey
33.     {
34.         get { return ContextTokenValidTo > DateTime.UtcNow ? this._cacheKey : null; }
35.     }
36.  
37.     ///<summary>
38.     /// The context token's "refreshtoken" claim.
39.     ///</summary>
40.     publicstring RefreshToken
41.     {
42.         get { return ContextTokenValidTo > DateTime.UtcNow ? this._refreshToken : null; }
43.     }
44.  
45.     publicoverridestring UserAccessTokenForSPHost
46.     {
47.         get
48.         {
49.             return GetAccessTokenString(refthis.userAccessTokenForSPHost,
50.                                         () => TokenHelper.GetAccessToken(this._refreshToken, this._targetPrincipalName, this.SPHostUrl.Authority, this._targetRealm));
51.         }
52.     }
53.  
54.     publicoverridestring UserAccessTokenForSPAppWeb
55.     {
56.         get
57.         {
58.             if (this.SPAppWebUrl == null)
59.             {
60.                 returnnull;
61.             }
62.  
63.             return GetAccessTokenString(refthis.userAccessTokenForSPAppWeb,
64.                                         () => TokenHelper.GetAccessToken(this._refreshToken, this._targetPrincipalName, this.SPAppWebUrl.Authority, this._targetRealm));
65.         }
66.     }
67.  
68.     publicoverridestring AppOnlyAccessTokenForSPHost
69.     {
70.         get
71.         {
72.             return GetAccessTokenString(refthis.appOnlyAccessTokenForSPHost,
73.                                         () => TokenHelper.GetAppOnlyAccessToken(TokenHelper.SharePointPrincipal, this.SPHostUrl.Authority, TokenHelper.GetRealmFromTargetUrl(this.SPHostUrl)));
74.         }
75.     }
76.  
77.     publicoverridestring AppOnlyAccessTokenForSPAppWeb
78.     {
79.         get
80.         {
81.             if (this.SPAppWebUrl == null)
82.             {
83.                 returnnull;
84.             }
85.  
86.             return GetAccessTokenString(refthis.appOnlyAccessTokenForSPAppWeb,
87.                                         () => TokenHelper.GetAppOnlyAccessToken(TokenHelper.SharePointPrincipal, this.SPAppWebUrl.Authority, TokenHelper.GetRealmFromTargetUrl(this.SPAppWebUrl)));
88.         }
89.     }
90.  
91.     public SharePointAcsSerializableContext(Uri spHostUrl, Uri spAppWebUrl, string spLanguage, string spClientTag, string spProductNumber, string contextToken, SharePointContextToken contextTokenObj)
92.         : base(spHostUrl, spAppWebUrl, spLanguage, spClientTag, spProductNumber)
93.     {
94.         if (string.IsNullOrEmpty(contextToken))
95.         {
96.             thrownewArgumentNullException("contextToken");
97.         }
98.  
99.         if (contextTokenObj == null)
100.         {
101.             thrownewArgumentNullException("contextTokenObj");
102.         }
103.  
104.         this._contextToken = contextToken;
105.  
106.         this._refreshToken = contextTokenObj.RefreshToken;
107.         this._targetPrincipalName = contextTokenObj.TargetPrincipalName;
108.         this._targetRealm = contextTokenObj.Realm;
109.         this._contextTokenValidTo = contextTokenObj.ValidTo;
110.         this._cacheKey = contextTokenObj.CacheKey;
111.     }
112.  
113.     ///<summary>
114.     /// Ensures the access token is valid and returns it.
115.     ///</summary>
116.     ///<param name="accessToken">The access token to verify.</param>
117.     ///<param name="tokenRenewalHandler">The token renewal handler.</param>
118.     ///<returns>The access token string.</returns>
119.     privatestaticstring GetAccessTokenString(refTuple<string, DateTime> accessToken, Func<OAuth2AccessTokenResponse> tokenRenewalHandler)
120.     {
121.         RenewAccessTokenIfNeeded(ref accessToken, tokenRenewalHandler);
122.  
123.         return IsAccessTokenValid(accessToken) ? accessToken.Item1 : null;
124.     }
125.  
126.     ///<summary>
127.     /// Renews the access token if it is not valid.
128.     ///</summary>
129.     ///<param name="accessToken">The access token to renew.</param>
130.     ///<param name="tokenRenewalHandler">The token renewal handler.</param>
131.     privatestaticvoid RenewAccessTokenIfNeeded(refTuple<string, DateTime> accessToken, Func<OAuth2AccessTokenResponse> tokenRenewalHandler)
132.     {
133.         if (IsAccessTokenValid(accessToken))
134.         {
135.             return;
136.         }
137.  
138.         try
139.         {
140.             OAuth2AccessTokenResponse oAuth2AccessTokenResponse = tokenRenewalHandler();
141.  
142.             DateTime expiresOn = oAuth2AccessTokenResponse.ExpiresOn;
143.  
144.             if ((expiresOn - oAuth2AccessTokenResponse.NotBefore) > AccessTokenLifetimeTolerance)
145.             {
146.                 // Make the access token get renewed a bit earlier than the time when it expires
147.                 // so that the calls to SharePoint with it will have enough time to complete successfully.
148.                 expiresOn -= AccessTokenLifetimeTolerance;
149.             }
150.  
151.             accessToken = Tuple.Create(oAuth2AccessTokenResponse.AccessToken, expiresOn);
152.         }
153.         catch (WebException)
154.         {
155.         }
156.     }
157. }

For the most part, this is the same implementation as the out of box SharePointAcsContext class.  Besides the changed logic for checking the context token’s expiration date (lines 13-43) and setting the field properties (lines 104-110), it’s the same code.  One additional thing to call out is that we changed the logic to obtain a new access token (lines 50 and 64) to use the individual parameters instead of passing a SharePointContextToken object.

SharePointAcsSerializableContextProvider

Now that we’ve created the new class, how do we use it?  As mentioned previously, SharePointContext uses a provider pattern.  The provider class is responsible for creating the appropriate SharePointContext derived type.  Our new class is shown in the diagram.

The implementation here is identical to the SharePointAcsContextProvider class, we just replace any references to SharePointAcsContext with SharePointAcsSerializableContext.

Provider

1. publicclassSharePointAcsSerializableContextProvider : SharePointContextProvider
2.     {
3.         privateconststring SPContextKey = "SPContext";
4.         privateconststring SPCacheKeyKey = "SPCacheKey";
5.  
6.         protectedoverrideSharePointContext CreateSharePointContext(Uri spHostUrl, Uri spAppWebUrl, string spLanguage, string spClientTag, string spProductNumber, HttpRequestBase httpRequest)
7.         {
8.             string contextTokenString = TokenHelper.GetContextTokenFromRequest(httpRequest);
9.             if (string.IsNullOrEmpty(contextTokenString))
10.             {
11.                 returnnull;
12.             }
13.  
14.             SharePointContextToken contextToken = null;
15.             try
16.             {
17.                 contextToken = TokenHelper.ReadAndValidateContextToken(contextTokenString, httpRequest.Url.Authority);
18.             }
19.             catch (WebException)
20.             {
21.                 returnnull;
22.             }
23.             catch (AudienceUriValidationFailedException)
24.             {
25.                 returnnull;
26.             }
27.  
28.             returnnewSharePointAcsSerializableContext(spHostUrl, spAppWebUrl, spLanguage, spClientTag, spProductNumber, contextTokenString, contextToken);
29.         }
30.  
31.         protectedoverridebool ValidateSharePointContext(SharePointContext spContext, HttpContextBase httpContext)
32.         {
33.             SharePointAcsSerializableContext spAcsContext = spContext asSharePointAcsSerializableContext;
34.  
35.             if (spAcsContext != null)
36.             {
37.                 Uri spHostUrl = SharePointContext.GetSPHostUrl(httpContext.Request);
38.                 string contextToken = TokenHelper.GetContextTokenFromRequest(httpContext.Request);
39.                 HttpCookie spCacheKeyCookie = httpContext.Request.Cookies[SPCacheKeyKey];
40.                 string spCacheKey = spCacheKeyCookie != null ? spCacheKeyCookie.Value : null;
41.  
42.                 return spHostUrl == spAcsContext.SPHostUrl &&
43.                        !string.IsNullOrEmpty(spAcsContext.CacheKey) &&
44.                        spCacheKey == spAcsContext.CacheKey &&
45.                        !string.IsNullOrEmpty(spAcsContext.ContextToken) &&
46.                        (string.IsNullOrEmpty(contextToken) || contextToken == spAcsContext.ContextToken);
47.             }
48.  
49.             returnfalse;
50.         }
51.  
52.         protectedoverrideSharePointContext LoadSharePointContext(HttpContextBase httpContext)
53.         {
54.             return httpContext.Session[SPContextKey] asSharePointAcsSerializableContext;
55.         }
56.  
57.         protectedoverridevoid SaveSharePointContext(SharePointContext spContext, HttpContextBase httpContext)
58.         {
59.             SharePointAcsSerializableContext spAcsContext = spContext asSharePointAcsSerializableContext;
60.  
61.             if (spAcsContext != null)
62.             {
63.                 HttpCookie spCacheKeyCookie = newHttpCookie(SPCacheKeyKey)
64.                 {
65.                     Value = spAcsContext.CacheKey,
66.                     Secure = true,
67.                     HttpOnly = true
68.                 };
69.  
70.                 httpContext.Response.AppendCookie(spCacheKeyCookie);
71.             }
72.  
73.             httpContext.Session[SPContextKey] = spAcsContext;
74.         }
75.     }

Registering the Provider

Now that we have a context implementation and provider, we need to register the provider for our web application.  We do that in Global.asax.cs.  Rather than add the code there, we follow the pattern for other customizations by creating a new class, SharePointConfig.cs located in the App_Start folder of our web application.

SharePointConfig

1. publicclassSharePointConfig
2. {
3.     publicstaticvoid RegisterProvider()
4.     {
5.         //Register the serializable context provider as the current
6.         SharePointContextProvider.Register(newSharePointAcsSerializableContextProvider());
7.     }
8. }

Now in Global.asax.cs in Application_Start, we reference our new class.

Application_Start

1. protectedvoid Application_Start()
2. {
3.     AreaRegistration.RegisterAllAreas();
4.     IdentityConfig.ConfigureIdentity();
5.     FilterConfig.RegisterGlobalFilters(GlobalFilters.Filters);
6.     RouteConfig.RegisterRoutes(RouteTable.Routes);
7.     BundleConfig.RegisterBundles(BundleTable.Bundles);
8.     SharePointConfig.RegisterProvider();
9. }

When the application starts, our provider is registered as the current provider.  That’s it, the rest of our code is untouched.

The Payoff

The payoff is that we don’t have to make any changes to our implementation code.  We can write the same code that can easily be used with multiple providers just by switching configuration.  Notice that we do not directly reference the SharePointAcsSerializableContext because the provider model handles this for us.

Code Snippet

1. publicvoid foo()
2.         {
3.             SharePointContext spContext = SharePointContextProvider.Current.GetSharePointContext(HttpContext.Current);
4.  
5.             string listUrl = "";
6.  
7.             using (var clientContext = spContext.CreateUserClientContextForSPHost())
8.             {
9.  
10.                 if (clientContext != null)
11.                 {
12.                     //Do something with context here
13.                 }
14.             }
15.         }

Very cool, the implementation details are abstracted away from our code.

Using Azure Redis Cache

Because the types are serializable, we can now use Azure Redis Cache as a custom session state provider.  First, create a new Azure Redis Cache by going to https://portal.azure.com.

Create a new Azure Redis Cache (this process takes around 30 minutes).

Once created, go to the cache and obtain the key.

 

Add the NuGet package for RedisSessionStateProvider to your web application.

Change the system.web/sessionState configuration section as shown, using your own cache host and key.

Web.config

1. <sessionStatemode="Custom"customProvider="CacheProvider">
2.   <providers>
3.     <add
4.       name="CacheProvider"
5.       type="Microsoft.Web.Redis.RedisSessionStateProvider"
6.       host="kirkedemo.redis.cache.windows.net"
7.       accessKey="asdfhgdrthtrsawefdaserhy7jsergwaeg="
8.       ssl="true" />
9.   </providers>
10. </sessionState>

You have now moved your session state to a scalable, distributed, secure cache, allowing your web site to scale to many instances.

Summary

This post showed how you how to use Azure Redis Cache for session state and how to create a SharePointContext provider that can be serialized using Azure Redis Cache. 

For More Information

MVC Movie App with Azure Redis Cache in 15 Minutes

Building Cloud Apps with Microsoft Azure: Best practices for DevOps, data storage, high availability, and more

How to use Azure Redis Cache

Comments

• Anonymous
  October 02, 2014
  SharePoint 2013 uses AppFabric as a distributed cache, with admittedly "interesting" results, although not for session state management. It would be interesting to find a way to replace AppFabric with Redis in these scenarios as well. Having multiple caching products on the same farm isn't ideal.

• Anonymous
  October 02, 2014
  You don't have multiple caching products on the same farm.  AppFabric remains a dependency for SharePoint 2013, while Azure Redis Cache is an Azure service that is a dependency for my app.  The app is completely separate from SharePoint, apps and SharePoint share zero infrastructure.  My app continues to live wherever I deem the best fit, and it communicates remotely to your SharePoint farm.  

• Anonymous
  October 02, 2014
  Hi Kirk, thank for sharing! Interesting info.

• Anonymous
  October 15, 2014
  Kirk, why when I click a button, the key is in the cookie or the session is not recreated after 20 min of inactivity? Know what can be?

• Anonymous
  October 16, 2014
  A big thank you to Kirk! This saved us a lot or time in overcoming the issue. Hope Microsoft will update their built in code in visual studio 2013.

• Anonymous
  October 16, 2014
  A big THANK YOU to Kirk! This saved us a lot of time! Hope Microsoft will update visual studio 2013...

• Anonymous
  October 19, 2014
  Kirk, Thanks for the post.  I think that an interesting side effect of this is that you use a SharePoint context in a custom REST api of your provider hosted app.  I did something similar to this using the .Net memory cache in order to re-hydrate a acs context from a context token and that spcachekey cookie, allowing me to get a context in a 'stateless' call.  It works well, but it's not quite as elegant as what you have here and it won't scale as well either.    I'll definitely have to try this out and see how it feels. Thanks again.

• Anonymous
  November 25, 2014
  Very interesting post, sure I will use it soon.

• Anonymous
  November 25, 2014
  Kirk, do you know whats the difference between  StackExchange.Redis (azure.microsoft.com/.../cache-dotnet-how-to-use-azure-redis-cache)  and RedisStateSessionProvider packages?

• Anonymous
  January 12, 2015
  I have problems to implement. The SharePointContext class returns an error because it can not be serialized. I followed step by step, but I can not resolve the error.

• Anonymous
  January 13, 2015
  @Rafael - make sure to edit the SharePointContext class and add the [Serializable] attribute to the class.

• Anonymous
  August 28, 2015
  The comment has been removed

• Anonymous
  August 28, 2015
  Please ignore I have it building now thanks!

• Anonymous
  August 31, 2015
  @Laurence - thanks for pointing this out.  Yes, you have to mark the SharePointContext class serializable.  I mention this in the beginning, but didn't do a good job highlighting how to do this.  Thanks for pointing this out for others that may have the same issue.

• Anonymous
  December 08, 2015
  Wouldn't using node affinity on a load balancer address the issue of scaling out to more than 1 instance? That way we don't have to use external caching infrastructure and rewrite the SharePointContextToken class?