Hybrid site collection provisioning from Azure to on-premises SharePoint
Hybrid models are more and more common with SharePoint and especially with larger enterprises we see sites being often hosted in on-premises and cloud. If you are planning to have self service site collection creation experience for end users, the pattern is pretty identical regardless of the location where you are hosting your site collections. Classic approach for these kind of deployments is that you host the engine to provision site collections in on-premises and based on the rules you either provision site collections to cloud or on-premises.
Down side for this approach is that you will have then customizations hosted in the on-premises side. If you’d implement this using app model techniques you’d have to have additional hosting infrastructure for these customizations (provider hosted), which is certainly not optimal, especially if you do not have any other customization requirements. You could certainly implement this also as full trust code (farm solution), but that would still cause you some customizations which you’d need to maintain in the farm with maintenance down times.
One option for this is to turn things slightly upside down and start hosting customizations mainly in Azure with connectivity to on-premises. This means that we will host the end user logic and the main business logic in Azure and fully minimize the code and complexity added to on-premises farm. We keep on hearing requests towards this model from many of the enterprise customers we work on daily bases assisting on moving to cloud or to more hybrid models.
To fully understand the techniques covered in here, I’d suggest to check following previous blog posts, which are explaining some of the key concepts, which we basically just extend with on-premises dimension with this particular post.
- Step 1 – Using Azure Storage Queues and WebJobs for asynchronous actions in Office 365
- Step 2 - Asynchronous on-demand site collection provisioning to Office 365 with Azure WebJobs
In this blog post we will take still advantage of the Azure Storage Queues and Azure WebJobs, but we will further extend the solution by adding support to target new site collection creation from Azure towards on-premises SharePoint farm using Azure Service Bus Relay Service.
Notice that this is conceptual architecture and even though this is coming from real life project with enterprise customer, like majority of samples and solutions Office 365 Developer Patterns and Practices guidance, you should be always mindful for the used technologies and only apply this level of complexity to your designs when truly needed. Simple is beautiful and designing architecture based on requirements, not based on technology is important.
Logical design for our solution
- Actual form to request site collections is hosted in Microsoft Azure as provider hosted app
- Request for site collection creation are added to Azure storage queue for asynchronous processing
- Azure WebJob is automatically called for new items in the queue which checks the target environment and starts executing the right route
- Site collections targeted to be created to cloud are created directly by the WebJob using app only token access
- If request is for on-premises, request is passed to Azure Service Bus which has on-premises service connected to it. This means that the on-premises code is executed and the on-premises site collection creation can be started
- Site collections are created based on the business and functional requirements to on-premises
Notice that like in the case of our reference implementation, you do not necessarily need to setup any app model infrastructure to on-premises side. We can use directly dedicated service account to perform the needed operations in the on-premises which means that there’s no need for addition infrastructure or any app model related security configurations.
“Wait wait! – wouldn't this require end point to be exposed from on-premises to Internet?” – No.
That’s actually the beauty of Azure Service Bus. We do not need to have any exposed end points to Internet from on-premises to make this happen. What will basically happen is that the service in on-premises will open up two directional and secure communication channel from on-premises to Azure, which is also used to receive messages from the Azure to on-premises. This is fully secured channel where we only exposed end points which are needed for the business requirements. In this particular case, only exposed end point is method to request site collection creation where we pass in the needed elements what end user filled in to the form.
Here’s network level design for this particular functionality from on-premises perspective, which will hopefully explain slightly more on where things are hosted and how communications work.
- Actual UI in the Azure for end users to request new site collections
- Can be deployed and secured as provider hosted app, so that there’s no other authentication needed than the access token with the SharePoint hosted app infrastructure
- Normal site collection creation towards Office 365 either using app only token or “service account” model
- Connection to on-premises using Azure Services Bus
- Service bus service in on-premises can be hosted for example as Windows service in the web front end servers of local SharePoint farm. This way the processing is high load balanced
- Notice that there’s no Internet facing end points needed, all communications are through secured service bus connection
- Actual creation of the on-premises site collections using suitable technology. Since this is on-premises, you could certainly use full trust code to achieve this or use on-premises CSOM.
Azure configuration for reference solution
Create storage account
Before you can make this sample work, you will need to have storage account in your Azure tenant. You can create one by logging to Azure management portal and then creating new service account using the creation wizard.
Multiple projects in the solution accesses this storage account, so we need to copy the account name and key information for reuse. After the storage account has been created, you can access this needed information by clicking the Manage Access Keys button for the storage account. This will show you needed details for solution configuration.
Actual configuration of the individual projects is dependent on the project (details below), but as a rule of thumb, you will need to update storage connection string from the app config or from the web.config based on your Azure storage details.
Create Service Bus namespace with ACS support
Since we will take advantage of the Service Bus Relay Service and to be able to make this work cross network we will have to create the service bus with ACS support. This is not unfortunately available from the Azure UI, but you can achieve this with Azure PowerShell. Here’s the needed steps for creating service bus namespace with ACS support.
- Install Azure PowerShell - https://azure.microsoft.com/en-us/documentation/articles/install-configure-powershell/
- Start Azure PowerShell
- Connect to your Azure account using Add-AzureAccount command
- Create new service bus namespace with following syntax
- New-AzureSBNamespace yournamespace "West Europe" -CreateACSNamespace $true -NamespaceType Messaging
- Notice that you want to update the region accordingly based on your own location
Here’s reference execution for above lines where we are creating a new Service Bus namespace as ‘o365pnpblogpost’
After above lines are executed, you can see your service bus namespace in the Azure UI and you can copy the
If you click Connection Information from the Azure you can see the Service Bus connectivity details, which are needed for the our solution when we setup connectivity cross different modules in the solution. Notice that now that we used the CreateACSNamespace switch, we also have connection string under ACS element. Copy namespace and default key information from Azure, since we will need those for configuration (app.config and web.config). purposes
Reference solution structure
Here’s our Visual studio solution structure and the configuration details one-by-one for each of the projects and other files.
Provisioning.Hybrid.Simple.Console.SendMessage
This is helper project to test Azure Service Bus connection with simplistic implementation. It actually calls our business logic to add message to the service bus located in the Common project. Purpose of this helper project is to be able to ensure that that messages can be sent cross the service bus connection.
<appSettings>
<!-- Service Bus specific app setings for messaging connections -->
<add key="ServiceBus.NameSpace" value="" />
<add key="ServiceBus.Secret" value="" />
</appSettings>
IncreaseOnePremTimeOuts.ps1
PowerShell to adjust client side call time out in the on-premises. By default client side time out setting is 90 seconds, but dependent on your environment, this might not be sufficient timeout for site collection creation and the customizations what you would be applying to the site, so in many cases you’d have to adjust the setting. This PowerShell script should be executed in our on-premises environment as a administrator to adjust the time outs accordingly.
#
# By default time out setting is 90 seconds, which might not be enough
# for site collection creation. This setting can be controlled from the
# SPWebApplication.ClientCallableSettings.
#
# This script will increase the setting to 5 minutes to ensure that
# site collection creation is successful.
#
$webAppUrl = "https://dev.contoso.com"
$timeoutInMinutes = 5;
$snapin = Get-PSSnapin | Where-Object {$_.Name -eq 'Microsoft.SharePoint.Powershell'}
if ($snapin -eq $null)
{
Write-Host "Loading SharePoint Powershell Snapin"
Add-PSSnapin "Microsoft.SharePoint.Powershell"
}
Get web application
wa = Get-SPWebApplication -Identity $webAppUrl
Increase time out for CSOM calls - by default this is 90 seconds
wa.ClientCallableSettings.ExecutionTimeout = [System.Timespan]::FromMinutes($timeoutInMinutes);
wa.Update();
Output current setting
wa = Get-SPWebApplication -Identity $webAppUrl
wa.ClientCallableSettings
Provisioning.Hybrid.Simple
This is the app project for introducing app for SharePoint. App is asking manage permissions rights when installed, so that we are able to dynamically to solve the identity of the caller in provider hosted app side.
Provisioning.Hybrid.Simple.Common
This is our business logic component which has also shared data objects and interfaces definitions for the service bus. Key files and their meaning as follows:
- ISiteRequest – this is the interface definition for the site requests sent from Azure to on-premises
- ServiceBusMessageManager – business logic for Service Bus messaging
- SiteCollectionRequest – data object for site requests. Used with Azure storage queues and in Service bus communications
Provisioning.Hybrid.Simple.OnPrem
This is the on-premises service responsible of creating connection from on-premises towards Azure and also acts as the process which is creating the site collections towards on-premises farm, when requests are received.
You will need to configure right values to service bus information and also the connectivity information towards on-premises farm. Current implementation is using so called service account approach which means that you do not have to install any apps in your on-premises farm, we are rather using just identity with proper permission to call the CSOM operations to create the requested site collection.
Notice that you will need to have at least April 2014 CU installed on your SP2013 farm and you will specifically need to enable CSOM support for on-premises site collection creation by following steps defined in following blog post: Provisioning site collections using SP App model in on-premises with just CSOM
<appSettings>
<!-- Service Bus specific app settings for messaging connections -->
<add key="ServiceBus.Namespace" value=""/>
<add key="ServiceBus.Secret" value=""/>
<!--
Provisioning settings for on-premises with service account model
This assumes that you have enabled CSOM based site collection creation
in on premises.
-->
<add key="ProvisioningAccount" value="administrator"/>
<add key="ProvisioningDomain" value="contoso"/>
<add key="ProvisioningPassword" value="pass@word1"/>
<add key="AdminSiteCollectionUrl" value="https://dev.contoso.com"/>
<add key="LeadingURLForSiteCollections" value="https://dev.contoso.com"/>
</appSettings>
After you have updated the app.config accordingly, you can start the solution. If you do not get any exceptions, connection to Azure is working properly. Below is a picture from on-premises farm where the Provisioning.Hybrid.Simple.OnPrem.exe.config configuration file has been updated accordingly and the solution is now running and has successfully made connection to Azure Service Bus. Whenever it will receive a message, it will also output the message to console for debugging purposes.
Provisioning.Hybrid.Simple.WebJob
This is the actual Azure WebJob in the Azure, which acts as the worker process and either creates the site collection to Office 365 or calls Azure Service Bus connection if requested target was on-premises. You will need to update following setting from the app.config.
<appSettings>
<!-- prefix from the contoso.sharepoint.com -->
<add key="Office365Tenant" value="contoso" />
<!-- App only tenant id and secret for creating site collections in cloud -->
<add key="ClientId" value="[App ID for tenant access]" />
<add key="ClientSecret" value="[App Secret for tenant access]" />
<!-- Service Bus specific app settings for messaging connections towards on-premise -->
<add key="ServiceBus.Namespace" value="[Your Service Bus namespace]" />
<add key="ServiceBus.Secret" value="[Your Service Bus Key]" />
</appSettings>
Notice that to be able for this code to create site collections in the Office 365, you will need to register app id with tenant permissions. You can do this manually by moving to /_layouts/15/appreginv.aspx page under any of the sites in your tenant. You can generate the client id and secret using the buttons, title can be anything you want and the app domain information, including the redirect URL is not really relevant, so we can use just localhost for them like in following picture.
After clicking Create you get confirmation as follows on the provided information. Copy this client ID and secret to some safe location.
Move to /_layouts/15/appreginv.aspx page under the layouts url under your site and lookup your app id from previous step. App’s permissions Request XML is not filled initially when you do the lookup operation.
Add following to permission App’s Permission Request XML text box for asking tenant level permissions for the id and secret combination.
<AppPermissionRequests AllowAppOnlyPolicy="true">
<AppPermissionRequest Scope="https://sharepoint/content/tenant" Right="FullControl" />
</AppPermissionRequests>
After clicking Create you are being asked to explicitly trust the requested permission and you will need to have the requested permission to do this, which in this case means that only a tenant administrator can provide this high permissions. You should definitely secure this registration information properly for security reasons.
Second section to update in the app.config for the WebJob is the connection strings to Azure Storage, so that your WebJob connectivity to storage queues and to logging will work as planned. Here’s the lines from the app config to update.
<connectionStrings>
<!-- The format of the connection string is "DefaultEndpointsProtocol=https;AccountName=NAME;AccountKey=KEY" -->
<!-- For local execution, the value can be set either in this config file or through environment variables -->
<add name="AzureWebJobsDashboard" connectionString="DefaultEndpointsProtocol=https;AccountName=[Storage Account];AccountKey=[Storage Key]" />
<add name="AzureWebJobsStorage" connectionString="DefaultEndpointsProtocol=https;AccountName=[Storage Account];AccountKey=[Storage Key]" />
</connectionStrings>
Provisioning.Hybrid.Simple.SimpleWeb
This is the actual user interface for self service site collection creation.
<appSettings>
<add key="ClientId" value="7c821ec4-b3a3-409b-a322-e8263b9dcc2b" />
<add key="ClientSecret" value="7ViPeoOq2cVzntOfS2yaM48DeIPw7sC6X9LGgUxH7Ds=" />
<add key="StorageConnectionString" value="DefaultEndpointsProtocol=https;AccountName=[StorageAccount];AccountKey=[StorageKey]" />
</appSettings>
User interface is as simple as it gets. In this sample case we do not even ask requested URL information, since that’s being randomized for each of the requests. You can select the target of the site collection from Site Target drop down.
Here’s confirmation message shown for end user after the request for new site collection creation to cloud or on-premises is stored as a request to the Azure Storage Queue.
How does the provisioned site collection look like?
Here’s a screenshot from on-premises side on created site collection. In this case we are creating always simple out of the box team site with custom branding, but you could just as well included all the other needed elements to the provisioning, like provisioning needed site columns, content types and other elements using CSOM.
Would this be a real life solution?
Right now this reference solution is pretty simple, but like with majority of the solutions what we release from the Office 365 Developer Patterns and Practices (PnP), this model is already in use with some of our customers. Majority of the PnP guidance is directly coming from our sibling program (JDP) where we have group of highly trained SMEs working full time with large customers on their farm solution (FTC) transition to app model. These customers are pretty large, so not all of the things what we release in the PnP are suitable for small or mid-size customers. We do still want you to have access on this IP, which is coming from real life projects, since it certainly can be beneficial for your projects as well.
When you are looking any of the samples in the PnP, remember that they are reference implementations and you definitely should always align your development to functional and business requirements for your particular deployment. No point adding additional complexity for nothing, simplicity aligned with the deployment specific quality requirements is the key for successful projects.
This reference sample is in the end relatively simple and you might want to consider few suggestions, if you are planning to use this kind of setup in the production environment. We wanted to keep our reference sample as simple as possible, so that you can get started on this model, if that’s needed for your deployment. Here’s some suggested updates for the solution before using it in production, there’s many other aspects also to consider depending on your environment and requirement specifics. We might get more fully baked version included as a Solution at some point in PnP guidance.
- You should have proper logging implemented to your solution for execution and exception tracking
- Relocate business logic to business logic components
- Notice that since CSOM redistributables are different between on-prem and cloud, you will end up having actually 3 components. One for cloud business logic, one for on-premises business logic and the third which as the shared objects, which we pass through Service Bus Relay.
- You might implement the on-premises service as a Window service, so that you can easily monitor that behaviour with typical monitoring solutions
- UI changes in the request form, like people picker for administrator and URL request
- Check existing URLs from on-premises to avoid overlap with requested sites
- Collect created site collections to one centralized location as a site directory
Keep also in mind that you will need to host the on-premises service bus end point in somewhere. You could use for example your on-premises SharePoint farm web front end servers for this hosting, so that you would have high available layout for incoming requests.
Video on the solution usage in practice
Here’s a video on using this solution in practice, which explains the solution structure, configuration and purpose of each project.
Office 365 Developer Patterns and Practices
Techniques showed in this blog post are part of the Provisioning.Hybrid.Simple (as simple as it gets) solution in the Office 365 Developer Patterns and Practices guidance, which contains more than 100 samples and solutions demonstrating different patterns and practices related on the app model development together with additional documentation related on the app model techniques.
Check the details directly from the GitHub project at https://aka.ms/OfficeDevPnP. Please join us on sharing patterns and practices for the community for the benefit of the community. If you have any questions, comments or feedback related on this sample, blog post or anything on PnP, please join us and more than 1900 others who are using Office 365 Developer Patterns and Practices Yammer group at https://aka.ms/OfficeDevPnPYammer.
“From the community for the community” – “Sharing is caring”
Comments
Anonymous
March 05, 2015
Since the code executed on-premise is done under a service account, does that mean there is no way to impersonate the end user? Not that you would want to for site collection creation. But further interaction might require the user identity.Anonymous
March 06, 2015
Hi Jonathan, since we are using specific account, we actually are impersonating to end user or we are applying the customizations using account which could be normal end user account. You cannot however anyway impersonate to the account which originally requested the site, but that's no what we used to do in the classic full trust code solutions either, so this should be fine.Anonymous
July 26, 2017
The comment has been removed