Asynchronous site collection provisioning for Office 365
One of the highly requested samples related on the Office 365 and Azure usage is the asynchronous self-service site collection provisioning. This is pretty much a follow up for the similar blog post concentrating for the on-premises and Office 365 Dedicated side, but this one concentrates on the normal Office 365 with the provider hosted apps hosted in Microsoft Azure platform. There are small differences between the on-prem and the cloud, but the logical design is pretty much the same as in following picture, but for this one I wanted to indicate where the individual elements are actually hosted and using which technique.
- End users are exposed self service site collection creation UI
- Request are saved to SharePoint list, which could be optionally associated with a workflow for approval purposes. You could also store the requests to some other centralized location, like to Azure Storage Queue, which is then accessible by the remote timer job. Details on this slightly depends on the exactly business requirements.
- Remote timer job is schedule as a web job or as a worker process, which is checking if there’s any site collections to be created in the request queue. If there are and the request have been approved in the case of approval workflow, remote timer jobs actually starts the creation of the site collection
- Needed assets like images and themes are stored in some location where the remote timer job can provision them to the newly created sites
- Actual creation starts by creating out of the box site and then site can be specialized based on end user selection, like for example by providing multiple different adaptations of the team sites, which can differentiate for example with the created document library or content type perspective. Whatever is needed for the particular deployment.
How do I handle branding for the sites? Do I use web templates?
This is pretty common question and the recommendation is to use directly CSOM as part of the remote timer job to deploy the needed assets to the just created site collection. The reason why we don’t recommend web templates is that they are not automatically updated whenever the introduce any new capabilities ot the out of the box site definitions. This means that after a while, your web templates are falling behind of the updates which are applied to the out of the box sites.
Ok – so how should I approach this then?
You should always start by provisioning out of the box site, which means basically either team site or publishing site. Next step is then to apply the needed changes to the provisioned sites based on the selected template. This means branding, content types, libraries and whatever is needed, which we used to provision to the sites using the web template technique. That does not mean that you are limited to just few templates, you can still have many templates like with web templates side, but you just start with by creating the site using out of the box site.
- Provisioning engine which could be hosted in Azure or in on-premises. Could be as simple as set of PowerShell scripts which are provisioning the site collection and uploading the needed elements.
- We always start with the out of the box site provisioning, so that you get all the latest improvements for the sites.
- Actual specialization based on the user selection. Is it collaboration, project or organizational site. This is pretty much the stage which can be compared to the web templates. Main difference is however that we apply the needed settings and configurations to the sites using CSOM/REST. Uploading of the assets or creating the needed content types. This way the site has no dependencies on any external elements, which will help with the future updates on the service.
How is it developed?
You can download the code from the Office 365 Patterns and Practices guidance (PnP) and for detailed explanation, please check this video from Channel 9 by Andrew Connell where he explains the code structure and how things work based on the code we provided in the PnP project.
How would I deploy this to Azure and Office 365?
Like explained in AC’s video, this sample consists from two different part. One part is the UI part which exposes the self service site collection request form, which does not actually provision anything. Second part is then the remote timer job part. UI part will be deployed to the Microsoft Azure as a web site and the remote timer part will be added there as a web job. This way you can easily schedule how often your code is executed.
Following video is showing deployment to the Office 365 and Microsoft Azure in practice.
Like mentioned in the video, the normal timeout for the web job execution in the Microsoft Azure is 121 seconds. Since the site collection creation and applying the customizations will take longer than that, you’ll need to increase this timeout. This can be done by adding SCM_COMMAND_IDLE_TIMEOUT key to the application settings using the Azure console or you could also add this to the app.config of the console application when you deploy that to Azure.
Office 365 Developer Patterns and Practices
Techniques showed in this blog post are part of the Provisioning.Cloud.Async sample in the Office 365 Developer Patterns and Practices guidance, which contains more than 50 projects demonstrating different patterns and practices related on the app model development. Check the details directly from the GitHub project at https://aka.ms/OfficeDevPnP. Please join us on sharing patterns and practices for the community.
“From the community for the community” – “Sharing is caring”
Comments
Anonymous
August 28, 2014
I've built a similar App a few weeks ago. But instead of storing the requests in SharePoint I've used Azure Storage Queue. Using a Queue for this situation seems a little bit more natural to me. I'll try to sumarize it also in a blog post :)Anonymous
August 29, 2014
this... is... AWESOME!Anonymous
August 29, 2014
Hi Thorsten. that would be indeed more reliable and more sophisticated process. With this particular example we intentially wanted to keep things as simple as possible around the asycn option, but having the queue in the Azure Storage Queue would be indeed good option for storing the data in SP list. That would be indeed great thing to blog about with the details. Looking forward on seeing it.Anonymous
August 30, 2014
Great post, Vesa! I still have a question though... One of my customers currently uses a web template (save site as template) to setup new site collections in the O365 admin center (select a template later). They now want to automate this process because not all items are included in the web template (things like security, property bag items, ...). I will have to create a provider hosted app to set this up. If I understand your post correctly it's NOT recommended to perform a SPWeb.ApplyWebTemplate after site collection provisioning to provision branding, lists, CT & fields elements? It would break possible updates to the existing site definitions? In SP2007/2010 you could create a blank site and apply a webtemplate programmatically. In SP2013 I don't know if this even possible because a blank site template isn't available anymore? On the other hand it would be nice to have the flexibility of a web template. The customer is then able to build the 'basic' web template himself and the provider hosted app then only have to provision the web template and do the 'harder' work. This way, when the customer wants to add an additional list or some other content in the newly created site collections, he just have to create a new web template - and don't need to hire a developer to modify the provisioning engine (provider hosted app). It would be nice to read your thoughts about his. Kind regardsAnonymous
August 31, 2014
The comment has been removedAnonymous
September 01, 2014
Thx Vesa, clear for me... appreciate your feedback.Anonymous
September 01, 2014
Great post! <rant> BUT I hope that the team will soon deliver a solution that doesn't require me to provision a customized site using CODE and a provider hosted app. It is mind-boggling that we don't have a "simple" declarative solution for this scenario... And I hope that this "upcoming" solution will also cover Publishing sites. I'm tired of having to open up Visual Studio just to package a publishing site, the product has been around for 12 years now :-) </rant> Thanks, JonasAnonymous
September 01, 2014
The comment has been removedAnonymous
September 04, 2014
Hi Vesa, Great article! I'm working for a Microsoft Solution Partner called CTB in the Netherlands. We're using the web templates for years now in many projects at many customers, and didn't have any major issues with it yet. The main reason we're using them is because a functional consultant (or even the customer itself) can create or update such templates by a few clicks in the SharePoint UI and then use 'save site as template'. We then provision the web or site collection and apply the web template and other customizations which are not included in the web template afterwards. We've built an engine to support all those functionality, which is fully customizable without coding, by just using XML mappings and functions. We're currently porting our On-Premises full-trust provisioning solutions to an online variant. The Office 365 Patterns and Practices are a great help for our development team! Can we expect more issues on the SharePoint Online platform in combination with web templates because of the constant updates? Do you have some samples of 'Improvements' which are not pushed to the sites based on those custom web templates (which are in fact based on the Team Site template). Regards, Ad van den Berg Technical SharePoint Consultant at CTB xRM www.linkedin.com/.../advandenbergAnonymous
September 07, 2014
The comment has been removedAnonymous
September 10, 2014
Hi Vesa, Thanks for your reply. In most cases we do keep a 'Template site' available in the customer's web application which we use to create new versions of the web template every now and then. So this site is getting the updates targeted to the OOB site templates. I think we should stick to that procedure to prevent most of the possible challenges. In most cases this will fit for our customers/solutions. But again thanks for your explanation!Anonymous
September 15, 2014
Hi Vesa, Good article. It would be great if Microsoft provide normal way to provision third party apps to the sites. because now you need to activate Side Loading feature to do it but it's not best way because of this feature is not recommended for production..Anonymous
September 15, 2014
The comment has been removedAnonymous
October 09, 2014
Hi Vesa, Are you familiar with this bug with provisioning site template in SP2013? social.technet.microsoft.com/.../custom-web-template-fails-list-does-not-exist We're having this issue since the April 2014 CU. Still no fix available! This update makes the save site as template with content feature completely useless.Anonymous
October 16, 2014
Hi Ad van den Berg, I have not actually seen that, but looks something which should be reported as a bug to Office 365 Support or to customer support if it's on-prem environment. Key point is to get that reported using official channels, so that there's tracking on getting it fixed. Personally I have not seen that, since with remote provisioning you'd be not using that capability in first place and have not really used it earlier either.Anonymous
October 21, 2014
Hi Vesa, We're trying to do a provider hosted app for provisioning sites to SP online (async). Whenever we run the console solution to provision the sites, sometimes it works, sometimes we get the 'request timed out error'. It errors out on the ExecuteQuery after the web.applywebtemplate. Is there a different configuration or approach we should be taking for provisioning sites with custom templates? ThanksAnonymous
October 21, 2014
The comment has been removedAnonymous
October 21, 2014
Thanks Vesa. Even with Timeout.Infinite, I still get the timeout error.Anonymous
October 22, 2014
Hi Jane, I also faced that problem, but I never experienced it again after setting RequestTimeout = -1 I mentioned that I had two open client contexts that were wrapped into each other (one for checking new jobs, and one on the tenant-administration, for actually creating the site. I had to set the requesttimeout on both to infinite - maybe you are facing a similar issue? Regards MartinAnonymous
October 22, 2014
Thanks Martin, that's what we are seeing as well in our implementations for customers. You do need to assing the time out for each specific instances like you mentioned. Otherwise default value will be used and the calls will timeout if they take took long.Anonymous
November 07, 2014
Thanks Martin. I did miss setting the time out for the other client context. I do not get a timeout error anymore when I'm running the code on my machine. However, for some reason, I still get the time out error when I try to run the console solution as a webjob in Azure. I've set the SCM_COMMAND_IDLE_TIMEOUT key as noted in the article above. Is there a different setting that I've to do to run it in Azure? Thanks again.Anonymous
November 08, 2014
Hello Vesku, We need your help in fixing one issue. We are provisioning site collections using CSOM for 1) Provider hosted App and 2) Console application(for initial setup of multiple site collection for Intra). for 1) Getting error sometimes like "429: too many requests error"
- Getting error like "request uses too many resources error" We found below article which says about CSOM limitations msdn.microsoft.com/.../jj163082%28v=office.15%29.aspx So need your suggestion what other way or how to overcome this error? also atleast for on-prem we can increase request size. but what about the office 365? Thanks in advance!
Anonymous
November 09, 2014
Hi Jane, I'm sorry, I don't have experience with the Azure web jobs until now. but since I planned to do the same like you, I'd appreciate if you post it somewhere if you find out. Regards MartinAnonymous
November 25, 2014
Hi, The solution was working fine last week. Now I am getting the error below when the code tries the run ctx.ExecuteQuery in the code snippet below from the console app part of the solution. ctx.Load(listItems); ctx.ExecuteQuery(); ======================================================================= Exception:Thrown: "Access denied. You do not have permission to perform this action or access this resource." (Microsoft.SharePoint.Client.ServerUnauthorizedAccessException) A Microsoft.SharePoint.Client.ServerUnauthorizedAccessException was thrown: "Access denied. You do not have permission to perform this action or access this resource." ======================================================================= Any advise will be much appreciated. Thanks, JaneAnonymous
January 14, 2015
Sorry for the delayed response, completely missed the comments. As a rule of a thumb would suggest to post any questions or comments to Office 365 Developer Patterns and Practices Yammer group at the below address. We have 1500 members in this group with app model interest and it's really active group to assist with any questions.
- aka.ms/officedevpnpyammer Jane, Martin - We are not facing that time out issue with production deployment after setting the timeouts propertly and using web jobs. K - 429 is throttling warning, you should be looking following PnP sample for assistance - Jan - Working previous week, not now usually means expiration of tokens. Easiest way to force re-trusting is to update the version number of the solution to 1.0.0.1 and you should be good to go.