Clarifying Guidance on SharePoint Security Groups versus Active Directory Domain Services Groups

I received the following question on the SPYam group on Yammer, and decided to share the information with the broader audience via my blog.

For SharePoint security Microsoft always recommended to use Active directory group inside SharePoint group. But this guidance now changed Microsoft now recommend:

"We do not recommend SharePoint groups to assign permissions to sites. When a SharePoint group is used to assign permissions, a full crawl of the index occurs. Instead, we recommend Active Directory Domain Services (AD DS) groups."

[from https://technet.microsoft.com/en-us/library/ff758656(v=office.14).aspx]

Can someone from Microsoft confirm this please?!

 

Happy to try

Configuring the Environment

To demonstrate the effect of adding a user to a SharePoint group, adding an Active Directory group, and adding users to an Active Directory group, I want to isolate any changes as much as possible from the rest of my environment.  I configured my environment with a new web application CrawlDemo.Contoso.local, and a root site collection using the Team Site template.  The site collection administrator is a single user, “contoso\administrator”. 

In order to isolate the effect of search crawls to just this web application, I configured a new content source that contains my new web application’s URL.

This lets me isolate any crawls to just that one web application.  The rest of the settings for the content source:

• Crawl Settings: Crawl everything under the hostname for each start address
• Crawl Schedules: Enable Incremental Crawls
• Content Source Priority: Normal

I do not have continuous crawls configured for this environment, and none of my content sources are currently scheduled, so I can be sure that crawl information will be isolated to this single web application. 

Before continuing, I make sure that the crawl account has Full Read permission to the web application by checking the User Policy for the web application in Central Administration.

Note the text in red:

Adding or updating Web application policy with new users or groups will trigger a SharePoint Search crawl over all content covered by that policy. This can reduce search crawl freshness and increase crawl load. Consider using security groups at the policy level and add/remove users from security groups to avoid this.

We’ll come back to that later in this post as it is directly relevant to what we are discussing.

Once everything is configured, I initiate a full crawl of the content source.  Reviewing the Crawl Health report in Central Administration for Crawl Rate, I see that all 37 items were crawled.

If we focus our attention on the Summary table to the right of the screen, we can see the total items crawled was 37 items.

This means that the full number of items crawled was 37 items.  When we run an incremental crawl, we should only crawl the changed items and not every item.  The rest of this post focuses on the impact that security changes have on crawling.

A Note on Performance

I am in no way indicating that my demonstrations in this post be used for any type of performance benchmarking.  I am running all of these tests on a single laptop on a single drive using virtualized images that are not optimized for performance.  Further, I have other processes running on the machine, so the numbers that I show do not represent a production environment.  That said, we can make some general assumptions based on the data represented.

Understanding the EventCache

In each content database, there is a table named “EventCache” which contains a change log for everything in that content database.  Once the crawl is complete, I query it using the following SQL statement:

 select EventType, ObjectType, EventTime
from EventCache with (nolock)
where EventTime > '2013-05-06 16:37:36.750'

Note that it is unsupported to query SharePoint’s databases directly and should not be done in production without the direct involvement of a Microsoft support engineer. To minimize any potential effects of the query, I use the SQL NOLOCK hint to read uncommitted records and avoid introducing a table lock.  

Initially this query returns zero results because the time stamp used in the query is the last time stamp from when I created the site collection.  Let’s add a few documents to the Documents library in our new site.

We query the table again, and this time we see results in the EventCache table in the content database where the documents were added.

That makes complete sense, 4 documents added, 4 rows in the database.  We initiate an incremental crawl (not a full crawl) and wait for it to complete.  Once complete, click the Crawl Health Reports link in the search service application in Central Administration. 

The important part to call out is the Summary table on the right of the screen that shows 12 items were crawled during this incremental crawl, none of which were security items.

When the search crawler tries to determine what changes were applied, it reads the information in the EventCache table to calculate what needs to be crawled. The thing to learn from this section is that there is a table, EventCache, which the crawler uses to determine changes from each of the content databases. 

I run another incremental crawl, and the number of items decreases to 6 crawled items, and another decreases it to 5 items crawled.  No new rows were in the EventCache table, so SharePoint is simply crawling the top level pages to determine if any content changes appear.

Effect of Adding a User to a SharePoint Group

Let’s add a single domain user to the “Crawl Demo Members” group.

We now run our SQL query and see that a row has been added to the EventCache table. 

The new value, EventType 2097152, indicates a change where a member was added to a group (actually, 3 new rows were added, but I am only showing the relevant new row).  Recall that the search crawler reads the information in the EventCache table to calculate what needs to be crawled.  We run another incremental crawl and then inspect the health reports to see what happened.  Notice the red bar that now shows, the legend for the chart indicates this is a “security only” crawl.

Because we added a user to the group, everything in the site needs to be crawled in order to re-calculate the ACLs for the site.  Focusing on the Summary table at the right, we see that 36 security items were crawled simply because we added a user to a group.

We next turn to the crawl log to inspect a summary of our crawls so far.

Notice the value in the “Type” column has a letter “F” (indicating full crawl) or “I” (indicating incremental crawl).  The time for the incremental crawl went from 2:30 to 2:10 to 1:40 before we added a user to the group.  When we added the user to the group, it jumped up to 2:50 again because the ACLs for the items had to be recalculated.  The entire index had to be crawled to calculate ACLs.

In a large SharePoint environment with many documents, adding a user to a SharePoint group can have a significant impact on crawl performance.

Effect of Adding an Active Directory Group to a SharePoint Group

We just saw what happens when adding Active Directory Domain Services users to a SharePoint group.  Now let’s see the impact of adding an Active Directory group to a SharePoint group.  The effect will be less educational than before because the results will look nearly identical to what we just saw.  I create a new AD group, “MyCrawlGroup1” and add 12 users to the group in Active Directory.

Next, I add “MyCrawlGroup1” to the “Crawl Demo Members” SharePoint group.

We then go to the EventCache table to see what the changes were.  I get 3 new rows, the most important one is highlighted.

As we saw before, this change indicates that a member was added to a group.  We initiate an incremental crawl again, and just like before we see that we have the same number of security-only updates in the index.  Notice the red bar again.

Focusing on the Summary table, we see that we again have 36 security items, the same number as when we added a single user to a SharePoint group. 

If we check the crawl times, we can see that the time increased again.  This time, the time increased because SharePoint had to calculate the ACLs for the members of the domain group.  While the difference between the time spent calculating the members of the domain group versus adding a user directly is not statistically relevant (the difference of 10 seconds can be attributed to a number of factors), the fact that adding a domain group caused a security crawl is significant.

 

The point to understand in this section is that any time you add users or AD groups to SharePoint groups, your next incremental crawl causes the entire index to be crawled in order to calculate the ACLs. 

Effect of Adding an Active Directory Group to a Site

This should be predictable by now.  We will add an Active Directory group directly to a site instead of adding to a SharePoint group.  This time, I create a group with 12 different users in it called “MyCrawlGroup2”. 

Next, we will add the group directly to the site, granting this domain group Edit permissions just like the “Crawl Demo Members” group has.  When I go into “Site settings / Site permissions” and click the “Grant Permissions” button, an interesting thing happens, highlighted below.

SharePoint defaults to adding the new user or AD group to an existing SharePoint group.  Since we already proved what happens when you add a domain group to a SharePoint group, let’s change the drop-down to “Edit” instead of “Crawl Demo Members [Edit]”, meaning the group now is added to the site with the same level of permissions as the SharePoint group.

We query the EventCache table and highlight the interesting row.

EventType 524288 indicates “change assignment add”, and object type 4 means “Web”, so we are changing the permission on the web itself.  As expected, we run the incremental crawl, and you can guess what happens… a security crawl again.  Notice the red bar.

Inspecting the crawl log, we see that the time increased again.  Hopefully not statistically relevant, but the time has increased a bit in our crawl logs.

This should come as no surprise, adding a security object directly to the site permissions will cause the entire index to be crawled to calculate the ACLs.

Effect of Adding Users to Active Directory Groups

By now, you’re hopefully wondering, where’s the payoff for trudging through this post and all the pictures and data so far?  Well, you’re here!  Let’s see what happens when I add 12 more users to the security group “MyCrawlGroup1”, testing if adding users to a domain group that is contained within a SharePoint group causes security crawls. 

First, we query the EventCache table to see if any changes were recorded since the last record was written.  Nope!

With no changes in the EventCache table, our new users will be able to access the site because we added them to the Active Directory group, but no changes were recorded to SharePoint to trigger a security crawl.  We check the report, and see that the “Security only” red bar no longer shows in the graph.

Further, looking at the crawl log, our crawl time significantly decreased from 3:31 to just 1:50.

The lesson to learn is that adding users to an Active Directory group that already has permission in a SharePoint group does not trigger a full crawl of the index to calculate changes in permissions.

Now let’s try adding users to our “MyCrawlGroup2”, testing to see if adding users to an Active Directory group that already has permissions to the site causes a full crawl of the index.  I add 12 users to the AD group.

Of course, there are no new records in SQL.

And our incremental crawl runs again.  The time stays consistent with zero security updates.

The lesson here is that adding users to an Active Directory group that has already been granted permission to a SharePoint site directly does not initiate a full crawl of the index to calculate security changes.

Bonus Material: Adding a User Policy to the Web Application

By now, you’ve seen that adding a user or AD group to a SharePoint group or site causes a full index to calculate the permissions.  At the beginning of this post, I called out the User Policy button in Central Administration, noting the text in red:

Adding or updating Web application policy with new users or groups will trigger a SharePoint Search crawl over all content covered by that policy.

Let’s see what happens when we try to add a user.  In a previous post, I covered PowerShell that configures the object cache accounts.  Configuring the object cache accounts requires that you add Full Read permission to the Portal Super Reader and Full Control to the Portal Super Reader account.  I add the two accounts as part of the configuration steps necessary.

Next, I check SQL.

EventType 8192 means change or modify, and object type 2048 means Security Policy.  As you can reasonably expect by now, this causes a full index to calculate permissions as described in the red text on that page.  Notice the red bar is back.

The lesson to learn is that it is important that you configure the object cache accounts so they have permission to the cache (see my blog posting Setting Object Cache Accounts in SharePoint 2010, the same steps apply for SharePoint 2013) while understanding the impact that adding the permission can have on crawl performance.  Adding a user policy to the web application causes a full index to calculate permissions of all objects covered by that policy.  You still need to configure the object cache accounts, but make sure that you plan for the potential impact on crawl performance when you apply this change.

Summary

Hopefully this post shed some light on how security works with SharePoint and the effect that adding users or Active Directory groups can have on crawl performance. Adding users to Active Directory groups that have already been granted permissions to SharePoint does not affect crawl performance.  Adding users to Active Directory groups is the recommendation for managing security permissions in SharePoint.

Comments

• Anonymous
  May 06, 2013
  Hi  Kirk,Thank you very much for answering me and the great clarification.Regards,Mahmoud Hamed

• Anonymous
  May 06, 2013
  Kirk, this is a great post! Keep it up mate.Tob.

• Anonymous
  May 06, 2013
  Kirk, very insightful post!I am missing the general guidance though as the linked MSDN article does not refer to NOT using active directory groups in SharePoint groups. Also the article "Assign Permission Levels in SP 2013" on MSDN clearly states to use Security groups? technet.microsoft.com/.../cc261972.aspxI'm confused now what the best practice is and what the change from SP2010 may be.

• Anonymous
  May 06, 2013
  ... found the last sentence: "Adding users to Active Directory groups is the recommendation for managing security permissions in SharePoint."Perfect :-) Though I don't understand what changed with SP2013 - I thought this was also the recommendation for SP2010?

• Anonymous
  May 06, 2013
  @moontear - It's actually how SharePoint has functioned the entire time.  The guidance of using AD groups versus SharePoint groups deserves a bit of context.  Self-service permissions in SharePoint works wonderfully for small SharePoint farms with few individual permissions.  It breaks in large farms where, inevitably, someone conceives a solution with individual item permissions that suddenly becomes business critical and grows to millions of documents in a single library.  It is this latter case that is the context of this article, and is also the context of the TechNet article that was quoted in the original question.  Understanding how the product works is a much better approach than relying on absolute statements.  That's why I wrote the article in the manner I did, I wanted to help readers educate themselves on the potential impact and decide for themselves what risk a full crawl of the index poses for their environment.

• Anonymous
  May 07, 2013
  Kirk - this is a great post! Cheers, Craig.

• Anonymous
  May 08, 2013
  Excellent post !! I like the format in which you arrived at the conclusion. Keep it up..

• Anonymous
  May 08, 2013
  Great post, thanks Kirk. I do have a question though. As a workflow guy (predominately using Nintex Workflow) my clients and I struggle with permissions set this way. To reduce maintenance and management we usually try and use the same groups used for permissions as groups to assign tasks to in workflow. This way, if there is an HR team which need permission to a particular site and that same team needs to approve workflow tasks, we can use a single SP Group, with the HR users in it. If the SP Group contains an AD group, then the tasks do not get properly assigned and members of the AD group can't respond. Similarly if you assign a task to an AD group the task can't be completed. In some circumstances I create SP groups that have no specific permissions on a site, and then use this group to assign tasks too. In this case, if users are added or removed from this group (that has no permissions) what is the load on the DB and servers? Typically, the membership of these kinds of groups can change quite frequently so it would be good to know.Thanks again.

• Anonymous
  May 08, 2013
  Kevin - See the following articles for more information on fine-grained permissions.  technet.microsoft.com/.../gg128955.aspxtechnet.microsoft.com/.../dn169566.aspx

• Anonymous
  May 09, 2013
  Kirk, thanks for the detailed post. I wasn't aware of this impact on crawling but now that I do I think we may rething our direction on where we use AD groups versus SharePoint groups. We're not using SharePoint 2013 yet but having seen a lot of demos and having had a play myself the 'Share' option seems to really be getting pushed and is much more obvious to users now. Am I missing something or does this not seem to fly in the face of some of the MS best practice, though I do appreciate the balance between business functionality and admin headaches :)

• Anonymous
  May 09, 2013
  Derek - actually, this is the same behavior that SharePoint has always had, it's not new to SharePoint 2013.  There are areas where you will likely want to utilize SharePoint groups for specific functionality, such as Approvers for WCM, workflows, alerts, and other capabilities.  This is not an either/or approach, and understanding how the product works is key to making balanced decisions.  The point of this post was to explain why the recommendation is to use AD groups for securing content and the implication of fine-grained permissions.

• Anonymous
  May 28, 2013
  Great post, Kirk!  I love the way you lay everything out so thoroughly!

• Anonymous
  May 31, 2013
  The article is INVALUABLE. Many thanks for saving many of us these days.Regards,-T.s

• Anonymous
  April 01, 2014
  Great post Kirk!

• Anonymous
  April 25, 2014
  Hey, the first reference at the top of the blog does not seem to contain the quote attributed to it.  More importantly, the first FGP article referenced in the Q&A above now updates the guidance, at least in one sense:(technet.microsoft.com/.../gg128955.aspx)"Due to a change in SharePoint Server 2013 search continuous crawl capabilities that now include security information, we no longer recommend that you avoid using SharePoint groups to manage dynamic user and group memberships. Prior to SharePoint Server 2013, using dynamic memberships in SharePoint groups could cause search results to not show up correctly for all members until a full crawl occurred after the membership change. Now with the continuous crawl capability including security information a dynamic membership or other security change will be picked up sooner (default value is 15 minutes) and be used by search query result trimming."I also vaguely recall hearing that SP 2013 Search handles security changes in a more optimal way than 2010...something to the effect of Search is able to update the security information on items in the search index, without requiring re-processing of items on which only the security has changed.  It's still work, but I imagine it is a fraction of the work that is required to re-index a document.  Thoughts?

• Anonymous
  April 25, 2014
  SharePoint 2013 search index supports the use of index groups, one of which is the security update group. This can be updated independently of content and means security changes have less of an,impact on crawls than they did,in 2010. Continuous crawl now supports this partial update mechanism too which adds to,its effectiveness.I would still use AD groups added,to,SharePoint groups whenever possible. Kirk's data, findings and logic are perfectly sound. The product is just better than,it used to be!

• Anonymous
  May 12, 2014
  But, if I am changing permissions, especially removing permissions, then don't I want to run a full crawl anyway?

• Anonymous
  May 12, 2014
  Ignore my previous question, I am thinking of SP 2010, didn't realise 2013 was different.

• Anonymous
  September 11, 2014
  Adding users to an existing AD group in sharepoint with permissions..doesnt work..the user continues to get access denied..

  • Anonymous
    July 12, 2016
    Hi John,Your issue here will be the setup for SPSecurityToken.Use the Get-SPSecurityTokenServiceConfig to see the setup of your Environment, there are many articles on how to amend the values based on your requirements, I would suggest doing this in a Dev Env first so you can test thoroughly then implement in Live to reduce any user impact. I had the same issue and now have the permissions working within 3 minutes instead of up to 10 hours as it is by default.

• Anonymous
  March 20, 2015
  I am setting up both SharePoint and Project Server.  Does this apply the same way to Project Server?

• Anonymous
  June 30, 2015
  Very well written article!! So just to clarify, if we were to add an AD group to the web application policy and then add a user to this ad group, is it safe to assume that a full crawl wont happen?

• Anonymous
  March 03, 2016
  As always great post - thanks Kirk!

• Anonymous
  March 03, 2016
  @Ramana, adding users to an existing AD group won't trigger a crawl in SharePoint. I know it's an old question but I thought I'd post my comment hoping it may benefit others. :-)

• Anonymous
  July 20, 2016
  Okay, that's all great, but... what if you need to constrain a column in a list to a single group? You can't choose the AD group, and SharePoint can't interrogate AD groups.

  • Anonymous
    September 14, 2016
    I have the same concern here...I need to set target audiences on web parts, but none of my AD Groups that were added outside of created SharePoint are available. Have you learned anything new on your problem @LoriH ?

• Anonymous
  August 01, 2016
  As per Microsoft article, it is not recommended to apply permissions on AD groups. Please suggest if I am missing something. Check Note under Introduction Section. Copying the extract for your reference.https://technet.microsoft.com/en-us/library/cc261972.aspxFor ease of security management, we do not recommend the following items when you manage AD DS groups.Assign permission levels directly to AD DS groups.Adding security groups that contain nested security groups, contacts, or distribution lists.

  • Anonymous
    August 02, 2016
    @Deepak - see the section "Effect of Adding an Active Directory Group to a SharePoint Group".

• Anonymous
  August 10, 2016
  I understand that adding users to a SharePoint group individually has an impact on the crawler each time.I understand that adding a SharePoint group itself has an impact on the crawler.I understand that adding an AD group to the SP group creates an similar overhead.However once the AD group has been added to the SP Group , there is no further impact on crawler each time a new member is added to the AD group.So is the recommendation of the article to use AD groups rather than individual AD users to manage permissions? (this was my understanding of best practice in any case from the perspective of ease of management/maintenance of the permission groups rather than for search performance reasons)

  • Anonymous
    August 10, 2016
    The best practice is to understand exactly how it works and apply it to your farm accordingly.When this post was written (noting it was over 3 years ago), I was in Premier Field Engineering for SharePoint supporting multiple customers who were using SharePoint for document archival, each had over 50TB of document content each (one of them was over 30TB for one content database). Those customers typically had very short incremental crawl times, usually done within minutes. One day, someone posted a "best practices" article that said to only use SharePoint groups, and each of them changed the security structure of their document archive. Each customer contacted me letting me know their search crawler had been running for days and this was impacting their business. For these customers, obviously using SharePoint groups is not the answer because it triggers a massive crawl. The impact of that crawl on your farm is what needs to be considered. Further, SharePoint 2013 and subsequent patches reduced the impact of security crawls. To what extent it reduced the impact, I don't know, I haven't worked with SharePoint in a few years. There were changed in the product because they innovated within the O365 service. The whole point of this article is to explain exactly what is happening so that you can decide what the "best practice" is for your environment. If you assign individual AD users permission to a SharePoint object (item, list, site, site collection, or application), then you will trigger a security crawl. If you add a SharePoint group to a SharePoint object, you will trigger a security crawl. If you add an AD user to an existing AD group, you do not trigger a security crawl whether that AD group is assigned directly or if it is contained within a SharePoint group. If a security crawl is not too impactful on your environment and using SharePoint groups eases your management overhead, then do it. If a security crawl would significantly impact your farm, then don't. This article wasn't to define a single best practice that applies across the board, but to explain how SharePoint works. Someone else's "best practice" could be detrimental to your implementation. The best practice is to understand exactly how it works and apply it to your farm accordingly.