Share via

How user profile synchronization works in SharePoint 2010

  • Article

While we have a bunch of documents (and blogs) on provisioning the user profile synchronization service, configuring it and running it, we don’t address the larger conceptual question of “what’s going on under the hood” when synchronizing data from a directory source into SharePoint. This blog entry is meant to address exactly that question.

But before we dive, let’s first look at how we think of the User in the User Profile Service, and how that leads us to solving this seemingly simple but fundamental need for rich user data in the profile store.

The “User” in the User Profile Service
In a typical business system, a user is determined by the identity presented to the system. Once this identity is confirmed, business system follows a certain set of rules and policies to give appropriate access. But there is rich information about this user’s activity in the system that the corporation overall can benefit from and the user can benefit from. In the User Profile Service, we think of the user at the center of it all as a “person” in the profile store who has a face -through the Profile, My Site, Newsfeed; has a web service that can be crawled - for example for Social search; a store backing the person up and; relationships and a networks in the enterprise.

Need for the User Profile Synchronization Service
With that mindset, that is “person” at the center, as the cornerstone of the User Profile Service architecture, the question becomes, how does one get started in accumulating interesting information about a user in the user profile store. That is where the User Profile Synchronization service comes in. User Profile Synchronization service facilitates bringing user and group information from directory sources, such as Active Directory. If there is additional interesting data in a business system, say employee ID or hire date, a Business Data Connectivity model can be used to augment the user records with this data. SharePoint 2010 utilizes the synchronization engine from Forefront Identity Manager (FIM) 2010. With SharePoint 2010, data can be written back to the directory source (AD or LDAP), if appropriate permissions are present. A popular scenario for this capability is to write back pictures uploaded in SharePoint back to say, Active Directory, which than can be used by Microsoft Outlook.

Inner workings of the User Profile Synchronization Service
User Profile Synchronization service undergoes a series of steps to stage data from the directory source, process it and bring it over to the user profile store.

Before understanding how data flows from the directory source to/from user profile store, let’s look at two key storage areas for the synchronization service:

1. Connector Space (CS) is a staging area that contains representations of the designated objects from a connected data source and the attributes. Synchronization Service uses the connector space to determine what has changed in the connected data source and to stage incoming changes. Synchronization Service also uses the connector space to stage outgoing changes for export to the connected data source. Synchronization Service maintains a distinct connector space as a staging area for each management agent (connection in SharePoint 2010).

2. Metaverse (MV) is a storage area that contains the aggregated identity information from multiple connected data sources, providing a single global, integrated view of all combined objects.

Now let’s consider a few examples to illustrate the data flow in common scenarios.

Scenario 1 – Full Sync: if Active Directory was the directory source and a full AD synchronization was being performed to bring users, groups and group membership information into the user profile store. 

 

Step 1: Import from Active Directory into Active Directory CS
In this step, data is brought into the Connector Space from the directory source.

Step 2: Import from User Profile Store into the SharePoint CS
Similar to step 1, in this step, data is brought into the Connector Space from the User Profile Store. Note that this step is conditional skipped for the very first full sync but is performed for any subsequent full syncs.

Step 3: Synchronization of data with the MV
This step moves and reconciles the data and its references into the MV. If there was existing data present in the MV, or if there were additional sources of data, for example, multiple forests, this is where reconciling of data and references will occur. This is also the step that transfers data to the SharePoint Connector Space to gets it ready for export to user profile store.

Step 4: Export of data to user profile store
Once data is available in the SharePoint CS, it can now be moved to the user profile store. Note that some of this data requires further processing, for example, an employee’s manager is a reference to an incoming record, which might or might not have arrived before the employee. So SharePoint stages this data for further post-processing. Similar post-processing is required for groups and memberships.

Export is the most interesting and critical step from a SharePoint perspective, as this is the step that brings the data into the profile store. Before this step is reached, while much work is being done, there is no data flowing into the user profile store.

Step 5: Import and confirmation that the data has been received
Once the data is added to the profile store, a file is created to confirm its receipt back to the FIM synchronization engine. This file is also used to send data back, if there is additional data being sent back to AD, such as a picture.

Step 6: Synchronization of the data with MV
This step is similar in outcome as the previous synchronization Step 2. Now the data is being sent back to the MV from user profile store, including any data needed to be written back to AD. During this step, along with reconciling the data in the MV, any data needing to go back to AD is also prepared and marked ready for export in AD CS.

Step 7: Export to AD
Similar to Step 3, this step carries data back from AD CS to AD. This step will be a no-op if there is nothing being written back to AD.

Step 8: Import and confirmation that the data has been received
Similar to Step 5, this step closes the loop on data sent to AD.

 

Scenario 2 – Full Sync with BCS data: if Active Directory was the directory source and along with AD data, additional data from business systems, such as SAP, was being brought in.

Along with bringing in the identity data for the user from a directory source, often additional interesting data might reside in business systems. For example, employee ID might exist in another system, which if brought into the user’s profile, can be used for interesting scenarios, for example combined with the Audience functionality in SharePoint.

While a directory source, such as AD or LDAP, is required to create users (rows) in profile store, additional data (columns) can be brought in using Business Connectivity Service BCS (previously Business Data Catalog) models. Documentation on how to create these models or configure them for synchronization is elsewhere and is beyond the scope of this document. What we describe here is how the data flow works in such scenarios.


For bringing in the data through a BCS model, in addition to the steps outlined in Scenario 1 above, following steps are performed, after Step 8.

Step 9: Import from BCS
BCS Data is joined, using the attribute selected for BCS connection creation, and then the property values for mapped properties are brought into the BCS Connector Space

Step 10: Sync BCS data to MV
BCS Data is synchronized into the MV as well as setup for export from SP CS to the Profile Store. Notice the one-way sync arrow, as the BCS data is only brought into the profile store. We do not write the data back to the respective BCS business system.

Step 11: Export of data to user profile store
Similar to step 4, BCS data is written into the user profile store.

Step 12: Import and confirmation of data receipt
Same as step 5, we close the loop with the SharePoint CS that the data has been received.

Scenario 3 – Incremental Sync: if Active Directory was the directory source, a full synchronization was completed earlier, and an incremental synchronization is being run.

The steps performed for incremental sync are exactly the same as full sync, except wherever a sync or import step were performed, it’s now replaced by Delta sync or Delta import, respectively. Only the changes flow through, not everything. Note that regardless of incremental or full synchronization, BCS synchronization will bring in all user data from the BCS source system.

 

Few final thoughts
As discussed here, user profile synchronization is a key element of populating the ‘person’ in your profile store. It’s critical to understand and plan what information resides in the profile store, where it comes from, how it gets there and how it’s maintained. Behind each of these key questions is the complex process of bringing data to-from directory sources such as AD and business systems such as SAP. Our hope is that this discussion helped you gain a better understanding of how it all works and how the data flows, so you can manage and maintain the profile store more effectively.

Comments

  • Anonymous
    May 05, 2010
    Thanks for the post. This is really helpful. A couple of questions:
  1. From your description it sounds like if you want to bring in data via BCS to the user profile store, you must also use AD or another directory for the initial user creation. Is this correct or can you bring in all the data via the BCS?
  2. I was under the impression that the BCS had 2 way sync capability (supposed to be one of the advantages over the BDC) but you state that sync from the BCS data is one way. Can you please clarify? Thanks!
  • Anonymous
    May 07, 2010
    Hi Caedmon,
  1. You can't create new users (rows) through BCS, you can only augment existing user's properties (columns).
  2. Yes, BCS does have two-way capability, to read and write. But that's the BCS service/models, independently of sync. While the sync engine and BCS both are capable of doing so, the 2010 BCS based sync is scoped to read-only and will not write back the data to the business sytem through BCS.
  • Tanujb

  • Anonymous
    May 15, 2010
    Hi, I have one question I'm sure many others have too i.e. how does the above AD example differ if you choose to use a generic LDAP store like OpenLDAP? Thanks for the great info. Regards, Paul

  • Anonymous
    June 03, 2010
    Hi - very useful article. Is it possible to write a custom connector? I have an internal user master that is not AD (and is not LDAP at all) that I need to use to synchronize with the Sharepoint user profile store. I'm just looking for some pointers to docs or is this something that is just Not Possible?

  • Anonymous
    June 21, 2010
    If you have already FIM 2010 deployed within your organisation, can you leverage that for Sharepoint 2010 provisioning or does Sharepoint 2010 needs it''s own FIM instance?

  • Anonymous
    September 30, 2010
    Please see the following technet.microsoft.com/.../ff959234.aspx Configure profile synchronization using a Lightweight Directory Interchange Format (LDIF) file (SharePoint Server 2010) Published: September 16, 2010 This article describes how to use a Lightweight Directory Interchange Format (LDIF) file to synchronize user and group profile information between Microsoft SharePoint Server 2010 and a Lightweight Directory Access Protocol (LDAP) provider not directly supported by Microsoft SharePoint Server 2010. For a list of directly supported LDAP providers, see Identify directory services and business systems.

  • Anonymous
    November 09, 2010
    I have a different issue : on my user profile sync service i have 2 connections ( from AD, and From BCS) , i am using BCS to augment the users properties , on my BCS i am using  a WCF that i created which invoking a method getemployee(string unique id)  //my unique id is caming from my user profile so i can get the rest of my properties to sync , my issue is that my unique id is in this format (domainuserid) thats how its stored on Active directory ,so the WCF method is throwing an error because of the special caractere("") , when i use WCFclientTest and i invoke the method using a string in the following (domain\uniqueId) it works , the prob is that i was thinking to run a timer job  to manipulate the unique id after the Sync is done , but the as i said i have 2 connections when the sync is done  the bcs sync will be done too , i want to stop the sync after AD sync is done , run the timer job then continue the sync with BCS. any idea?? thanks.

  • Anonymous
    February 04, 2011
    Hello, I am not sure if my issue is related to the above coments, but any help would be greatly appreciated.  As the Admin of our SharePoint site I deleted some files that were listed under the "List".  The delete was successfully from my view, but all other users on the Site are still seeing the files that were deleted.  Please, please help! Is this a sync issue?

  • Anonymous
    February 04, 2011
    I wanted to make a change...I was changing the content site and the content site was a list... Thank you

  • Anonymous
    February 17, 2011
    Excellent post; very informative. I have one question: Do you know if it's possible to use a BCS connection by itself just to update existing profiles from an external system without having an AD or other type of primary connection? I've set up an ECT which returns data from the external system into an external list without problem. Profile synchronization completes all steps in FIM successfully. But no profiles ever get updated. And I missing something here? Thanks.

  • Anonymous
    April 21, 2011
    Wonderful article. Good to know about profile import and what happens in the background. But I have a query, I read this article www.ekhichdi.com/.../How-to-give-Replicating-Directory-Changes-permission-270.html Now I do not have permission on the Active directory and the AD team is not going to make the changes. I am testing SP 2010 profile import on my test machine. Is there a way I can test import apart from importing from AD? Maybe for 10-20 users? Regards

  • Anonymous
    May 23, 2011
    We have full installation of FIM 2010 synchronization service in our organization. How does this information for SharePoint differ from the sychronization service in FIM 2010? Can we use FIM 2010 for SharePoint instead of using what is being described here?

  • Anonymous
    May 24, 2011
    I have a question Does it have any relation Sharepoint FAST I have unistalled, reinstalled and configured FAST since then i noticed Membership details are not updating i became member of few more sites but these sites are not listed in Membership section of mysite what could be the possible reason user profile sync, crawl of search everything is working fine

  • Anonymous
    June 30, 2011
    The comment has been removed

  • Anonymous
    October 09, 2011
    I am doing a SharePoint user profile sync from AD, the process is working fine, from "Synchronization service manager" 2373 profiles were added. but only 9 were added to SharePoint. i checked AD and found that the 9 accounts have type "USER" and others have "inetOrgPerson". How can i add these marked as "inetOrgPerson" to SharePoint user profile?? please advice

  • Anonymous
    December 08, 2011
    So we can't use BCS to create the profiles? User Profiles can only be sync from LDAP source? Actually I have a CSV file containing user info extract from AD and I need to enable them on the user profile sync. I imported the file into a table in a database and created BCS, but some how after I do a sync no profiles can be created. and it is not allowed to connect to the AD to do the user profile sync Any thoughts? please help

  • Anonymous
    February 08, 2012
    If you are updating user profiles using SharePoint 2010 APIs then please remember following -

  1. Profile update takes lot of time
  2. More the number of properties in each profile, more the time it will take to update
  3. Update only the properties which are changing
  4. Try to load configuration values before running the main logic (of updating the properties)
  5. Reason it takes more time to update profile/properties is internal to SP 2010
  1. I have a BCS connection to a custom source.
  2. It is mapped with an attribute, say Employee Id
  3. The Profile Property import works fine for all the custom Properties mapped to the BCS Source. But, the problem arises when I change the connection string of the source. I do not get any of the Properties populated. The only difference between the old and the new source is that, the Value of Employee Id is different. I have updated the Employee Id value for the users in SharePoint. But still I don't get the required values from the Data Source. Can anyone throw some light on this problem? Is there a caching happening anywhere?

  • Anonymous
    March 05, 2014
    Thanks for explaining this complex process. I've got a sync with BCS augmentation working, but it takes FOREVER! It took 20 hours to sync 30,000 users and nearly all of the time is spent during the DS_FULLIMPORT phase of the BCS import. We added an index to the table and that took it down to 15 hours but that is still brutal. It seems to process each user one at a time, running a SELECT * FROM table WHERE User_ID = [the user id]. Each select takes only milliseconds to return but when profiling it seems to take 2.5 seconds per user. That time seems to be spent by SharePoint doing something after the record is returned. Why the heck does it select each user one at a time? Why doesn't it pull in all the data from the table or view and then join the data to the AD data in the sync database or in memory? This seems very inefficient.

  • Anonymous
    March 06, 2014
    The comment has been removed

  • Anonymous
    March 24, 2014
    Thanks for the UPS explanation. Faisal Masood http://www.FaisalMasood.com

  • Anonymous
    May 21, 2014
    Hi, i was wondering what is the basic difference between full sync v's incremental sync? in our organization we have just noticed that the updates from AD are not reflected in SharePoint directory search for 1 week but new profiles does reflect the next day. i checked the schedule and noticed that we have only incremental import scheduled once every night. do we need full import to see the modifications of existing user profiles? searched several places and most of them say incremental is sufficient and full import is just a one time deal? any clarifications is greatly appriciated