What’s going on under the hood when synchronizing data from a directory source into SharePoint?
The “User” in the User Profile Service
In a typical business system, a user is determined by the identity presented to the system. Once this identity is confirmed, business system follows a certain set of rules and policies to give appropriate access. But there is rich information about this user’s activity in the system that the corporation overall can benefit from and the user can benefit from. In the User Profile Service, we think of the user at the center of it all as a “person” in the profile store who has a face -through the Profile, My Site, Newsfeed; has a web service that can be crawled – for example for Social search; a store backing the person up and; relationships and a networks in the enterprise.
Need for the User Profile Synchronization Service
With that mindset, that is “person” at the center, as the cornerstone of the User Profile Service architecture, the question becomes, how does one get started in accumulating interesting information about a user in the user profile store. That is where the User Profile Synchronization service comes in. User Profile Synchronization service facilitates bringing user and group information from directory sources, such as Active Directory. If there is additional interesting data in a business system, say employee ID or hire date, a Business Data Connectivity model can be used to augment the user records with this data. SharePoint 2010 utilizes the synchronization engine from Forefront Identity Manager (FIM) 2010. With SharePoint 2010, data can be written back to the directory source (AD or LDAP), if appropriate permissions are present. A popular scenario for this capability is to write back pictures uploaded in SharePoint back to say, Active Directory, which than can be used by Microsoft Outlook.
Inner workings of the User Profile Synchronization Service
User Profile Synchronization service undergoes a series of steps to stage data from the directory source, process it and bring it over to the user profile store.
Before understanding how data flows from the directory source to/from user profile store, let’s look at two key storage areas for the synchronization service:
1. Connector Space (CS) is a staging area that contains representations of the designated objects from a connected data source and the attributes. Synchronization Service uses the connector space to determine what has changed in the connected data source and to stage incoming changes. Synchronization Service also uses the connector space to stage outgoing changes for export to the connected data source. Synchronization Service maintains a distinct connector space as a staging area for each management agent (connection in SharePoint 2010).
2. Metaverse (MV) is a storage area that contains the aggregated identity information from multiple connected data sources, providing a single global, integrated view of all combined objects.
Now let’s consider a few examples to illustrate the data flow in common scenarios.
Scenario 1 – Full Sync: if Active Directory was the directory source and a full AD synchronization was being performed to bring users, groups and group membership information into the user profile store. 
Step 1: Import from Active Directory into Active Directory CS
In this step, data is brought into the Connector Space from the directory source.
Step 2: Import from User Profile Store into the SharePoint CS
Similar to step 1, in this step, data is brought into the Connector Space from the User Profile Store. Note that this step is conditional skipped for the very first full sync but is performed for any subsequent full syncs.
Step 3: Synchronization of data with the MV
This step moves and reconciles the data and its references into the MV. If there was existing data present in the MV, or if there were additional sources of data, for example, multiple forests, this is where reconciling of data and references will occur. This is also the step that transfers data to the SharePoint Connector Space to gets it ready for export to user profile store.
Step 4: Export of data to user profile store
Once data is available in the SharePoint CS, it can now be moved to the user profile store. Note that some of this data requires further processing, for example, an employee’s manager is a reference to an incoming record, which might or might not have arrived before the employee. So SharePoint stages this data for further post-processing. Similar post-processing is required for groups and memberships.
Export is the most interesting and critical step from a SharePoint perspective, as this is the step that brings the data into the profile store. Before this step is reached, while much work is being done, there is no data flowing into the user profile store.
Step 5: Import and confirmation that the data has been received
Once the data is added to the profile store, a file is created to confirm its receipt back to the FIM synchronization engine. This file is also used to send data back, if there is additional data being sent back to AD, such as a picture.
Step 6: Synchronization of the data with MV
This step is similar in outcome as the previous synchronization Step 2. Now the data is being sent back to the MV from user profile store, including any data needed to be written back to AD. During this step, along with reconciling the data in the MV, any data needing to go back to AD is also prepared and marked ready for export in AD CS.
Step 7: Export to AD
Similar to Step 3, this step carries data back from AD CS to AD. This step will be a no-op if there is nothing being written back to AD.
Step 8: Import and confirmation that the data has been received
Similar to Step 5, this step closes the loop on data sent to AD.
Scenario 2 – Full Sync with BCS data: if Active Directory was the directory source and along with AD data, additional data from business systems, such as SAP, was being brought in.
Along with bringing in the identity data for the user from a directory source, often additional interesting data might reside in business systems. For example, employee ID might exist in another system, which if brought into the user’s profile, can be used for interesting scenarios, for example combined with the Audience functionality in SharePoint.
While a directory source, such as AD or LDAP, is required to create users (rows) in profile store, additional data (columns) can be brought in using Business Connectivity Service BCS (previously Business Data Catalog) models. Documentation on how to create these models or configure them for synchronization is elsewhere and is beyond the scope of this document. What we describe here is how the data flow works in such scenarios.
For bringing in the data through a BCS model, in addition to the steps outlined in Scenario 1 above, following steps are performed, after Step 8.
Step 9: Import from BCS
BCS Data is joined, using the attribute selected for BCS connection creation, and then the property values for mapped properties are brought into the BCS Connector Space
Step 10: Sync BCS data to MV
BCS Data is synchronized into the MV as well as setup for export from SP CS to the Profile Store. Notice the one-way sync arrow, as the BCS data is only brought into the profile store. We do not write the data back to the respective BCS business system.
Step 11: Export of data to user profile store
Similar to step 4, BCS data is written into the user profile store.
Step 12: Import and confirmation of data receipt
Same as step 5, we close the loop with the SharePoint CS that the data has been received.
Scenario 3 – Incremental Sync: if Active Directory was the directory source, a full synchronization was completed earlier, and an incremental synchronization is being run.
The steps performed for incremental sync are exactly the same as full sync, except wherever a sync or import step were performed, it’s now replaced by Delta sync or Delta import, respectively. Only the changes flow through, not everything. Note that regardless of incremental or full synchronization, BCS synchronization will bring in all user data from the BCS source system.
Few final thoughts
As discussed here, user profile synchronization is a key element of populating the ‘person’ in your profile store. It’s critical to understand and plan what information resides in the profile store, where it comes from, how it gets there and how it’s maintained. Behind each of these key questions is the complex process of bringing data to-from directory sources such as AD and business systems such as SAP. Our hope is that this discussion helped you gain a better understanding of how it all works and how the data flows, so you can manage and maintain the profile store more effectively.
