FIM Troubleshooting: Generic Synchronization Errors
With declarative provisioning, FIM has introduced a very powerful technology to manage distributed identity data in your environment.
In some cases, you might experience issues during the synchronization process that require you to troubleshoot your configuration. If your issue is related to a specific error, you can use the related error message to start with a root cause analysis. However, there are issues that don’t have a specific error but an unexpected result as a starting point.
An example for this is a scenario in which accounts are supposed to be provisioned to an external system and the expected identity data is not available after running the related run profiles. Scenarios that don’t raise errors but do also not produce the expected results are also known as generic error scenarios.
The objective of this document is to walk you through the process of troubleshooting generic synchronization errors that are related to declarative provisioning.
Introduction to scoping generic synchronization errors
Troubleshooting generic synchronization errors requires an understanding of how the synchronization engine processes identity data under ideal conditions.
You can find a detailed description of how the process works in Understanding Data Synchronization with External Systems.
It is recommended that you familiarize yourself with the content of this document before reading the next sections.
A generic synchronization error doesn’t have a specific error as a starting point.
Typically, these errors are first identified in a target system.
For example, a specific object or specific attributes of an object do not appear in an external system or in the database of the FIM Service.
Your troubleshooting analysis should start with a picture that outlines your expected data flow under ideal conditions.
The following illustration outlines the data flow from a source to a target during the synchronization process from the FIM Service database to an AD DS:
In this scenario, for a successfully synchronized object called Britta Simon, the following must be true:
- A representation of Britta Simon must exist in the FIM service database
- A representation of Britta Simon must be staged in the FIM connector space
- The object in the FIM connector space must be linked to a representation of Britta Simon in the metaverse
- The metaverse representation of Britta Simon must be linked to a representation in the AD DS connector space
- A representation of Britta Simon must exist in AD DS.
To troubleshoot a generic synchronization error, you should compare your expected data flow with the actual data flow in your system.
Your comparison should start at the beginning of the complete synchronization process.
In the case of the current example, this is the representation of Britta Simon in the FIM service database.
In addition to that, you should map your expected data flow parts to the processing phases.
This is necessary because your verification steps might be dependent on the processing phase.
In FIM, synchronizing identity data between external systems (ES) is the objective of the FIM synchronization service.
The complete end-to-end process from a source to a target consists of three distinct phases:
- Import – Identity data from an external system ES1 is staged in a connector space CS1
- Synchronization – Identity data is synchronized between the connector spaces CS1 and CS2
- Export – identity data that is staged in a connector space CS2 is exported to an external system ES2
The following illustration outlines the process flow in these phases:
In the case of a generic synchronization error, it is very unlikely for the error to be related to the import or export process.
If something goes wrong in these phases, you will usually see a related error reported by the system in the Synchronization Service Manager.
In other words, if an expected object does not appear in a target system, it is very likely that nothing was staged for an export in the target connector space.
In this case, the export process has done what it is supposed to do.
As a consequence of this, you can remove import and export from your troubleshooting related analysis and set the focus on the synchronization process.
The synchronization process from a source to a target involves a source connector space, the metaverse and at least one target connector space. Technically, the synchronization process consists of two distinct phases:
- Inbound synchronization
- Outbound synchronization
The objective of the inbound synchronization phase is to bring identity data that has been staged in a connector space (CS) into the metaverse (MV).
When the inbound synchronization phase results in updates to the metaverse, the outbound synchronization phase is triggered.
The following illustration outlines the related process flow:
You can use the architecture of the synchronization process to scope your troubleshooting analysis.
For example, using the scenario outlined above, in which an object in the FIM service database is supposed to be provisioned to an AD DS, you should start your error analysis in the connector space of this FIM management agent and follow the data flow chain from the related source object to the connector space of the AD DS management agent.
Using the architecture of the synchronization process, you can draw the following conclusions:
- If the required identity data does not appear in the connector space of the FIM management agent, you have an issue with the import process
- If the required identity data does not appear in the metaverse, you have an issue with the inbound synchronization process
- If the required identity data does not appear in the connector space of the ADDS management agent, you have an issue with the outbound synchronization process
As stated earlier in this document, it is very unlikely that you have a generic error issue with the import process, which means, you can focus your attention on the inbound and outbound synchronization process.
The second scoping level for your troubleshooting strategy is the part of the identity data that is missing.
In other words, is you should scope your analysis based on whether a complete object or are attribute values are missing.
When you are done scoping your issue, you are ready to use the tools that are available to troubleshoot your issue.
Troubleshooting Tools
In addition to understanding the synchronization process, you should also familiarize yourself with the available tools to troubleshoot synchronization related issues. The available troubleshooting tools belong into the following categories:
- Configuration inspection / determination
- Object state inspection / determination
- Synchronization process analysis
In addition to the existing tools, FIM also provides a WMI and PowerShell based scripting interface that can be used to gather data and create reports.
You can find a collection of related scripts in the FIM ScriptBox.
Tools to verify your configuration
The synchronization process is governed by the configuration of your synchronization rules.
Some configuration settings are located in the user interface of the FIM Synchronization Service Manager.
For example, you use the FIM Synchronization Manager to configure your attribute flow precedence configuration.
There are also some configuration settings located on the FIM server.
For example, to configure your provisioning policy, you use the related dialog pages in the FIM portal.
While it is possible to implement verification routines directly into the user interface, for example, a routine that verifies whether a number was provided when a number is required, some valid configuration settings can turn into configuration mistakes when they are used in a specific context.
In other words, the user interface cannot detect all possible configuration mistakes.
To troubleshoot configuration mistakes, you can use, in addition to the user interface, the FIM PowerShell cmdlet that enables you to export object information as XML data.
The FIM ScriptBox provides a collection of verification scripts that are based on this method.
Some of these scripts indicate directly if a configuration mistake was detected.
The most relevant script in the context of configuration related issues is the FIM Object Visualizer, which enables you to create reports of your current configuration.
Tools to verify the current state of an object
The FIM synchronization process is state based.
This means, the synchronization engine uses the state information that is stored on a connector space object to make efficient processing decisions.
In addition to information that has been imported from an external system, FIM also stores information about the data that has been processed towards the metaverse and exported to an external system on each connector space object.
Examining the state based information stored on a connector space object is in many cases elementary to detect the root cause of an issue.
To provide information about the current state of a connector space object, FIM provides the following tools:
- Connector Space Export Tool
- Connector Space Search Feature
- Metaverse Search Feature
Connector Space Export Tool
The Connector Space Export Tool (CSExport) is a command line tool to export objects from the connector space of a management agent as XML data.
The tool enables you to configure filters to narrow down the scope of the exported objects.
| http://apfhrw.bay.livefilestore.com/y1pCed6u9dxDV3LOdzgtYt8xFaDWmIf_thMiOiFb3SmUARdxwIei5b6sPCHGruZWYphrJEU8j2BZEd51ZoAkp_ONkG8moMODvLC/Note.gif Note | |
|---|---|
| For more details about CSExport, see the Connector Space Export Tool in the Microsoft Forefront Identity Manager Help. |
To simplify the process of using CSExport, the FIM ScriptBox provides a graphical script called FIM CSExport Viewer that is a wrapper around CSExport.
Connector Space Search Feature
In the case of the initialization of a scenario or in the case of a troubleshooting exercise, before you synchronize objects, you should first get an overview of what is staged in a source connector space.
Next to CSExport, you can also use the Search Connector Space feature of the Synchronization Service Manager to do this.
The Search Connector Space feature provides an option to limit the scope of the returned objects.
For example, you can set the scope of a search to only return objects with a specific name or state.
The following screenshot shows an example for the available scopes in the Search Connector Space dialog:
The search results are returned in form of a table with configurable column settings.
The table view enables you to get an overview of the settings for a group of objects.
Using Sub-Tree a scope without specifying a distinguished name (DN) returns all of the objects within the connector space.
If you have many objects staged in your connector space, you can reduce the amount of returned data by specifying a scope.
| http://apfhrw.bay.livefilestore.com/y1pCed6u9dxDV3LOdzgtYt8xFaDWmIf_thMiOiFb3SmUARdxwIei5b6sPCHGruZWYphrJEU8j2BZEd51ZoAkp_ONkG8moMODvLC/Note.gif Note | |
|---|---|
| For more details about the scope options, see Search for a Connector Space Object in the Microsoft Forefront Identity Manager Help. |
The following screenshot shows an example for the results returned by a connector space search:
More details about a result can be obtained by double-clicking the result:
The connector space search result returns:
- The attribute values of a connector space object.
- Important state information regarding the synchronization process.
- The metaverse object the connector space object is linked to.
In other words, you can use the connector space object properties to troubleshoot synchronization process related issues.
Metaverse Search Feature
The metaverse search feature is similar to the connector space search.
The main difference is the perspective used to look at your identity data.
For a metaverse search, the entry point is a metaverse object.
The objects returned by a metaverse search include information about the connector space objects a metaverse object is linked to as well as the method that was used to establish the connection:
| http://apfhrw.bay.livefilestore.com/y1pCed6u9dxDV3LOdzgtYt8xFaDWmIf_thMiOiFb3SmUARdxwIei5b6sPCHGruZWYphrJEU8j2BZEd51ZoAkp_ONkG8moMODvLC/Note.gif Note | |
|---|---|
| For more details about the scope options, see Using Metaverse Search in the Microsoft Forefront Identity Manager Help. |
Tools to analyze the synchronization process
Verifying your configuration is the first step to detect configuration mistakes.
However, this does not always help to find the error.
In this case, you can either initiate a test synchronization run for a specific object or you can analyze the reported statistics of an actual synchronization run
For both methods to work, you should always start with an expectation.
For example, if you plan to synchronize 100 new objects from a source to a target, you should review the reported values and compare them with your expectation.
As a best practice, you should write your expectation and the reported values down.
When you use the related tools to analyze the synchronization process, you should always keep the architecture of the synchronization process in mind:
The available tools to analyze the synchronization process are structured in a way that provides the related information in alignment with this process.
To provide information about the synchronization process, FIM provides the following tools:
- Management Agent Statistics
- Synchronization Statistics
- Synchronization Process Preview
Management Agent Statistics
The management agent statistics dialog of the Synchronization Service Manager gives you a high level report about the data that has been staged in the connector space of an object.
This data gives you a first impression about the health of a connector space.
For example, if the number of disconnectors is high you might have an issue with your inbound synchronization rule.
The following screenshot shows an example of the management agent statistics:
| http://apfhrw.bay.livefilestore.com/y1pCed6u9dxDV3LOdzgtYt8xFaDWmIf_thMiOiFb3SmUARdxwIei5b6sPCHGruZWYphrJEU8j2BZEd51ZoAkp_ONkG8moMODvLC/Note.gif Note | |
|---|---|
| For more details about the management agent statistics, see Management Agent Run Statistics in the Microsoft Forefront Identity Manager Help. |
With using PowerShell to display the statistics of a management agent, the FIM ScriptBox provides a script you can use to create a related report.
Synchronization Statistics
When you run a run profile, you can use the Synchronization Statistics in the user interface of the Synchronization Service Manager to retrieve processing information.
Reported updates are indicated in form of a link you can use to review the affected objects.
The following screenshot shows an example of the reported statistics for a delta import synchronization run:
| http://apfhrw.bay.livefilestore.com/y1pCed6u9dxDV3LOdzgtYt8xFaDWmIf_thMiOiFb3SmUARdxwIei5b6sPCHGruZWYphrJEU8j2BZEd51ZoAkp_ONkG8moMODvLC/Note.gif Note | |
|---|---|
| For more details about the run statistics, see Management Agent Run Statistics in Help in the Microsoft Forefront Identity Manager Help. |
Synchronization statistics are grouped according to the architecture of the synchronization process into inbound and outbound synchronization.
While inbound synchronization has always one connector space as source and the metaverse as a target, outbound synchronization statistics have to be grouped by target connector spaces.
The following screenshot shows an example for the synchronization statistics of a synchronization run:
Synchronization Process Preview
In the Synchronization Service Manager, you can get information about the current state of your objects and you can also use it to simulate (“what if”) a synchronization run for an object.
In the FIM terminology, this is also known as preview.
All dialogs that are displaying connector space object details provide the option to run a preview.
This includes, for example, the Search Connector Space feature that was introduced in a previous section.
To launch the Preview dialog, you click the Preview button on the properties dialog page of a connector space object:
During a preview run, your synchronization rules are applied to the selected object and the processing results are displayed.
The following screenshot shows an example for this:
The option to apply your synchronization rules to an object makes the preview feature a very powerful tool in the troubleshooting process.
| http://apfhrw.bay.livefilestore.com/y1pCed6u9dxDV3LOdzgtYt8xFaDWmIf_thMiOiFb3SmUARdxwIei5b6sPCHGruZWYphrJEU8j2BZEd51ZoAkp_ONkG8moMODvLC/Note.gif Note | |
|---|---|
| Next to simulating an actual synchronization run, you can also use the preview to physically apply a synchronization run (“commit preview”) to an object.
If your issue is related to the data of your objects and not the configuration, this option is helpful if you need to isolate an object that is causing problems from a group of objects. |
| http://apfhrw.bay.livefilestore.com/y1pCed6u9dxDV3LOdzgtYt8xFaDWmIf_thMiOiFb3SmUARdxwIei5b6sPCHGruZWYphrJEU8j2BZEd51ZoAkp_ONkG8moMODvLC/Note.gif Note | |
|---|---|
| For more details about the preview feature, see Using Preview in the Microsoft Forefront Identity Manager Help. |
Generic Troubleshooting Operations
In the previous sections, you have been introduced to the tools you can use to start troubleshooting a configuration issue.
The objective of this section is to provide you with specific troubleshooting steps for generic synchronization issues.
Provisioning Configuration Verification
The Synchronization Service Manager provides you with an option to enable or disable the provisioning process.
The following screenshot shows the related dialog:
As a first step to troubleshoot generic synchronization issues, you should review and document the current provisioning process related configuration in your environment.
Especially, in cases where you can’t get objects pushed out to an external system, it is possible that you can fix the issue simply by enabling synchronization rule provisioning.
The FIM ScriptBox provides a script called FIM Provisioning Configuration Documenter that generates a report of your current synchronization rule provisioning configuration.
ERL Population Verification
A basic requirement for object information to be pushed out to an external system is that the object is indirectly linked to the related outbound synchronization rule.
Indirectly means that the relationship between the affected object and the outbound synchronization rule includes a third object called Expected Rule Entry (ERE).
In other words, the affected object points to an ERE and the ERE points to the outbound synchronization rule.
The attribute that links a resource to an ERE is known as Expected Rules List (ERL), which is a multi-valued attribute.
The following illustration shows an example for the related architecture:
The architecture is also known as outbound synchronization triple.
You can find more details about the outbound synchronization triple management process in Understanding Data Synchronization with External Systems.
From a troubleshooting perspective, you should examine the current state of the outbound synchronization triple.
In FIM, an object is brought into the scope of a synchronization rule by the FIM service.
If an object has been brought into the scope of synchronization rule, the Synchronization Rule Status is Pending.
The following screenshot shows an example for this.
While objects are brought into the scope of a synchronization rule by the FIM Service, a synchronization rule is applied to an object by the FIM synchronization service.
An actively applied synchronization rule is indicated by a Synchronization Rule Status of Applied.
The following screenshot shows an example for this:
The FIM ScriptBox provides scripts to display the value of the ERL attribute for a user and a group.
If the affected object does not have an outbound synchronization triple, you need to determine why it is missing.
It is possible that the outbound synchronization rule triple is missing due to a mistake in your synchronization policy configuration.
A synchronization policy configuration consists of several components such as workflows, management policy rules (MPR) and synchronization rules that have link relationships.
If the required relationship has not been configured correctly, the FIM service cannot create an outbound synchronization triple.
The FIM ScriptBox provides a script that generates a report of your current synchronization policy configuration.
The following screenshot shows an example of a report that was generated by this script:
You can use the report that is generated by this script to verify the accuracy of your synchronization policy configuration and to detect misconfiguration.
To enable the synchronization engine to apply synchronization rule to an object, the following must be true:
- An inbound attribute flow mapping for ERL attribute must exist
- The related ERL object must be in the metaverse
- The related synchronization rule object must be in the metaverse
After you have verified that the outbound synchronization triple exists, you should use a connector space search to determine whether the triple components have been staged in the FIM connector space.
The following screenshot shows an example of this:
The identity object and the associated ERE objects are tied together by the ERL attribute.
To preserve the link relationship between these objects during the transition from the FIM connector space to the metaverse, an import attribute flow mapping must be configured for the ERL attribute for the affected object type in the FIM management agent’s configuration.
The following screenshot shows an example of this:
The FIM ScriptBox provides a script you can use to determine the current configuration of the ERL attribute flow mapping configuration.
The referential relationship between a resource and the ERE should be available in the metaverse if the import attribute flow mapping for the ERL attribute exists.
You can verify the existence of the relationship by using the Metaverse Search.
The following screenshot shows an example of this:
If you have verified that ERL attribute has been successfully populated in the metaverse, you have also successfully completed the ERL population verification process.
This means that your scenario is correctly configured to provision objects to an external system.
Synchronization Rule Configuration Verification
The configuration of your outbound synchronization rules is typically dependent on your scenario requirements.
However, there is one outbound attribute flow mapping you must always configure if your outbound synchronization rule is configured to create objects in an external system – a mapping for the DN attribute.
The following screenshot shows the related configuration setting of an outbound synchronization rule:
If an outbound synchronization rule is supposed to create resources in external systems, you must configure an initial outbound attribute flow mapping for the DN attribute.
A potential indicator for a missing initial outbound attribute flow mapping for the DN attribute is a Synchronization Rule Status of Not Applied.
The following screenshot shows an example for this:
The DN attribute has, in the context of the synchronization process, a special function and updates to this attribute are not treated as regular attribute flows but as rename operations.
The following screenshot shows an example for this:
If your outbound synchronization rule handles DN updates, you need to configure an initial outbound attribute flow mapping and a regular outbound attribute flow mapping for the DN attribute.
When you implement the rename scenario by using an initial and a regular attribute flow for the DN attribute, you should keep in mind, how both flows are applied during the synchronization process.
Initial attribute flow mappings belong to the object creation or provisioning process.
When the synchronization process provisions a new connector to a target connector space, all initial attribute flow mappings are used to initialize the newly created object.
After the object level operations have been applied, the synchronization engine also applies attribute level operations such as attribute flows.
This has a consequence for newly provisioned objects.
If you have an initial and a regular attribute flow mapping for the same attribute configured, both flows are applied to the object.
Your attribute flow configuration should include the required logic to handle this scenario to avoid surprises with the final value of your DN attribute.
In many cases, the value of the DN attribute is calculated based on a string concatenation that starts with a fixed string such as “DN=”; however, there are also cases where the entire DN is tracked in an attribute of an object. In this case, you should always apply the Trim function to the attribute flow mapping to avoid whitespace related issues. The following screenshot shows a configuration example:
Object Dependencies Verification
Relationships between objects are typically expressed in the form of reference attributes.
This means the referencing object has the “address” of the referenced objects as value.
The member attribute of a group is a common example of this.
FIM preserves referential integrity during a synchronization run of an object.
This means existing relationships between objects are preserved during a transition from a connector space to the metaverse and vice versa.
However, in order to preserve the referential integrity of objects, the referencing and the referenced objects must be in the same storage segment such as a connector space and the metaverse.
The following illustration outlines the related architecture:
When, for example, a member of a group was lost during a transition, you should first verify, whether a representation exists for the referenced object.
If a representation does not exist, the synchronization engine is not able to calculate the related value for the reference attribute of the referencing object.
Losing a reference to an object during a transition does not generate errors.
To detect whether a missing reference is caused by the absence of the referenced object, you can use the connector space or the metaverse search feature of the Synchronization Service Manager.
Your search results should return the object architecture shown in the picture above.
Attribute Flow Precedence Configuration Verification
During the synchronization process, alleged data loss can also be a result of your attribute flow precedence configuration.
The attribute flow precedence process determines whether a source is granted permission to flow an attribute value to a target.
If the source does not have a sufficient precedence value, the precedence process rejects an attempt to flow an attribute value.
The following illustration outlines how this process works:
An attribute flow precedence check is applied during inbound and outbound attribute flows. The best method to detect attribute flow precedence related issues is using the preview feature. When the flow precedence process rejects an attribute flow, you will find a skipped not precedent status for the related flow. The following screenshot shows an example for this:
In the FIM ScriptBox, you can find a script called FIM Object Visualizer that contains amongst other reports also a report of your current attribute flow precedence configuration.
Generic Synchronization Troubleshooting Steps
The objective of this section is to provide you with specific steps to troubleshoot common generic synchronization errors and lists the related scripts that can assist you in a troubleshooting task.
In FIM, to flow attributes, you first need the required infrastructure of connector space and metaverse objects that are linked to each other.
Before you can look into attribute deployment related issues, you should first verify that the required object architecture is in place.
Troubleshooting generic object provisioning issues
The focus of this section is the healthy processing architecture for provisioning objects to an external system.
A healthy architecture has the following structure:
When you troubleshoot a generic provisioning issue, you can use the following table as guideline. As soon as the answer to one of these questions is “no”, you should fix the configuration before you move on with the next step.
| Step | Question | Answer |
| 1 | Is Synchronization Rule Provisioning enabled? | |
| 2 | Does the object have a reference to the right synchronization rule? | |
| 3 | Is the state of your Expected Rules List entry Applied? | |
| 4 | Is Create Resource in External System enabled in the outbound synchronization rule? | |
| 5 | Is an outbound attribute flow mapping for the DN attribute configured in your synchronization rule? | |
| 6 | Are the source and the related ERE object in the FIM connector space? | |
| 7 | Is an ERL attribute flow mapping configured in the Synchronization Service manager? | |
| 8 | Are the source and the related ERE object in the metaverse? |
Related Scripts
This section lists the scripts that assist you in your troubleshooting exercise:
Troubleshooting attribute deployment issues
The effect of an attribute deployment related issues is relatively simple – an expected attribute value does not show up on a target object.
The basic requirement for deploying attribute values is an active attribute flow mapping.
The first troubleshooting sequence of an attribute deployment related issue includes all the object provisioning related steps that are listed in the previous section.
In other words, to deploy attribute values, you also need to have the following architecture in place:
| http://apfhrw.bay.livefilestore.com/y1pCed6u9dxDV3LOdzgtYt8xFaDWmIf_thMiOiFb3SmUARdxwIei5b6sPCHGruZWYphrJEU8j2BZEd51ZoAkp_ONkG8moMODvLC/Note.gif Note | |
|---|---|
| Any kind of outbound facing activity requires an outbound synchronization triple to be deployed in the metaverse. This includes the cases in which you need to update attribute values on already existing objects. |
Deploying Reference attributes
In the context of attribute deployments, reference attributes play a special role.
As outlined under “Object Dependencies Verification”, for reference attributes to flow, the referencing and the referenced object must be in the same data segment in the FIM synchronization service.
The following illustration outlines the related architecture:
If the related architecture does not exist, the FIM synchronization service cannot resolve the value of a reference attribute.
Analyzing Attribute Flow Precedence Issues
The most common issue in conjunction with attribute deployment is related to attribute flow precedence.
An attribute flow precedence evaluation is applied during inbound attribute flow and outbound attribute flow.
The first troubleshooting step is to review your current attribute flow precedence configuration.
The following screenshot shows the related dialog in the Synchronization Service Manager:
In addition to this, you can also use the FIM Object Visualizer to create a related report for your environment.
If a report doesn’t reveal the actual issue, you should run a preview on one of the affected objects to determine the reason for the attribute population issue.
Summary
Troubleshooting generic synchronization errors is not that difficult if you are familiar with the synchronization process and its components.
It is important to start at the right place with your analysis.
The right place is the source connector space of your object.
From the source connector space, you follow the processing chain through the metaverse to the target connector space.
By using the related search tool such as connector space and metaverse search, you examine the attributes and the connection state of the affected object to determine what the issue is.
If analyzing the current state of the objects doesn’t provide the required information to solve the issue, you should run synchronization in preview mode, which will tell you how the synchronization service would process the objects.
Key for your analysis is to write down your expectations first, and then to compare them with your actual results. Recording your troubleshooting steps is helpful – even if you can’t solve the issue.
You can use the data in a post to the FIM forum or when you contact PSS to speed up the troubleshooting process.
In many cases, it is advisable to also include a report of your current configuration.
The FIM ScriptBox provides with the FIM Object Visualizer a reporting tool you can use to create configuration reports.
Community Resources
- Troubleshooting FIM 2010 Roadmap
- Understanding Data Synchronization with External Systems (TechNet Library)
- About Reference Attributes (TechNet Forums)
- About Attribute Flow Precedence (TechNet Forums)