FIM 2010: Providing confirming imports to the sync engine when your target system doesn’t support delta imports
Introduction
There are many systems out there that just don't support delta imports. Deltas are important for ensuring speedy and efficient operation of the FIM synchronization engine. While we can't invent true deltas when the connected system doesn't provide the information, sometimes its enough to just provide FIM the deltas of what it has changed. This is especially relevant for systems where FIM is mostly responsible for the data in the connected system.
What would be handy, is if at export time, we could provide FIM with the confirmed changes made in the target system. While FIM allows us to say that an export was successful, it still expects to confirm those changes took place on the next import operation. What if we could construct our import CSEntryChange objects at export time?
The Lithnet.MetadirectoryServices library contains a set of helper classes to make writing management agents and rules extensions a little bit easier.
One of the tools it contains is a CSEntryChangeQueue class, that gives you the ability to serialize CSEntryChange objects to XML and back again. CSEntryChange objects are placed in the queue on export and saved to disk at the end of the operation. On a delta import operation, the queue is reloaded from disk, and the CSEntryChanges are passed back to the sync engine. There are two scenarios we can explore that can take advantage of this functionality.
Scenario 1: Re-playing the exported CSEntryChange back to the sync engine on delta import
This is a really quick and easy way to get confirming imports. After the successful export of an item to the target system, we can simply pass the CSEntryChange exported by FIM into the CSEntryChangeQueue, and get it back from the queue when the next import operation is called. Provided nothing went wrong with the export, we have all the information needed to provide the confirming import to FIM on the next import operation.
 Caution |
|---|
| Important: you must ensure that you only replay CSEntryChanges for successful exports, otherwise you may misrepresent the state of your connected system to the sync engine. Anyway, a full import will be required for FIM to get the correct information. |
Scenario 2: Constructing the delta import CSEntryChange at export time, and saving it for the next delta import
This is a really cool way to provide deltas. Let's say you update a resource, and upon update, the system provides you a copy of (or link to) the resource after your update. This is quite a common practice with REST APIs. After a successful PUT or POST operation, you may receive a new copy of the resource as it appears after your modification. You actually have a true delta representation that you want to provide to the sync engine, its just that you are in the middle of an export, and the sync engine doesn't want it yet! So, you can construct your CSEntryChange for import as you normally would with the information returned by the system, and submit it to the delta queue. The next time an import is called, the correct data will be passed to the sync engine, without making another potentially expensive call to the target system.
Even if the target system doesn't automatically provide you an updated copy of the resource, there's nothing stopping you from getting the object yourself after export and constructing your import CSEntryChange. After all, at this point in time, you know the resource has been changed - once the export operation is complete, you've lost that information.
Remember, its not a true delta
Both these scenarios can potentially save a call to the target system and each allows you to clear the pending export confirmation without having to do a full import.
What you don't get however, is changes made outside of FIM. These will still need to be obtained via a full import process.
However, if your target system is only updated by the sync engine, then this process will work well. In a worst case scenario, you can have confirming imports immediately after an export, and run regular (perhaps less regular) full imports to obtain other changes.
Using the Lithnet.MetadirectoryServices.CSEntryChangeQueue
Let's have a look at how to use the built-in CSEntryChangeQueue object to load and save deltas to a file after export. The following code shows a call-based ECMA 2 MA, that stores the delta information at export time in the queue. At the end of the export operation, the CSEntryChanges in queue are saved to an XML file.
public class MASample : IMAExtensible2CallExport, IMAExtensible2CallImport
{
private string filename = @"D:\MAData\MyMA\Delta.xml";
private Schema operationSchema;
private OpenImportConnectionRunStep importRunStepParameters;
public void OpenExportConnection(KeyedCollection<string, ConfigParameter> configParameters, Schema types, OpenExportConnectionRunStep exportRunStep)
{
// The schema types are required for serialization, so save them into a local variable for use in CloseExportConnection
this.operationSchema = types;
// Load the existing items from the queue. We want to add to this collection, rather than overwrite it as an import operation may not have been run yet
CSEntryChangeQueue.LoadQueue(this.filename);
// ... Export initialization code
}
public PutExportEntriesResults PutExportEntries(IList<CSEntryChange> csentries)
{
foreach (CSEntryChange item in csentries)
{
try
{
// Export Code
// ...
// ...
// On successful export, add the CSEntryChange to the queue
CSEntryChangeQueue.Add(item);
}
catch (Exception)
{
// If the export failed, do not add the CSEntryChange to the queue
}
}
return new PutExportEntriesResults();
}
public void CloseExportConnection(CloseExportConnectionRunStep exportRunStep)
{
// Save the items in the queue to the specified file
CSEntryChangeQueue.SaveQueue(this.filename, this.operationSchema);
}
}
Upon import, we first check to see if we are doing a full or delta import. If a full import has been requested, we import directly from the source system. If a delta import is requested, we load the queue from the disk, and replay the CSEntryChanges back to the sync engine. Once either a delta or full import operation is completed, we clear the queue, and save the empty list back to the disk.
public OpenImportConnectionResults OpenImportConnection(KeyedCollection<string, ConfigParameter> configParameters, Schema types, OpenImportConnectionRunStep importRunStep)
{
// The schema types are required for serialization, so save them into a local variable for use in CloseImportConnection
this.operationSchema = types;
this.importRunStepParameters = importRunStep;
if (this.importRunStepParameters.ImportType == OperationType.Delta)
{
// Load the items from the queue
CSEntryChangeQueue.LoadQueue(this.filename);
}
return new OpenImportConnectionResults();
}
public GetImportEntriesResults GetImportEntries(GetImportEntriesRunStep importRunStep)
{
GetImportEntriesResults results = new GetImportEntriesResults();
if (this.importRunStepParameters.ImportType == OperationType.Delta)
{
int count = 0;
// Add items to the result collection until the page size has been exceeded or
// The queue is empty
while (CSEntryChangeQueue.Count > 0 && (count < this.importRunStepParameters.PageSize))
{
results.CSEntries.Add(CSEntryChangeQueue.Take());
count++;
}
// If the queue is not yet empty, tell the sync engine that we have more to import
results.MoreToImport = CSEntryChangeQueue.Count > 0;
}
else
{
// Perform normal full import
}
return results;
}
public CloseImportConnectionResults CloseImportConnection(CloseImportConnectionRunStep importRunStep)
{
if (this.importRunStepParameters.ImportType == OperationType.Full)
{
// We just performed a full import so clear the queue
CSEntryChangeQueue.Clear();
}
// Save the remaining items in the queue. If we have done a full import, the queue will be empty, and will clear any existing delta records
CSEntryChangeQueue.SaveQueue(this.filename, this.operationSchema);
return new CloseImportConnectionResults();
}
Summary
While this pattern is not necessarily applicable when dealing with 'source' systems, it does have a place for 'target' systems that are predominately managed by FIM. Even in systems that do have a lot of changes that aren't made by the sync engine, there is still a net gain. If you have 10,000 objects in your target, and are doing hourly full imports to get those changes, these methods allow you to supplement your full imports with fast, frequent confirming delta imports. Keep in mind the following points;
- It doesn't negate the need to do full imports
- It does negate the need to do a full import after export purely to confirm the last export run
- If there are no changes made in the target system that FIM doesn't care about, it's as good as having delta support in the target system
- You need to ensure that the 'delta' CSEntryChange accurately reflects the state of the target system
Downloads
Get the nuget package today and read the documentation for full details.