Post-ADC Problems In Federations

Okay, so these problems could potentially exist anywhere. The reason I draw attention to them is that, often times in federations, a consultant, such as myself, might only be working with one of the groups (the ones leading the initiative). As such, visibility into what is occuring in the other departments doesn't always exist. Most of the problems we encountered could've been avoided. In fact, we put some procedures in place to try and prevent them, but couldn't enforce the requirement that other departments would successfully complete the tasks. 

The ADC Tools consist of two components. The first component is part of the ADC snap-in for the Microsoft Management Console. The second component is a stand-alone utility titled Exdeploy.exe, which is available on the Microsoft Web Site for download. The purpose of the ADC Tools is to prevent Exchange 2003 customers (upgrading from 5.5) from making many of the directory synchronization mistakes that were common during Exchange 2000 rollouts.

The tools cover four basic functions: (1) General health check on servers and the environment, (2) Resource Mailboxes that were not matched 1:1 with user accounts, (3) Configuration of connection agreements, and (4) Verification that all objects have been replicated between Exchange 5.5 and Active Directory.

In federated environments, the ADC Tools present the following problems:

1. Validity checks assume that one group or administration has complete control over the entire Exchange environment.
2. The ADC will treat existing duplicate objects (a custom recipient in Exchange 5.5 and a mail-enabled user in Active Directory) as two distinct objects and won't flag them as problematic.
3. Without proper administrative control, the ADC cannot see hidden objects in Exchange 5.5 that belong to other administrations (this actually shows up as a warning in the tool).

Therefore, although the information provided by these tools is essential when dealing with the complexities of an Exchange 2003 rollout, the required overhead to manage the federated model mustn't be neglected.

Prior to turning on synchronization with the ADC, we attempted on several occasions to run the verification tools in the ADC and exdeploy.exe. Many of the problems reported by the tool (particularly resource mailbox issues) were valid and assigned for resolution. However, a number of issues were a bit more challenging. For example, the ADC tool designed to fix resource mailboxes requires credentials for each domain/site pair where those mailboxes reside. Asking folks to surrender their admin passwords was certainly not going to happen, so we could only report the problems from central IT and leave the resolution to agency IT departments. 

To ensure that we would have a clean deployment of the ADC, in spite of these limitations, we trained all agencies on the proper use of the tool and required them to sign off that they had completed their portion of the cleanup work. Signatures were centrally gathered and reported during the weekly status meetings. This provided protection for the central IT from advancing the schedule too aggressively while simultaneously pushing departmental IT to complete the appropriate tasks so that all could move forward.

The most common problem we experienced post-ADC deployment was the creation of duplicate objects. In most cases, administrators at several of the agencies had somehow mail-enabled user objects. The ADC interpreted this as the administrator's desire to add this user to the Exchange 5.5 GAL (as it would've shown up in a 200X GAL anyway) and created a custom recipient. However, a custom recipient of the same SMTP address already existed in Exchange 5.5 which created an un-resolvable mail object. To correct these types of problems, we typically recommended deleting the custom recipient in Exchange 5.5, and allowed that to replicate into the deletion of the contact in AD. 

This problem occurred for any number of reasons. Many times, it was an administrator with two accounts. One was mapped to an Exchange mailbox (mailbox-enabled user) and one was an administrator account that had its SMTP address filled in, but no mailbox in Exchange 5.5. The solution to this problem was easily resolved by removing the Exchange attributes from the administrator account and allowing it to delete the associated custom recipient. Another scenario that was common and more like the picture above was the case of contract employees. In many cases, they were given user accounts in AD (with SMTP address filled in) and a Custom Recipient in 5.5 was also created for them so they could participate in the GAL (as a recipient, anyway). 

The best way to resolve these types of problems upfront is to do an Active Directory search of objects that have SMTP addresses associated with them and ensure that they either map to a specific mailbox, or that the SMTP address is deleted before turning on the ADC. This can be a somewhat daunting task with 50,000 objects across multiple administrations, but generally, federations give us one good thing: a large number of people willing to do remedial tasks for the greater good (and their own good).

The final issue we encountered was that of hidden recipients. This one actually took an interesting turn during our rollout of the ADC. Prior to turning on the ADC, several objects throughout the Exchange 5.5 environment that were hidden were largely forgotten about. While this caused a few problems with duplicate resource mailboxes that could not be centrally detected, it actually created more problems post ADC. While we were attempting to validate the synchronization results we noticed that the NTObjectCheck.xml file grew to roughly 8K (it began as empty). When we looked at the objects reportedly un-matched, we could see that the MSExchAdcGlobalNames attribute was populated and it turned out to be matched to a hidden object in Exchange 5.5 that was under the control of another administration (and hence credentials). 

What had happened, in effect, is that the object post synchronization was mail-enabled from the initial sync. Because it was mail-enabled, the ADC tools verification sought a match in Exchange 5.5 but couldn't see it because it was hidden. This required us to launch yet another stage of federated validation, requiring each agency to validate their own results before we installed the first Exchange 2003 server. Although we had good confidence centrally that these 8K worth of problems were identical in nature, we could take the risk of progressing in a highly charged political environment.

Lesson learned: make no assumptions that what peer agencies or departments in a federation are telling you is true. Find ways to validate everything for yourself. In this case, we required every agency to sign off that they had done their homework, so they were responsible. That didn't, however, prevent me from supporting them through the issues discussed above.