The Case of the ADAM/AD LDS error A referral was returned from the server
My customer uses ADAM/AD LDS for directory aware applications instead up extending schema for Active Directory Domain Services. In short, they prefer to isolate line of business or 3rd party applications in their own sandbox and leave Active Directory schema alone when possible. The definition for ADAM\AD LDS from technet is "Active Directory Application Mode (ADAM) is a part of Microsoft’s fully integrated directory services available with Windows Server 2003, and is built specifically to address directory-enabled application scenarios. ADAM runs as a non-operating-system service, and, as such, it does not require deployment on a domain controller. Running as a non-operating-system service means that multiple instances of ADAM can run concurrently on a single server, and each instance can be configured independently."
In the past for their business, each application required a single application partition. As a result, their standard procedure included provisioning a new ADAM instance with a single application partition. The problem started when the requirements slightly changed. The new line of business application required two application partitions not one. They used their existing procedures for setting up one application partition for two application partitions scenario. Lab certification was failing due to intermittant errors in lab with error "0000202B: RefErr: DSID-03100697 A referral was returned from the server." when attempting to write to second application partition. They could not recreate the error when writing to first partition in a consistent manner.
For clarity let's walk through lab setup. Let's assume for the moment we have a domain called ACME.COM and our ADAM servers are called ADAM1.ACME.COM and ADAM2.ACME.COM. Let's create our ADAM instance and call it something simple and straight foward like "BlogInstance". Additionally, we can call our application partitions "BlogPartition1" and "BlogPartition2". Here are a few screen shots for initial setup to create the partition and first partition from which we can leverage the ADAM wizard.
[Use wizard to create ADAM instance]
[Specify ports]
[Create and name the application partition, first one of course]
[Import in box ldif files to create classes etc.]
OK. For brevity, I'll skip the rest of the wizards, next button throughout. So now in services on ADAM1.ACME.COM we can verify that new NT Service is installed.
As I mentioned, process is very important as it should be to the customer. Next step is to add ADAM2.ACME.COM to replica. From ADAM2.ACME.COM, let's run the ADAM wizard and create replica of existing instance.
[From ADAM2]
[Notice the application partitions available to copy, we have only created one and therefore this is only one presented]
At this point, the wizard completes successfully and we verify replication between ADAM1.ACME.COM and ADAM2.ACME.COM is working using command such as C:\windows\adam>repadmin /syncall ADAM1:389. So far so good. Next step which is new territory includes creation of second application partition for instance "BlogInstance". Let's say the second application partition is called "BlogPartition2" and is created on ADAM1.ACME.COM.
If you need to create the second partition for the application you can use dsmgmt.exe to accomplish task.
[Create second partition called BlogPartition2]
So now we have the same setup as the customer. Sure enough, if I try to create an object in the second partition from ADAM2.ACME.COM I get a referral error. Being able to reproduce the problem is huge in finding a resolution. Through research and collaboration with teammates an attribute, msDS-NC-Replica-Locations, was identified which could explain issue. This attribute was populated with both ADAM1.ACME.COM and ADAM2.ACME.COM for first partition where no problems were encountered. However, this same attribute for the second partition only listed one server where second partition was originally created, in our case ADAM1.ACME.COM. I think we are on to something here as our referral error goes away when executed from server that is populated in attribute msDS-NC-Replica-Locations.We need to understand how and why this value gets populated.
From MSDN:
2.194 Attribute msDS-NC-Replica-Locations
This attribute specifies a list of servers that are the replica set for the corresponding non-domain naming context.
cn: ms-DS-NC-Replica-Locations
ldapDisplayName: msDS-NC-Replica-Locations
attributeId: 1.2.840.113556.1.4.1661
attributeSyntax: 2.5.5.1
omSyntax: 127
omObjectClass: 1.3.12.2.1011.28.0.714
isSingleValued: FALSE
schemaIdGuid: 97de9615-b537-46bc-ac0f-10720f3909f3
systemOnly: FALSE
searchFlags: 0
linkID: 1044
systemFlags: FLAG_SCHEMA_BASE_OBJECT
[Partition 1, notice partition 1 lists both ADAM1.ACME.COM and ADAM2.ACME.COM]
[Partition 2 only contains one entry for ADAM1.ACME.COM]
How does referral error factor into this? The application making the query is reponsible for "chasing" a referral. In our case, ldifde.exe does not chase referrals, it's simply saying in order to make a change to second application partition, the change is required to be made on the server defined by msDS-NC-Replica-Locations attribute.
So how can we resolve the issue, customer preference is command line so setup can be automated.
1. C:\windows\ADAM>dsmgmt.exe
2. partition management
3. Connections
4. server connections: connect to server ADAM1:389 (Where ADAM1 is server where second partition was created and port is 389)
5. server connections: q
6. partition management: add NC Replica "DC=BlogPartition2,DC=ACME,DC=COM" ADAM2:389 (Where BlogPartition2 is second partition and you want to add ADAM2 to host NC replica)
8. partition management: q
8. dsmgmt: q
Let's verify that second application partition msDS-NC-Replica-Locations attribute includes both servers. Simply go back to ADSI-Edit and check second partition, it will now show both servers. Time to see if we can create object in second partition from ADAM2.ACME.COM as previously we received referral error.
Using LDIFDE.exe, placing ldf file in root of C:\ for simplicity.
[LDF - CreateContainer.ldf]
dn: CN=CreatOnADAM2,CN=Roles,DC=BlogPartition2,DC=ACME,DC=COM
changetype: add
objectClass: container
[ACME2.ACME.COM]
Success!
I hope this is helpful, I certainly learned quite a bit about ADAM\AD LDS and specifically purpose of Attribute msDS-NC-Replica-Locations as well as "Add NC Replica"