Debugging address book issues with Debug-CsAddressBookReplication
As I explained in a previous post, the address book component of Lync underwent major changes in Lync Server 2013, but the basic functionality remains the same. Address Book is a deceivingly simple component. While how it works is easy to explain, there are a great number of things that may go wrong. Debugging issues can therefore become very difficult.
In Lync 2010, we introduced Powershell cmdlets such as Test-CsAddressBookWebQuery and Test-CsAddressBookService. The former is used to test the web query while the latter is used for file download. Documentation may be found within the management console (get-help cmdlet-name) and elsewhere online. These cmdlets are useful for determining whether a problem exists but are not as useful for determining why.
For this reason we have Debug-CsAddressBookReplication. This cmdlet provides a wealth of information about the replication process itself and provides the ability to process an individual user. There are several ways to use it.
This is the simplest use.
PS C:\> Debug-CsAddressBookReplication
Pool Fqdn : pool0.vdomain.com
Replication State : NotTested
Task Owner Fqdn : SERVER.vdomain.com
Backup Fqdn : fakepool.vdomain.com
Server Heartbeats : {Microsoft.Rtc.SyntheticTransactions.Activities.Database.AbServerHeartbe
at}
Indexed Object Count : 122
Objects not indexed that should be : 0
Abandoned objects : 0
Normalization Failure Count : Not checked
Normalization Failures :
When run without parameters, the cmdlet uses the default DC and whatever pool the machine is in. It provides the following general info.
Pool Fqdn – This is the pool that was checked.
Replication State – I will get to this one later.
Task Owner Fqdn – If this is an enterprise pool, this states which front end owns the address book task. This is the machine that should be checked for event logs for address book replication.
Backup Fqdn – Unfortunately due to a bug this should be ignored.
Indexed Object Count – This is the number of users searchable from this pool.
Objects not indexed that should be – This should always be 0. If it is not 0, then there is a problem indexing some users. This is a good time to call support.
Abandoned objects – This should also always be 0, but there are a few scenarios where non-zero numbers of acceptable. Proper understanding of this value and the previous one will require a future blog post.
Normalization Failure Count – This is the number of users who had a number that failed to normalize. In Lync 2010 this information was provided by a text file on the file share. It is now available from Powershell, but I will explain more shortly.
Normalization Failures – When requested, this will provide the users and fields that failed to normalize.
Server Heartbeats – This provides information on the current status of all front ends in the pool. If a front end has not heartbeated for some time, the backend will consider it down and will not assign tasks to it. To view more information, do something similar to the following.
PS C:\> $a = Debug-CsAddressBookReplication
PS C:\> $a.ServerHearbeats
Server Last Hearbeat Last Register Last Unregister
------ ------------- ------------- ---------------
SERVER.vdomain.com 10/10/2012 10:10:48 PM 10/3/2012 5:31:21 PM 1/1/0001 12:00:00 AM
So what can you determine from this information? This will verify that address book replication is up and running and where to look for event logs. The objects not indexed value is perhaps the most important. A non-zero number there indicates trouble. It means there are users that should be searchable but are not.
The output differs significantly with the –User parameter. This parameter accepts one of the following
- SIP address
- Active Directory object guid
- SamAccountName
It will search for the user in Active Directory and return results similar to the following.
PS C:\> Debug-CsAddressBookReplication -user sip:iggy@vdomain.com
Pool Fqdn : pool0.vdomain.com
Replication State : NotTested
Task Owner Fqdn : SERVER.vdomain.com
Backup Fqdn : fakepool.vdomain.com
Server Heartbeats : {Microsoft.Rtc.SyntheticTransactions.Activities.Database.AbServerHeartbe
at}
Indexed Object Count : 122
Objects not indexed that should be : 0
Abandoned objects : 0
User Guid : 98fea012-915c-4545-9ccf-6d981fc1dcd1
Distinguished Name : CN=iggy Flipopovich,CN=Users,DC=vdomain,DC=com
Sip Address : sip:iggy@vdomain.com
Attribute Values : {entryid=98fea012-915c-4545-9ccf-6d981fc1dcd1, sn=flipopovich,
givenName=iggy, msRTCSIP-PrimaryUserAddress=iggy...}
Is Indexed : True
Should Be Indexed : True
Is Processed : True
Normalization Succeeded : True
Manager :
Normalization Failure Count : Not checked
Normalization Failures :
With the –User parameter the cmdlet provides the following additional information.
User Guid – This is the guid of the object in Active Directory.
Distinguished Name – The DN of the object in Active Directory.
Sip Address – Should be obvious
Is Indexed – If true, then the user is searchable. If false, other users will not be able to search for this user. Note that there are some users (those will attributes set specifying Address Book should ignore them) where false is the valid value.
Should be Indexed – This indicates whether the user should be searchable. If this value is true and is indexed is false, then there is a problem. If this value is false but you expect the user to be searchable, then you have likely set some attribute for the user causing address book to ignore it or the user does not have a Sip address or phone number.
Is Processed – Whether address book processed this user at all. If the user has a value indicating address book should ignore it, this will be false.
Normalization succeeded – Whether the user has any phone numbers that failed to normalize.
Manager – The distinguished name of the user’s manager, if set.
Attribute Values - Key value pairs of the information we have for the user in the database. This is what is available for search for the user. To see them better, do something similar to the following.
PS C:\> $a = Debug-CsAddressBookReplication -User sip:iggy@vdomain.com
PS C:\> $a.AttributeValues
Name Value
---- -----
entryid 98fea012-915c-4545-9ccf-6d981fc1dcd1
sn flipopovich
givenName iggy
msRTCSIP-PrimaryUserAddress iggy
displayName iggy flipopovich
msRTCSIP-PrimaryUserAddress iggy@vdomain.com
msRTCSIP-PrimaryUserAddress sip:iggy@vdomain.com
The following other parameters are useful.
DomainController – Specifies the DC to use instead of the default.
PoolFqdn – Specifies the pool to examine instead of the one the current machine belongs to.
VerifyNormalization – This will provide information about all users in the database that failed to normalize. It is a potentially expensive operation because it must pull apart binary blobs in the database to provide the information. Therefore it should not be run at mission critical times. The following is an example of its use.
PS C:\> Debug-CsAddressBookReplication -VerifyNormalization
Normalization failures
Return information on users that failed to normalize.
[Y] Yes [N] No [S] Suspend [?] Help (default is "Y"): Y
Pool Fqdn : pool0.vdomain.com
Replication State : NotTested
Task Owner Fqdn : SERVER.vdomain.com
Backup Fqdn : fakepool.vdomain.com
Server Heartbeats : {Microsoft.Rtc.SyntheticTransactions.Activities.Database.AbServerHeartbe
at}
Indexed Object Count : 122
Objects not indexed that should be : 0
Abandoned objects : 0
Normalization Failure Count : 40
Normalization Failures : {orgSearchU3@vdomain.com(942e3b3d-e0f0-4dc2-b836-0193e075e49c)
otherTelephone=288-2062 x8865,
orgSearchU3@vdomain.com(942e3b3d-e0f0-4dc2-b836-0193e075e49c)
ipPhone=616-3862 x4841,
InetOUser@vdomain.com(5c49c430-df18-4e2a-8371-088c12c5a44a)
telephoneNumber=708-1283 x7773,
InetOUser@vdomain.com(5c49c430-df18-4e2a-8371-088c12c5a44a) ipPhone=+1
(441) 475-3351 EXT3825...}
To better view the normalization failures do something similar to the following.
PS C:\> $a = Debug-CsAddressBookReplication -VerifyNormalization
Normalization failures
Return information on users that failed to normalize.
[Y] Yes [N] No [S] Suspend [?] Help (default is "Y"): y
PS C:\> $a.NormalizationFailures
AttributeName AttributeValue SipAddress UserGuid
------------- -------------- ---------- --------
otherTelephone 288-2062 x8865 orgSearchU3@vdomain.com 942e3b3d-e0f0-4dc2-b836...
ipPhone 616-3862 x4841 orgSearchU3@vdomain.com 942e3b3d-e0f0-4dc2-b836...
telephoneNumber 708-1283 x7773 InetOUser@vdomain.com 5c49c430-df18-4e2a-8371...
ipPhone +1 (441) 475-3351 EXT3825 InetOUser@vdomain.com 5c49c430-df18-4e2a-8371...
telephoneNumber 700-8800 x0360 Grp_manager2@vdomain.com f0ed5c70-c279-4955-956b...
ipPhone 706-7007 x2837 Grp_manager2@vdomain.com f0ed5c70-c279-4955-956b...
telephoneNumber 707-8044 x0088 MoveUserTest@vdomain.com f800e93d-945c-48f1-89e1...
otherTelephone 704-4374 x5370 MoveUserTest@vdomain.com f800e93d-945c-48f1-89e1...
telephoneNumber 170-6181 x7880 orgSearchU1@vdomain.com 590e071a-8e4a-41d9-8522...
ipPhone +1 (022) 152-4752 EXT5172 orgSearchU1@vdomain.com 590e071a-8e4a-41d9-8522...
otherTelephone 318-1125 x1133 orgSearchU16@vdomain.com 9ef32825-2a4b-4d97-b17b...
otherTelephone 463-4133 x2783 abwsTenantTester@user 77b2b3f3-302c-44b8-a12a...
ipPhone 705-4041 x6023 abwsTenantTester@user 77b2b3f3-302c-44b8-a12a...
otherTelephone +1 (520) 125-4267 EXT5563 orgSearchU15@vdomain.com 5dbb9fbe-c026-40ae-bbcb...
telephoneNumber 157-1563 x4363 vt1_user66NT@vdomain.com 872d2a5b-48ea-4159-b016...
ipPhone 705-5734 x5503 vt1_user66NT@vdomain.com 872d2a5b-48ea-4159-b016...
otherTelephone 708-5235 x8175 orgSearchU6@vdomain.com 4d545448-8059-4ec4-8415...
ipPhone 704-8184 x3017 orgSearchU6@vdomain.com 4d545448-8059-4ec4-8415...
telephoneNumber +1 (445) 577-7031 EXT0542 orgSearchU5@vdomain.com 1fcc7136-9715-4dab-b811...
As you can see, this will provide the specific attribute, value, Sip address and user guid in Active Directory for the problem accounts.
VerifyReplication – This will do the following for a user.
- Append a value to the sn in Active Directory
- Wait for replication to occur
- Verify the new sn appears in the database
- Changes the sn back to the previous value
For most of you, this is most useful for obtaining logs to send to support as this forces replication for one user – making the logs far easier to read. For most other purposes you should use Update-CsAddressBook.
-Verbose – Similar to all other cmdlets, this provides more information in the console. Some may prefer the output here to that above.
While certainly our hope is you will never need to use this cmdlet, it is there to provide useful information that can help you better understand what is happening when a user fails to appear within search results.