Exchange 2013 domainprep (prepareAD) fails with Microsoft.Exchange.Data.Directory.AdminLimitExceededException [ERROR_DS_ADMIN_LIMIT_EXCEEDED]
We've seen 2 cases in support with this issue that affected hosting companies. I anticipate we'll see more instances of this issue as more customers migrate to Exchange 2013, especially hosting companies, so I've decided to blog it out.
As with any installation issue, you start off by inspecting C:\ExchangeSetupLogs\ExchangeSetup.log. In there, you'll find that we were in the midst of creating Microsoft Exchange Security Groups (setup is inside the function initialize-ExchangeUniversalGroups) and then this error happens:
[10/15/2014 19:49:01.0934] [2] [ERROR] The administrative limit for this request was exceeded.
[10/15/2014 19:49:01.0934] [2] [ERROR] The administration limit on the server was exceeded.
[10/15/2014 19:49:01.0965] [2] Ending processing initialize-ExchangeUniversalGroups
[10/15/2014 19:49:02.0059] [1] The following 1 error(s) occurred during task execution:
[10/15/2014 19:49:02.0106] [1] 0. ErrorRecord: The administrative limit for this request was exceeded.
[10/15/2014 19:49:02.0106] [1] 0. ErrorRecord: Microsoft.Exchange.Data.Directory.AdminLimitExceededException: The administrative limit for this request was exceeded. ---> System.DirectoryServices.Protocols.DirectoryOperationException: The administration limit on the server was exceeded.
at System.DirectoryServices.Protocols.LdapConnection.ConstructResponse(Int32 messageId, LdapOperation operation, ResultAll resultType, TimeSpan requestTimeOut, Boolean exceptionOnTimeOut)
at System.DirectoryServices.Protocols.LdapConnection.SendRequest(DirectoryRequest request, TimeSpan requestTimeout)
at Microsoft.Exchange.Data.Directory.PooledLdapConnection.SendRequest(DirectoryRequest request, LdapOperation ldapOperation, Nullable`1 clientSideSearchTimeout, IActivityScope activityScope, String callerInfo)
at Microsoft.Exchange.Data.Directory.ADDataSession.ExecuteModificationRequest(ADObject entry, DirectoryRequest request, ADObjectId originalId, Boolean emptyObjectSessionOnException, Boolean isSync)
--- End of inner exception stack trace ---
at Microsoft.Exchange.Data.Directory.ADDataSession.AnalyzeDirectoryError(PooledLdapConnection connection, DirectoryRequest request, DirectoryException de, Int32 totalRetries, Int32 retriesOnServer)
at Microsoft.Exchange.Data.Directory.ADDataSession.ExecuteModificationRequest(ADObject entry, DirectoryRequest request, ADObjectId originalId, Boolean emptyObjectSessionOnException, Boolean isSync)
at Microsoft.Exchange.Data.Directory.ADDataSession.ExecuteModificationRequest(ADObject entry, DirectoryRequest request, ADObjectId originalId, Boolean emptyObjectSessionOnException)
at Microsoft.Exchange.Data.Directory.ADDataSession.Save(ADObject instanceToSave, IEnumerable`1 properties, Boolean bypassValidation)
at Microsoft.Exchange.Management.Tasks.InitializeExchangeUniversalGroups.CreateWKGuid(ADContainer container, ADObjectId dn, Guid wkGuid)
at Microsoft.Exchange.Management.Tasks.InitializeExchangeUniversalGroups.CreateGroup(ADOrganizationalUnit usgContainer, String groupName, Int32 groupId, Guid wkGuid, String groupDescription, GroupTypeFlags groupType, Boolean createAsRoleGroup)
at Microsoft.Exchange.Management.Tasks.InitializeExchangeUniversalGroups.CreateGroup(ADOrganizationalUnit usgContainer, String groupName, Int32 groupId, Guid wkGuid, String groupDescription)
at Microsoft.Exchange.Management.Tasks.InitializeExchangeUniversalGroups.InternalProcessRecord()
at Microsoft.Exchange.Configuration.Tasks.Task.<ProcessRecord>b__b()
at Microsoft.Exchange.Configuration.Tasks.Task.InvokeRetryableFunc(String funcName, Action func, Boolean terminatePipelineIfFailed)
What does this error mean? If you do some digging, you'll eventually find out that every attribute in Active Directory needs to fit into a 8KB ESE page. 8KB fits about 1200-1300 values in a nonlinked multivalued attribute. The question then becomes what attribute is causing our problem? We take an LDAP trace with the help of https://support.microsoft.com/kb/2221529 to determine the LDAP query causing the error and we see the following:
[0]11D4.13D0::10/22/2014-12:33:06.489 [Microsoft-Windows-LDAP-Client/Debug] ldap_modify called for connection 0x234e0898: DN is CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=test,DC=lab. Synchronous is 0x0.
<redacted for brevity>
[0]11D4.13D0::10/22/2014-12:33:06.489 [Microsoft-Windows-LDAP-Client/Debug] Unencrypted dump of Data sent on connection 0x234e0898
[0]11D4.13D0::10/22/2014-12:33:06.489 [Microsoft-Windows-LDAP-Client/Debug] 30 84 00 00 00 fb 02 02 10 3f 66 84 00 00 00 f1 0........?f.....
[0]11D4.13D0::10/22/2014-12:33:06.489 [Microsoft-Windows-LDAP-Client/Debug] 04 41 43 4e 3d 4d 69 63 72 6f 73 6f 66 74 20 45 .ACN=Microsoft.E
[0]11D4.13D0::10/22/2014-12:33:06.489 [Microsoft-Windows-LDAP-Client/Debug] 78 63 68 61 6e 67 65 2c 43 4e 3d 53 65 72 76 69 xchange,CN=Servi
[0]11D4.13D0::10/22/2014-12:33:06.489 [Microsoft-Windows-LDAP-Client/Debug] 63 65 73 2c 43 4e 3d 43 6f 6e 66 69 67 75 72 61 ces,CN=Configura
[0]11D4.13D0::10/22/2014-12:33:06.489 [Microsoft-Windows-LDAP-Client/Debug] 74 69 6f 6e 2c 44 43 3d 73 68 31 31 2c 44 43 3d tion,DC=test,DC=
[0]11D4.13D0::10/22/2014-12:33:06.489 [Microsoft-Windows-LDAP-Client/Debug] 6c 61 62 30 84 00 00 00 a8 30 84 00 00 00 a2 0a lab0.....0......
[0]11D4.13D0::10/22/2014-12:33:06.489 [Microsoft-Windows-LDAP-Client/Debug] 01 00 30 84 00 00 00 99 04 15 6f 74 68 65 72 57 ..0.......otherW
[0]11D4.13D0::10/22/2014-12:33:06.489 [Microsoft-Windows-LDAP-Client/Debug] 65 6c 6c 4b 6e 6f 77 6e 4f 62 6a 65 63 74 73 31 ellKnownObjects1
[0]11D4.13D0::10/22/2014-12:33:06.489 [Microsoft-Windows-LDAP-Client/Debug] 84 00 00 00 7c 04 7a 42 3a 33 32 3a 30 32 35 32 ....|.zB:32:0252
[0]11D4.13D0::10/22/2014-12:33:06.489 [Microsoft-Windows-LDAP-Client/Debug] 32 45 43 46 39 39 38 35 39 38 34 41 39 32 33 32 2ECF9985984A9232
[0]11D4.13D0::10/22/2014-12:33:06.489 [Microsoft-Windows-LDAP-Client/Debug] 30 35 36 46 43 37 30 34 43 43 38 42 3a 43 4e 3d 056FC704CC8B:CN=
[0]11D4.13D0::10/22/2014-12:33:06.489 [Microsoft-Windows-LDAP-Client/Debug] 4d 61 6e 61 67 65 64 20 41 76 61 69 6c 61 62 69 Managed.Availabi
[0]11D4.13D0::10/22/2014-12:33:06.489 [Microsoft-Windows-LDAP-Client/Debug] 6c 69 74 79 20 53 65 72 76 65 72 73 2c 4f 55 3d lity.Servers,OU=
[0]11D4.13D0::10/22/2014-12:33:06.489 [Microsoft-Windows-LDAP-Client/Debug] 4d 69 63 72 6f 73 6f 66 74 20 45 78 63 68 61 6e Microsoft.Exchan
[0]11D4.13D0::10/22/2014-12:33:06.489 [Microsoft-Windows-LDAP-Client/Debug] 67 65 20 53 65 63 75 72 69 74 79 20 47 72 6f 75 ge.Security.Grou
[0]11D4.13D0::10/22/2014-12:33:06.489 [Microsoft-Windows-LDAP-Client/Debug] 70 73 2c 44 43 3d 73 68 31 31 2c 44 43 3d 6c 61 ps,DC=test,DC=la
[0]11D4.13D0::10/22/2014-12:33:06.489 [Microsoft-Windows-LDAP-Client/Debug] 62 b
[0]11D4.13D0::10/22/2014-12:33:06.489 [Microsoft-Windows-LDAP-Client/Debug] End of Unencrypted dump of send buffer.
<redacted for brevity>
[0]11D4.13D0::10/22/2014-12:33:06.490 [Microsoft-Windows-LDAP-Client/Debug] Unencrypted dump of Data received on connection 0x234e0898
[0]11D4.13D0::10/22/2014-12:33:06.490 [Microsoft-Windows-LDAP-Client/Debug] 30 84 00 00 00 63 02 02 10 3f 67 84 00 00 00 59 0....c...?g....Y
[0]11D4.13D0::10/22/2014-12:33:06.490 [Microsoft-Windows-LDAP-Client/Debug] 0a 01 0b 04 00 04 52 30 30 30 30 32 30 32 34 3a ......R00002024:
[0]11D4.13D0::10/22/2014-12:33:06.490 [Microsoft-Windows-LDAP-Client/Debug] 20 53 76 63 45 72 72 3a 20 44 53 49 44 2d 30 32 .SvcErr:.DSID-02
[0]11D4.13D0::10/22/2014-12:33:06.490 [Microsoft-Windows-LDAP-Client/Debug] 30 35 30 35 35 45 2c 20 70 72 6f 62 6c 65 6d 20 05055E,.problem.
[0]11D4.13D0::10/22/2014-12:33:06.490 [Microsoft-Windows-LDAP-Client/Debug] 35 30 30 38 20 28 41 44 4d 49 4e 5f 4c 49 4d 49 5008.(ADMIN_LIMI
[0]11D4.13D0::10/22/2014-12:33:06.490 [Microsoft-Windows-LDAP-Client/Debug] 54 5f 45 58 43 45 45 44 45 44 29 2c 20 64 61 74 T_EXCEEDED),.dat
[0]11D4.13D0::10/22/2014-12:33:06.490 [Microsoft-Windows-LDAP-Client/Debug] 61 20 2d 31 30 32 36 0a 00 a.-1026..
[0]11D4.13D0::10/22/2014-12:33:06.490 [Microsoft-Windows-LDAP-Client/Debug] End of Unencrypted dump of receive buffer.
[0]11D4.13D0::10/22/2014-12:33:06.490 [Microsoft-Windows-LDAP-Client/Debug] CheckForWaiter setting MsgNo 0x103f, wait 0x234192a8
[0]11D4.13D0::10/22/2014-12:33:06.490 [Microsoft-Windows-LDAP-Client/Debug] LdapGetResponseFromServer : searching for messages for msgid 0x103f
[0]11D4.13D0::10/22/2014-12:33:06.490 [Microsoft-Windows-LDAP-Client/Debug] LdapGetResponseFromServer Message 0x103f marked received for connection 0x234e0bc8
[0]11D4.13D0::10/22/2014-12:33:06.490 [Microsoft-Windows-LDAP-Client/Debug] ldap_result found results at 0x233f41b8 for request 0x233eaca8 msgid 0x103f
[0]11D4.13D0::10/22/2014-12:33:06.490 [Microsoft-Windows-LDAP-Client/Debug] HrPeekTag ran out of data, length 0x69, offset 0x69.
[0]11D4.13D0::10/22/2014-12:33:06.490 [Microsoft-Windows-LDAP-Client/Debug] ldap_result returned 0x67 for connection 0x234e0898.
[0]11D4.13D0::10/22/2014-12:33:06.490 [Microsoft-Windows-LDAP-Client/Debug] HrPeekTag ran out of data, length 0x69, offset 0x69.
[0]11D4.13D0::10/22/2014-12:33:06.490 [Microsoft-Windows-LDAP-Client/Debug] HrPeekTag ran out of data, length 0x69, offset 0x69.
[2]11D4.13D0::10/22/2014-12:33:06.499 [Microsoft-Windows-LDAP-Client/Debug] ldap_delete called for connection 0x233ddb58: DN is CN=Managed Availability Servers,OU=Microsoft Exchange Security Groups,DC=test,DC=lab. Synchronous is 0x0.
Basically, setup was trying to modify otherWellKnownObjects on CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=test,DC=lab and got DSID-0205055E, problem 5008 (ADMIN_LIMIT_EXCEEDED) as a response. We consult an AD engineer and he tells us that when an attribute is "full," you won't be able to modify the object at all. We immediately check otherWellKnownObjects and see it only has 17 values; that can't be our issue. We leverage the script from https://www.open-a-socket.com/index.php/2014/09/19/multivalued-attribute-limits-in-active-directory/ and realize that several attributes, including globalAddressList and globalAddressList2 have over 1200 values.
Now for a brief history lesson. Exchange hosters create a lot of Global Address Lists - usually 1 for each tenant it hosts. globalAddressList is a nonlinked multivalued attribute that enumerates all the GALs in an Exchange organization. Exchange 2007 hit this 8KB limit and thus a tool named makeGalLinked.exe was made that converted the attribute from nonlinked to linked. For an explanation as to why linked attributes aren't subject to the 8KB limit, look under the Replication of Linked and Nonlinked Attributes in this TechNet article How the Active Directory Replication Model Works . Starting with Exchange 2010, our schema created a new value named globalAddressList2 which is a linked multivalued attribute and thus is not subject to the 8KB limit. In Exchange 2010 onwards, any new GAL will be stored into both attributes for legacy interop but once the limit is exceeded, a warning will appear informing administrators that any user who isn't on Exchange 2010 might have issues accessing their GAL (because the pointer to the GAL might not exist).
With that history in mind, think about why Exchange 2013 is failing. globalAddressList is hitting the limit because it is a full nonlinked attribute and thus causes AD to throw the error ADMIN_LIMIT_EXCEEDED. The resolution in both support cases was to remove some values from the globalAddressList and try /prepareAD again. There should be no data loss because globalAddressList2 is a superset of globalAddressList (it contains all the data plus more). All Exchange 2010 and future code should use globalAddressList2 and thus globalAddressList is a legacy attribute that eventually will be deprecated if it hasn't already been.
Cheers,
Matt