Active Directory Troubleshooting: server has returned the following error - invalid enumeration context
I ran into a situation similar to the scenario below; although, I caught the problem during testing and not the production implementation. I was faced with a decision to rework my code, or understand the error so I could address it effectively. In this article I will elaborate on what I found to be the cause of this error, and what situation brings it about. I will then show you what to do about it without having to revert to older methodologies to handle your task.
Scenario
If you have been working in an Active Directory environment for any length of time, there is a good possibility that you have run into a situation where you needed to run some process against all, or at least a large portion, of your Users or Computers. Managing each account individually is very time consuming and inefficient, so you turn to the almighty PowerShell to automate this process. You start by using Get-ADUser or Get-ADComputer to retrieve the Objects you want from AD, then you use the powerful Pipeline of PowerShell to send those AD Objects to another cmdlet to do the additional work you need. You may even take the results of the secondary cmdlet and use the pipeline to send them to tertiary, and quaternary cmdlets. The pipeline makes this all very easy. You've tested your process on a few objects and are ready to process against your entire domain. Your change control is approved, you are in the driver's seat, and you hit the enter button. This wonderful process takes off; it's pure magic. Flying along, PowerShell is processing each object and doing exactly what it was designed to do, but you have thousands of objects so this is going to take a while. Your part has been done, you created the process and kicked it off. PowerShell is doing its part, running your creation. Might as well go to the kitchen and warm up some leftover pizza and grab a nice cold Dr Pepper. You glance over at your laptop on your way to the La-Z-Boy and see that everything is running smoothly, so you flip on an old rerun of Gilligan's Island. You watch as they put together some crazy contraption, which in the end utterly fails. If only they had you and PowerShell, they would be home sipping their own Dr Peppers by the end of the show. You glance back over at your laptop and the unthinkable has occurred. In your PowerShell console where your process should be running, you see this:
get-aduser : The server has returned the following error: invalid enumeration context.
At line:1 char:1
+ get-aduser -Filter {msExchRecipientTypeDetails -eq "2147483648" -AND -NOT(UserAc .
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Get-ADUser], ADException
+ FullyQualifiedErrorId : The server has returned the following error: invalid enumeration context.,Microsoft.ActiveDirectory.Management.Commands.GetADUser
What happened? A quick web search of "The server has returned the following error: invalid enumeration context." returns lots of content on the topic with two prevailing recommendations.
- One of the attributes being queried is not indexed in AD. You can resolve this by modifying the Attribute in the AD Schema enabling the index flag.
- Ditch get-aduser and get-adcomputer and use ADSI DirectorySearcher instead.
IE:
You don't really want to manually modify your Active Directory schema, but this process is important so you give it a shot. Looking at all of the attributes you are pulling you notice that they are all already indexed, or you enable indexing but the problem persists. You really like using native PowerShell functionality and you don't really want to rework your code to use ADSI DirectorySearcher, but you really have no other options right? Wrong!
Digging for the Roots
In order to find a solution to this error, we have to understand why it happens in the first place. Let's don our troubleshooting hat and see what's going on. We will start by looking at the error message itself.
Note: for the purpose of the below illustration, I will only talk about Get-ADUser, but the same applies to Get-ADComputer.
The error message gives clear indication that the cmdlet which threw the error was Get-ADUser. Since our process is using Get-ADUser to send objects down the pipeline to other cmdlets, we run the Get-ADUser cmdlet by itself to see if we can get any new info. Surprisingly the Get-ADUser command completes with no errors, and fairly quickly; however, every time we run the cmdlet and pass objects to other cmdlets through the pipeline, we generate the error. That is interesting because Get-ADUser is supposed to just get the full results and hand them down the pipeline as soon as they come in right? Well, let's see.
What can we find out about how Get-ADUser works? If we look at the help of Get-ADUser, we see two parameters that are of interest.
-ResultPageSize <int>
Specifies the number of objects to include in one page for an Active Directory Domain Services query.
The default is 256 objects per page.
The following example shows how to set this parameter.
-ResultPageSize 500
-ResultSetSize <System.Nullable[System.Int32]>
Specifies the maximum number of objects to return for an Active Directory Domain Services query. If you want to receive all of the objects, set this parameter to $null (null value). You can use Ctrl+c to stop the query and return of objects.
The default is $null.
The following example shows how to set this parameter so that you receive all of the returned objects.
-ResultSetSize $null
So based on these parameters, we know that by default, Get-ADUser will return all matched objects from Active Directory, but it will do it in pages consisting of 256 objects per page.
Side Note: What exactly is paging? Simply put, it's breaking a large query result down into smaller more manageable pieces. By default, if a Query results in 1000 objects, rather than sending all 1000 object back to the client, the server will send 256 object at a time (in separate pages) until all objects are sent. 1000 objects would result in 4 pages. It is up to the client to tell the server when it is ready for the next page of results.
(Paging Search Results: https://msdn.microsoft.com/en-us/library/aa367011(v=vs.85).aspx)
Now let's look at Get-ADUser in action.
Using the sysinternals tool, TCPView.exe, we monitor the TCP communication of our Get-ADUser cmdlet while using Measure-Command to capture our runtime.
http://blogs-fb4x.rhcloud.com/wp-content/uploads/2015/10/1.png
PS C:\> Measure-Command -Expression {get-aduser -Filter {-NOT(UserAccountControl -band 2)} -ErrorAction Stop}
Days : 0
Hours : 0
Minutes : 0
Seconds : 28
Milliseconds : 904
Ticks : 289045151
TotalDays : 0.000334542998842593
TotalHours : 0.00802903197222222
TotalMinutes : 0.481741918333333
TotalSeconds : 28.9045151
TotalMilliseconds : 28904.5151
From the results we see that it took us 28.9 seconds to get 14.79 MBytes (15,504,491 Rcvd Bytes) of returned data from our query. We also see that we are connecting to TCP port 9389 on our domain controller. A quick web search on this TCP port reveals that this is the port used by Active Directory Web Services (http://www.t1shopper.com/tools/port-number/9389/).
In order to see how many return objects we are dealing with, we run the command again, piping the results into Measure-Object
PS C:\> get-aduser -Filter {-NOT(UserAccountControl -band 2)} -ErrorAction Stop | Measure-Object
Count : 14163
The first thing we notice is that our process does not error out this time, even though we are using the pipeline. So maybe the problem relates to how long it takes to process our other pipeline commands. Since Measure-Object takes very little time to process, it does not result in the error being generated. In order to test this theory we will use start-sleep to inject some processing time down the pipeline.
PS C:\> get-aduser -Filter {-NOT(UserAccountControl -band 2)} -ErrorAction Stop | ForEach-Object {Start-Sleep -Milliseconds 200; $_}
get-aduser : The server has returned the following error: invalid enumeration context.
At line:1 char:1
+ get-aduser -Filter {-NOT(UserAccountControl -band 2)} -ErrorAction Stop | ForEac ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Get-ADUser], ADException
+ FullyQualifiedErrorId : The server has returned the following error: invalid enumeration context.,Microsoft.ActiveDirectory.Management.Commands.GetADUser
After a while of running, we once again get our "The server has returned the following error: invalid enumeration context." error, but why? The code runs fine without the pipeline, and it runs fine with the pipeline as long as the subsequent cmdlets complete quickly. We know that the Get-ADUser cmdlet completes in about 30 seconds without the pipeline. Is it the same when we have slower processing further down the pipe?
Let's do two things to see what is going on. We will use TCPView and Measure-Command, as we did before, to see what is going on from a network perspective and how long our code is running before generating the error.
PS C:\>Measure-Command -Expression {get-aduser -Filter {-NOT(UserAccountControl -band 2)} -ErrorAction Stop | ForEach-Object {Start-Sleep -Milliseconds 200; $_}}
After 5 minutes of running, we see that we have only received 2.15 Mbytes of data.
http://blogs-fb4x.rhcloud.com/wp-content/uploads/2015/10/2.png
After 10 minutes of running, we are only sitting at 4.09 Mbytes of data.
http://blogs-fb4x.rhcloud.com/wp-content/uploads/2015/10/3.png
Our slow data rate return continues until we finally error out after 30 minutes, and we also see that we did not receive our expected full data size of around 14.79 Mbytes.
http://blogs-fb4x.rhcloud.com/wp-content/uploads/2015/10/4.png
get-aduser : The server has returned the following error: invalid enumeration context.
At line:1 char:30
+ Measure-Command -Expression {get-aduser -Filter {-NOT(UserAccountControl -band 2 ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Get-ADUser], ADException
+ FullyQualifiedErrorId : The server has returned the following error: invalid enumeration context.,Microsoft.ActiveDirectory.Management.Commands.GetADUser
Days : 0
Hours : 0
Minutes : 30
Seconds : 49
Milliseconds : 886
Ticks : 18498861406
TotalDays : 0.0214107192199074
TotalHours : 0.513857261277778
TotalMinutes : 30.8314356766667
TotalSeconds : 1849.8861406
TotalMilliseconds : 1849886.1406
So our data is returning a lot slower, and we are erring out before receiving our full dataset. This is indicative of a timeout. Since we know that we are connecting to Active Directory Web Services, a quick web search for "Active Directory Web Services Timeout 30 minutes" gets us here: https://technet.microsoft.com/en-us/library/dd391908(v=ws.10).aspx
Excerpt from above site:
MaxEnumContextExpiration
30 minutes
Specifies the maximum allowed time period during which the ADWS service processes and retrieves the results of a query request from a client computer.
Caution
Changing the default value of this parameter is strongly discouraged. Most of the search results are returned within 30 minutes.
Ok, lets put together what we've learned.
- Get-ADUser uses paging (256 object per page by default)
- It is up to the client to request new Pages
- When piping out AD Objects, the longer the code down the pipeline takes to process, the slower we retrieve data from Active Directory Web Services
- If that slower processing causes the retrieval time to run over 30 minutes, then the Enumeration Context Expires
Now lets fill in the blanks for the full, end to end process.
- Get-ADUser executes sending the query to Active Directory Web Services
- Active Directory Web Service receives the query, creates an enumeration context for the results, and returns the first page of results containing 256 objects
- Get-ADUser receives those results and sends them down the pipeline, but does not query Active Directory Web Services for the next page until the pipeline has finished processing all 256 objects already received.
- The pipeline processes each object placed in it one at a time down the entire pipeline. So all pipeline processes are completed on the first object before the second object begins it journey down the pipeline. (For 256 objects with 200 millisecond pipeline processing per item, that's 51.2 seconds per page. With 14163 object, we are looking at 55.32 pages and 47.21 minutes.)
- Once all 256 objects have been processed for the current page, Get-ADUser requests the next page of results from Active Directory Web Services.
- Active Directory Web Services returns the next 256 objects.
- Steps 3 - 6 repeat until all objects are retrieve or the processing time goes longer than 30 minutes. After 30 minutes, Active Directory Web Services expires the enumeration context it created in step 2. Once expired, Active Directory Web Services returns the error, "invalid enumeration context", when Get-ADUser request the next page because the enumeration context has expired and is no longer valid.
So mystery solved.
Solutions
There are two possible solutions to this issue, the first of which I would not recommend.
- Increase the "MaxEnumContextExpiration" value for Active Directory Web Service. There are many reasons not to take this approach.
- First and foremost, Microsoft recommends again this. "Changing the default value of this parameter is strongly discouraged. Most of the search results are returned within 30 minutes."
- This change would have to be made on all Domain Controllers that the script would possibly hit.
- The target value to increase to would be a moving target depending on number of objects returned and length of time required to process all pipeline code.
- Increasing this value increases the potential impact of long running processes on your Domain Controllers.
- If you choose this method, it can be done by modifying the "Microsoft.ActiveDirectory.WebServices.exe.config" file in the ADWS directory. By default: "C:\Windows\ADWS".http://blogs-fb4x.rhcloud.com/wp-content/uploads/2015/10/5.png
- Retrieve your Active Directory objects to a variable first, then send it down the pipeline using the variable. This method is easy to implement in your code without lots of configuration changes in your Active Directory environment. It works because writing the objects to a variable is very fast so your Get-ADUser and Get-ADComputer cmdlets can quickly write all object to the variable and request the next page until all object are received.
It's this simple:
Instead of doing this:
get-aduser -Filter {-NOT(UserAccountControl -band 2)} -ErrorAction Stop | ForEach-Object {Start-Sleep -Milliseconds 200; $_}
Do this:
$adobjects = get-aduser -Filter {-NOT(UserAccountControl -band 2)} -ErrorAction Stop
$adobjects | ForEach-Object {Start-Sleep -Milliseconds 200; $_}
Or if you wanna keep it on a single commandline for the PowerShell console, just separate it with a semicolon:
$adobjects = get-aduser -Filter {-NOT(UserAccountControl -band 2)} -ErrorAction Stop; $adobjects | ForEach-Object {Start-Sleep -Milliseconds 200; $_}
My hope is that you will find this useful. Please feel free to leave a comment and let me know how it has helped. Thanks for reading!