SharePoint 2013 On-Prem reset index stuck resolution

Recently I have had a few customers with index out of sync errors, the customers have tried everything possible to get the indexers to sync and they are at the last resort, nuclear option, to reset the search index. The reason can very for how this has got to this state such as network related or AntiVirus or the 808 firewall issue, I will cover more of these issues in some more blog posts.

*Reset index is the last resort as you must complete a full crawl of your data again and users will not be able to retrieve search results until you create a new search index.

 

In my cases the index is out of sync and never catches up, the index components are yellow in the topology view.

In the SP management console run:

get-spenterprisesearchstatus -SearchApplication $ssa -Detailed -Text

Name : IndexComponent6

State : Degraded

State : List of degraded cells: Cell:IndexComponent6-SP34fdebde8866I.1.3(Index out of sync - catching up);

catch_up : True

Partition : 3

Host : SOMESERVER1

 

Name : Cell:IndexComponent6-SP34fdebde8866I.1.3

State : Degraded

State : Components OK. Index out of sync - catching up

Primary : False

Partition : 3

catch_up : True

in_sync : False

left_behind : True

 

What I have found is that sometimes the reset index process hangs and there is no documentation as to what to do about it.

On the SSA you will see something like this:

Administrative status Paused for:Index reset

Or:

Administrative status Paused for:External request, Index reset

Or:

Administrative status Paused for:External request

 

 

 

You can run this command in the SSA management console to verify if the SSA is paused. If this command returns False, the SSA is running. If this command returns True, the SSA is paused.

$ssa.IsPaused() -ne 0

 This is the most common I have been seeing with indexers out of sync and reset index stuck:

  ($ssa.IsPaused() -band 0x100) -ne 0

If True returns the search index is being deleted and you must “Wait until the search index is deleted.”

Well if your indexers are out of sync this can take many many hours or never return!

(If False you may want to look at the link at the bottom for more commands on this to see why the SSA is still paused, could be a backup or topo change, etc)

 

Here is the process I have been able to use to fix this issue:

#1 First try to force a resume, if this doesn’t work that’s fine and go to #2 anyways, just more of a try a last ditch force.

$ssa.ForceResume($ssa.IsPaused())

 

#2 On all of the nodes that contain an indexer component, stop the timer services and delete the indexes:

Stop “SharePoint Timer Service”

Stop “SharePoint Search Host Controller”

Delete data directory, by default install it will look like this:

C:\Program Files\Microsoft Office Servers\15.0\Data\Office Server\Applications\Search\Nodes\[Hex ID\IndexComponentN\storage\data\*

Start “SharePoint Timer Service”

Start “SharePoint Search Host Controller”

 

#3 Once they directories are deleted, run reset index again (which should complete very quickly)

 

 

For more information on reset index:

https://technet.microsoft.com/en-us/library/jj219652(v=office.15).aspx

The PowerShell to complete this is here as the UI sometimes times out:

$ssa = Get-SPEnterpriseSearchServiceApplication

$disableAlerts = $true

$ignoreUnreachableServer = $true

$ssa.reset($disableAlerts, $ignoreUnreachableServer)

if (-not $?) {

 Write-Error "Reset failed"

}

 

To find why the SSA is paused for other reasons check here:

Manage a paused Search service application in SharePoint Server 2013

https://technet.microsoft.com/en-us/library/dn745901(v=office.15).aspx