Search SSA backup is failing- SP2013

Search service application backup fails when doing a backup of the service application from Central Administration

Microsoft.Ceres.SearchCore.Services.Seeding.BackupRestoreSeedingException: Member I.0.1 was lost from cluster SPa86253565da7.1' at    at Microsoft.Ceres.SearchCore.IndexController.BackupService.ThrowOnFailure(JobStatus status)

Cause

SSA was not resumed and from analysis found crawls were resumed during backup.

Resolution

Went through the backup log file.  [2/15/2016 7:57:40 PM] Verbose: [.....] SQL command completed at: 2/15/2016 7:57:40 PM.

[2/15/2016 7:57:40 PM] Verbose: [IndexController] Waiting for first phase backup of index SPa86253565da7.

[2/15/2016 7:57:41 PM] Progress: [IndexController] 13 percent complete.

[2/15/2016 8:28:04 PM] Progress: [IndexController] 16 percent complete.

[2/15/2016 8:52:44 PM] Progress: [IndexController] 19 percent complete.

[2/15/2016 9:27:25 PM] Progress: [IndexController] 22 percent complete.

[2/15/2016 9:57:50 PM] Progress: [IndexController] 25 percent complete.

[2/15/2016 10:32:28 PM] Progress: [IndexController] 28 percent complete.

[2/15/2016 10:47:58 PM] Progress: [IndexController] 31 percent complete.

[2/15/2016 11:23:16 PM] Progress: [IndexController] 34 percent complete.

[2/16/2016 12:08:34 AM] Progress: [IndexController] 37 percent complete.

[2/16/2016 1:36:57 AM] Progress: [IndexController] 40 percent complete.

[2/16/2016 2:19:53 AM] Verbose: [IndexController] Aborting backup job for index SPa86253565da7.

[2/16/2016 2:19:54 AM] Verbose: [Enterprise Search] Resuming crawls and feeding.

[2/16/2016 2:19:56 AM] Verbose: [Enterprise Search] Resuming crawls and feeding completed in 0.0 minutes.

[2/16/2016 2:19:56 AM] Verbose: [Enterprise Search] Resuming master merge.

[2/16/2016 2:24:26 AM] Verbose: [Enterprise Search] Resuming master mergex completed in 4.5 minutes.

[2/16/2016 2:24:26 AM] FatalError: Object Enterprise Search failed in event OnBackup. For more information, see the spbackup.log or sprestore.log file located in the backup directory.

FaultException: Management called failed with System.InvalidOperationException: 'Job failed: Have tried to perform backup/restore operation twice on all in-sync members in cluster SPa86253565da7.1, but none succeeded. Last failure message: Microsoft.Ceres.SearchCore.Services.Seeding.BackupRestoreSeedingException: Member I.0.1 was lost from cluster SPa86253565da7.1' at    at Microsoft.Ceres.SearchCore.IndexController.BackupService.ThrowOnFailure(JobStatus status)

at Microsoft.Ceres.SearchCore.IndexController.BackupService.ProgressFirstPhase(String handle)

at Microsoft.Ceres.SearchCore.IndexController.IndexControllerManagementAgent.WrapCall[T](Func`2 original)

Clearly looks from highlighted above that crawls were resumed during backup causing failures. This is expected as the Index files will not be consistent across partitions and to avoid inconsistencies in backup the backup job just fails.

We should just manually pause the SSA through CA or powershell. The whole idea on pausing SSA is to ensure that no new changes are made to the topology/query, crawl rules/ or any data on index files in the file system as this is part of  the search configuration that we backup.

$ssa = Get-SPEnterpriseSearchServiceApplication <SearchServiceApplicationName>
Suspend-SPEnterpriseSearchServiceApplication -Identity $ssa

Recommendations on Backups in Sharepoint-

 

1.       If you are backing up search for the first time, you must use the Full option. You must perform a full backup before you can perform a differential backup.

The Differential option only applies to the search databases. The search index files are always fully backed up, even when you use the Differential option.  

 

2.       Backing up a service application does not affect the state of the farm. However, it does require resources. Therefore, backing up a service application might affect farm performance while the backup is running. You can avoid performance issues by backing up the service application during hours when farm use is lowest.

 

3.  You must create a folder on the local computer or the network in which to store the backups. For better performance, we recommend that you back up to the local computer and then move the backup files to a network folder. For more information about how to create a backup folder, see Prepare to back up and restore farms in SharePoint 2013.

 

Ref: https://technet.microsoft.com/en-us/library/ee428318.aspx