Troubleshooting Content Index State Issues for Database Copy in DR Site - Exchange 2016 DAG

Question

Hello Community,

I am currently facing an issue with the Content Index State in my Exchange 2016 DAG environment, and I would greatly appreciate your assistance. Here’s a detailed description of my environment, the problem, and the steps I’ve taken so far.

---

Environment Details

In my Exchange 2016 setup, I have the following configuration:

• Primary Datacentre (DC) Site: Two Exchange 2016 servers configured as DAG members, hosting multiple database copies. Everything in this site is functioning correctly, including database health and content indexing.
• Disaster Recovery (DR) Site: One Exchange 2016 server, also a DAG member. This server is meant to host database copies as a backup for the DC site.

---

The Issue

After adding a database copy to the Exchange server in the DR site, the copy status shows as Mounted, Passive, Healthy, but the Content Index State is consistently Failed and Suspended. This issue only occurs for existing databases added to the DR server.

Interestingly, when I created a test database in the DR site and added a copy to one of the DC servers, the Content Index State showed as Healthy, and the database worked fine even after activation in the DC site.

---

Troubleshooting Steps Taken

Here are the steps I have taken to resolve the issue, organized for clarity:

1. Rebuilding the Content Index I stopped the MSExchangeFastSearch and HostControllerService services on the DR server. After that, I deleted the folder containing the content index files for the problematic database and restarted the services. However, this did not resolve the issue, and the Content Index State remained in the Failed and Suspended status.
2. Reseeding the Content Index Using the Update-MailboxDatabaseCopy cmdlet with the -CatalogOnly parameter, I attempted to reseed the content index for the problematic database copy. Despite this effort, the status of the Content Index did not improve.
3. Removing and Re-Adding the Database Copy I removed the affected database copy from the DR server and re-added it to the DAG. Unfortunately, the Content Index State persisted as Failed and Suspended.
4. Creating a Test Database As a diagnostic step, I created a new test database in the DR site and added a database copy to one of the DC site servers. To my surprise, this configuration worked perfectly. The test database’s Content Index State showed as Healthy, and I was able to activate and operate the database from the DC site without any issues.

---

Additional Checks

To rule out other potential causes, I conducted the following checks:

• Network Ports: Verified that all necessary ports for DAG communication are open between the DC and DR servers.
• Antivirus Software: Temporarily disabled antivirus software on the DR server to eliminate interference.
• Permissions: Confirmed that the ContentSubmitters and Administrators groups have the required permissions.
• Event Logs: Checked the DR server’s event logs and found the following entries:
  • Event ID 1009 (Source: MSExchangeFastSearch) - This event cannot be found.
    • Event ID 1310 (Source: ASP.NET 4.0.30319.0) - This event cannot be found.

These event log entries seem unusual and could be related to the issue, but I am not entirely sure how to interpret their relevance to the Content Index State problem.

---

Observations and Questions

From my observations, the issue appears to be isolated to existing databases added to the DR server, as the test database works fine in both sites. I suspect there could be a deeper configuration or environmental issue specific to the DR server or these particular databases.

I would like to understand:

1. Could this be related to the existing database configuration rather than the DR server itself?
2. How can I address the Event IDs mentioned above, and are they relevant to the Content Index issue?
3. Are there additional troubleshooting steps I can perform to resolve this?

---

I look forward to your insights and suggestions.

-Moshiur

Answer 1

Hi @Moshiur (Moshiur Khan) ,Welcome to the Microsoft Q&A platform!Thank you for providing such a detailed description of the environment and the steps you have taken. Content Index State issues can sometimes be tricky, but let's try to solve it step by step.

1. The fact that the new test database works fine suggests that the problem may indeed be related to a specific existing database.

Solution:

Compare the settings and configuration of the problematic database (including properties, folder structure, and any custom configuration) with the new test database that works well.

2. Since you mentioned checking permissions, let's make sure that all necessary permissions are applied correctly, not only to the database, but also to the files and folders associated with the content index.

Solution:

a. Double-check permissions:

• Make sure the 'Network Service' account has the necessary permissions on the index folder.
• Verify that the 'ContentSubmitters' and 'Administrators' groups have the correct access rights.

b. Make sure that the MSExchangeFastSearch and HostControllerService services are running under the correct service accounts.

3. The event IDs you mentioned (1009 and 1310) can give us some clues.

Event ID 1009 (MSExchangeFastSearch):**

This usually indicates a problem with the FastSearch service that handles content indexing.

Solution:

• Check the details of the event if possible. Sometimes it includes a more specific error.
• Look in the logs for any related events that occurred around the same time.

Event ID 1310 (ASP.NET 4.0.30319.0):**

This may indicate a problem with the ASP.NET application that FastSearch depends on.

Solution:

Check the detailed logs for ASP.NET errors and compare them to the Exchange specific logs.

4. Ports and Network Configuration

You have already checked the ports required for DAG communication, but sometimes firewall and network issues may still be an issue.

Solution:

• Perform a network connectivity test between the DC and DR site to ensure there are no intermittent issues.
• Use the "Test-ExchangeSearch" and "Test-ReplicationHealth" cmdlets to verify there are no underlying network issues.

5. Shared Components and Dependencies

Sometimes, underlying dependencies or components that are shared between different services can cause Content Index issues.

Solution:

Evaluate if there are any shared components, such as system libraries, services, or configuration settings, that could cause conflicts.

6. Full Reseed

As a last resort, consider full reseed of the database copy instead of just reseed of the content index.

Solution:

Use the "Update-MailboxDatabaseCopy" cmdlet without any parameters to perform a full reseed of the problematic database copy.

Here is a high-level approach to the solution:

a. Permission check:

Get-MailboxServer |Get-ADPermission -User “Domain\Exchange Servers” # Verify permissions

b. Service restart and reseat:

# Restart service
Restart service MSExchangeFastSearch
Restart-Service HostControllerService
 
# Full reseed
Update-MailboxDatabaseCopy -Identity “DatabaseName\ServerName”

c. Dig deeper into the Event Viewer logs for more detailed error messages with the same timestamp as the above events.

 

By following these steps, you should be able to determine if the issue is specific to the database configuration or related to the server itself, and resolve the associated event ID logs

---

Please feel free to contact me for any updates. And if this helps, don't forget to mark it as an answer.

Best,

Jake Zhang