Backups NIC Causes Exchange DAG Clustering Issues and Exchange Information Store Worker Process Alert

Question

1. Product & Service Pack:

1. Windows Server DC 2019 - Version 1809 (OS Build 17763.4377)
2. Exchange 2019 – CU 15 with November 2024 SU (15.02.1748.010)

2. Cluster Type (For WINDOWS only):

1. DAG

3. Troubleshooting Performed:

1. Unregistering DNS Addresses: Unchecking the "Register this connection’s addresses in DNS" checkbox for the backups NIC was attempted but has not definitively resolved the issue.
2. Database and Server Removal: The mailbox database copy was removed, and the server was removed from the DAG. Then, the backups NIC was disabled. After re-adding the server to the DAG and creating a new mailbox database copy, the issue was resolved until the backups NIC was reenabled.
3. Temporary Network Interface Disabling: If the mailbox database copy is still valid, disabling the NIC and waiting 15 minutes allows the server to start the cluster service again.
4. Metric Adjustment Attempt: Adjusting the network metric for the backups NIC using PowerShell and netsh commands was attempted to deprioritize the network.
5. Manual Configuration: Ensuring "ManualDagNetworkConfiguration" is set to True and running the DiscoverNetworks task to see if automatic configuration aligns with the intended network settings.
6. Cluster Module Verification: Ensuring the Failover Clustering feature and its tools are properly installed and the correct module is imported for consistent command execution.
7. Network Interface Review: Listing all cluster network interfaces and verifying their states to identify any discrepancies or misconfigurations.

4. Recent Changes:

1. Installed Exchange 2019 CU 14
2. Installed Exchange 2019 CU 14 SU February 2024
3. Installed Exchange 2019 CU 14 SU November 2024
4. Installed Exchange 2019 CU 15

5. When did the problem start?

1. Tuesday 05 March 2023 (initial build out, noticed with backups started)

6. Problem Frequency:

Permanent (if the NIC is enabled)

ISSUE:

The Backups Network Interface Card (NIC) is causing clustering issues in the Exchange Database Availability Group (DAG). The DAG keeps using the subnets assigned to the backups NIC for replication, even when the "ManualDagNetworkConfiguration" setting is set to True. This problem persists after a reset or when the DiscoverNetworks task is run. Additionally, there is an alert (msg252) indicating that the Exchange Information Store Worker Process is not being found. The server currently has a backups NIC, which is definitively causing the issue. The multiple NICs with conflicting settings can cause network misconfigurations, disrupting communication between cluster nodes and services. This misconfiguration prevents the Exchange Information Store Worker Process from functioning correctly, leading to the alert. This issue only occurs while the NIC is enabled on msg252. The other server, msg251, also has the backups NIC on the same subnet but does not experience this issue unless it is not the DAG manager and the other server's NIC is functioning. msg251 can interchangeably experience the same issue depending on when its Backups NIC is activated or when the Backups NIC on msg252 is activated. One always works while the other fails in the cluster.

Replication Issues: The Exchange DAG is using the subnets assigned to the backups NIC for replication instead of the intended network, leading to unexpected replication paths and potential network congestion.

Persistent Problems: Despite setting "ManualDagNetworkConfiguration" to True, the issue persists after running the DiscoverNetworks task or resetting the configuration.

Network Misconfiguration Alerts: There is an alert on the msg252 server indicating that the Exchange Information Store Worker Process is not being found, pointing to underlying network misconfiguration issues.

Service Disruptions: The Information Store Worker Process is not functioning correctly, disrupting communication between cluster nodes and services.

Temporary Fixes: Temporary fixes, such as removing/re-adding the mailbox database/Server DAG Membership and disabling the backups NIC, resolve the issue only until the backups NIC is reenabled.

Answer 1

Why the need for a backup NIC?

Seriously, Im not trying to be a smart a$$. :)

I think that over complicates things and if you are going to do backups ( those are not really needed however : https://learn.microsoft.com/en-us/exchange/high-availability/disaster-recovery/disaster-recovery?view=exchserver-2019

then I would run with just one NIC if possible