RPC Filtering and Exchange 2010 Database Availability Groups
Recently I’ve had the opportunity to work with customers who were having issues seeding databases using update-mailboxdatabasecopy in Exchange 2010. When attempting to perform an update the following sample error was returned:
A source-side operation failed. Error An error occurred while performing the seed operation. Error: Failed to open a log truncation context to source server 'SOURCE-SERVER'. Hresult: 0xc7ff07d7. Error: Failed to open a log truncation context because the Microsoft Exchange Information Store service is not running.. [Database: MailboxDatabase2, Server: TARGET-SERVER]
*Note that the HResult maybe different in the error even though the root of the issue is the same.
In each instance the server we were trying to run the update for was located across a WAN link or separated by firewall devices.
In the reference cases I worked we found that the devices providing the WAN connectivity were performing RPC packet inspection. For example, Threat Management Gateway has an RPC inspection agent and Cisco devices have a setting to enable DCERPC filtering. It would appear that certain RPCs that originate from Windows 2008 and Windows 2008 R2 do not conform to the expected format that these filters use. When a non-conforming packet is identified it is subsequently dropped.
We have also observed RPC filtering cause the following issues:
- Continuous replication circular logging fails to trigger log truncation across nodes.
- Log truncation does not occur in a DAG when a backup is successful on a member that has traffic between nodes subject to RPC filtering.
- Backup header information for databases does not update on active database copies when a backup is successful on a member that has traffic between nodes subject to RPC filtering.
To correct the issue RPC filtering had to be disabled on both the source and target devices providing the WAN connectivity between sites.
Comments
Anonymous
November 06, 2013
Confirmed! Thanks for this. I have been working on this issue for 2 weeks with all the exact same issues. We also had DCE/RPC filtering enabled on the Cisco ASA firewall. Once we turned this off, immediately all transaction logs were truncated and I was able to add a database copy. For the truncation of the transaction logs I observed the following event log ids: Level: Information Source: MSExchangeIS Task Category: Exchange VSS Writer Event Id: 9827 Message: Exchange VSS writer (instance <guid>) has successfully completed the full or incremental backup of replicated database <db name>. The log files will be truncated aster they have been replayed. Level: Information Source: ESE Event ID: 225 Task Category: ShadowCopy Message: Information Store (5868) <dbname>: No log files can be truncated.Anonymous
November 06, 2013
@FlackMonkey: Thanks for the comment. The 225 event here will always be thrown. When we moved log truncation out of the information store to the replication service the 225 event still fires. It's often confusing but can be ignored. TIMMCMIC