Legacy Backup API Failing on HrESEBackupReadFile with HR=0xC7FF1004
Recently I had the opportunity to dig into an interesting failure with the Exchange Legacy Backup API (ESEBCLI2.DLL). A backup vendor was seeing a reproducible error when they tried to remotely backup certain Exchange servers from their backup software when running on Windows Server 2008. They found that their call to HrESEBackupReadFile was returning 0xC7FF1004 instead of the expected S_OK but they could not figure out why.
Let me start by pointing out that the Legacy Backup API is a technology that is on its way out. Exchange 2010 no longer supports legacy API backups, and there are many caveats to using the API against Exchange 2007. With all that said up front, there are still quite a few backup applications that make use of this interface.
At its core, the legacy backup API uses Remote Procedure Calls to connect to the Exchange server and control the flow of backups and restores. When you actually go to read a file, it does something a bit different though. Instead of trying to package the data up and deliver it via returns to RPCs, the server actually creates a TCP socket connection back to the Backup Client.
If something prevents the return socket connection, such as the Windows Firewall, Network Address Translation or an External Firewall, you will receive the 0xC7FF1004 result code when you call HrESEBackupReadFile. In this particular case, we ruled out all of these “external” factors, but still were seeing the error. Furthermore, looking at a Netmon trace of the issue showed that the server was never even attempting (on the wire at least) to connect back to the client.
The mechanism by which this return socket connection is established relies on an address that is provided by the client. This would seem to be a valid process, but the introduction of changes to support IPv6 in Exchange Server 2007 SP1, and the inclusion of an IPv6 aware network stack on Windows Server 2008 collided to induce a failure.
The simplified sequence of events is:
- The Backup Client asks the OS for an address and includes a flag that says “Give me any IP address, IPv6 or IPv4”
- The OS gives the Backup Client an address, and includes a flag that indicates it could be IPv6 or IPv4
- The Client API sends this address as part of an RPC to the Server.
- The Server sees the flag and tries to create a socket, and includes a flag that indicates that it might eventually use an IPv6 address with the socket.
- The Server, in this case a Windows Server 2003 machine without IPv6 refuses to create the socket, returns a status code to the client that tells it to us a different backup method that unfortunately is not applicable for a remote backup.
From here, you will see an Event logged in the Exchange Server’s Application Log that reads:
Source: ESE BACKUP
Event ID: 909
Description: Information Store (PID) Backup data transfer method is RPC
And subsequently when the client calls HrESEBackupReadFile, the result you get back is 0xC7FF1004, which is a generic ESE backup error code.
So how do you fix it? One way is to follow the instructions in the following KB article to really disable IPv6 on the backup client:
https://support.microsoft.com/kb/929852
It is not sufficient to unbind IPv6 from the interface or disable IPv6 through other means. Note that the DisabledComponents Value should be 0xFFFFFFFF to completely disable IPv6.
This method has been noted previously in relation to MAPI issues with IPv6, and it appears that it is relevant here as well. Remember that you have to reboot for these changes to take effect, and also that changes to your network configuration may impact the Windows Firewall, as well as Network Location Awareness. Since 0xC7FF1004 is the same error that will occur if your backup client blocks the return socket connection, make certain that this change doesn’t end up blocking the return socket connection for a different reason (Such as a network connection that was previously identified as Private or Domain defaulting to Public).