Exchange 2010: Page Zeroing and VSS Based Backups
In Exchange Server 2010 Service Pack 1, page zeroing is enabled by default and there is no method to disable it. Page zeroing is a process that takes pages that exist in whitespace within the database and marks them with a pattern of zeros making them forensically unrecoverable. The page zeroing process runs as part of the background maintenance process.
Recently we have investigated cases where page zeroing has led to larger than expected VSS backup data sets. Specifically these are for VSS-based products that perform a delta backup from a previous snapshot versus transferring a full data set with each backup. These might include but are not limited to:
- System Center Data Protection Manager
- Windows Server Backup (when leveraging non-network storage)
- Hardware-based VSS providers
In all cases investigated there was an event that increased the amount of whitespace within the database. For example, multiple mailboxes were moved to another database leaving a more noticeable amount of whitespace within a given database.
Let’s look at an example:
The anchor backup is taken of a 500 GB database with 10 GB of associated log files. This results in a transfer to backup medium of 510 GB. Over the course of the day 6 GB of changes occur within the database (5 GB of actual user changes / 1 GB of page zeroing changes) with 10 GB of associated logs. This yields a delta transfer to backup medium of 16 GB. Over the course of time, delta transfers all float around 16 GB with standard usage patterns etc. At some point the administrator migrates a group of users out of the database and this accounts for 30 GB of change. The resulting backup to medium is now 30 GB + log files. At this point, we expected the increase in delta transfer as there was an event that caused an increase in database activity. What is not expected though, is that from this point forward, the delta transfers to backup medium continue to be greater than 30 GB. In many cases this exceeds the expected snapshot size and storage allocated on the medium server.
In these instances, the page zeroing process continuously zeros pages that are already zeroed, thereby causing a daily delta change rate that corresponds with whitespace.
The following actions can be taken to correct this condition:
1) Adjust snapshot storage to accommodate the larger delta data sets.
2) Migrate all mailboxes out of the mailbox database and remove the mailbox database.
3) Allow the whitespace to be recycled as additional mailboxes are added to the database.
4) Offline defragment the database.
Exchange 2010 Service Pack 3 Rollup Update 1 has a code change that corrects the page zeroing behavior.
Comments
Anonymous
May 21, 2013
Cheers Tim :)Anonymous
June 24, 2013
Great post! it appears we're hitting the same issue here, and we have no direction until reading this.. :)