Freigeben über

Exchange 2007 / 2010: Windows Server Backup and Performance Issues

  • Artikel

Some administrators choose Windows Server Backup to backup their Exchange 2007 and Exchange 2010 mailbox databases.  In some instances administrators report that during the backup process they experience performance issues on the server and issues with the length of time it takes for a backup to complete.  It should go without saying that it is preferred that backups not be taken during production hours when resources should be dedicated to end user access.  There maybe times though where this is not feasible.

 

In many cases the performance issues coincide with the consistency check portion of the Exchange backup.  Remember that using Windows Server Backup to backup Exchange is essentially a 4 step process:

1)  Volume Shadow copy is made of the Exchange data on the host where the backup is initiated.

2)  All Exchange log files and database files in the backup set must have a consistency check performed.

3)  The data on the drives is written to the desired backup media.

4)  Backup complete is performed allowing for log truncation activities (if applicable).

 

The consistency check process is utilizing eseutil on the local box in order to perform log file and database verification.  (The administrator can run a similar command using eseutil /k).  When the verification process is running it is technically occurring at two different locations:

1)  Against the changed blocks in the shadow copy storage.

2)  Against the production database and log files (via the shadow copy).

 

An important note here is that eseutil is not throttled during this verification routine which can incur performance impacts on your disks.  (It is ultimately this performance impact which causes performance issues with client access etc).

 

Let’s take a look at an example.   In our example we will backup a Windows 2008 R2 / Exchange 2010 SP1 installation on server MBX-1. 

 

Let’s inventory the information we are going to backup.  In this instance all public folder databases and mailbox databases that reside on the server.

 

[PS] C:\Windows\system32>Get-MailboxDatabase -Server MBX-1 | fl name,logfolderpath,edbfilepath

Name : MBX-1-DB0
LogFolderPath : D:\MBX-1\MBX-1-DB0\MBX-1-DB0-Logs
EdbFilePath : E:\MBX-1\MBX-1-DB0\MBX-1-DB0-Database\MBX-1-DB0.edb

[PS] C:\Windows\system32>Get-PublicFolderDatabase -Server MBX-1 | fl name,logfolderpath,edbfilepath

Name : MBX-1-DB1
LogFolderPath : d:\MBX-1\MBX-1-DB1\MBX-1-DB1-Logs
EdbFilePath : e:\MBX-1\MBX-1-DB1\MBX-1-DB1-Database\MBX-1-DB1.edb

Since all of the Exchange files reside on the D and E volumes I can expect to see the consistency check occurring against shadow storage created on the root of those volumes and the production paths.

 

Using VSSAdmin and can verify the shadow storage configuration for each volume on my server. 

 

C:\>vssadmin list shadowstorage
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2005 Microsoft Corp.

Shadow Copy Storage association
For volume: (\\?\Volume{a693b28b-a5f0-11de-813d-806e6f6e6963}\)\\?\Volume{a693b28b-a5f0-11de-813d-806e6f6e6963}\
Shadow Copy Storage volume: (\\?\Volume{a693b28b-a5f0-11de-813d-806e6f6e6963}\)\\?\Volume{a693b28b-a5f0-11de-813d-806e6f6e6963}\
Used Shadow Copy Storage space: 0 B (0%)
Allocated Shadow Copy Storage space: 0 B (0%)
Maximum Shadow Copy Storage space: 32 MB (32%)

Shadow Copy Storage association
For volume: (D:)\\?\Volume{cd015911-a5f1-11de-9f02-00155d00010c}\
Shadow Copy Storage volume: (D:)\\?\Volume{cd015911-a5f1-11de-9f02-00155d00010c}\
Used Shadow Copy Storage space: 0 B (0%)
Allocated Shadow Copy Storage space: 0 B (0%)
Maximum Shadow Copy Storage space: 2.5 GB (10%)

Shadow Copy Storage association
For volume: (E:)\\?\Volume{cd015918-a5f1-11de-9f02-00155d00010c}\
Shadow Copy Storage volume: (E:)\\?\Volume{cd015918-a5f1-11de-9f02-00155d00010c}\
Used Shadow Copy Storage space: 0 B (0%)
Allocated Shadow Copy Storage space: 0 B (0%)
Maximum Shadow Copy Storage space: 5 GB (10%)

Shadow Copy Storage association
For volume: (C:)\\?\Volume{a693b28c-a5f0-11de-813d-806e6f6e6963}\
Shadow Copy Storage volume: (C:)\\?\Volume{a693b28c-a5f0-11de-813d-806e6f6e6963}\
Used Shadow Copy Storage space: 0 B (0%)
Allocated Shadow Copy Storage space: 0 B (0%)
Maximum Shadow Copy Storage space: 7.19 GB (10%)

 

This output is telling us what we expect to see.  As of now shadow storage is configured for volume D  [(D:)\\?\Volume{cd015911-a5f1-11de-9f02-00155d00010c}\] to use a shadow storage location on volume D [(D:)\\?\Volume{cd015911-a5f1-11de-9f02-00155d00010c}\].

 

At this time we will initiate a Windows Server Backup (Backup once) job of the volumes D and E.

 

As indicated the first step of the backup is creating the shadow copy.  We can see that shadow copies by running vssadmin list shadows while the backup is in progress.  Here is a sample output.

 

C:\>vssadmin list shadows
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2005 Microsoft Corp.

Contents of shadow copy set ID: {f267c79e-cfd3-4952-94f3-c6384dc83c28}
Contained 2 shadow copies at creation time: 1/9/2011 12:53:44 PM
Shadow Copy ID: {b8b737a1-8765-4896-9526-b069294458aa}
Original Volume: (D:)\\?\Volume{cd015911-a5f1-11de-9f02-00155d00010c}\
Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy64
Originating Machine: MBX-1.domain.com
Service Machine: MBX-1.domain.com
Provider: 'Microsoft Software Shadow Copy provider 1.0'
Type: ApplicationRollback
Attributes: Persistent, No auto release, Differential

      Shadow Copy ID: {50e65eca-6b2e-457c-8ccf-5673598e7f96}
Original Volume: (E:)\\?\Volume{cd015918-a5f1-11de-9f02-00155d00010c}\
Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy65
Originating Machine: MBX-1.domain.com
Service Machine: MBX-1.domain.com
Provider: 'Microsoft Software Shadow Copy provider 1.0'
Type: ApplicationRollback
Attributes: Persistent, No auto release, Differential

From this output you can see that shadow copies have been created for information on both the D and E drives.

 

The next step of the backup is the consistency check. 

image

 

Using process monitor we can verify that eseutil is running against the databases and log files in shadow storage.

Log Files:

12:54:31.1154221 PM    eseutil.exe    5248    ReadFile    \Device\HarddiskVolumeShadowCopy64\MBX-1\MBX-1-DB0\MBX-1-DB0-Logs\E0000006088.log    SUCCESS    Offset: 4,096, Length: 262,144, I/O Flags: Non-cached, Priority: Normal

image

Database File

12:54:29.1153106 PM    eseutil.exe    2304    ReadFile    \Device\HarddiskVolumeShadowCopy65\MBX-1\MBX-1-DB0\MBX-1-DB0-Database\MBX-1-DB0.edb    SUCCESS    Offset: 4,434,755,584, Length: 65,536, I/O Flags: Non-cached, Priority: Normal

image

 

 

The administrator may choose to change the shadow copy storage location.  For example, for the D and E volumes the administrator may select to change the shadow storage location to the C drive.  In some instances the administrator performs this activity in an attempt to alleviate the performance issues that may occur when the consistency check is running.  Let’s take a look at what impact this has:

 

In this example I will utilize vssadmin to move the shadow storage for the D volume to the C volume:

 

vssadmin add shadowstorage /for=D: /on=C: /maxsize=unbounded

 

The same command can be utilized for other disks.  Using VSSAdmin List ShadowStorage I can verify that the C volume is now being utilized as shadow copy space for both D and E.  The following is example output:

 

Shadow Copy Storage association
For volume: (D:)\\?\Volume{cd015911-a5f1-11de-9f02-00155d00010c}\
Shadow Copy Storage volume: (C:)\\?\Volume{a693b28c-a5f0-11de-813d-806e6f6e6963}\
Used Shadow Copy Storage space: 0 B (0%)
Allocated Shadow Copy Storage space: 0 B (0%)
Maximum Shadow Copy Storage space: UNBOUNDED (100%)

Shadow Copy Storage association
For volume: (E:)\\?\Volume{cd015918-a5f1-11de-9f02-00155d00010c}\
Shadow Copy Storage volume: (C:)\\?\Volume{a693b28c-a5f0-11de-813d-806e6f6e6963}\
Used Shadow Copy Storage space: 0 B (0%)
Allocated Shadow Copy Storage space: 0 B (0%)
Maximum Shadow Copy Storage space: UNBOUNDED (100%)

 

At the time the consistency check is performed the administrator can observe in performance monitor disk activity now at two locations.  The first location is the volume shadow copy location and the second location is the production location of Exchange data.  Remember that a shadow copy only contains changed data (copy on write data).  When accessing shadow storage, if the shadow storage does not contain a changed block then the IO is passed to the original location.  Therefore it is not possible to completely eliminate the disk impact of a consistency check on the storage hosting the production data.

 

The last stage of the backup is the copying of the backup data from the VSS shadow copy location to the backup destination.  In our example process monitor shows a read occurring from the VSS shadow copy location and a subsequent write occurring to the backup destination – in this case the B: drive.

 

12:54:55.0625394 PM    wbengine.exe    372    ReadFile    \Device\HarddiskVolumeShadowCopy65    SUCCESS    Offset: 514,981,888, Length: 65,536, I/O Flags: Non-cached, Priority: Normal

12:54:55.0627246 PM    wbengine.exe    372    WriteFile    B:\WindowsImageBackup\MBX-1\Backup 2011-01-09 175320\cd015918-a5f1-11de-9f02-00155d00010c.vhd    SUCCESS    Offset: 134,617,600, Length: 65,536, I/O Flags: Non-cached, Priority: Normal

 

image

 

At this point we should have a basic understanding of how the backup process can cause performance issues on a disk.  First the checksum runs against both the databases and log files without any throttling using both the shadow copy location and the production storage locations.  Second the streaming of data to the backup media occurs off the same shadow storage and production data locations to the backup media (although this process is less disk intensive then eseutil non the less it is additional IO to the volumes in question).

Comments

  • Anonymous
    January 01, 2003
    @Ravi...

    I'm glad you like it.

    TIMMCMIC

  • Anonymous
    January 01, 2003
    @Matthew:

    I'm glad you found it helpful.

    TIMMCMIC

  • Anonymous
    December 11, 2013
    What an EXCELLENT post, Thank you for taking the time to document this.

  • Anonymous
    December 11, 2013
    @Adam: I'm glad you found it helpful. TIMMCMIC

  • Anonymous
    March 29, 2014
    Superb document... Thanks a lot :)

  • Anonymous
    April 02, 2014
    Very helpful, thank you very much!

  • Anonymous
    April 02, 2014
    Very helpful, thank you very much!

  • Anonymous
    February 23, 2016
    very helpful. thank you!

  • Anonymous
    February 23, 2016
    @MortonChen...

    I'm glad you found it helpful.

    TIMMCMIC