Exchange 2010 - Using VSS to perform an online offline database seed.

Artikel
07/12/2011

When implementing an Exchange 2010 Database Availability Group it may become necessary to perform a database seeding operation. The operation is typically performed as part of adding a mailbox database copy to a DAG member but may also be performed to recover from database divergence.

Seeding is most often performed using the update-mailboxdatabasecopy command. During seeding, the target replication service sends a seeding request to the source replication service on the DAG replication port (64327 by default). The source replication service then initiates a local ESE streaming backup session to the Information Store service. Pages are read from the source database by the source replication service and transmitted to the target replication service. The target replication service then writes the pages to the target database. There are sometimes where this process fails or for various reasons cannot be utilized. This means an alternate way of seeding the database is needed.

One method is to perform a manual offline seeding. In this operation, the source database is dismounted, verified, to be in a clean shutdown state, and then manually copied offline to the target. This can obviously be inconvenient, since the source database has to be down while the copy procedure is being performed.

Another method is to use a VSS backup of the database to seed the database copy. You can use VSS to backup the database, and VSS to restore the database. (There are no longer streaming backups for Exchange 2010).

When using an Exchange-aware VSS application, there are typically four destinations for a restore (note, your backup software may not enable all the options):

1. Original mailbox database.

2. Alternate mailbox database.

3. Recovery mailbox database.

4. File system.

To use the VSS backup and restore method, you would choose to restore to the file system.

The following steps outline a high level process on how to utilize a VSS backup and restore to file system to complete an online offline database seed operation.

======================================

The first step is to enable replication for the mailbox database. This step is accomplished by utilizing the add-mailboxdatabasecopy command with the –seedingPostponed parameter. This command will add the copy and inform all replication services the copy is present. Log truncation will also be suspended since copy status is not healthy for the database. If –seedingPostponed is not specified the database seeding operation will automatically be performed.

add-mailboxdatabasecopy –identity <DBNAME> –mailboxServer <DAGMember> –seedingPostponed:$TRUE

[PS] C:\Windows\system32>Add-MailboxDatabaseCopy -Identity DAG-DB3 -MailboxServer MBX-3 -SeedingPostponed:$TRUE
WARNING: Replication is suspended for database copy 'DAG-DB3' because the database copy needs to be seeded.

If you have already added a mailbox database copy for the database proceed to the second step.

======================================

The second step is to ensure that the mailbox database copy is in a suspended state. Mailbox database copies can be suspended in bulk or one at a time. The following is an example command to suspend a single mailbox database copy:

Suspend-mailboxdatabasecopy –identity <Database\Server>

[PS] C:\Windows\system32>Suspend-MailboxDatabaseCopy -Identity DAG-DB3\MBX-3

Confirm
Are you sure you want to perform this action?
Suspending mailbox database copy "DAG-DB3" on server "MBX-3".
[Y] Yes [A] Yes to All [N] No [L] No to All [?] Help (default is "Y"): a

Prior to proceeding the get-mailboxdatabasecopy status command should be utilized to verify a status of suspended. The following is an example command to verify copy status for a single database copy:

Get-mailboxdatabasecopystatus –identity <Database\Server>

[PS] C:\Windows\system32>Get-MailboxDatabaseCopyStatus DAG-DB3\MBX-3

Name Status CopyQueue ReplayQueue LastInspectedLogTime ContentIndex
Length Length State
---- ------ --------- ----------- -------------------- ------------
DAG-DB3\MBX-3 Suspended 16 191 11/29/2010 7:49:53 AM Failed

======================================

The third step is to note the important paths that are necessary to complete the rest of these steps. Specifically, we are interested in the mailbox database log path and database file path. To get all the paths for the mailbox database on the source server, use the following command:

get-mailboxdatabase –identity <Database> | fl name,logFilePrefix,logFolderPath,edbFilePath

[PS] C:\Windows\system32>Get-MailboxDatabase -Identity DAG-DB3 | fl name,logFilePrefix,logFolderPath,edbFilePath

Name : DAG-DB3
LogFilePrefix : E04
LogFolderPath : c:\DAG\DAG-DB3
EdbFilePath : c:\DAG\DAG-DB3\DAG-DB3.edb

======================================

The forth step is to verify that the source log file sequence is in order. If the source log file sequence has been manually manipulated, and if any log gaps are present, this results in a failure of the seed operation. This step ensures that log files are in sequence on the source machine.

To ensure that the log sequence on the source machine is in the correct order, perform the following operations:

1) Open a command prompt and navigate to the log file directory of the mailbox database. This path can be found from the output gathered in step 3 above.

2) Run the following eseutil command:

eseutil /ml <LogFilePrefix>

The log file prefix can be found from the output gathered in step 3.

When you run this command it will scan every log file found in the source directory. If any gaps or errors are identified, you cannot continue with these steps. If the command completes and errors on the last log file in the series this is expected, as the EXX.log is currently open for writing and cannot be scanned. The following is sample output that you should receive for a mailbox database that is online.

[PS] C:\DAG\DAG-DB3>eseutil /ml E04

Extensible Storage Engine Utilities for Microsoft(R) Exchange Server
Version 14.01
Copyright (C) Microsoft Corporation. All Rights Reserved.

Initiating FILE DUMP mode...

Verifying log files...
Base name: E04

Log file: C:\DAG\DAG-DB3\E04000000BF.log - OK
Log file: C:\DAG\DAG-DB3\E04000000C0.log - OK
Log file: C:\DAG\DAG-DB3\E04000000C1.log - OK
Log file: C:\DAG\DAG-DB3\E04000000C2.log - OK
Log file: C:\DAG\DAG-DB3\E04000000C3.log - OK
Log file: C:\DAG\DAG-DB3\E04000000C4.log - OK
Log file: C:\DAG\DAG-DB3\E04000000C5.log - OK
Log file: C:\DAG\DAG-DB3\E04000000C6.log - OK
Log file: C:\DAG\DAG-DB3\E04000000C7.log - OK
Log file: C:\DAG\DAG-DB3\E04000000C8.log - OK
Log file: C:\DAG\DAG-DB3\E04000000C9.log - OK
Log file: C:\DAG\DAG-DB3\E04000000CA.log - OK
Log file: C:\DAG\DAG-DB3\E04000000CB.log - OK
Log file: C:\DAG\DAG-DB3\E04000000CC.log - OK
Log file: C:\DAG\DAG-DB3\E04000000CD.log - OK
Log file: C:\DAG\DAG-DB3\E04000000CE.log - OK
Log file: C:\DAG\DAG-DB3\E04000000CF.log - OK
Log file: C:\DAG\DAG-DB3\E04.log
ERROR: Cannot open log file (C:\DAG\DAG-DB3\E04.log). Error -1032.

Operation terminated with error -1032 (JET_errFileAccessDenied, Cannot access file, the file is locked or in use) after
1.469 seconds.

======================================

The fifth step is to perform a VSS backup of the database. Please consult with your backup vendor to ensure that a successful FULL backup is performed. Please also make sure that a consistency check of the backup is performed.

======================================

The sixth step is to restore the VSS backup. When you perform the restore, you should select the option to restore to file system (it may be necessary to consult your backup vendor). This may require that you restore to the file system of the Exchange Server, so it may be necessary to ensure that sufficient free space exists on a volume on the Exchange Server where the restore will be performed.

If multiple databases are being restored I recommend that databases be restored individually.

Please ensure that no recovery operations are performed on the database (options like roll forward recovery / replay logs / etc should be avoided).

At this point we now have the EDB file on the file system and we will use it for the seeding operation.

In our example we’ll restore to c:\RESTORE.

======================================

The seventh step is to ensure that the target paths are ready to have the database moved in place. The paths referenced in this step can be obtained from step 3.

In this example we will ensure the path c:\DAG\DAG-DB3 is empty on the target server.

If the paths already existed they are now ready to have the restored database moved to them.

If the paths do not exist they should be manually created. If you are using nested folders you need to create the entire directory structure.

======================================

The eighth step is to move the restored database to the target directory. This can be accomplished in a few different ways, but I will make a recommendation below.

From the source server map to the drive$ share of the target. For example, I would map the Y drive to \\MBX-3\C$\DAG\DAG-DB3 using our example.

net use y: <path>

C:\>net use y: \\MBX-3\c$\DAG\DAG-DB3
The command completed successfully.

On the source server open a command prompt to the directory where the data was restored. In this example c:\Restore

Use eseutil to copy the database from the source directory to the target directory. A sample command:

eseutil /y <Source.EDB> /d <Target.EDB> Here is the output expected from the command using our example:

C:\Restore>eseutil /y DAG-DB3.edb /d y:\DAG-DB3.edb

Extensible Storage Engine Utilities for Microsoft(R) Exchange Server
Version 14.01
Copyright (C) Microsoft Corporation. All Rights Reserved.

Initiating COPY FILE mode...
Source File: DAG-DB3.edb
Destination File: y:\DAG-DB3.edb

Copy Progress (% complete)

0 10 20 30 40 50 60 70 80 90 100
|----|----|----|----|----|----|----|----|----|----|
...................................................

Total bytes read = 0x18810000 (411107328) (392 MB)
Total bytes written = 0x18810000 (411107328) (392 MB)

Operation completed successfully in 29.16 seconds.

At this point the copy has been seeded on the target server.

In place of the network administrators may consider a portable storage device for the transportation of the database to the target server. (Note: In this case eseutil /y would be used to copy the data to the portable storage and from the portable storage to the target server).

======================================

The ninth step is to verify the health of the copied database. We need to ensure that the database was not corrupted as a part of the copy process.

On the target server open a command prompt and navigate to the location of the database file. In our example this is c:\DAG\DAG-DB3.

Use the eseutil /k to perform a checksum of the database:

eseutil /k <Database.edb>

The following output will be observed when the command completes:

C:\DAG\DAG-DB3>eseutil /k DAG-DB3.edb

Initiating CHECKSUM mode...
Database: DAG-DB3.edb
Temp. Database: TEMPCHKSUM4804.EDB

File: DAG-DB3.edb

Checksum Status (% complete)

0 10 20 30 40 50 60 70 80 90 100
|----|----|----|----|----|----|----|----|----|----|
...................................................

12546 pages seen
0 bad checksums
0 correctable checksums
5571 uninitialized pages
0 wrong page numbers
0x62c97 highest dbtime (pgno 0x7c)

6273 reads performed
392 MB read
6 seconds taken
65 MB/second
5895209 milliseconds used
939 milliseconds per read
3437 milliseconds for the slowest read
93 milliseconds for the fastest read

Operation completed successfully in 6.672 seconds.

We are interested in ensuring that there are 0 bad checksums.

======================================

The last step in the process is to resume the mailbox database copy.

The following command can be used to resume the mailbox database copy:

Resume-MailboDatabaseCopy –identity <Database\Server>

[PS] C:\>Resume-MailboxDatabaseCopy DAG-DB3\MBX-3

Post a resume the following events can be noted in the application log on the target server.

Log Name: Application
Source: MSExchange Search Indexer
Date: 12/6/2010 5:30:16 AM
Event ID: 109
Task Category: General
Level: Information
Keywords: Classic
User: N/A
Computer: MBX-3.exchange.msft
Description:
Exchange Search Indexer has created a new search index and will perform a full crawl for the Mailbox Database DAG-DB3 (GUID = 124ba5d0-b20e-4297-8ac7-e1613dc86225). Reason for full crawl: Catalog doesn't exist.

Log Name: Application
Source: MSExchangeRepl
Date: 12/6/2010 5:30:17 AM
Event ID: 2114
Task Category: Service
Level: Information
Keywords: Classic
User: N/A
Computer: MBX-3.exchange.msft
Description:
The replication instance for database DAG-DB3 has started copying log files. The first log file copied was generation 191.

Log Name: Application
Source: MSExchange Search Indexer
Date: 12/6/2010 5:30:18 AM
Event ID: 108
Task Category: General
Level: Information
Keywords: Classic
User: N/A
Computer: MBX-3.exchange.msft
Description:
Exchange Search Indexer has enabled indexing for the Mailbox Database DAG-DB3 (GUID = 124ba5d0-b20e-4297-8ac7-e1613dc86225).

Log Name: Application
Source: MSExchangeIS Mailbox Store
Date: 12/6/2010 5:30:27 AM
Event ID: 1000
Task Category: General
Level: Information
Keywords: Classic
User: N/A
Computer: MBX-3.exchange.msft
Description:
Attempting to start the Information Store "DAG-DB3".

Log Name: Application
Source: MSExchangeRepl
Date: 12/6/2010 5:30:37 AM
Event ID: 2157
Task Category: Service
Level: Information
Keywords: Classic
User: N/A
Computer: MBX-3.exchange.msft
Description:
The replication instance for database DAG-DB3 has copied and replayed multiple logs.

Log Name: Application
Source: MSExchange Search Indexer
Date: 12/6/2010 5:32:56 AM
Event ID: 110
Task Category: General
Level: Information
Keywords: Classic
User: N/A
Computer: MBX-3.exchange.msft
Description:
Exchange Search Indexer completed a full crawl (indexing) of Mailbox Database DAG-DB3 (GUID = 124ba5d0-b20e-4297-8ac7-e1613dc86225).

The administrator can monitor the post seeding activity using the get-mailboxdatabasecopystatus <Database\Server> command.

In this example we will use get-mailboxdatabasecopystatus DAG-DB3\MBX-3.

Initially the administrator will note a STATUS of RESYNCHRONIZING and a CONTENTINDEXSTATE of CRAWLING.

[PS] C:\>Get-MailboxDatabaseCopyStatus DAG-DB3\MBX-3

After all delta log files have copied to the target and replay has begun, the administrator will note a STATUS of HEALTHY and a CONTENTINDEXSTATE of CRAWLING.

[PS] C:\>Get-MailboxDatabaseCopyStatus DAG-DB3\MBX-3

Due to the online offline database seed the content index files were not copied from the target server and were not restored from backup. Therefore the search service on the target will initialize a new content index for this store and being indexing from the source. If the database is large content indexing could be in a CRAWLING state for some time.

After the content index has been successfully built the administrator will note a STATUS of HEALTHY and a CONTENTINDEXSTATE of HEALTHY.

[PS] C:\>Get-MailboxDatabaseCopyStatus DAG-DB3\MBX-3

At this point the database seeding operations are now 100% complete and the database and fully participate in DAG functions.

======================================

Comments

Anonymous
January 01, 2003
@Suzan... You can just delete the file if it is no longer necessary for replication etc. TIMMCMIC
Anonymous
March 29, 2012
Excellent article to undersantd offline database seeding when we enable DAG replication over WAN. We used to follow the offline Exchange Database copying of SCR source server to target server and enable the SCR replication in Exchange 2007 SCR. Should we use same method in exchange 2010 to copy the offline database to target DAG server and run add-mailboxdatabasecopy cmd let to enable replication ??.
Anonymous
June 06, 2013
I want to use this procedure to offline seed a failed database replication in a remote DAG member. Step 7 says that the target path has to be empty but the old copy of the database is there. How do I remove it?
Anonymous
October 25, 2013
Hi Tim - what if the second step does not show suspended, but failed and suspended instead? Will this still work?
Anonymous
October 25, 2013
@Brian: What a great question. Surprisingly this is the second time I've had this question this week - go figure!. Suspended, failed and suspended, or failed would all be valid status here. The real point of having one of these status is to ensure that no log truncation can occur either through circular logging <or> through the backup process. Any of these status will not allow for log truncation. TIMMCMIC

Freigeben über

Exchange 2010 - Using VSS to perform an online offline database seed.

Comments

Zusätzliche Ressourcen