Share via

BizTalk Server Disaster Recovery

  • Article

Disaster Site Set Up

1 - Setting up BizTalk Server at DR Site

  1. Configure these BizTalk Server run-time servers using the BizTalk Configuration Wizard to join them to the production BizTalk group. When configuring the BizTalk Server run-time servers at the disaster recovery site (including the disaster recovery Enterprise Single Sign-On Master Secret server) select No to the question “Is this the master secret server?” and select “Join” to the question “Do you want to create or join a BizTalk Server group?”
  2. After configuring the BizTalk Server run-time disaster recovery servers, create BizTalk host instances on the disaster recovery site that correspond to the production site host instances, but do not start these host instances. For example, if the production site has three Hosts Send, Receive, and Orchestration with instances on server1, server2, and server3, create three corresponding host instances on DRserver1, DRserver2, DRserver3.
  3. All BizTalk Server-related Windows services such as the BizTalk Host Instance and Rules Engine Service at the disaster recovery site should be set to “disabled” in the Services Manager to prevent the disaster recovery site from performing any processing.
  4. Create a set of SQL Server database instances in the disaster recovery site. To ensure that the disaster recovery SQL Server database instances can provide the same level of performance as the production SQL Server database instances, the disaster recovery SQL Server database instances should be configured with similar hardware and number of physical computers running SQL Server.
  5. Configure log shipping for each production SQL Server database instance to apply to a corresponding SQL Server database instance at the disaster recovery site.
  6. Make sure drive letter(s) on the production site where the database files are stored match the drive letter(s) at the disaster recovery site where the database files are restored. So if the SQL Server database file group is located on G:\data in production, there must be a G:\data directory on the destination (DR) server or the restore will fail.
  7. Create SQL Server security logins for the disaster recovery site that correspond to the production site so that in the event that a failover to the disaster recovery site is required, all required security logins are present on the destination system.
  8. Once installation of the disaster recovery SQL Server instances is completed, perform a full backup of master and msdb databases so that a clean system can be restored in the event that a switch to the disaster recovery site fails.
  9.  Set up log shipping for SQL Server. Refer Appendix 2.

2 - Setting up SQL Server

 

3 - BizTalk Application Backup and SQL Server backup

  1. Always keep the latest back up of the applications
  2. Make sure the SQL backup jobs are running and we have the latest backup
  3. Keep the backup of machine.config, registry settings
  4. Keep the MSMQ configuration backup
  5. Back up master secret

4 - Backing up IIS

– Use the backup and restore scripts.

5 - IIS certificates

– install IIS 6.0 Resource Kit.

6 - Certificate store certificates

 – Take the back up of all the certificates from the certificate store for below services, applications.

  1. Loud Cloud
  2. StudentEmail Hosting

 

 

Recovering from a Disaster

7 - Restoring the BizTalk Group

  i.      Stop all the processing(Refer appendix 1)

8 - Restore SQL Server and BizTalk applications

                    i.      Restore SQL Server from Log Shipping – Refer Appendix 3

                   ii.      Restore the Master Secret Server. Appendix 5.

                  iii.      Restoring SQL Server Jobs. Refer Appendix 4.

                  iv.      Restore BAM database by using the BAM backup.

                 v.     Resolve all the incomplete BAM activity

9 - IIS and certificates restore

                      i.      IIS restore – use IISRestore batch file to restore the IIS.
                     ii.      IIsCertDeploy.vbs will be located in %programfiles%\IIS Resources\IIsCertDeploy.  Use below query to restore the certficates.

                                    - iiscertdeploy.vbs -e d:\webbackups\certs\site1.pfx

 -i w3svc/1 -p site1pwd -q On.

10 - Certificate Restore

 

Restore certificates from the back up of certs taken as part of certificate back up. Use the scripts attached.

11 - Recovering the Runtime Computers

 

                       i.      Update Runtime Computers 
                   ii.      If receive locations and send ports are not affected by the loss of the production site, it may not be necessary to reconfigure the application with disaster recovery-specific locations.

 

Appendix 1 – Stopping BizTalk Server Processing

To stop application processing on the source system, ensure that no connections are open between the production BizTalk Server runtime computers and the SQL Server computers that house the BizTalk Server databases. Follow these steps to stop application processing on the production BizTalk Server runtime computers:

  1. Stop the IIS so that no new request should come to Service Bus.
  2. Disable all receive locations on the BizTalk Server computers in the BizTalk group. Make a note of all receive locations that are disabled so that these receive locations can be re-enabled later. This will stop BizTalk Server from processing incoming messages.
  3. Make sure that BizTalk has finished processing any messages or orchestrations. Therefore you have to query for Active orchestrations/messages in the Group hub from the BizTalk Server Administration Console.
  4. Make sure that BizTalk has stopped sending messages to its Send ports/Send Port Groups. You could use the Tracked Messages queries to see if messages are being send recently.
  5. Stop all host instances from running on the BizTalk Server computers in the group. This can be done from the BizTalk Server Administration console. Make a note of all the host instances that were stopped so that these host instances can be restarted later.
  6. Stop all SQL Server Agent jobs related to BizTalk Server on the SQL Server computers that house BizTalk Server databases.
  7. Stop any other BizTalk Server services in Services Manager that may be running on the BizTalk Server computers in the group, for example, the Enterprise Single Sign-On Service and the Rule Engine Update Service. Make a note of the services that are stopped so that they can be restarted later.
  8. Close all applications that connect to the SQL Server computers that house BizTalk Server databases. This includes instances of the BizTalk Server Administration console, Visual Studio 2010 and any other installed BizTalk applications.
  9. Verify that there is no database activity generated by BizTalk Server. Use SQL Server Enterprise Manager or SQL Server 2005 Management Studio to see what processes are connected to the SQL Server computers that house BizTalk Server databases. This can be done by clicking to expand Management, Current Activity in SQL Server Enterprise Manager or by clicking to expand Management and Activity Monitor in SQL Server 2005 Management Studio. Then click to select Process Info. Alternatively use the system stored procedures sp_who or sp_who2 to identify any open connections to the SQL Server computers that house BizTalk Server databases. If there are any processes connected, locate them and terminate them normally; or as a last resort, right-click each process in the Process Info pane in SQL Server Enterprise Manager or SQL Server 2005 Management Studio and click Kill process to terminate the connection.
  10. Additional database processing may be occurring in application databases. If these databases will be restored, ensure that all processing is stopped.

Appendix 2 – Log Shipping

 

  1. Log in as a member of the BizTalk Server Administrators group to perform this procedure. You must have the same version of SQL Server on both the source and destination systems. SQL Server must be installed in the same relative location on both the source and destination systems.
  2. The directory for SQL transaction log (.LDF files) on the source system must also exist on the destination system. If this directory is not on the destination system, create the directory with the same name and permissions as on the source system.
  3. Open the SQL Server management studio.
  4. Connect to appropriate SQL server
  5. Run below script on master database
    1. C:\Program Files\Microsoft BizTalk Server 2009\Schema\LogShipping_Destination_Schema.sql
    2. C:\Program Files\Microsoft BizTalk Server 2010\Schema\LogShipping_Destination_Logic.sql
    3. Ad Hoc Distributed Queries on destination system by using below query

 

sp_configure 'show advanced options', 1;

RECONFIGURE;

sp_configure 'Ad Hoc Distributed Queries', 1;

RECONFIGURE;

GO

 

SELECT a.*

FROM OPENROWSET('SQLNCLI', 'Server=Seattle1;Trusted_Connection=yes;',

     'SELECT GroupName, Name, DepartmentID

      FROM AdventureWorks2008R2.HumanResources.Department

      ORDER BY GroupName, Name') AS a;

GO

 

  1. Please make sure that query should execute in first attempt else you need to execute all the steps from step one. Execute below query. For more than one server please repeat this for all the servers.

exec bts_ConfigureBizTalkLogShipping @nvcDescription = 'LogShippingSolution_ENV',

@nvcMgmtDatabaseName = 'BizTalkMgmtDB',

@nvcMgmtServerName = 'BizTalk Database Server',

@SourceServerName = null, -- null indicates that this destination server restores all databases

@fLinkServers = 1 -- 1 automatically links the server to the management database

 

 

 

  1. Verify below jobs are created as in SQL Server Agent jobs. We need to monitor these jobs and these jobs should not fail.

                    i.      BTS Log Shipping Get Backup History

                   ii.      BTS Server Log Shipping Restore Databases

                  iii.      BTS Log Shipping Restore To Mark

  1. Open C:\Program Files (x86)\Microsoft BizTalk Server 2010\Bins32\Schema\Restore on the BizTalk server.
  2. Edit SchemaUpdateInfo.xml  replace the name of the Source System and Destination System.
  3. Add two more entries for BAM.

<Database Name="BAM Alerts Application DB" oldDBName="BAMAlertsApplication" oldDBServer="SourceServer" newDBName=" BAMAlertsApplication" newDBServer="DestinationServer"/>

<Database Name="BAM Alerts Instance DB" oldDBName="BAMAlertsNSMain" oldDBServer="SourceServer" newDBName="BAMAlertsNSMain" newDBServer="DestinationServer"/>

  1. Since we have more than two databases add other two messageboxes and set isMaster=0
  2. SAVE the file and exist.

Appendix 3 – Restoring Database

1-      If there is only one server in the destination system, make sure that all of the log backup sets (except for the most recent set) have been restored. For more information, see Viewing the History of Restored Backups. If all the log backup sets have not been restored, and the restore job is not currently running, run the restore job (manually if necessary). If there are outstanding backup sets that can be restored, the job will process them until they are all restored.

2-      If there are multiple servers in the destination system, all servers must be restored to the same backup set. You must view the restore history on each server and make sure that the most recent log backup set restored is the same on all servers. If it is not, you must manually run the restore job on each server that needs the most recent log backup set restored.

3-      The adm_BackupHistory table is the central history point for the log shipping process for the source system. All backup work performed is recorded to this table.

          i.      Connect to SQL Server management studio Disable three Log shipping related jobs i.e. “BTS Log Shipping – Get Backup History”, “BTS Server Log Shipping Restore Databases”

4-      Start the job and “BTS Log Shipping - Restore To Mark” at step 1. If the job is successful, SQL Server Agent jobs and BizTalk Server databases are restored to the destination system.

Appendix 4 – Backing up and Restoring SQL Server Jobs

**1-     ** In SQL Server Management Studio Expand SQL Server Agent and expand the list of Jobs. Right click and say script job as Create To or Drop To and save it with .sql extension.
**2-     ** To restore a job in SQL Server management studio, locate the file and execute the scripted job to create the job.               

Appendix 5 – Restoring the Master Secret Server

**1-     ** Make sure that Master Secret has been backed up - this step needs to happen before the Production ENTSSO server is down/before start of disaster recovery process. If not back up the mater secret. 

  1. On the Start menu, click Programs, click Microsoft Enterprise Single Sign-On, and then click SSO Administration.
  2. In the scope pane of the ENTSSO MMC Snap-In, expand the Enterprise Single Sign-On node.
  3. Right-click System, and then click Back up Master Secret.

2. Change the master secret server name in the SSO database to reference the new master secret server. For example, the name of the new master secret server may be NewMSSServer. To do this, follow these steps on the original master secret server:

   a. Paste the following code in a text editor such as notepad.exe:

    <sso>
    <globalInfo>
    <secretServer>NewMSSServer</secretServer>
    </globalInfo>
    </sso>

    b. Save the file as .xml file. For example, save the file as NewMSSServer.xml.

    c. At a command prompt, change to the Enterprise SSO installation folder. By default, the installation folder is <drive>:\Program Files\Common Files\Enterprise Single Sign-On.

    d. Type ssomanage -updatedb XMLFile to update the master secret server name in the database.
     
    Replace XMLFile with the name of the .xml file that you saved.
    On a system that supports User Account Control (UAC), you may need to run the tool with Administrative privileges.

  Note: may need to run below command (before a,b and c) if DR SSo still have any reference to Prod server 
    SSomanage" -server <DR SSO Server>

3. Restart the Enterprise Single Sign-On service on the new master secret server.

4. Restore the backed-up master secret onto the new master secret server by following the steps in How to Restore the Master Secret on the new master secret server.

Appendix 6 – Update the Run Time Computers

1-      Navigate to “C:\Program Files\Microsoft BizTalk Server 2010\Schema\Restore”. Execute cscript UpdateDatabase.vbs SampleUpdateInfo.xml on the BizTalk Server.

2-      Run the script mention in Step 5 on each BizTalk Server.

3-      Restart all of the BizTalk Server services.

4-      After restoring your databases, you must restart the Windows Management Instrumentation service.

5-      Login to BizTalk Server Admin. Remove the existing group.

6-      Connect to existing Group. In the Connect to Existing BizTalk Server Configuration Database dialog box, in the SQL Server name drop-down list box, select the name of the Microsoft SQL Server instance that hosts the BizTalk Management database. When you select the instance of SQL Server, BizTalk Server automatically attempts to detect BizTalk Server databases on that computer.

7-      Update the SSO Server name as displayed in the Group Properties dialog box available in the BizTalk Server Administration console. To update the SSO Server name, launch the BizTalk Server Administration console, click to expand BizTalk Server Administration, right-click the BizTalk Group node and select Properties to display the General tab of the BizTalk Server Administration console. Then enter the name of the Enterprise Single Sign-On server that this computer will use to access the configuration information for the adapters. This is the name of the SSO server used to connect to the SSO database.

8-      In the Database name drop-down list box, select the BizTalk Management database (BizTalkMgmtDb) to which you want to connect, and then click OK.

9-      Restart the following services

  1. Enterprise SSO Service
  2. Rule Engine Update Service
  3. BizTalk Host Instances

10-   Run the BizTalk Server Application Installation MSI file on each BizTalk server in the group.

11-   Run the BizTalk Server Application Configuration MSI file on one server in the group to configure the application for the disaster recovery site. In our case the names for receive locations and send ports remain the same so we do not need to update the bindings. These information are already present in the BizTalkMgmt database.

12-   Restore application processing by enabling all application receive locations, send ports, and host instances.

13-   Restart the following services

  1. Enterprise SSO Service
  2. Rule Engine Update Service
  3. BizTalk Host Instances
  4. Verify that BizTalk is restored and running.

 

Troubleshooting help

**1-     ** Troubleshooting Log Shipping http://msdn.microsoft.com/en-us/library/cc296675(BTS.10).aspx

 

Scripts

PowerShell script for Certificate installation:

 

 

write-host "Starting Cert Import"
function Import-PfxCertificate {     
    
   param([String]$certPath,[String]$certRootStore = “CurrentUser”,[String]$certStore = “My”,$pfxPass = $null)     
   $pfx = new-object System.Security.Cryptography.X509Certificates.X509Certificate2     
    
   if ($pfxPass -eq $null) {$pfxPass = read-host "Enter the pfx password"  -assecurestring}     
    
   $pfx.import($certPath,$pfxPass,"Exportable,PersistKeySet")     
    
   $store = new-object System.Security.Cryptography.X509Certificates.X509Store($certStore,$certRootStore)     
   $store.open("MaxAllowed")     
   $store.add($pfx)     
   $store.close()  
   write-host "Success!!"      
     
} 
  
function Import-509Certificate {     
    
   param([String]$certPath,[String]$certRootStore,[String]$certStore)     
    
   $pfx = new-object System.Security.Cryptography.X509Certificates.X509Certificate2     
   $pfx.import($certPath)     
    
   $store = new-object System.Security.Cryptography.X509Certificates.X509Store($certStore,$certRootStore)    
   $store.open("MaxAllowed")     
   $store.add($pfx)     
   $store.close()  
   write-host "Success!!"   
}    
        
  
Import-509Certificate "C:\DR\loudcloud.cer"  "CurrentUser" "My"

 

 

IIS 6.0 Backup and Recovery : I will share these soon.