Edit

Share via

Stretch cluster replication using shared storage

This evaluation example allows you to configure computers and their storage in a single stretch cluster, where two nodes share one set of storage and two nodes share another set of storage, then replication keeps both sets of storage mirrored in the cluster to allow immediate failover. These nodes and their storage should be located in separate physical sites, although it isn't required. There are separate steps for creating Hyper-V and File Server clusters as sample scenarios.

Important

Servers in different sites must be able to communicate with the other servers via a network, but not have any physical connectivity to the other site's shared storage. This scenario doesn't make use of Storage Spaces Direct.

This walkthrough uses the following environment as an example:

  • Four servers named SR-SRV01, SR-SRV02, SR-SRV03, and SR-SRV04.

  • The four servers are formed into a single cluster called SR-SRVCLUS.

  • A pair of logical "sites" that represents two different data centers. One called Redmond and the other called Bellevue.

  • Servers SR-SRV01 and SR-SRV02 belong to the Redmond site.

  • Servers SR-SRV03 and SR-SRV04 belong to the Bellevue site.

Diagram showing two nodes in Redmond replicating with two nodes of the same cluster in the Bellevue site.

Note

You can use only as few as two nodes, where one node each is in each site. However, you can't perform intra-site failover with only two servers. You can use as many as 64 nodes.

Prerequisites

  • Software:

    • 2-64 servers running Windows Server 2016 Datacenter Edition or later. If you're running Windows Server 2019, you can use Standard Edition if you're comfortable replicating only a single volume up to 2 TB in size.
    • The Active Directory Domain Services role, Failover Clustering, and Storage Replica features must be installed on your device. To learn more, see Install or Uninstall Roles, Role Services, or Features.
    • Your device must be part of an Active Directory (AD) forest.
    • Appropriate firewall and router rules to allow ICMP, SMB port 445, SMB Direct port 5445, and WS-MAN port 5985 bidirectional traffic between all nodes.

  • Hardware:

    • At least 2 GB of RAM and two cores per server. You'll need more RAM and cores for virtual machine environments.
    • Two sets of shared storage, using SAS JBODs (such as with Storage Spaces), Fibre Channel SAN, Shared VHDX, or iSCSI Target.
    • The storage should contain a mix of HDD and SSD media and must support Persistent Reservation. Each storage set should be available to two of the servers only (asymmetric).
      • The physical storage must be identical in capacity, have the same sector sizes on all the data disks and log disks.
      • The data and log disks must be initialized as GPT.
      • The volumes must be formatted as NTFS or ReFS.
      • The log volumes should use flash-based storage and high performance resiliency settings. Microsoft recommends that the log storage be faster than the data storage.
      • The log volume must be at least 9 GB, or larger or smaller based on log requirements.
    • Each set of storage must allow creation of at least two virtual disks, one for replicated data and one for logs.
    • The replicated storage can't be located on the drive containing the Windows operating system folder.

  • Disk configuration:

    • JBOD enclosures

      • Ensure that each set of paired server nodes can see that site's storage enclosures only (asymmetric storage) and that the SAS connections are correctly configured.

      • Provision the storage using Storage Spaces by following steps 1-3 provided in Deploy Storage Spaces on a stand-alone server using PowerShell or Server Manager.

    • iSCSI storage

      • Ensure that each set of paired server nodes can see that site's storage enclosures only, such as asymmetric storage. You should use more than one single network adapter if using iSCSI.

      • Provision the storage using your vendor documentation. If using Windows-based iSCSI Targeting, see the iSCSI Target Server overview.

    • For FC SAN storage:

      • Ensure that each set of paired server nodes can see that site's storage enclosures only (asymmetric storage) and that you properly zoned the hosts.

      • Provision the storage using your vendor documentation.

  • Network:

    • At least one 1-GbE connection on each server for synchronous replication.
    • A network between servers with sufficient bandwidth to contain your IO write workload and an average of ~5ms round trip latency for synchronous replication. Asynchronous replication doesn't have a latency recommendation.

Note

Data disks can use either mirrored or parity spaces or RAID 1 or 10, RAID 5 or RAID 50.

Log volumes must never be used for other workloads.

Important

While it's possible to attach a storage device to a single server and use this for replication, Windows Failover Clustering still relies on SCSI Persistent Reservations. Therefore, the storage must still be a Shared Storage type such as a SAN technology. Local disks or disks presented by a hypervisor might not be compatible. In Azure, the disks must be a premium SSD size that supports sharing, even if only one VM is to be attached to it.

Many of these requirements can be determined by using the Test-SRTopology cmdlet. You get access to this tool if you install Storage Replica or the Storage Replica Management Tools features on at least one server. There's no need to configure Storage Replica to use this tool, only to install the cmdlets.

Provision your environment

Always sign in as a domain user who is a member of the built-in administrator group on all servers. Always run PowerShell or the command prompt with elevated privileges.

  1. Install Windows Server on all server nodes, using either the Server with Desktop Experience or Server Core installation options.

  2. Ensure that BIOS/UEFI settings for servers are set to high performance, such as disabling C-State, setting QPI speed, enabling NUMA, and setting the highest memory frequency. Ensure power management in Windows Server is set to high performance. Restart as required.

  3. Add the network information and join the nodes to the domain, then restart them.

    Note

    This guide presumes you have two pairings of servers for use in a stretch cluster. A WAN or LAN network separates the servers and the servers belong to either physical or logical sites.

  4. Connect the first set of shared JBOD storage enclosure, Shared VHDX, iSCSI target, or FC SAN to the servers in Redmond site.

  5. Connect the second set of storage to the servers in Bellevue site.

  6. Install the latest vendor storage and enclosure firmware, all necessary drivers, HBA drivers, and BIOS/UEFI firmware on all four nodes. Restart the nodes as needed.

    Note

    Consult your hardware vendor documentation for configuring shared storage and networking hardware.

In the next step, the servers will be added and configured with the necessary roles and features:

  1. In Server Manager, select Manage, then select Add Servers.

  2. In the Add Servers window, select your servers through the following methods:

    1. AD (the servers must be joined to the domain)

    2. DNS (computer name or IP)

    3. Import (from text file)

  3. Once you select your servers, import them using the "→" button, then select OK.

  4. On SR-SRV04 or a remote management computer, run the following command in an elevated PowerShell window:

    $Servers = 'SR-SRV01','SR-SRV02','SR-SRV03','SR-SRV04'

$Servers | foreach { Install-WindowsFeature -ComputerName $_ -Name Storage-Replica,Failover-Clustering,FS-FileServer -IncludeManagementTools -Restart }

Configure a Hyper-V Failover Cluster or File Server cluster

After you set up your server nodes, the next step is to create a Hyper-V failover cluster or a File Server cluster. If the nodes reside in different subnets, then an IP Address for the additional site must be created using the "OR" dependency. To learn more, see Configuring IP Addresses and Dependencies for Multi-Subnet Clusters – Part III.

If you're creating a two-node stretch cluster, you must add all storage before continuing. This is a by-design behavior in Windows Server 2016. Run the following command to add available storage:

Get-ClusterAvailableDisk -All | Add-ClusterDisk

When using a test server with no write IO load on a specified source volume, consider adding a workload as Test-SRTopology doesn't generate a useful report. You should test with production-like workloads in order to see real world numbers and recommended log sizes.

Alternatively, copy some files into the source volume during the test or download and run diskspd to generate write IOs. For example, to sample a low write IO workload for 10 minutes on the D: volume, run the following command:

diskspd -c1g -d600 -W5 -C5 -b4k -t2 -o2 -r -w5 -i100 D:\Test.dat

No option is available to configure site awareness using Failover Cluster Manager in Windows Server 2016.

Note

Configure a File Share Witness or Cloud Witness to provide quorum if a site goes down. Windows Server now includes an option for Cloud (Azure)-based Witness. You can choose this quorum option instead of the file share or disk witness.

For more information about quorum configuration, see the Configure and Manage the Quorum in a Windows Server 2012 Failover Cluster guide's Witness Configuration. For more information on the cluster quorum cmdlets, see the FailoverClusters module set.

Tip

Review Network Recommendations for a Hyper-V Cluster in Windows Server 2012 to ensure cluster networking is optimally configured. Configure cluster networking and AD for faster DNS site failover. You can utilize Hyper-V software defined networking, stretched VLANs, network abstraction devices, lowered DNS TTL, and other common techniques.

You can also configure VM resiliency so that guests don't pause for long during node failures. Instead, they fail over to the new replication source storage within 10 seconds. To perform this action, run the (Get-Cluster).ResiliencyDefaultPeriod=10 command.

  1. In Server Manager, select Tools, then select Failover Cluster Manager.

  2. In the right pane, under Actions, select Validate Configuration to ensure you can proceed.

    Note

    You should expect storage errors from cluster validation due to the use of asymmetric storage.

  3. Create the Hyper-V compute cluster with 15 characters or fewer using the Deploy a Hyper-V Cluster guide. Follow steps 7-10 within the Redmond site to create a test virtual machine only to ensure the cluster is working normally within the two nodes sharing the storage in the first test site.

  4. Add one disk in the Redmond site to the cluster CSV. Right-click a source disk in the Disks node of the Storage section, and then select Add to Cluster Shared Volumes.

  5. The Test-SRTopology cmdlet is used to determine if the Storage Replica requirements are met. For example, to validate two of the proposed stretch cluster nodes where each have a D: and E: volume for 30 minutes, perform the following steps:

    1. Move all available storage to SR-SRV01.

    2. In Failover Cluster Manager, expand your cluster, right-click on Roles, then select Create Empty Role.

    3. Add the online storage to the empty role named New Role.

    4. Move all available storage to SR-SRV03.

    5. Right-click on Roles again, then select Create Empty Role.

    6. Move New Role (2) to SR-SRV03.

    7. Add the online storage to that empty role named New Role (2).

    8. Evaluate the cluster with by running the following command:

      $params = @{
   SourceComputerName       = 'SR-SRV01'
   SourceVolumeName         = 'D:'
   SourceLogVolumeName      = 'E:'
   DestinationComputerName  = 'SR-SRV03'
   DestinationVolumeName    = 'D:'
   DestinationLogVolumeName = 'E:'
   DurationInMinutes        = 30
   ResultPath               = 'C:\Temp'
}
MD C:\Temp | Test-SRTopology @params

  6. Examine the TestSrTopologyReport-<date>.html report to ensure you meet the Storage Replica requirements and note the initial sync time prediction and log recommendations.

    A screenshot of the Storage Replica replication report for a Hyper-V Failover Cluster.

  7. Return the disks to Available Storage and remove the temporary empty roles.

  8. Remove the test virtual machine. Add any real test virtual machines needed for further evaluation to a proposed source node.

  9. Configure stretch cluster site awareness so that servers SR-SRV01 and SR-SRV02 are in the Redmond site, servers SR-SRV03 and SR-SRV04 are in the Bellevue site. Ensure that Redmond is the primary node that has ownership of the source storage and VMs:

    New-ClusterFaultDomain -Name Redmond -Type Site -Description "Primary" -Location "Redmond Datacenter"

New-ClusterFaultDomain -Name Bellevue -Type Site -Description "Secondary" -Location "Bellevue Datacenter"

Set-ClusterFaultDomain -Name SR-SRV01 -Parent Redmond
Set-ClusterFaultDomain -Name SR-SRV02 -Parent Redmond
Set-ClusterFaultDomain -Name SR-SRV03 -Parent Bellevue
Set-ClusterFaultDomain -Name SR-SRV04 -Parent Bellevue

(Get-Cluster).PreferredSite="Redmond"

Manage stretched cluster replication

You can perform all of the steps on the cluster nodes directly or from a remote management computer that contains the RSAT. You can also use Failover Cluster Manager to determine the current source and destination of replication and their status. Managing stretched cluster replication can be performed using the GUI or PowerShell.

To alter replication source and destination within the stretch cluster, use the following methods:

  1. To move the source replication between nodes in the same site, right-click the source CSV, select Move Storage, select Select Node, and then select a node in the same site. If using non-CSV storage for a role assigned disk, you move the role.

  2. To move the source replication from one site to another, right-click the source CSV, select Move Storage, select Select Node, and then select a node in another site. If you configured a preferred site, you can use best possible node to always move the source storage to a node in the preferred site. If using non-CSV storage for a role assigned disk, you move the role.

  3. To perform planned failover the replication direction from one site to another, shut down both nodes in one site using Server Manager or SConfig.

  4. To perform unplanned failover the replication direction from one site to another, cut power to both nodes in one site.

    Note

    In Windows Server 2016, you may need to use Failover Cluster Manager or Move-ClusterGroup to move the destination disks back to the other site manually after the nodes come back online.

    Storage Replica dismounts the destination volumes. This is by design.

  5. To change the log size from the default 8 GB, right-click both the source and destination log disks, select the Replication Log tab, then change the sizes on both the disks to match.

  6. To add another pair of replicated disks to the existing replication group, you must ensure that there is at least one extra disk in available storage. You can then right-click the Source disk and select Add replication partnership.

    Note

    This need for an additional 'dummy' disk in available storage is due to a regression and not intentional.

To remove the existing replication:

  • Right-click the source CSV disk and select Replication, then select Remove. Accept the warning prompt.

    Optionally, remove the storage from CSV to return it to available storage for further testing.

    Note

    You may need to use Disk Management or Server Manager to add back drive letters to volumes after return to available storage.

To measure the replication performance, you can utilize the Performance Monitor (perfmon.exe) tool on both the source and destination nodes. To learn more about the Performance Monitor, see Using Performance Monitor and Add Counters Dialog Box.

On the destination node:

  1. Add the Storage Replica Statistics objects with all their performance counters for the data volume.

  2. Examine the results.

On the source node:

  1. Add the Storage Replica Statistics and Storage Replica Partition I/O Statistics objects with all their performance counters for the data volume (the latter is only available with data on the current source server).

  2. Examine the results.

See also