Storage Quality of Service
Storage Quality of Service (QoS) in Windows Server 2016 provides a way to centrally monitor and manage storage performance for virtual machines using Hyper-V and the Scale-Out File Server roles. The feature automatically improves storage resource fairness between multiple virtual machines using the same file server cluster and allows policy-based minimum and maximum performance goals to be configured in units of normalized IOPS.
You can use Storage QoS in Windows Server 2016 to accomplish the following:
Mitigate noisy neighbor issues. By default, Storage QoS ensures that a single virtual machine cannot consume all storage resources and starve other virtual machines of storage bandwidth.
Monitor end to end storage performance. As soon as virtual machines stored on a Scale-Out File Server are started, their performance is monitored. Performance details of all running virtual machines and the configuration of the Scale-Out File Server cluster can be viewed from a single location
Manage Storage I/O per workload business needs Storage QoS policies define performance minimums and maximums for virtual machines and ensures that they are met. This provides consistent performance to virtual machines, even in dense and overprovisioned environments. If policies cannot be met, alerts are available to track when VMs are out of policy or have invalid policies assigned.
This document outlines how your business can benefit from the new Storage QoS functionality. It assumes that you have a previous working knowledge of Windows Server, Windows Server Failover Clustering, Scale-Out File Server, Hyper-V, and Windows PowerShell.
Overview
This section describes the requirements for using Storage QoS, an overview of a software-defined solution using Storage QoS, and a list of Storage QoS related terminologies.
Storage QoS Requirements
Storage QoS supports two deployment scenarios:
Hyper-V using a Scale-Out File Server This scenario requires both of the following:
Storage cluster that is a Scale-Out File Server cluster
Compute cluster that has least one server with the Hyper-V role enabled
For Storage QoS, the Failover Cluster is required on Storage servers, but the compute servers are not required to be in a failover cluster. All servers (used for both Storage and Compute) must be running Windows Server 2016.
If you do not have a Scale-Out File Server cluster deployed for evaluation purposes, for step by step instructions to build one using either existing servers or virtual machines, see Windows Server 2012 R2 Storage: Step-by-step with Storage Spaces, SMB Scale-Out and Shared VHDX (Physical).
Hyper-V using Cluster Shared Volumes. This scenario requires both of the following:
Compute cluster with the Hyper-V role enabled
Hyper-V using Cluster Shared Volumes (CSV) for storage
Failover Cluster is required. All servers must be running the same version of Windows Server 2016.
Using Storage QoS in a software-defined storage solution
Storage Quality of Service is built into the Microsoft software-defined storage solution provided by Scale-Out File Server and Hyper-V. The Scale-Out File Server exposes file shares to the Hyper-V servers using the SMB3 protocol. A new Policy Manager has been added to the File Server cluster, which provides the central storage performance monitoring.
Figure 1: Using Storage QoS in a software-defined storage solution in Scale-Out File Server
As Hyper-V servers launch virtual machines, they are monitored by the Policy Manager. The Policy Manager communicates the Storage QoS policy and any limits or reservations back to the Hyper-V server, which controls the performance of the virtual machine as appropriate.
When there are changes to Storage QoS policies or to the performance demands by virtual machines, the Policy Manager notifies the Hyper-V servers to adjust their behavior. This feedback loop ensures that all virtual machines VHDs perform consistently according to the Storage QoS policies as defined.
Glossary
| Term | Description |
|---|---|
| Normalized IOPS | All of the storage usage is measured in "Normalized IOPS." This is a count of the storage input/output operations per second. Any IO that is 8KB or smaller is considered as one normalized IO. Any IO that is larger than 8KB is treated as multiple normalized IOs. For example, a 256KB request is treated as 32 normalized IOPS. Windows Server 2016 includes the ability to specify the size used to normalize IOs. On the storage cluster, the normalized size can be specified and take effect on the normalization calculations cluster wide. The default remains 8 KB. This setting affects all virtual machines. (The virtual machines created on local volumes are also affected.) |
| Flow | Each file handle opened by a Hyper-V server to a VHD or VHDX file is considered a "flow". If a virtual machine has two virtual hard disks attached, it will have 1 flow to the file server cluster per file. If a VHDX is shared with multiple virtual machines, it will have 1 flow per virtual machine. |
| InitiatorName | Name of the virtual machine that is reported to the Scale-Out File Server for each flow. |
| InitiatorID | An identifier matching the virtual machine ID. This can always be used to uniquely identify individual flows virtual machines even if the virtual machines have the same InitiatorName. |
| Policy | Storage QoS policies are stored in the cluster database, and have the following properties: PolicyId, MinimumIOPS, MaximumIOPS, ParentPolicy, and PolicyType. |
| PolicyId | Unique identifier for a policy. It is generated by default, but can be specified if desired. |
| MinimumIOPS | Minimum normalized IOPS that will be provided by a policy. Also known as "Reservation". |
| MaximumIOPS | Maximum normalized IOPS that will be limited by a policy. Also known as "Limit". |
| Aggregated | A policy type where the specified MinimumIOPS & MaximumIOPS and Bandwidth are shared among all flows assigned to the policy. All VHDs assigned the policy on that storage system have a single allocation of I/O bandwidth for them to all share. |
| Dedicated | A policy type where the specified Minimum & MaximumIOPS and Bandwidth are managed for individual VHD/VHDx. |
How to set up Storage QoS and monitor basic performance
This section describes how to enable the new Storage QoS feature and how to monitor storage performance without applying custom policies.
Set up Storage QoS on a Storage Cluster
This section discusses how to enable Storage QoS on either a new or an existing Failover Cluster and Scale-Out File Server that is running Windows Server 2016.
Set up Storage QoS on a new installation
If you have configured a new Failover Cluster and configured a Cluster Shared Volume(CSV) on Windows Server 2016, then the Storage QoS feature will be set up automatically.
Verify Storage QoS installation
After you have created a Failover Cluster and configured a CSV disk, Storage QoS Resource is displayed as a Cluster Core Resource and visible in both Failover Cluster Manager and Windows PowerShell. The intent is that the failover cluster system will manage this resource and you should not have to do any actions against this resource. We display it in both Failover Cluster Manager and PowerShell to be consistent with the other failover cluster system resources like the new Health Service.
Figure 2: Storage QoS Resource displayed as a Cluster Core Resource in Failover Cluster Manager
Use the following PowerShell cmdlet to view the status of Storage QoS Resource.
PS C:\> Get-ClusterResource -Name "Storage Qos Resource"
Name State OwnerGroup ResourceType
---- ----- ---------- ------------
Storage Qos Resource Online Cluster Group Storage QoS Policy Manager
Set up Storage QoS on a Compute Cluster
The Hyper-V role in Windows Server 2016 has built-in support for Storage QoS and is enabled by default.
Install Remote Administration Tools to manage Storage QoS policies from remote computers
You can manage Storage QoS policies and monitor flows from compute hosts using the Remote Server Administration Tools. These are available as optional features on all Windows Server 2016 installations, and can be downloaded separately for Windows 10 at the Microsoft Download Center website.
The RSAT-Clustering optional feature includes the Windows PowerShell module for remote management of Failover Clustering, including Storage QoS.
- Windows PowerShell: Add-WindowsFeature RSAT-Clustering
The RSAT-Hyper-V-Tools optional feature includes the Windows PowerShell module for remote management of Hyper-V.
- Windows PowerShell: Add-WindowsFeature RSAT-Hyper-V-Tools
Deploy virtual machines to run workloads for testing
You will need some virtual machines stored on the Scale-Out File Server with relevant workloads. For some tips in how to simulate load and do some stress testing, see the following page for a recommended tool (DiskSpd) and some example usage: DiskSpd, PowerShell and storage performance: measuring IOPS, throughput and latency for both local disks and SMB file shares.
The example scenarios shown in this guide include five virtual machines. BuildVM1, BuildVM2, BuildVM3 and BuildVM4 are running a desktop workload with low to moderate storage demands. TestVm1 is running an online transaction processing benchmark with high storage demand.
View current storage performance metrics
This section includes:
How to query flows using the
Get-StorageQosFlowcmdlet.How to view performance for a volume using the
Get-StorageQosVolumecmdlet.
Query flows using the Get-StorageQosFlow cmdlet
The Get-StorageQosFlow cmdlet shows all current flows initiated by Hyper-V servers. All data is collected by the Scale-Out File Server cluster, hence the cmdlet can be used on any node in the Scale-Out File Server cluster, or against a remote server using the -CimSession parameter.
The following sample command shows how to view all files opened by Hyper-V on server using Get-StorageQoSFlow.
PS C:\> Get-StorageQosFlow
InitiatorName InitiatorNodeNam StorageNodeName FilePath Status
e
------------- ---------------- --------------- -------- ------
plang-fs3.pla... C:\ClusterSt... Ok
plang-fs2.pla... C:\ClusterSt... Ok
plang-fs1.pla... C:\ClusterSt... Ok
plang-fs3.pla... C:\ClusterSt... Ok
plang-fs2.pla... C:\ClusterSt... Ok
plang-fs1.pla... C:\ClusterSt... Ok
TR20-VMM plang-z400.pl... plang-fs1.pla... C:\ClusterSt... Ok
BuildVM4 plang-c2.plan... plang-fs1.pla... C:\ClusterSt... Ok
WinOltp1 plang-c1.plan... plang-fs1.pla... C:\ClusterSt... Ok
BuildVM3 plang-c2.plan... plang-fs1.pla... C:\ClusterSt... Ok
BuildVM1 plang-c2.plan... plang-fs1.pla... C:\ClusterSt... Ok
TR20-VMM plang-z400.pl... plang-fs1.pla... C:\ClusterSt... Ok
BuildVM2 plang-c2.plan... plang-fs1.pla... C:\ClusterSt... Ok
TR20-VMM plang-z400.pl... plang-fs1.pla... C:\ClusterSt... Ok
plang-fs3.pla... C:\ClusterSt... Ok
plang-fs2.pla... C:\ClusterSt... Ok
BuildVM4 plang-c2.plan... plang-fs2.pla... C:\ClusterSt... Ok
WinOltp1 plang-c1.plan... plang-fs2.pla... C:\ClusterSt... Ok
BuildVM3 plang-c2.plan... plang-fs2.pla... C:\ClusterSt... Ok
WinOltp1 plang-c1.plan... plang-fs2.pla... C:\ClusterSt... Ok
plang-fs1.pla... C:\ClusterSt... Ok
The following sample command is formatted to show virtual machine name, Hyper-V host name, IOPS, and VHD file name, sorted by IOPS.
PS C:\> Get-StorageQosFlow | Sort-Object StorageNodeIOPS -Descending | ft InitiatorName, @{Expression={$_.InitiatorNodeName.Substring(0,$_.InitiatorNodeName.IndexOf('.'))};Label="InitiatorNodeName"}, StorageNodeIOPS, Status, @{Expression={$_.FilePath.Substring($_.FilePath.LastIndexOf('\')+1)};Label="File"} -AutoSize
InitiatorName InitiatorNodeName StorageNodeIOPS Status File
------------- ----------------- --------------- ------ ----
WinOltp1 plang-c1 3482 Ok IOMETER.VHDX
BuildVM2 plang-c2 544 Ok BUILDVM2.VHDX
BuildVM1 plang-c2 497 Ok BUILDVM1.VHDX
BuildVM4 plang-c2 316 Ok BUILDVM4.VHDX
BuildVM3 plang-c2 296 Ok BUILDVM3.VHDX
BuildVM4 plang-c2 195 Ok WIN8RTM_ENTERPRISE_VL_BU...
TR20-VMM plang-z400 156 Ok DATA1.VHDX
BuildVM3 plang-c2 81 Ok WIN8RTM_ENTERPRISE_VL_BU...
WinOltp1 plang-c1 65 Ok BOOT.VHDX
18 Ok DefaultFlow
12 Ok DefaultFlow
WinOltp1 plang-c1 4 Ok 9914.0.AMD64FRE.WINMAIN....
TR20-VMM plang-z400 4 Ok DATA2.VHDX
TR20-VMM plang-z400 3 Ok BOOT.VHDX
0 Ok DefaultFlow
0 Ok DefaultFlow
0 Ok DefaultFlow
0 Ok DefaultFlow
0 Ok DefaultFlow
0 Ok DefaultFlow
0 Ok DefaultFlow
The following sample command shows how to filter flows based on InitiatorName to easily find the storage performance and settings for a specific virtual machine.
PS C:\> Get-StorageQosFlow -InitiatorName BuildVm1 | Format-List
FilePath : C:\ClusterStorage\Volume2\SHARES\TWO\BUILDWORKLOAD\BUILDVM1.V
HDX
FlowId : ebfecb54-e47a-5a2d-8ec0-0940994ff21c
InitiatorId : ae4e3dd0-3bde-42ef-b035-9064309e6fec
InitiatorIOPS : 464
InitiatorLatency : 26.2684
InitiatorName : BuildVM1
InitiatorNodeName : plang-c2.plang.nttest.microsoft.com
Interval : 300000
Limit : 500
PolicyId : 1aaaaaa1-2bb2-3cc3-4dd4-5eeeeeeeeee5
Reservation : 500
Status : Ok
StorageNodeIOPS : 475
StorageNodeLatency : 7.9725
StorageNodeName : plang-fs1.plang.nttest.microsoft.com
TimeStamp : 2/12/2015 2:58:49 PM
VolumeId : 4d91fc3a-1a1e-4917-86f6-54853b2a6787
PSComputerName :
MaximumIops : 500
MinimumIops : 500
The data returned by the Get-StorageQosFlow cmdlet includes:
The Hyper-V hostname (InitiatorNodeName).
The virtual machine's name and its ID (InitiatorName and InitiatorId)
Recent average performance as observed by the Hyper-V host for the virtual disk (InitiatorIOPS, InitiatorLatency)
Recent average performance as observed by the Storage cluster for the virtual disk (StorageNodeIOPS, StorageNodeLatency)
Current policy being applied to the file, if any, and the resulting configuration (PolicyId, Reservation, Limit)
Status of the policy
Ok - No issues exist
InsufficientThroughput- A policy is applied, but the Minimum IOPS cannot be delivered. This can happen if the minimum for a VM, or all VMs together, are more than the storage volume can deliver.
UnknownPolicyId - A policy was assigned to the virtual machine on the Hyper-V host, but is missing from the file server. This policy should be removed from the virtual machine configuration, or a matching policy should be created on the file server cluster.
View performance for a volume using Get-StorageQosVolume
Storage performance metrics are also collected on a per-storage volume level, in addition to the per-flow performance metrics. This makes it easy to see the average total utilization in normalized IOPS, latency, and aggregate limits and reservations applied to a volume.
PS C:\> Get-StorageQosVolume | Format-List
Interval : 300000
IOPS : 0
Latency : 0
Limit : 0
Reservation : 0
Status : Ok
TimeStamp : 2/12/2015 2:59:38 PM
VolumeId : 434f561f-88ae-46c0-a152-8c6641561504
PSComputerName :
MaximumIops : 0
MinimumIops : 0
Interval : 300000
IOPS : 1097
Latency : 3.1524
Limit : 0
Reservation : 1568
Status : Ok
TimeStamp : 2/12/2015 2:59:38 PM
VolumeId : 4d91fc3a-1a1e-4917-86f6-54853b2a6787
PSComputerName :
MaximumIops : 0
MinimumIops : 1568
Interval : 300000
IOPS : 5354
Latency : 6.5084
Limit : 0
Reservation : 781
Status : Ok
TimeStamp : 2/12/2015 2:59:38 PM
VolumeId : 0d2fd367-8d74-4146-9934-306726913dda
PSComputerName :
MaximumIops : 0
MinimumIops : 781
How to create and monitor Storage QoS Policies
This section describes how to create Storage QoS policies, apply these policies to virtual machines, and monitor a storage cluster after policies are applied.
Create Storage QoS policies
Storage QoS policies are defined and managed in the Scale-Out File Server cluster. You can create as many policies as needed for flexible deployments (up to 10,000 per storage cluster).
Each VHD/VHDX file assigned to a virtual machine can be configured with a policy. Different files and virtual machines can use the same policy or they can each be configured with separate policies. If multiple VHD/VHDX files or multiple virtual machines are configured with the same policy, they will be aggregated together and will share the MinimumIOPS and MaximumIOPS fairly. If you use separate policies for multiple VHD/VHDX files or virtual machines, the minimum and maximums are tracked separately for each.
If you create multiple similar policies for different virtual machines and the virtual machines have equal storage demand, they will receive a similar share of IOPS. If one VM demands more and the other less, then IOPS will follow that demand.
Types of Storage QoS Policies
There are two types of policies: Aggregated (previously known as SingleInstance) and Dedicated (previously known as MultiInstance). Aggregated policies apply maximums and minimum for the combined set of VHD/VHDX files and virtual machines where they apply. In effect, they share a specified set of IOPS and bandwidth. Dedicated policies apply the minimum and maximum values for each VHD/VHDx, separately. This makes it easy to create a single policy that applies similar limits to multiple VHD/VHDx files.
For instance, if you create an Aggregated policy with a minimum of 300 IOPS and a maximum of 500 IOPS. If you apply this policy to 5 different VHD/VHDx files, you are making sure that the 5 VHD/VHDx files combined will be guaranteed at least 300 IOPS (if there is demand and the storage system can provide that performance) and no more than 500 IOPS. If the VHD/VHDx files have similar high demand for IOPS and the storage system can keep up, each VHD/VHDx file will get about 100 IOPS.
However, if you create a Dedicated policy with similar limits and apply it to VHD/VHDx files on 5 different virtual machines, each virtual machine will get at least 300 IOPS and no more than 500 IOPS. If the virtual machines have similar high demand for IOPS and the storage system can keep up, each virtual machine will get about 500 IOPS. If one of the virtual machines has multiple VHD/VHDx files with the same MulitInstance policy configured, they will share the limit so that the total IO from the VM from files with that policy will not exceed the limits.
Hence, if you have a group of VHD/VHDx files that you want to exhibit the same performance characteristics and you don't want the trouble of creating multiple, similar policies, you can use a single Dedicated policy and apply to the files of each virtual machine.
Keep the number of VHD/VHDx files assigned to a single Aggregated policy to 20 or less. This policy type was meant to do aggregation with a few VMs on a cluster.
Create and apply a Dedicated policy
First, use the New-StorageQosPolicy cmdlet to create a policy on the Scale-Out File Server as shown in the following example:
$desktopVmPolicy = New-StorageQosPolicy -Name Desktop -PolicyType Dedicated -MinimumIops 100 -MaximumIops 200
Next, apply it to the appropriate virtual machines' hard disk drives on the Hyper-V server. Note the PolicyId from the previous step or store it in a variable in your scripts.
On the Scale-Out File Server, using PowerShell, create a Storage QoS policy and get its Policy ID as shown in the following example:
PS C:\> $desktopVmPolicy = New-StorageQosPolicy -Name Desktop -PolicyType Dedicated -MinimumIops 100 -MaximumIops 200
C:\> $desktopVmPolicy.PolicyId
Guid
----
1aaaaaa1-2bb2-3cc3-4dd4-5eeeeeeeeee5
On the Hyper-V server, using PowerShell, set the Storage QoS Policy using the Policy ID as shown in the following example:
Get-VM -Name Build* | Get-VMHardDiskDrive | Set-VMHardDiskDrive -QoSPolicyID 1aaaaaa1-2bb2-3cc3-4dd4-5eeeeeeeeee5
Confirm that the policies are applied
Use Get-StorageQosFlow PowerShell cmdlet to confirm that the MinimumIOPS and MaximumIOPS have been applied to the appropriate flows as shown in the following example.
PS C:\> Get-StorageQoSflow | Sort-Object InitiatorName |
ft InitiatorName, Status, MinimumIOPS, MaximumIOPS, StorageNodeIOPS, Status, @{Expression={$_.FilePath.Substring($_.FilePath.LastIndexOf('\')+1)};Label="File"} -AutoSize
InitiatorName Status MinimumIops MaximumIops StorageNodeIOPS Status File
------------- ------ ----------- ----------- --------------- ------ ----
BuildVM1 Ok 100 200 250 Ok BUILDVM1.VHDX
BuildVM2 Ok 100 200 251 Ok BUILDVM2.VHDX
BuildVM3 Ok 100 200 252 Ok BUILDVM3.VHDX
BuildVM4 Ok 100 200 233 Ok BUILDVM4.VHDX
TR20-VMM Ok 33 666 1 Ok DATA2.VHDX
TR20-VMM Ok 33 666 5 Ok DATA1.VHDX
TR20-VMM Ok 33 666 4 Ok BOOT.VHDX
WinOltp1 Ok 0 0 0 Ok 9914.0.AMD6...
WinOltp1 Ok 0 0 5166 Ok IOMETER.VHDX
WinOltp1 Ok 0 0 0 Ok BOOT.VHDX
On the Hyper-V server, you can also use the provided script Get-VMHardDiskDrivePolicy.ps1 to see what policy is applied to a virtual hard disk drive.
PS C:\> Get-VM -Name BuildVM1 | Get-VMHardDiskDrive | Format-List
Path : \\plang-fs.plang.nttest.microsoft.com\two\BuildWorkload
\BuildVM1.vhdx
DiskNumber :
MaximumIOPS : 0
MinimumIOPS : 0
QoSPolicyID : 1aaaaaa1-2bb2-3cc3-4dd4-5eeeeeeeeee5
SupportPersistentReservations : False
ControllerLocation : 0
ControllerNumber : 0
ControllerType : IDE
PoolName : Primordial
Name : Hard Drive
Id : Microsoft:AE4E3DD0-3BDE-42EF-B035-9064309E6FEC\83F8638B
-8DCA-4152-9EDA-2CA8B33039B4\0\0\D
VMId : ae4e3dd0-3bde-42ef-b035-9064309e6fec
VMName : BuildVM1
VMSnapshotId : 00000000-0000-0000-0000-000000000000
VMSnapshotName :
ComputerName : PLANG-C2
IsDeleted : False
Query for Storage QoS Policies
Get-StorageQosPolicy lists all configured policies and their status on a Scale-Out File Server.
PS C:\> Get-StorageQosPolicy
Name MinimumIops MaximumIops Status
---- ----------- ----------- ------
Default 0 0 Ok
Limit500 0 500 Ok
SilverVm 500 500 Ok
Desktop 100 200 Ok
Limit500 0 0 Ok
VMM 100 2000 Ok
Vdi 1 100 Ok
Status can change over time based on how the system is performing.
Ok - All flows using that policy are receiving their requested MinimumIOPS.
InsufficientThroughput - One or more of the flows using this policy are not receiving the Minimum IOPS
You can also pipe a policy to Get-StorageQosPolicy to get the status of all flows configured to use the policy as follows:
PS C:\> Get-StorageQosPolicy -Name Desktop | Get-StorageQosFlow | ft InitiatorName, *IOPS, Status, FilePath -AutoSize
InitiatorName MaximumIops MinimumIops InitiatorIOPS StorageNodeIOPS Status FilePat
h
------------- ----------- ----------- ------------- --------------- ------ -------
BuildVM4 100 50 187 17 Ok C:\C...
BuildVM3 100 50 194 25 Ok C:\C...
BuildVM1 200 100 195 196 Ok C:\C...
BuildVM2 200 100 193 192 Ok C:\C...
BuildVM4 200 100 187 169 Ok C:\C...
BuildVM3 200 100 194 169 Ok C:\C...
Create an Aggregated Policy
Aggregated policies should be used if you want multiple virtual hard disks to share a single pool of IOPS and bandwidth. For example, if you apply the same Aggregated policy to hard disks from two virtual machines, the minimum will be split between them according to demand. Both disks will be guaranteed a combined minimum, and together they will not exceed the specified maximum IOPS or bandwidth.
The same approach could also be used to provide a single allocation to all VHD/VHDx files for the virtual machines comprising a service or belonging to a tenant in a multihosted environment.
There is no difference in the process to create Dedicated and Aggregated policies other than the PolicyType that is specified.
The following example shows how to create an Aggregated Storage QoS Policy and get its policyID on a Scale-Out File Server:
PS C:\> $highPerf = New-StorageQosPolicy -Name SqlWorkload -MinimumIops 1000 -MaximumIops 5000 -PolicyType Aggregated
[plang-fs]: PS C:\Users\plang\Documents> $highPerf.PolicyId
Guid
----
1aaaaaa1-2bb2-3cc3-4dd4-5eeeeeeeeee5
The following example shows how to apply the Storage QoS Policy on Hyper-V server using the policyID obtained in the preceding example:
PS C:\> Get-VM -Name WinOltp1 | Get-VMHardDiskDrive | Set-VMHardDiskDrive -QoSPolicyID 1aaaaaa1-2bb2-3cc3-4dd4-5eeeeeeeeee5
The following example shows how to view effects of the Storage QoS policy from file server:
PS C:\> Get-StorageQosFlow -InitiatorName WinOltp1 | format-list InitiatorName, PolicyId, MinimumIOPS, MaximumIOPS, StorageNodeIOPS, FilePath
InitiatorName : WinOltp1
PolicyId : 1aaaaaa1-2bb2-3cc3-4dd4-5eeeeeeeeee5
MinimumIops : 250
MaximumIops : 1250
StorageNodeIOPS : 0
FilePath : C:\ClusterStorage\Volume2\SHARES\TWO\BASEVHD\9914.0.AMD64FRE.WIN
MAIN.141218-1718_SERVER_SERVERDATACENTER_EN-US.VHDX
InitiatorName : WinOltp1
PolicyId : 2bbbbbb2-3cc3-4dd4-5ee5-6ffffffffff6
MinimumIops : 250
MaximumIops : 1250
StorageNodeIOPS : 0
FilePath : C:\ClusterStorage\Volume3\SHARES\THREE\WINOLTP1\BOOT.VHDX
InitiatorName : WinOltp1
PolicyId : 3cccccc3-4dd4-5ee5-6ff6-7aaaaaaaaaa7
MinimumIops : 1000
MaximumIops : 5000
StorageNodeIOPS : 4550
FilePath : C:\ClusterStorage\Volume3\SHARES\THREE\WINOLTP1\IOMETER.VHDX
PS C:\> Get-StorageQosFlow -InitiatorName WinOltp1 | for
mat-list InitiatorName, PolicyId, MinimumIOPS, MaximumIOPS, StorageNodeIOPS, FilePath
InitiatorName : WinOltp1
PolicyId : 4dddddd4-5ee5-6ff6-7aa7-8bbbbbbbbbb8
MinimumIops : 250
MaximumIops : 1250
StorageNodeIOPS : 0
FilePath : C:\ClusterStorage\Volume2\SHARES\TWO\BASEVHD\9914.0.AMD64FRE.WIN
MAIN.141218-1718_SERVER_SERVERDATACENTER_EN-US.VHDX
InitiatorName : WinOltp1
PolicyId : 5eeeeee5-6ff6-7aa7-8bb8-9cccccccccc9
MinimumIops : 250
MaximumIops : 1250
StorageNodeIOPS : 0
FilePath : C:\ClusterStorage\Volume3\SHARES\THREE\WINOLTP1\BOOT.VHDX
InitiatorName : WinOltp1
PolicyId : 6ffffff6-7aa7-8bb8-9cc9-0dddddddddd0
MinimumIops : 1000
MaximumIops : 5000
StorageNodeIOPS : 4550
FilePath : C:\ClusterStorage\Volume3\SHARES\THREE\WINOLTP1\IOMETER.VHDX
Each virtual hard disk will have the MinimumIOPS and MaximumIOPS and MaximumIobandwidth value adjusted based on its load. This ensures that the total amount of bandwidth used for the group of disks stays within the range defined by policy. In the example above, the first two disks are idle, and the third one is allowed to use up to the maximum IOPS. If the first two disks start issuing IO again, then the maximum IOPS of the third disk will be lowered automatically.
Modify an existing policy
The properties of Name, MinimumIOPS, MaximumIOPS, and MaximumIoBandwidthcan be changed after a policy is created. However, the Policy Type (Aggregated/Dedicated) cannot be changed once the policy is created.
The following Windows PowerShell cmdlet shows how to change the MaximumIOPS property for an existing policy:
[DBG]: PS C:\demoscripts>> Get-StorageQosPolicy -Name SqlWorkload | Set-StorageQosPolicy -MaximumIops 6000
The following cmdlet verifies the change:
PS C:\> Get-StorageQosPolicy -Name SqlWorkload
Name MinimumIops MaximumIops Status
---- ----------- ----------- ------
SqlWorkload 1000 6000 Ok
[plang-fs1]: PS C:\Users\plang\Documents> Get-StorageQosPolicy -Name SqlWorkload | Get-Storag
eQosFlow | Format-Table InitiatorName, PolicyId, MaximumIops, MinimumIops, StorageNodeIops -A
utoSize
InitiatorName PolicyId MaximumIops MinimumIops StorageNodeIops
------------- -------- ----------- ----------- ---------------
WinOltp1 1aaaaaa1-2bb2-3cc3-4dd4-5eeeeeeeeee5 1500 250 0
WinOltp1 2bbbbbb2-3cc3-4dd4-5ee5-6ffffffffff6 1500 250 0
WinOltp1 3cccccc3-4dd4-5ee5-6ff6-7aaaaaaaaaa7 6000 1000 4507
How to identify and address common issues
This section describes how to find virtual machines with invalid Storage QoS policies, how to recreate a matching policy, how to remove a policy from a virtual machine, and how to identify virtual machines that do not meet the Storage QoS policy requirements.
Identify virtual machines with invalid policies
If a policy is deleted from the file server before it's removed from a virtual machine, the virtual machine will keep running as if no policy were applied.
PS C:\> Get-StorageQosPolicy -Name SqlWorkload | Remove-StorageQosPolicy
Confirm
Are you sure you want to perform this action?
Performing the operation "DeletePolicy" on target "MSFT_StorageQoSPolicy (PolicyId =
"1aaaaaa1-2bb2-3cc3-4dd4-5eeeeeeeeee5")".
[Y] Yes [A] Yes to All [N] No [L] No to All [?] Help (default is "Y"):
The status for the flows will now show "UnknownPolicyId"
PS C:\> Get-StorageQoSflow | Sort-Object InitiatorName | ft InitiatorName, Status, MinimumIOPS, MaximumIOPS, StorageNodeIOPS, Status, @{Expression={$_.FilePath.Substring($_.FilePath.LastIndexOf('\')+1)};Label="File"} -AutoSize
InitiatorName Status MinimumIops MaximumIops StorageNodeIOPS Status File
------------- ------ ----------- ----------- --------------- ------ ----
Ok 0 0 0 Ok Def...
Ok 0 0 10 Ok Def...
Ok 0 0 13 Ok Def...
Ok 0 0 0 Ok Def...
Ok 0 0 0 Ok Def...
Ok 0 0 0 Ok Def...
Ok 0 0 0 Ok Def...
Ok 0 0 0 Ok Def...
Ok 0 0 0 Ok Def...
BuildVM1 Ok 100 200 193 Ok BUI...
BuildVM2 Ok 100 200 196 Ok BUI...
BuildVM3 Ok 50 64 17 Ok WIN...
BuildVM3 Ok 50 136 179 Ok BUI...
BuildVM4 Ok 50 100 23 Ok WIN...
BuildVM4 Ok 100 200 173 Ok BUI...
TR20-VMM Ok 33 666 2 Ok DAT...
TR20-VMM Ok 25 975 3 Ok DAT...
TR20-VMM Ok 75 1025 12 Ok BOO...
WinOltp1 UnknownPolicyId 0 0 0 UnknownPolicyId 991...
WinOltp1 UnknownPolicyId 0 0 4926 UnknownPolicyId IOM...
WinOltp1 UnknownPolicyId 0 0 0 UnknownPolicyId BOO...
Recreate a matching Storage QoS policy
If a policy was unintentionally removed, you can create a new one using the old PolicyId. First, get the needed PolicyId
PS C:\> Get-StorageQosFlow -Status UnknownPolicyId | ft InitiatorName, PolicyId -AutoSize
InitiatorName PolicyId
------------- --------
WinOltp1 1aaaaaa1-2bb2-3cc3-4dd4-5eeeeeeeeee5
WinOltp1 2bbbbbb2-3cc3-4dd4-5ee5-6ffffffffff6
WinOltp1 3cccccc3-4dd4-5ee5-6ff6-7aaaaaaaaaa7
Next, create a new policy using that PolicyId
PS C:\> New-StorageQosPolicy -PolicyId 1aaaaaa1-2bb2-3cc3-4dd4-5eeeeeeeeee5 -PolicyType Aggregated -Name RestoredPolicy -MinimumIops 100 -MaximumIops 2000
Name MinimumIops MaximumIops Status
---- ----------- ----------- ------
RestoredPolicy 100 2000 Ok
Finally, verify that it was applied.
PS C:\> Get-StorageQoSflow | Sort-Object InitiatorName | ft InitiatorName, Status, MinimumIOPS, MaximumIOPS, StorageNodeIOPS, Status, @{Expression={$_.FilePath.Substring($_.FilePath.LastIndexOf('\')+1)};Label="File"} -AutoSize
InitiatorName Status MinimumIops MaximumIops StorageNodeIOPS Status File
------------- ------ ----------- ----------- --------------- ------ ----
Ok 0 0 0 Ok DefaultFlow
Ok 0 0 8 Ok DefaultFlow
Ok 0 0 9 Ok DefaultFlow
Ok 0 0 0 Ok DefaultFlow
Ok 0 0 0 Ok DefaultFlow
Ok 0 0 0 Ok DefaultFlow
Ok 0 0 0 Ok DefaultFlow
Ok 0 0 0 Ok DefaultFlow
Ok 0 0 0 Ok DefaultFlow
BuildVM1 Ok 100 200 192 Ok BUILDVM1.VHDX
BuildVM2 Ok 100 200 193 Ok BUILDVM2.VHDX
BuildVM3 Ok 50 100 24 Ok WIN8RTM_ENTERPRISE_VL...
BuildVM3 Ok 100 200 166 Ok BUILDVM3.VHDX
BuildVM4 Ok 50 100 12 Ok WIN8RTM_ENTERPRISE_VL...
BuildVM4 Ok 100 200 178 Ok BUILDVM4.VHDX
TR20-VMM Ok 33 666 2 Ok DATA2.VHDX
TR20-VMM Ok 33 666 2 Ok DATA1.VHDX
TR20-VMM Ok 33 666 10 Ok BOOT.VHDX
WinOltp1 Ok 25 500 0 Ok 9914.0.AMD64FRE.WINMA...
Remove Storage QoS Policies
If the policy was removed intentionally, or if a VM was imported with a policy that you don't need, it can be removed.
PS C:\> Get-VM -Name WinOltp1 | Get-VMHardDiskDrive | Set-VMHardDiskDrive -QoSPolicyID $null
Once the PolicyId is removed from the virtual hard disk settings, the status will be "Ok" and no minimum or maximum will be applied.
PS C:\> Get-StorageQoSflow | Sort-Object InitiatorName | ft InitiatorName, MinimumIOPS, MaximumIOPS, StorageNodeIOPS, Status, @{Expression={$_.FilePath.Substring($_.FilePath.LastIndexOf('\')+1)};Label="File"} -AutoSize
InitiatorName MinimumIops MaximumIops StorageNodeIOPS Status File
------------- ----------- ----------- --------------- ------ ----
0 0 0 Ok DefaultFlow
0 0 16 Ok DefaultFlow
0 0 12 Ok DefaultFlow
0 0 0 Ok DefaultFlow
0 0 0 Ok DefaultFlow
0 0 0 Ok DefaultFlow
0 0 0 Ok DefaultFlow
0 0 0 Ok DefaultFlow
0 0 0 Ok DefaultFlow
BuildVM1 100 200 197 Ok BUILDVM1.VHDX
BuildVM2 100 200 192 Ok BUILDVM2.VHDX
BuildVM3 9 9 23 Ok WIN8RTM_ENTERPRISE_VL_BUILDW...
BuildVM3 91 191 171 Ok BUILDVM3.VHDX
BuildVM4 8 8 18 Ok WIN8RTM_ENTERPRISE_VL_BUILDW...
BuildVM4 92 192 163 Ok BUILDVM4.VHDX
TR20-VMM 33 666 2 Ok DATA2.VHDX
TR20-VMM 33 666 1 Ok DATA1.VHDX
TR20-VMM 33 666 5 Ok BOOT.VHDX
WinOltp1 0 0 0 Ok 9914.0.AMD64FRE.WINMAIN.1412...
WinOltp1 0 0 1811 Ok IOMETER.VHDX
WinOltp1 0 0 0 Ok BOOT.VHDX
Find virtual machines that are not meeting Storage QoS Policies
The InsufficientThroughput status is assigned to any flows that:
Have a minimum defined IOPS set by policy; and
Are initiating IO at a rate meeting or exceeding the minimum; and
Are not achieving minimum IOP rate
PS C:\> Get-StorageQoSflow | Sort-Object InitiatorName | ft InitiatorName, MinimumIOPS, MaximumIOPS, StorageNodeIOPS, Status, @{Expression={$_.FilePath.Substring($_.FilePath.LastIndexOf('\')+1)};Label="File"} -AutoSize
InitiatorName MinimumIops MaximumIops StorageNodeIOPS Status File
------------- ----------- ----------- --------------- ------ ----
0 0 0 Ok DefaultFlow
0 0 0 Ok DefaultFlow
0 0 15 Ok DefaultFlow
0 0 0 Ok DefaultFlow
0 0 0 Ok DefaultFlow
0 0 0 Ok DefaultFlow
0 0 0 Ok DefaultFlow
0 0 0 Ok DefaultFlow
0 0 0 Ok DefaultFlow
BuildVM3 50 100 20 Ok WIN8RTM_ENTE...
BuildVM3 100 200 174 Ok BUILDVM3.VHDX
BuildVM4 50 100 11 Ok WIN8RTM_ENTE...
BuildVM4 100 200 188 Ok BUILDVM4.VHDX
TR20-VMM 33 666 3 Ok DATA1.VHDX
TR20-VMM 78 1032 180 Ok BOOT.VHDX
TR20-VMM 22 968 4 Ok DATA2.VHDX
WinOltp1 3750 5000 0 Ok 9914.0.AMD64...
WinOltp1 15000 20000 11679 InsufficientThroughput IOMETER.VHDX
WinOltp1 3750 5000 0 Ok BOOT.VHDX
You can determine flows for any status, including InsufficientThroughput as shown in the following example:
PS C:\> Get-StorageQosFlow -Status InsufficientThroughput | fl
FilePath : C:\ClusterStorage\Volume3\SHARES\THREE\WINOLTP1\IOMETER.VHDX
FlowId : 1ca356ff-fd33-5b5d-b60a-2c8659dc803e
InitiatorId : 2ceabcef-2eba-4f1b-9e66-10f960b50bbf
InitiatorIOPS : 12168
InitiatorLatency : 22.983
InitiatorName : WinOltp1
InitiatorNodeName : plang-c1.plang.nttest.microsoft.com
Interval : 300000
Limit : 20000
PolicyId : 1aaaaaa1-2bb2-3cc3-4dd4-5eeeeeeeeee5
Reservation : 15000
Status : InsufficientThroughput
StorageNodeIOPS : 12181
StorageNodeLatency : 22.0514
StorageNodeName : plang-fs2.plang.nttest.microsoft.com
TimeStamp : 2/13/2015 12:07:30 PM
VolumeId : 0d2fd367-8d74-4146-9934-306726913dda
PSComputerName :
MaximumIops : 20000
MinimumIops : 15000
Monitor Health using Storage QoS
The new Health Service simplifies the monitoring of the Storage Cluster, providing a single place to check for any actionable events in any of the nodes. This section describes how to monitor the health of your storage cluster using the debug-storagesubsystem cmdlet.
View Storage Status with Debug-StorageSubSystem
Clustered Storage Spaces also provide information on the health of the storage cluster in a single location. This can help administrators quickly identify current problems in storage deployments and monitor as issues arrive or are dismissed.
VM with invalid policy
VMs with invalid policies are also reported through the storage subsystem health monitoring. Here is an example from the same state as described in Finding VMs with invalid policies section of this document.
C:\> Get-StorageSubSystem -FriendlyName Clustered* | Debug-StorageSubSystem
EventTime :
FaultId : 0d16d034-9f15-4920-a305-f9852abf47c3
FaultingObject :
FaultingObjectDescription : Storage QoS Policy 5d1bf221-c8f0-4368-abcf-aa139e8a7c72
FaultingObjectLocation :
FaultType : Storage QoS policy used by consumer does not exist.
PerceivedSeverity : Minor
Reason : One or more storage consumers (usually Virtual Machines) are
using a non-existent policy with id
5d1bf221-c8f0-4368-abcf-aa139e8a7c72. Consumer details:
Flow ID: 1ca356ff-fd33-5b5d-b60a-2c8659dc803e
Initiator ID: 2ceabcef-2eba-4f1b-9e66-10f960b50bbf
Initiator Name: WinOltp1
Initiator Node: plang-c1.plang.nttest.microsoft.com
File Path:
C:\ClusterStorage\Volume3\SHARES\THREE\WINOLTP1\IOMETER.VHDX
RecommendedActions : {Reconfigure the storage consumers (usually Virtual Machines)
to use a valid policy., Recreate any missing Storage QoS
policies.}
PSComputerName :
Lost redundancy for a storage spaces virtual disk
In this example, a Clustered Storage Space has a virtual disk created as a three-way mirror. A failed disk was removed from the system, but a replacement disk was not added. The storage subsystem is reporting a loss of redundancy with HealthStatus Warning, but OperationalStatus "OK because the volume is still online.
PS C:\> Get-StorageSubSystem -FriendlyName Clustered*
FriendlyName HealthStatus OperationalStatus
------------ ------------ -----------------
Clustered Windows Storage o... Warning OK
[plang-fs1]: PS C:\Users\plang\Documents> Get-StorageSubSystem -FriendlyName Clustered* | Deb
ug-StorageSubSystem
EventTime :
FaultId : dfb4b672-22a6-4229-b2ed-c29d7485bede
FaultingObject :
FaultingObjectDescription : Virtual disk 'Two'
FaultingObjectLocation :
FaultType : VirtualDiskDegradedFaultType
PerceivedSeverity : Minor
Reason : Virtual disk 'Two' does not have enough redundancy remaining to
successfully repair or regenerate its data.
RecommendedActions : {Rebalance the pool, replace failed physical disks, or add new
physical disks to the storage pool, then repair the virtual
disk.}
PSComputerName :
Sample script for continuous monitoring of Storage QoS
This section includes a sample script showing how common failures can be monitored using WMI script. It's designed as a starting part for developers to retrieve health events in real time.
Example script:
param($cimSession)
# Register and display events
Register-CimIndicationEvent -Namespace root\microsoft\windows\storage -ClassName msft_storagefaultevent -CimSession $cimSession
while ($true)
{
$e = (Wait-Event)
$e.SourceEventArgs.NewEvent
Remove-Event $e.SourceIdentifier
}
Frequently Asked Questions
How do I retain a Storage QoS policy being enforced for my virtual machine if I move its VHD/VHDx files to another storage cluster
The setting on the VHD/VHDx file that specifies the policy is the GUID of a policy ID. When a policy is created, the GUID can be specified using the PolicyID parameter. If that parameter is not specified, a random GUID is created. Therefore, you can get the PolicyID on the storage cluster where the VMs currently store their VHD/VHDx files and create an identical policy on the destination storage cluster and then specify that it be created with the same GUID. When the VMs files are moved to the new storage clusters, the policy with the same GUID will be in effect.
System Center Virtual Machine Manager can be used to apply policies across multiple storage clusters, which makes this scenario much easier.
If I change the Storage QoS Policy, why don't I see it take effect immediately when I run Get-StorageQoSFlow
If you have a flow that is hitting a maximum of a policy and you change the policy to either make it higher or lower, and then you immediately determine the latency/IOPS/BandWidth of the flows using the PowerShell cmdlets, it will take up to 5 minutes to see the full effects of the policy change on the flows. The new limits will be in effect within a few seconds, but the Get-StorgeQoSFlow PowerShell cmdlet uses an average of each counter using a 5 minute sliding window. Otherwise, if it was showing a current value and you ran the PowerShell cmdlet multiple times in a row, you might see vastly different values because values for IOPS and latencies can fluctuate significantly from one second to another.
What new functionality was added in Windows Server 2016
In Windows Server 2016 the Storage QoS Policy type names were renamed. The Multi-instance policy type is renamed as Dedicated and Single-instance was renamed as Aggregated. The management behavior of Dedicated policies is also modified - VHD/VHDX files within the same virtual machine that have the same Dedicated policy applied to them will not share I/O allocations.
There are two new Storage QoS features Windows Server 2016:
Maximum Bandwidth
Storage QoS in Windows Server 2016 introduces the ability to specify the maximum bandwidth that the flows assigned to the policy can consume. The parameter when specifying it in the StorageQosPolicy cmdlets is MaximumIOBandwidth and the output is expressed in bytes per second. If both MaximimIops and MaximumIOBandwidth are set in a policy, they will both be in effect and the first one to be reached by the flow(s) will limit the I/O of the flows.
IOPS normalization is configurable
Storage QoSin uses normalization of IOPS. The default is to use a normalization size of 8K. Storage QoS in Windows Server 2016 introduces the ability to specify a different normalization size for the storage cluster. This normalization size affects all flows on the storage cluster and takes effect immediately (within a few seconds) once it is changed. The minimum is 1KB and the maximum is 4GB (recommend not setting more than 4MB since it's unusual to have more than 4MB IOs).
Something to consider is that the same IO pattern/throughput shows up with different IOPS numbers in the Storage QoS output when you change the IOPS normalization due to the change in normalization calculation. If you are comparing IOPS between storage clusters, you might also want to verify what normalization value each is using since that will affect the normalized IOPS reported.
Example 1: Creating a new policy and viewing the maximum bandwidth on the storage cluster
In PowerShell, you can specify the units that a number is expressed in. In the following example, 10MB is used as the maximum bandwidth value. Storage QoS will convert this and save it as bytes per second Hence, 10MB is converted into 10485760 bytes per second.
PS C:\Windows\system32> New-StorageQosPolicy -Name HR_VMs -MaximumIops 1000 -MinimumIops 20 -MaximumIOBandwidth 10MB
Name MinimumIops MaximumIops MaximumIOBandwidth Status
---- ----------- ----------- ------------------ ------
HR_VMs 20 1000 10485760 Ok
PS C:\Windows\system32> Get-StorageQosPolicy
Name MinimumIops MaximumIops MaximumIOBandwidth Status
---- ----------- ----------- ------------------ ------
Default 0 0 0 Ok
HR_VMs 20 1000 10485760 Ok
PS C:\Windows\system32> Get-StorageQoSFlow | fL InitiatorName,FilePath,InitiatorIOPS,InitiatorLatency,InitiatorBandwidth
InitiatorName : testsQoS
FilePath : C:\ClusterStorage\Volume2\TESTSQOS\VIRTUAL HARD DISKS\TESTSQOS.VHDX
InitiatorIOPS : 5
InitiatorLatency : 1.5455
InitiatorBandwidth : 37888
Example 2: Get IOPS normalization settings and specify a new value
The following example demonstrates how to get the storage clusters IOPS normalization settings (default of 8KB), then set it to 32KB, and then show it again. Note, in this example, specify "32KB", since PowerShell allows specifying the unit instead of requiring the conversion to bytes. The output does show the value in bytes per second. This setting affects all virtual machines. (The virtual machines created on local volumes are also affected.)
PS C:\Windows\system32> Get-StorageQosPolicyStore
IOPSNormalizationSize
---------------------
8192
PS C:\Windows\system32> Set-StorageQosPolicyStore -IOPSNormalizationSize 32KB
PS C:\Windows\system32> Get-StorageQosPolicyStore
IOPSNormalizationSize
---------------------
32768