Working with NVMe drives
Applies to:
- Windows 10
- Windows Server 2016
Learn how to work with high-speed NVMe devices from your Windows application. Device access is enabled via StorNVMe.sys, the in-box driver first introduced in Windows Server 2012 R2 and Windows 8.1. It's also available to Windows 7 devices through a KB hot fix. In Windows 10, several new features were introduced, including a pass-through mechanism for vendor-specific NVMe commands and updates to existing IOCTLs.
This topic provides an overview of general-use APIs that you can use to access NVMe drives in Windows 10. It also describes:
- How to send a vendor-specific NVMe command with pass-through
- How to send an Identify, Get Features, or Get Log Pages command to the NVMe drive
- How to obtain temperature information from an NVMe drive
- How to perform behavior changing commands, such as setting temperature thresholds
APIs for working with NVMe drives
You can use the following general-use APIs to access NVMe drives in Windows 10. These APIs can be found in winioctl.h for user mode applications, and ntddstor.h for kernel mode drivers. For more information about header files, see Header files.
IOCTL_STORAGE_PROTOCOL_COMMAND : Use this IOCTL with the STORAGE_PROTOCOL_COMMAND structure to issue NVMe commands. This IOCTL enables NVMe pass-through and supports the Command Effects log in NVMe. You can use it with vendor-specific commands. For more info, see Pass-through mechanism.
STORAGE_PROTOCOL_COMMAND : This input-buffer structure includes a ReturnStatus field that can be used report the following status values.
- STORAGE_PROTOCOL_STATUS_PENDING
- STORAGE_PROTOCOL_STATUS_SUCCESS
- STORAGE_PROTOCOL_STATUS_ERROR
- STORAGE_PROTOCOL_STATUS_INVALID_REQUEST
- STORAGE_PROTOCOL_STATUS_NO_DEVICE
- STORAGE_PROTOCOL_STATUS_BUSY
- STORAGE_PROTOCOL_STATUS_DATA_OVERRUN
- STORAGE_PROTOCOL_STATUS_INSUFFICIENT_RESOURCES
- STORAGE_PROTOCOL_STATUS_NOT_SUPPORTED
IOCTL_STORAGE_QUERY_PROPERTY : Use this IOCTL with the STORAGE_PROPERTY_QUERY structure to retrieve device information. For more info, see Protocol-specific queries and Temperature queries.
STORAGE_PROPERTY_QUERY : This structure includes the PropertyId and AdditionalParameters fields to specify the data to be queried. In the PropertyId filed, use the STORAGE_PROPERTY_ID enumeration to specify the type of data. Use the AdditionalParameters field to specify more details, depending on the type of data. For protocol-specific data, use the STORAGE_PROTOCOL_SPECIFIC_DATA structure in the AdditionalParameters field. For temperature data, use the STORAGE_TEMPERATURE_INFO structure in the AdditionalParameters field.
STORAGE_PROPERTY_ID : This enumeration includes new values that allow IOCTL_STORAGE_QUERY_PROPERTY to retrieve protocol-specific and temperature information.
- StorageAdapterProtocolSpecificProperty: If ProtocolType = ProtocolTypeNvme and DataType = NVMeDataTypeLogPage, callers should request 512 byte chunks of data.
- StorageDeviceProtocolSpecificProperty
Use one of these protocol-specific property IDs in combination with STORAGE_PROTOCOL_SPECIFIC_DATA to retrieve protocol-specific data in the STORAGE_PROTOCOL_DATA_DESCRIPTOR structure.
- StorageAdapterTemperatureProperty
- StorageDeviceTemperatureProperty
Use one of these temperature property IDs to retrieve temperature data in the STORAGE_TEMPERATURE_DATA_DESCRIPTOR structure.
STORAGE_PROTOCOL_SPECIFIC_DATA : Retrieve NVMe-specific data when this structure is used for the AdditionalParameters field of STORAGE_PROPERTY_QUERY and a STORAGE_PROTOCOL_NVME_DATA_TYPE enum value is specified. Use one of the following STORAGE_PROTOCOL_NVME_DATA_TYPE values in the DataType field of the STORAGE_PROTOCOL_SPECIFIC_DATA structure:
- Use NVMeDataTypeIdentify to get Identify Controller data or Identify Namespace data.
- Use NVMeDataTypeLogPage to get log pages (including SMART/health data).
- Use NVMeDataTypeFeature to get features of the NVMe drive.
STORAGE_TEMPERATURE_INFO : This structure is used to hold specific temperature data. It's used in the STORAGE_TEMERATURE_DATA_DESCRIPTOR to return the results of a temperature query.
IOCTL_STORAGE_SET_TEMPERATURE_THRESHOLD : Use this IOCTL with the STORAGE_TEMPERATURE_THRESHOLD structure to set temperature thresholds. For more info, see Behavior changing commands.
STORAGE_TEMPERATURE_THRESHOLD : This structure is used as an input buffer to specify the temperature threshold. The OverThreshold field (boolean) specifies if the Threshold field is the over threshold value or not (otherwise, it's the under threshold value).
Pass-through mechanism
Commands which are not defined in the NVMe specification are the most difficult for the host OS to handle – the host has no insight into the effects that the commands may have on the target device, the exposed infrastructure (namespaces/block sizes), and its behavior.
To better carry such device specific commands through the Windows storage stack, a new pass-through mechanism allows vendor-specific commands to be piped through. This pass-through pipe will also aid in development of management and testing tools. However, this pass-through mechanism requires use of the Command Effects Log. Moreover, StoreNVMe.sys requires all commands, not just pass-through commands, to be described in the Command Effects Log.
Important
StorNVMe.sys and Storport.sys will block any command to a device if it is not described in the Command Effects Log.
Supporting the Command Effects Log
The Command Effects Log (as described in Commands Supported and Effects, section 5.10.1.5 of NVMe Specification 1.2) allows the description of the effects of vendor-specific commands together with specification-defined commands. This facilitates both command support validation as well as command behavior optimization, and therefore should be implemented for the entire set of commands that the device supports. The following conditions describe the result on how the command is sent based on its Command Effects Log entry.
For any specific command described in the Command Effects Log...
While:
Command Supported (CSUPP) is set to ‘1’ signifying that the command is supported by the controller (Bit 01)
Note
When CSUPP is set to ‘0’ (signifying that the command is not supported) the command will be blocked
And if any of the following is set:
Controller Capability Change (CCC) is set to ‘1’ signifying that the command may change controller capabilities (Bit 04)
Namespace Inventory Change (NIC) is set to ‘1’ signifying that the command may change the number, or capabilities for multiple namespaces (Bit 03)
Namespace Capability Change (NCC) is set to ‘1’ signifying that the command may change the capabilities of a single namespace (Bit 02)
Command Submission and Execution (CSE) is set to 001b or 010b, signifying that the command may be submitted when there is no other outstanding command to the same or any namespace, and that another command should not be submitted to the same or any namespace until this command is complete (Bits 18:16)
Then the command will be sent as the only command outstanding to the adapter.
Else if:
- Command Submission and Execution (CSE) is set to 001b, signifying that the command may be submitted when there is no other outstanding command to the same namespace, and that another command should not be submitted to the same namespace until this command is complete (Bits 18:16)
Then the command will be sent as the only command outstanding to the Logical Unit Number object (LUN).
Otherwise, the command is sent with other commands outstanding without inhibition. For example, if a vendor-specific command is sent to the device to retrieve statistical information that is not spec-defined, there should be no risk to changing the device’s behavior or capability to execute I/O commands. Such requests could be serviced in parallel to I/O and no pause-resume would be necessary.
Using IOCTL_STORAGE_PROTOCOL_COMMAND to send commands
Pass-through can be conducted using the IOCTL_STORAGE_PROTOCOL_COMMAND, introduced in Windows 10. This IOCTL was designed to have a similar behavior as the existing SCSI and ATA pass-through IOCTLs, to send an embedded command to the target device. Via this IOCTL, pass-through can be sent to a storage device, including an NVMe drive.
For example, in NVMe, the IOCTL will allow the sending down of the following command codes.
- Vendor Specific Admin Commands (C0h – FFh)
- Vendor Specific NVMe Commands (80h – FFh)
As with all other IOCTLs, Use DeviceIoControl to send the pass-through IOCTL down. The IOCTL is populated using the STORAGE_PROTOCOL_COMMAND input-buffer structure found in ntddstor.h. Populate the Command field with the vendor-specific command.
typedef struct _STORAGE_PROTOCOL_COMMAND {
ULONG Version; // STORAGE_PROTOCOL_STRUCTURE_VERSION
ULONG Length; // sizeof(STORAGE_PROTOCOL_COMMAND)
STORAGE_PROTOCOL_TYPE ProtocolType;
ULONG Flags; // Flags for the request
ULONG ReturnStatus; // return value
ULONG ErrorCode; // return value, optional
ULONG CommandLength; // non-zero value should be set by caller
ULONG ErrorInfoLength; // optional, can be zero
ULONG DataToDeviceTransferLength; // optional, can be zero. Used by WRITE type of request.
ULONG DataFromDeviceTransferLength; // optional, can be zero. Used by READ type of request.
ULONG TimeOutValue; // in unit of seconds
ULONG ErrorInfoOffset; // offsets need to be pointer aligned
ULONG DataToDeviceBufferOffset; // offsets need to be pointer aligned
ULONG DataFromDeviceBufferOffset; // offsets need to be pointer aligned
ULONG CommandSpecific; // optional information passed along with Command.
ULONG Reserved0;
ULONG FixedProtocolReturnData; // return data, optional. Some protocol, such as NVMe, may return a small amount data (DWORD0 from completion queue entry) without the need of separate device data transfer.
ULONG Reserved1[3];
_Field_size_bytes_full_(CommandLength) UCHAR Command[ANYSIZE_ARRAY];
} STORAGE_PROTOCOL_COMMAND, *PSTORAGE_PROTOCOL_COMMAND;
The vendor specific command desired to be sent should be populated in the highlighted field above. Note again that the Command Effects Log must be implemented for pass-through commands. In particular, these commands need to be reported as supported in the Command Effects Log (see previous section for more information). Also note that PRP fields are driver specific thus applications sending commands can leave them as 0.
Finally, this pass-through IOCTL is intended for sending vendor-specific commands. To send other admin or non-vendor specific NVMe commands such as Identify, this pass-through IOCTL should not be used. For example, IOCTL_STORAGE_QUERY_PROPERTY should be used for Identify or Get Log Pages. For more info, see the next section, Protocol-specific queries.
Don't update firmware through the pass-through mechanism
Firmware download and activation commands should not be sent using pass-through. IOCTL_STORAGE_PROTOCOL_COMMAND should only be used for vendor-specific commands.
Instead, use the following general storage IOCTLs (introduced in Windows 10) to avoid applications directly using the SCSI_miniport version of the Firmware IOCTL. Storage drivers will translate the IOCTL to either a SCSI command or the SCSI_miniport version of the IOCTL to the miniport.
These IOCTLs are recommended for developing firmware upgrade tools in Windows 10 and Windows Server 2016:
For getting storage information and updating firmware, Windows also supports PowerShell cmdlets for doing this quickly:
Get-StorageFirmwareInfoUpdate-StorageFirmware
Note
To update firmware on NVMe in Windows 8.1, use IOCTL_SCSI_MINIPORT_FIRMWARE. This IOCTL was not backported to Windows 7. For more information, see Upgrading Firmware for an NVMe Device in Windows 8.1.
Returning errors through the pass-through mechanism
Similar to SCSI and ATA pass-through IOCTLs, when a command/request is sent to the miniport or device, the IOCTL returns if it was successful or not. In the STORAGE_PROTOCOL_COMMAND structure, the IOCTL returns the status through the ReturnStatus field.
Example: sending a vendor-specific command
In this example, an arbitrary vendor-specific command (0xFF) is sent via pass-through to an NVMe drive. The following code allocates a buffer, initializes a query, and then sends the command down to the device via DeviceIoControl.
ZeroMemory(buffer, bufferLength);
protocolCommand = (PSTORAGE_PROTOCOL_COMMAND)buffer;
protocolCommand->Version = STORAGE_PROTOCOL_STRUCTURE_VERSION;
protocolCommand->Length = sizeof(STORAGE_PROTOCOL_COMMAND);
protocolCommand->ProtocolType = ProtocolTypeNvme;
protocolCommand->Flags = STORAGE_PROTOCOL_COMMAND_FLAG_ADAPTER_REQUEST;
protocolCommand->CommandLength = STORAGE_PROTOCOL_COMMAND_LENGTH_NVME;
protocolCommand->ErrorInfoLength = sizeof(NVME_ERROR_INFO_LOG);
protocolCommand->DataFromDeviceTransferLength = 4096;
protocolCommand->TimeOutValue = 10;
protocolCommand->ErrorInfoOffset = FIELD_OFFSET(STORAGE_PROTOCOL_COMMAND, Command) + STORAGE_PROTOCOL_COMMAND_LENGTH_NVME;
protocolCommand->DataFromDeviceBufferOffset = protocolCommand->ErrorInfoOffset + protocolCommand->ErrorInfoLength;
protocolCommand->CommandSpecific = STORAGE_PROTOCOL_SPECIFIC_NVME_ADMIN_COMMAND;
command = (PNVME_COMMAND)protocolCommand->Command;
command->CDW0.OPC = 0xFF;
command->u.GENERAL.CDW10 = 0xto_fill_in;
command->u.GENERAL.CDW12 = 0xto_fill_in;
command->u.GENERAL.CDW13 = 0xto_fill_in;
//
// Send request down.
//
result = DeviceIoControl(DeviceList[DeviceIndex].Handle,
IOCTL_STORAGE_PROTOCOL_COMMAND,
buffer,
bufferLength,
buffer,
bufferLength,
&returnedLength,
NULL
);
In this example, we expect protocolCommand->ReturnStatus == STORAGE_PROTOCOL_STATUS_SUCCESS if the command succeeded to the device.
Protocol-specific queries
Windows 8.1 introduced IOCTL_STORAGE_QUERY_PROPERTY for data retrieval. In Windows 10, the IOCTL was enhanced to support commonly requested NVMe features such as Get Log Pages, Get Features, and Identify. This allows for the retrieval of NVMe specific information for monitoring and inventory purposes.
The input buffer for the IOCTL, STORAGE_PROPERTY_QUERY (from Windows 10) is shown here.
typedef struct _STORAGE_PROPERTY_QUERY {
STORAGE_PROPERTY_ID PropertyId;
STORAGE_QUERY_TYPE QueryType;
UCHAR AdditionalParameters[1];
} STORAGE_PROPERTY_QUERY, *PSTORAGE_PROPERTY_QUERY;
When using IOCTL_STORAGE_QUERY_PROPERTY to retrieve NVMe protocol-specific information in the STORAGE_PROTOCOL_DATA_DESCRIPTOR, configure the STORAGE_PROPERTY_QUERY structure as follows:
Allocate a buffer that can contains both a STORAGE_PROPERTY_QUERY and a STORAGE_PROTOCOL_SPECIFIC_DATA structure.
Set the PropertyID field to StorageAdapterProtocolSpecificProperty or StorageDeviceProtocolSpecificProperty for a controller or device/namespace request, respectively.
Set the QueryType field to PropertyStandardQuery.
Fill the STORAGE_PROTOCOL_SPECIFIC_DATA structure with the desired values. The start of the STORAGE_PROTOCOL_SPECIFIC_DATA is the AdditionalParameters field of STORAGE_PROPERTY_QUERY.
The STORAGE_PROTOCOL_SPECIFIC_DATA structure (from Windows 10) is shown here.
typedef struct _STORAGE_PROTOCOL_SPECIFIC_DATA {
STORAGE_PROTOCOL_TYPE ProtocolType;
ULONG DataType;
ULONG ProtocolDataRequestValue;
ULONG ProtocolDataRequestSubValue;
ULONG ProtocolDataOffset;
ULONG ProtocolDataLength;
ULONG FixedProtocolReturnData;
ULONG Reserved[3];
} STORAGE_PROTOCOL_SPECIFIC_DATA, *PSTORAGE_PROTOCOL_SPECIFIC_DATA;
To specify a type of NVMe protocol-specific information, configure the STORAGE_PROTOCOL_SPECIFIC_DATA structure as follows:
Set the ProtocolType field to ProtocolTypeNVMe.
Set the DataType field to an enumeration value defined by STORAGE_PROTOCOL_NVME_DATA_TYPE:
- Use NVMeDataTypeIdentify to get Identify Controller data or Identify Namespace data.
- Use NVMeDataTypeLogPage to get log pages (including SMART/health data).
- Use NVMeDataTypeFeature to get features of the NVMe drive.
When ProtocolTypeNVMe is used as the ProtocolType, queries for protocol-specific information can be retrieved in parallel with other I/O on the NVMe drive.
Important
For an IOCTL_STORAGE_QUERY_PROPERTY that uses a STORAGE_PROPERTY_ID of StorageAdapterProtocolSpecificProperty, and whose STORAGE_PROTOCOL_SPECIFIC_DATA or STORAGE_PROTOCOL_SPECIFIC_DATA_EXT structure is set to ProtocolType=ProtocolTypeNvme and DataType=NVMeDataTypeLogPage, set the ProtocolDataLength member of that same structure to a minimum value of 512 (bytes).
The following examples demonstrate NVMe protocol-specific queries.
Example: NVMe Identify query
In this example, the Identify request is sent to an NVMe drive. The following code initializes the query data structure and then sends the command down to the device via DeviceIoControl.
BOOL result;
PVOID buffer = NULL;
ULONG bufferLength = 0;
ULONG returnedLength = 0;
PSTORAGE_PROPERTY_QUERY query = NULL;
PSTORAGE_PROTOCOL_SPECIFIC_DATA protocolData = NULL;
PSTORAGE_PROTOCOL_DATA_DESCRIPTOR protocolDataDescr = NULL;
//
// Allocate buffer for use.
//
bufferLength = FIELD_OFFSET(STORAGE_PROPERTY_QUERY, AdditionalParameters) + sizeof(STORAGE_PROTOCOL_SPECIFIC_DATA) + NVME_MAX_LOG_SIZE;
buffer = malloc(bufferLength);
if (buffer == NULL) {
_tprintf(_T("DeviceNVMeQueryProtocolDataTest: allocate buffer failed, exit.\n"));
goto exit;
}
//
// Initialize query data structure to get Identify Controller Data.
//
ZeroMemory(buffer, bufferLength);
query = (PSTORAGE_PROPERTY_QUERY)buffer;
protocolDataDescr = (PSTORAGE_PROTOCOL_DATA_DESCRIPTOR)buffer;
protocolData = (PSTORAGE_PROTOCOL_SPECIFIC_DATA)query->AdditionalParameters;
query->PropertyId = StorageAdapterProtocolSpecificProperty;
query->QueryType = PropertyStandardQuery;
protocolData->ProtocolType = ProtocolTypeNvme;
protocolData->DataType = NVMeDataTypeIdentify;
protocolData->ProtocolDataRequestValue = NVME_IDENTIFY_CNS_CONTROLLER;
protocolData->ProtocolDataRequestSubValue = 0;
protocolData->ProtocolDataOffset = sizeof(STORAGE_PROTOCOL_SPECIFIC_DATA);
protocolData->ProtocolDataLength = NVME_MAX_LOG_SIZE;
//
// Send request down.
//
result = DeviceIoControl(DeviceList[Index].Handle,
IOCTL_STORAGE_QUERY_PROPERTY,
buffer,
bufferLength,
buffer,
bufferLength,
&returnedLength,
NULL
);
ZeroMemory(buffer, bufferLength);
query = (PSTORAGE_PROPERTY_QUERY)buffer;
protocolDataDescr = (PSTORAGE_PROTOCOL_DATA_DESCRIPTOR)buffer;
protocolData = (PSTORAGE_PROTOCOL_SPECIFIC_DATA)query->AdditionalParameters;
query->PropertyId = StorageDeviceProtocolSpecificProperty;
query->QueryType = PropertyStandardQuery;
protocolData->ProtocolType = ProtocolTypeNvme;
protocolData->DataType = NVMeDataTypeLogPage;
protocolData->ProtocolDataRequestValue = NVME_LOG_PAGE_HEALTH_INFO;
protocolData->ProtocolDataRequestSubValue = 0;
protocolData->ProtocolDataOffset = sizeof(STORAGE_PROTOCOL_SPECIFIC_DATA);
protocolData->ProtocolDataLength = sizeof(NVME_HEALTH_INFO_LOG);
//
// Send request down.
//
result = DeviceIoControl(DeviceList[Index].Handle,
IOCTL_STORAGE_QUERY_PROPERTY,
buffer,
bufferLength,
buffer,
bufferLength,
&returnedLength,
NULL
);
//
// Validate the returned data.
//
if ((protocolDataDescr->Version != sizeof(STORAGE_PROTOCOL_DATA_DESCRIPTOR)) ||
(protocolDataDescr->Size != sizeof(STORAGE_PROTOCOL_DATA_DESCRIPTOR))) {
_tprintf(_T("DeviceNVMeQueryProtocolDataTest: Get Identify Controller Data - data descriptor header not valid.\n"));
return;
}
protocolData = &protocolDataDescr->ProtocolSpecificData;
if ((protocolData->ProtocolDataOffset < sizeof(STORAGE_PROTOCOL_SPECIFIC_DATA)) ||
(protocolData->ProtocolDataLength < NVME_MAX_LOG_SIZE)) {
_tprintf(_T("DeviceNVMeQueryProtocolDataTest: Get Identify Controller Data - ProtocolData Offset/Length not valid.\n"));
goto exit;
}
//
// Identify Controller Data
//
{
PNVME_IDENTIFY_CONTROLLER_DATA identifyControllerData = (PNVME_IDENTIFY_CONTROLLER_DATA)((PCHAR)protocolData + protocolData->ProtocolDataOffset);
if ((identifyControllerData->VID == 0) ||
(identifyControllerData->NN == 0)) {
_tprintf(_T("DeviceNVMeQueryProtocolDataTest: Identify Controller Data not valid.\n"));
goto exit;
} else {
_tprintf(_T("DeviceNVMeQueryProtocolDataTest: ***Identify Controller Data succeeded***.\n"));
}
}
Important
For an IOCTL_STORAGE_QUERY_PROPERTY that uses a STORAGE_PROPERTY_ID of StorageAdapterProtocolSpecificProperty, and whose STORAGE_PROTOCOL_SPECIFIC_DATA or STORAGE_PROTOCOL_SPECIFIC_DATA_EXT structure is set to ProtocolType=ProtocolTypeNvme and DataType=NVMeDataTypeLogPage, set the ProtocolDataLength member of that same structure to a minimum value of 512 (bytes).
Note that the caller needs to allocate a single buffer containing STORAGE_PROPERTY_QUERY and the size of STORAGE_PROTOCOL_SPECIFIC_DATA. In this example, it’s using the same buffer for input and output from the property query. That’s why the buffer that was allocated has a size of “FIELD_OFFSET(STORAGE_PROPERTY_QUERY, AdditionalParameters) + sizeof(STORAGE_PROTOCOL_SPECIFIC_DATA) + NVME_MAX_LOG_SIZE”. Although separate buffers could be allocated for both input and output, we recommend using a single buffer to query NVMe related-information.
identifyControllerData->NN is Number of Namespaces (NN). Windows detects a namespace as a physical drive.
Example: NVMe Get Log Pages query
In this example, based off of the previous one, the Get Log Pages request is sent to an NVMe drive. The following code prepares the query data structure and then sends the command down to the device via DeviceIoControl.
ZeroMemory(buffer, bufferLength);
query = (PSTORAGE_PROPERTY_QUERY)buffer;
protocolDataDescr = (PSTORAGE_PROTOCOL_DATA_DESCRIPTOR)buffer;
protocolData = (PSTORAGE_PROTOCOL_SPECIFIC_DATA)query->AdditionalParameters;
query->PropertyId = StorageDeviceProtocolSpecificProperty;
query->QueryType = PropertyStandardQuery;
protocolData->ProtocolType = ProtocolTypeNvme;
protocolData->DataType = NVMeDataTypeLogPage;
protocolData->ProtocolDataRequestValue = NVME_LOG_PAGE_HEALTH_INFO;
protocolData->ProtocolDataRequestSubValue = 0; // This will be passed as the lower 32 bit of log page offset if controller supports extended data for the Get Log Page.
protocolData->ProtocolDataRequestSubValue2 = 0; // This will be passed as the higher 32 bit of log page offset if controller supports extended data for the Get Log Page.
protocolData->ProtocolDataRequestSubValue3 = 0; // This will be passed as Log Specific Identifier in CDW11.
protocolData->ProtocolDataRequestSubValue4 = 0; // This will map to STORAGE_PROTOCOL_DATA_SUBVALUE_GET_LOG_PAGE definition, then user can pass Retain Asynchronous Event, Log Specific Field.
protocolData->ProtocolDataOffset = sizeof(STORAGE_PROTOCOL_SPECIFIC_DATA);
protocolData->ProtocolDataLength = sizeof(NVME_HEALTH_INFO_LOG);
//
// Send request down.
//
result = DeviceIoControl(DeviceList[Index].Handle,
IOCTL_STORAGE_QUERY_PROPERTY,
buffer,
bufferLength,
buffer,
bufferLength,
&returnedLength,
NULL
);
if (!result || (returnedLength == 0)) {
_tprintf(_T("DeviceNVMeQueryProtocolDataTest: SMART/Health Information Log failed. Error Code %d.\n"), GetLastError());
goto exit;
}
//
// Validate the returned data.
//
if ((protocolDataDescr->Version != sizeof(STORAGE_PROTOCOL_DATA_DESCRIPTOR)) ||
(protocolDataDescr->Size != sizeof(STORAGE_PROTOCOL_DATA_DESCRIPTOR))) {
_tprintf(_T("DeviceNVMeQueryProtocolDataTest: SMART/Health Information Log - data descriptor header not valid.\n"));
return;
}
protocolData = &protocolDataDescr->ProtocolSpecificData;
if ((protocolData->ProtocolDataOffset < sizeof(STORAGE_PROTOCOL_SPECIFIC_DATA)) ||
(protocolData->ProtocolDataLength < sizeof(NVME_HEALTH_INFO_LOG))) {
_tprintf(_T("DeviceNVMeQueryProtocolDataTest: SMART/Health Information Log - ProtocolData Offset/Length not valid.\n"));
goto exit;
}
//
// SMART/Health Information Log Data
//
{
PNVME_HEALTH_INFO_LOG smartInfo = (PNVME_HEALTH_INFO_LOG)((PCHAR)protocolData + protocolData->ProtocolDataOffset);
_tprintf(_T("DeviceNVMeQueryProtocolDataTest: SMART/Health Information Log Data - Temperature %d.\n"), ((ULONG)smartInfo->Temperature[1] << 8 | smartInfo->Temperature[0]) - 273);
_tprintf(_T("DeviceNVMeQueryProtocolDataTest: ***SMART/Health Information Log succeeded***.\n"));
}
Callers could use a STORAGE_PROPERTY_ID of StorageAdapterProtocolSpecificProperty, and whose STORAGE_PROTOCOL_SPECIFIC_DATA or STORAGE_PROTOCOL_SPECIFIC_DATA_EXT structure is set to ProtocolDataRequestValue=VENDOR_SPECIFIC_LOG_PAGE_IDENTIFIER to request 512 byte chunks of vendor specific data.
Example: NVMe Get Features query
In this example, based off of the previous one, the Get Features request is sent to an NVMe drive. The following code prepares the query data structure and then sends the command down to the device via DeviceIoControl.
//
// Initialize query data structure to Volatile Cache feature.
//
ZeroMemory(buffer, bufferLength);
query = (PSTORAGE_PROPERTY_QUERY)buffer;
protocolDataDescr = (PSTORAGE_PROTOCOL_DATA_DESCRIPTOR)buffer;
protocolData = (PSTORAGE_PROTOCOL_SPECIFIC_DATA)query->AdditionalParameters;
query->PropertyId = StorageDeviceProtocolSpecificProperty;
query->QueryType = PropertyStandardQuery;
protocolData->ProtocolType = ProtocolTypeNvme;
protocolData->DataType = NVMeDataTypeFeature;
protocolData->ProtocolDataRequestValue = NVME_FEATURE_VOLATILE_WRITE_CACHE;
protocolData->ProtocolDataRequestSubValue = 0;
protocolData->ProtocolDataOffset = 0;
protocolData->ProtocolDataLength = 0;
//
// Send request down.
//
result = DeviceIoControl(DeviceList[Index].Handle,
IOCTL_STORAGE_QUERY_PROPERTY,
buffer,
bufferLength,
buffer,
bufferLength,
&returnedLength,
NULL
);
if (!result || (returnedLength == 0)) {
_tprintf(_T("DeviceNVMeQueryProtocolDataTest: Get Feature - Volatile Cache failed. Error Code %d.\n"), GetLastError());
goto exit;
}
//
// Validate the returned data.
//
if ((protocolDataDescr->Version != sizeof(STORAGE_PROTOCOL_DATA_DESCRIPTOR)) ||
(protocolDataDescr->Size != sizeof(STORAGE_PROTOCOL_DATA_DESCRIPTOR))) {
_tprintf(_T("DeviceNVMeQueryProtocolDataTest: Get Feature - Volatile Cache - data descriptor header not valid.\n"));
return;
}
//
// Volatile Cache
//
{
_tprintf(_T("DeviceNVMeQueryProtocolDataTest: Get Feature - Volatile Cache - %x.\n"), protocolDataDescr->ProtocolSpecificData.FixedProtocolReturnData);
_tprintf(_T("DeviceNVMeQueryProtocolDataTest: ***Get Feature - Volatile Cache succeeded***.\n"));
}
Protocol-specific set
From Windows 10 19H1, the IOCTL_STORAGE_SET_PROPERTY was enhanced to support NVMe Set Features.
The input buffer for the IOCTL_STORAGE_SET_PROPERTY is shown here:
typedef struct _STORAGE_PROPERTY_SET {
//
// ID of the property being retrieved
//
STORAGE_PROPERTY_ID PropertyId;
//
// Flags indicating the type of set property being performed
//
STORAGE_SET_TYPE SetType;
//
// Space for additional parameters if necessary
//
UCHAR AdditionalParameters[1];
} STORAGE_PROPERTY_SET, *PSTORAGE_PROPERTY_SET;
When using IOCTL_STORAGE_SET_PROPERTY to set NVMe feature, configure the STORAGE_PROPERTY_SET structure as follows:
- Allocate a buffer that can contains both a STORAGE_PROPERTY_SET and a STORAGE_PROTOCOL_SPECIFIC_DATA_EXT structure;
- Set the PropertyID field to StorageAdapterProtocolSpecificProperty or StorageDeviceProtocolSpecificProperty for a controller or device/namespace request, respectively.
- Fill the STORAGE_PROTOCOL_SPECIFIC_DATA_EXT structure with the desired values. The start of the STORAGE_PROTOCOL_SPECIFIC_DATA_EXT is the AdditionalParameters field of STORAGE_PROPERTY_SET.
The STORAGE_PROTOCOL_SPECIFIC_DATA_EXT structure is shown here.
typedef struct _STORAGE_PROTOCOL_SPECIFIC_DATA_EXT {
STORAGE_PROTOCOL_TYPE ProtocolType;
ULONG DataType; // The value will be protocol specific, as defined in STORAGE_PROTOCOL_NVME_DATA_TYPE or STORAGE_PROTOCOL_ATA_DATA_TYPE.
ULONG ProtocolDataValue;
ULONG ProtocolDataSubValue; // Data sub request value
ULONG ProtocolDataOffset; // The offset of data buffer is from beginning of this data structure.
ULONG ProtocolDataLength;
ULONG FixedProtocolReturnData;
ULONG ProtocolDataSubValue2; // First additional data sub request value
ULONG ProtocolDataSubValue3; // Second additional data sub request value
ULONG ProtocolDataSubValue4; // Third additional data sub request value
ULONG ProtocolDataSubValue5; // Fourth additional data sub request value
ULONG Reserved[5];
} STORAGE_PROTOCOL_SPECIFIC_DATA_EXT, *PSTORAGE_PROTOCOL_SPECIFIC_DATA_EXT;
To specify a type of NVMe feature to set, configure the STORAGE_PROTOCOL_SPECIFIC_DATA_EXT structure as follows:
- Set the ProtocolType field to ProtocolTypeNvme;
- Set the DataType field to the enumeration value NVMeDataTypeFeature defined by STORAGE_PROTOCOL_NVME_DATA_TYPE;
The following examples demonstrate NVMe feature set.
Example: NVMe Set Features
In this example, the Set Features request is sent to an NVMe drive. The following code prepares the set data structure and then sends the command down to the device via DeviceIoControl.
PSTORAGE_PROPERTY_SET setProperty = NULL;
PSTORAGE_PROTOCOL_SPECIFIC_DATA_EXT protocolData = NULL;
PSTORAGE_PROTOCOL_DATA_DESCRIPTOR_EXT protocolDataDescr = NULL;
//
// Allocate buffer for use.
//
bufferLength = FIELD_OFFSET(STORAGE_PROPERTY_SET, AdditionalParameters) + sizeof(STORAGE_PROTOCOL_SPECIFIC_DATA_EXT);
bufferLength += NVME_MAX_LOG_SIZE;
buffer = new UCHAR[bufferLength];
//
// Initialize query data structure to get the desired log page.
//
ZeroMemory(buffer, bufferLength);
setProperty = (PSTORAGE_PROPERTY_SET)buffer;
setProperty->PropertyId = StorageAdapterProtocolSpecificProperty;
setProperty->SetType = PropertyStandardSet;
protocolData = (PSTORAGE_PROTOCOL_SPECIFIC_DATA_EXT)setProperty->AdditionalParameters;
protocolData->ProtocolType = ProtocolTypeNvme;
protocolData->DataType = NVMeDataTypeFeature;
protocolData->ProtocolDataValue = NVME_FEATURE_HOST_CONTROLLED_THERMAL_MANAGEMENT;
protocolData->ProtocolDataSubValue = 0; // This will pass to CDW11.
protocolData->ProtocolDataSubValue2 = 0; // This will pass to CDW12.
protocolData->ProtocolDataSubValue3 = 0; // This will pass to CDW13.
protocolData->ProtocolDataSubValue4 = 0; // This will pass to CDW14.
protocolData->ProtocolDataSubValue5 = 0; // This will pass to CDW15.
protocolData->ProtocolDataOffset = 0;
protocolData->ProtocolDataLength = 0;
//
// Send request down.
//
result = DeviceIoControl(m_deviceHandle,
IOCTL_STORAGE_SET_PROPERTY,
buffer,
bufferLength,
buffer,
bufferLength,
&returnedLength,
NULL
);
Temperature queries
In Windows 10, IOCTL_STORAGE_QUERY_PROPERTY can also be used to query temperature data from NVMe devices.
To retrieve temperature information from an NVMe drive in the STORAGE_TEMPERATURE_DATA_DESCRIPTOR, configure the STORAGE_PROPERTY_QUERY structure as follows:
Allocate a buffer that can contains a STORAGE_PROPERTY_QUERY structure.
Set the PropertyID field to StorageAdapterTemperatureProperty or StorageDeviceTemperatureProperty for a controller or device/namespace request, respectively.
Set the QueryType field to PropertyStandardQuery.
The STORAGE_TEMPERATURE_INFO structure (from Windows 10) is shown here.
typedef struct _STORAGE_TEMPERATURE_INFO {
USHORT Index; // Starts from 0. Index 0 may indicate a composite value.
SHORT Temperature; // Signed value; in Celsius.
SHORT OverThreshold; // Signed value; in Celsius.
SHORT UnderThreshold; // Signed value; in Celsius.
BOOLEAN OverThresholdChangable; // Can the threshold value being changed by using IOCTL_STORAGE_SET_TEMPERATURE_THRESHOLD.
BOOLEAN UnderThresholdChangable; // Can the threshold value being changed by using IOCTL_STORAGE_SET_TEMPERATURE_THRESHOLD.
BOOLEAN EventGenerated; // Indicates that notification will be generated when temperature cross threshold.
UCHAR Reserved0;
ULONG Reserved1;
} STORAGE_TEMPERATURE_INFO, *PSTORAGE_TEMPERATURE_INFO;
Behavior changing commands
Commands that manipulate device attributes or potentially impact device behavior are more difficult for the operating system to deal with. If device attributes change at run-time while I/O is being processed, synchronization or data integrity issues can arise if not properly handled.
The NVMe Set-Features command is a good example of a behavior changing command. It allows for the changing of the arbitration mechanism and the setting of temperature thresholds. To ensure that in-flight data is not at risk when behavior-affecting set commands are sent down, Windows will pause all I/O to the NVMe device, drain queues, and flush buffers. Once the set command has executed successfully, I/O is resumed (if possible). If I/O cannot be resumed, a device reset may be required.
Setting temperature thresholds
Windows 10 introduced IOCTL_STORAGE_SET_TEMPERATURE_THRESHOLD, an IOCTL for getting and setting temperature thresholds. You can also use it to get the current temperature of the device. The input/output buffer for this IOCTL is the STORAGE_TEMPERATURE_INFO structure, from the previous code section.
Example: Setting over-threshold temperature
In this example, an NVMe drive's over-threshold temperature is set. The following code prepares the command and then sends it down to the device via DeviceIoControl.
BOOL result;
ULONG returnedLength = 0;
STORAGE_TEMPERATURE_THRESHOLD setThreshold = {0};
setThreshold.Version = sizeof(STORAGE_TEMPERATURE_THRESHOLD);
setThreshold.Size = sizeof(STORAGE_TEMPERATURE_THRESHOLD);
setThreshold.Flags = STORAGE_TEMPERATURE_THRESHOLD_FLAG_ADAPTER_REQUEST;
setThreshold.Index = SensorIndex;
setThreshold.Threshold = Threshold;
setThreshold.OverThreshold = UpdateOverThreshold;
//
// Send request down.
//
result = DeviceIoControl(DeviceList[DeviceIndex].Handle,
IOCTL_STORAGE_SET_TEMPERATURE_THRESHOLD,
&setThreshold,
sizeof(STORAGE_TEMPERATURE_THRESHOLD),
NULL,
0,
&returnedLength,
NULL
);
Setting vendor-specific features
Without the Command Effects Log, the driver has no knowledge of the ramifications of the command. This is why the Command Effects Log is required. It helps the operating system determine if a command is high impact and if it can be sent in parallel with other commands to the drive.
The Command Effects Log is not yet granular enough to encompass vendor-specific Set-Features commands. For this reason, it is not yet possible to send vendor-specific Set-Features commands. However, it is possible to use the pass-through mechanism, discussed earlier, to send vendor-specific commands. For more info, see Pass-through mechanism.
Header files
The following files are relevant to NVMe development. These files are included with the Microsoft Windows Software Development Kit (SDK).
| Header file | Description |
|---|---|
| ntddstor.h | Defines constants and types for accessing the storage class drivers from kernel mode. |
| nvme.h | For other NVMe-related data-structures. |
| winioctl.h | For overall Win32 IOCTL definitions, including storage APIs for user mode applications. |