pcivirt.h header

Article
01/23/2023

Reference guide for using interfaces used exposing VFs to a Virtual Machine.

Devices that conform to the PCI Express Single-Root I/O Virtualization (SR-IOV) specification can provide multiple interfaces to the device. These interfaces, known as Virtual Functions (VFs), are independent and are provided through the initial device interface, known as the Physical Function (PF). For example, an Ethernet NIC that supports SR-IOV can be designed to have a switch with one physical Ethernet port (connected to the physical wire) and many virtual Ethernet ports.

The PF’s configuration space allows the PF driver to manage the VF's PCI resources, including memory-mapped I/O space and message-signaled interrupts. Since VFs are a subset of a complete device, they can be less expensive to expose in hardware than a traditional function in a multi-function package. This allows the device maker to build more interfaces, and to manage any shared resources centrally.

When Windows is running directly on the machine hardware, the device drivers participate in operations related to Plug and Play, power management, interrupt management and other tasks. A trusted Windows bus driver and Hardware Abstraction Layer (HAL) own bus configuration and configure the entire bus. The driver runs within the same privilege level and there are no trust boundaries in kernel mode.

When Windows is running on a virtual machine (VM), those assumptions do not apply. VFs can be put under the control of a non-privileged VM. However, the hardware must be security checked so that the security or performance of the system is not impacted.

When a driver running on the VF requests a configuration space read or write, the request is received by the virtualization stack and sent to the SR-IOV device’s PF driver. It is the responsibility of the PF driver to respond to those requests and provide details for the VF. The PF driver may occasionally require that a configuration read or write request that is passed down to the hardware.

The stack uses an I/O MMU to differentiate traffic coming from the various interfaces that the device exposes, enforcing policy about which regions of memory a device can access and which interrupts it can generate.

PCI virtualization.

Hardware requirements

The system to be used for SR-IOV device assignment must meet the requirements for SR-IOV networking and Direct Device Assignment. The system must have a IOMMU, that IOMMU must be configured to give control of devices over to the operating system, and PCIe ACS (Access Control Services) must be enabled and configured for use by the operating system. Finally, the device in question must not use line based interrupts, and must not require ATS (Address Translation Services).

More information here:

Everything you wanted to know about SR-IOV in Hyper-V. Part 1

Discrete Device Assignment — Description and background

To determine if a system supports device assignment, and if a particular PCI device will work for device assignment:

Discrete Device Assignment script

GUID_DEVINTERFACE_VIRTUALIZABLE_DEVICE is a device class interface that is provided by drivers for SR-IOV devices. This GUID provides a way to query for all the device stacks which expose the various function tables that are used to manage the virtualization-related features of the device. After the driver registers the GUID, individual capabilities are discovered by sending IRP_MN_QUERY_INTERFACE. The driver must respond to that request with GUID_SRIOV_DEVICE_INTERFACE_STANDARD. Drivers must also handle IOCTL_SRIOV_NOTIFICATION and IOCTL_SRIOV_EVENT_COMPLETE.

A driver for an SR_IOV device, which runs in a privileged VM is the host OS. It owns Plug-and-Play and power management for an entire machine, and exposes PCI Express SR-IOV Virtual Functions in non-privileged VMs, must provide the GUID_SRIOV_DEVICE_INTERFACE_STANDARD (defined in the header Pcivirt.h). That driver might be PCI Express SR-IOV Physical Function (PF) driver that creates the FDO, or it might be a lower filter on that device node in the case when the FDO is being managed by a port driver.

The device interface is required so that the driver can access the configuration space of the VFs.

In the PF driver's EVT_WDF_DRIVER_DEVICE_ADD implementation, perform these tasks:

After calling WdfDeviceCreate to create the function device object (FDO), call WdfDeviceCreateDeviceInterface to register GUID_DEVINTERFACE_VIRTUALIZABLE_DEVICE. This allows the virtualization stack to retrieve a device handle to the SR-IOV device.
Expose the GUID_SRIOV_DEVICE_INTERFACE_STANDARD.
- Initialize a SRIOV_DEVICE_INTERFACE_STANDARD structure and set members to function pointers of the callback functions implemented by the PF driver.
- Configure the structure by calling WDF_QUERY_INTERFACE_CONFIG_INIT.
- Register the interface with the FDO by calling WdfDeviceAddQueryInterface.

    // Make the device visible as an assignable device.
    //
    status = WdfDeviceCreateDeviceInterface(
        fdo,
        &GUID_DEVINTERFACE_VIRTUALIZABLE_DEVICE,
        NULL);
    if (!NT_SUCCESS(status))
    {
        TraceEvents(TRACE_LEVEL_ERROR, DBG_INIT,
                    "Failed to create interface: %!STATUS!",
                    status);
        goto Cleanup;
    }

    //
    // Expose SRIOV_DEVICE_INTERFACE_STANDARD
    //
    RtlZeroMemory(&sriovInterface, sizeof(sriovInterface));
    sriovInterface.Size = sizeof(sriovInterface);
    sriovInterface.Version = 1;
    sriovInterface.Context = deviceContext;
    sriovInterface.InterfaceReference = Virtualization_ReferenceInterface;
    sriovInterface.InterfaceDereference = Virtualization_DereferenceInterface;
    sriovInterface.ReadVfConfig = Virtualization_ReadConfig;
    sriovInterface.WriteVfConfig = Virtualization_WriteConfig;
    sriovInterface.ReadVfConfigBlock = Virtualization_ReadBlock;
    sriovInterface.WriteVfConfigBlock = Virtualization_WriteBlock;
    sriovInterface.ResetVf = Virtualization_ResetFunction;
    sriovInterface.SetVfPowerState = Virtualization_SetPowerState;
    sriovInterface.GetDeviceLocation = Virtualization_GetDeviceLocation;
    sriovInterface.GetVendorAndDevice = Virtualization_GetVendorAndDevice;
    sriovInterface.QueryProbedBars = Virtualization_QueryProbedBars;
    sriovInterface.QueryLuid = Virtualization_QueryLuid;


    WDF_QUERY_INTERFACE_CONFIG_INIT(&qiConfig,
                                    (PINTERFACE)&sriovInterface,
                                    &GUID_SRIOV_DEVICE_INTERFACE_STANDARD,
                                    NULL);

    status = WdfDeviceAddQueryInterface(fdo, &qiConfig);

    if (!NT_SUCCESS(status))
    {
        TraceEvents(TRACE_LEVEL_ERROR, DBG_INIT,
                    "WdfDeviceAddQueryInterface failed: %!STATUS!\n",
                    status);
        goto Cleanup;
    }

Handling Plug and Play events

The virtualization stack is responsible for sending the appropriate messages to the VM, waiting for the reply (with a timeout) and in case of non-responding VM, and applying appropriate action such as vetoing the PnP event, or surprise-removing the device from the non-privileged VM. PF drivers that implement GUID_DEVINTERFACE_VIRTUALIZABLE_DEVICE must also handle these I/O control requests that allow the virtualization stack to react to PnP events.

The virtualization stack first sends IOCTL_SRIOV_ATTACH to the device. This notifies the device that the virtualization stack needs to be notified about certain PnP events.
This is in effect until the virtualization stack sends IOCTL_SRIOV_DETACH.
The virtualization stack queries the devices about PnP events by sending IOCTL_SRIOV_NOTIFICATION requests. The PF driver can inform the virtualization stack of a PnP event by completing the IOCTL_SRIOV_NOTIFICATION request.
The virtualization stack unblocks those events by sending IOCTL_SRIOV_EVENT_COMPLETE.

pcivirt.h contains the following programming interfaces:

IOCTLs


IOCTL_SRIOV_ATTACH The request indicates that the virtualization stack wants to register for Plug and Play events received by the SR-IOV device.
IOCTL_SRIOV_DETACH The request indicates that the virtualization stack wants to unregister for Plug and Play events (previously registered through the IOCTL_SRIOV_ATTACH request).
IOCTL_SRIOV_EVENT_COMPLETE The request indicates that the virtualization stack or the SR-IOV device received one of the events listed in SRIOV_PF_EVENT.
IOCTL_SRIOV_INVALIDATE_BLOCK The IOCTL_SRIOV_INVALIDATE_BLOCK request indicates that the virtualization stack wants to reset the contents of the specified configuration block.
IOCTL_SRIOV_MITIGATED_RANGE_UPDATE The IOCTL_SRIOV_MITIGATED_RANGE_UPDATE request indicates that the virtualization stack wants to update to the mitigation ranges.
IOCTL_SRIOV_NOTIFICATION The request indicates that the virtualization stack wants to be notified when one of the events listed in SRIOV_PF_EVENT occurs.
IOCTL_SRIOV_PROXY_QUERY_LUID This request supplies the local unique identifier of the SR_IOV device implementing the interface.
IOCTL_SRIOV_QUERY_MITIGATED_RANGE_COUNT The request determines the ranges of memory-mapped I/O space that must mitigated.
IOCTL_SRIOV_QUERY_MITIGATED_RANGES The request determines the specific ranges on which intercepts must be placed.

Callback functions


READ_WRITE_MITIGATED_REGISTER Reads or writes to mitigated address spaces.
SRIOV_GET_DEVICE_LOCATION Retrieves information about the current location of the PCI device on the bus, such as PCI Segment, Bus, Device and Function number.
SRIOV_GET_MMIO_REQUIREMENTS This callback function is not supported.
SRIOV_GET_RESOURCE_FOR_BAR Gets the translated resource for a specific Base Address Register (BAR).
SRIOV_GET_VENDOR_AND_DEVICE_IDS Supplies the Vendor and Device ID for a PCI Express SR-IOV Virtual Function (VF) to be used for generating a more generic Plug and Play ID for the VF. These IDs cannot be read directly from the VF's configuration space.
SRIOV_QUERY_LUID Gets the local unique identifier of the SR-IOV device.
SRIOV_QUERY_LUID_VF Gets the PCI Express SR-IOV Virtual Function (VF) given a unique identifier.
SRIOV_QUERY_PROBED_BARS Queries the data read from the physical function's (PF) base address registers (BARs) if the value -1 were written to them first.
SRIOV_QUERY_PROBED_BARS_2 Queries the data read from the specified PCI Express SR-IOV Virtual Function (VF) base address registers (BARs) if the value -1 were written to them first.
SRIOV_QUERY_VF_LUID Gets the local unique identifier of the PCI Express SR-IOV Virtual Function (VF).
SRIOV_READ_BLOCK Reads data from the specified configuration block of a PCI Express SR-IOV Virtual Function (VF).
SRIOV_READ_CONFIG Reads data from the configuration space of the specified PCI Express SR-IOV Virtual Function (VF).
SRIOV_RESET_FUNCTION Resets the specified PCI Express SR-IOV Virtual Function (VF).
SRIOV_SET_POWER_STATE Sets the power state of the specified PCI Express SR-IOV Virtual Function (VF).
SRIOV_WRITE_BLOCK Writes data to the specified configuration block of a PCI Express SR-IOV Virtual Function (VF).
SRIOV_WRITE_CONFIG Writes configuration data to a PCI Express SR-IOV Virtual Function (VF).

Structures


MITIGABLE_DEVICE_INTERFACE Stores function pointers to callback functions implemented by the physical function (PF) driver for the mitigable device interface.
SRIOV_DEVICE_INTERFACE_STANDARD Stores function pointers to callback functions implemented by the physical function (PF) driver in the device stack for the of the SR-IOV device.
SRIOV_DEVICE_INTERFACE_STANDARD_2 Stores function pointers to callback functions implemented by the physical function (PF) driver in the device stack for the of the SR-IOV device. This is an extended version of SRIOV_DEVICE_INTERFACE_STANDARD.
SRIOV_INVALIDATE_BLOCK Contains the configuration block information. This structure is used in a IOCTL_SRIOV_INVALIDATE_BLOCK request.
SRIOV_MITIGATED_RANGE_COUNT_INPUT This structure is used as an input buffer to the IOCTL_SRIOV_QUERY_MITIGATED_RANGE_COUNT request to determine the ranges of memory-mapped I/O space that must be mitigated.
SRIOV_MITIGATED_RANGE_COUNT_OUTPUT This structures is the output buffer received by the IOCTL_SRIOV_QUERY_MITIGATED_RANGE_COUNT request that contains an array of ranges of memory-mapped I/O space that must be mitigated.
SRIOV_MITIGATED_RANGE_UPDATE_INPUT This structure is used as an input buffer to the IOCTL_SRIOV_MITIGATED_RANGE_UPDATE request to indicate the virtual function (VF) whose memory-mapped I/O space that must be mitigated.
SRIOV_MITIGATED_RANGE_UPDATE_OUTPUT This structures is the output buffer received by the IOCTL_SRIOV_MITIGATED_RANGE_UPDATE request that indicates the virtual function (VF) whose memory-mapped I/O space was mitigated.
SRIOV_MITIGATED_RANGES_INPUT This structure is the input buffer in the IOCTL_SRIOV_QUERY_MITIGATED_RANGES request to get the specific ranges on which intercepts must be placed.
SRIOV_MITIGATED_RANGES_OUTPUT This structure is the output buffer received by the IOCTL_SRIOV_QUERY_MITIGATED_RANGES request to get the specific ranges on which intercepts must be placed.
SRIOV_PNP_EVENT_COMPLETE Stores the status for an event that the SR-IOV Physical Function (PF) driver should set for Plug and Play even completion. This structure is used in the input buffer of the IOCTL_SRIOV_EVENT_COMPLETE request.
SRIOV_PROXY_QUERY_LUID_OUTPUT Stores the local unique identifier of the SR_IOV device implementing the interface. This structure is the output buffer for the IOCTL_SRIOV_PROXY_QUERY_LUID request.

Enumerations


SRIOV_PF_EVENT Defines event values for the SR-IOV device.

Share via