NVIDIA L4 GPU Not Detected in Guest Hyper-V VM After DDA

Question

Environment Details:

• Host Machine:
  • OS: Windows Server 2022
  • GPU: NVIDIA L4 x 3
  • CPU: Intel Xeon Gold 6542Y (with VT-d enabled)
  • IOMMU and SR-IOV: Enabled in BIOS
• Virtual Machine (VM):
  • OS: Linux ( Debian 12)
  • Hyper-V DDA Configurations completing without errors.
  • 

Steps Performed:

1. Check the Current State of the VM

    PS C:\Users\Administrator> Get-VM
   

   Output:

    Name                State CPUUsage(%) MemoryAssigned(M) Uptime   Status     Version
    ----                ----- ----------- ----------------- ------   ------     -------
    VM_4.5.12_Build Off   0           0                 00:00:00 正常稼働中 10.0
   

2. Set the Automatic Stop Action

    PS C:\Users\Administrator> Set-VM -Name VM_4.5.12_Build -AutomaticStopAction TurnOff
   

3. Enable Guest Controlled Cache Types

    PS C:\Users\Administrator> Set-VM -GuestControlledCacheTypes $true -VMName VM_4.5.12_Build
   

4. Configure Memory-Mapped I/O Space
   • Allocated low memory-mapped I/O space:

       PS C:\Users\Administrator> Set-VM -LowMemoryMappedIoSpace 3Gb -VMName VM_4.5.12_Build
     

   • Allocated high memory-mapped I/O space:

       PS C:\Users\Administrator> Set-VM -HighMemoryMappedIoSpace 33280Mb -VMName VM_4.5.12_Build
     

5. Dismount the GPU from the Host
   The GPU was removed from the host using the following command:

    PS C:\Users\Administrator> Dismount-VMHostAssignableDevice -LocationPath "PCIROOT(9E)#PCI(0100)#PCI(0000)" -Force
   

6. Assign the GPU to the VM
   The GPU was assigned to the VM using:

    PS C:\Users\Administrator> Add-VMAssignableDevice -LocationPath "PCIROOT(9E)#PCI(0100)#PCI(0000)" -VMName VM_4.5.12_Build
   

7. Start the VM
   After completing the DDA setup, the VM was started:

    PS C:\Users\Administrator> Start-VM -Name VM_4.5.12_Build
   

8. Verify the GPU in the Guest OS
   Inside the guest OS, the following command was used to check if the GPU is visible:

    lspci | grep -i nvidia
   

   

Observed Issue:

• The GPU does not appear in the output of lspci inside the Linux VM, despite all configurations completing without errors.
• No errors or warnings were observed during the DDA setup process.

Additional Notes:

• NVIDIA drivers are installed on Windows server host.
• Data Center L4 Driver for Windows 560.94 | Windows Server 2022
  Driver Details | NVIDIA
• 

Answer 1

---

Hi All,

I wanted to provide an update regarding the issue I faced with NVIDIA L4 GPUs on Windows Server Hyper-V VMs using Discrete Device Assignment (DDA). I was able to resolve this issue successfully by following the guidance provided in the NVIDIA documentation:

Bug #2812853: Microsoft DDA not working with some GPUs.

The problem occurred because GPUs with more than 16 GB of memory require additional MMIO (Memory-Mapped Input/Output) space for proper mapping in the guest VM. Without this configuration, the GPU wouldn't be detected properly in the VM.

---

The workaround involves allocating sufficient HighMemoryMappedIoSpace for the VM based on the GPU's BAR1 memory size and the number of GPUs assigned to the VM. Here's the step-by-step process:

1. Use the following formula to calculate the required MMIO space:

    MMIO space=2×gpu-bar1-memory×assigned-gpus 
   

   Where:
   • gpu-bar1-memory: The amount of BAR1 memory for one GPU (equal to total GPU memory if not specified).
   • assigned-gpus: The number of GPUs assigned to the VM.
2. Assign the calculated MMIO space to the VM using the Set-VM PowerShell command on the Hyper-V host.

For a VM with 1 GPU assigned, where each GPU has 23 GB of BAR1 memory:

PowerShell Command:

Run the following on the Hyper-V host to set the required MMIO space:

Set-VM –HighMemoryMappedIoSpace 46GB –VMName <VM_Name>

For 3 NVIDIA L4 GPUs, each with 23 GB of memory assigned to a single VM:

MMIO space=2×23 GB×3=138 GB

Run the following to set the MMIO space for the VM:

Set-VM –HighMemoryMappedIoSpace 138GB –VMName <VM_Name>

Once the MMIO space is configured, reboot the VM and check that the GPU is recognized correctly in the guest VM using tools like lspci or nvidia-smi after installing the driver (for Linux VMs).

I hope this helps anyone facing similar issues. If you have additional questions, feel free to ask!

Best Regards,
Samadhan

Answer 2

Hi,

Please run the SurveyDDA.ps1 PowerShell script in an elevated PowerShell window to check if the server is configured correctly and devices are capable of Discrete Device Assignment.

Since the guest is a Debian VM, you may need to give a try to the kernels available in the Debian backports repository.

Backports - Debian Wiki

Answer 3

Please run the SurveyDDA.ps1 PowerShell script in an elevated PowerShell window to check if the server is configured correctly and devices are capable of Discrete Device Assignment.

Here is the result of SurveyDDA.ps1

Executing SurveyDDA.ps1, revision 1.txt

Since the guest is a Debian VM, you may need to give a try to the kernels available in the Debian backports repository.

Backports - Debian Wiki

I am using the Debian backported kernel 6.11 with Debian 12 Bookworm. On another Windows Server host with the same configuration,

DDA is working perfectly with NVIDIA RTX GPU models. I am able to use the NVIDIA RTX GPU in the Hyper-V Debian 12 guest OS.