Get started with debugging Hyper-V Virtual Switch 1
This is the first post about the Hyper-V Virtual Switch from debugging point of view.
The Hyper-V Virtual Switch is a software-based layer-2 Ethernet network switch that is available in Hyper-V Manager when you install the Hyper-V server role. The switch includes programmatically managed and extensible capabilities to connect virtual machines to both virtual networks and the physical network. In addition, Hyper-V Virtual Switch provides policy enforcement for security, isolation, and service levels. Hyper-V Virtual Switch starts from windows server 2008 through Windows server 2012 R2. Although it keeps evolving, the fundamental architecture and concepts remain the same all the same. In a series of posts here, I will focus on debugging Windows server 2012 R2 Hyper-V Virtual Switch with real samples and troubleshooting best practices.
In the first place, we should have some basic ideas about the Virtual Switch in Windows server 2012 R2(and legacy OS too)
- Virtual Switch is not standalone or out-of-box, it runs inside the host OS of Hyper-V for it is built on top of existing Windows OS libraries like NDIS. From windows point of view, Virtual Switch is merely a kernel driver (vmswitch.sys) running on Hyper-V host OS. So it actually shares the prominent resources like CPU/Memory etc. with the other components running on the same OS.
A handful of offloading technologies are developed to mitigate the impact and resource dependency on Hyper-V OS. For example, VMQ https://blogs.technet.com/b/networking/archive/2013/09/24/vmq-deep-dive-1-of-3.aspx is a typical solution to speed up the data transmit that bypasses the Virtual Switch to some extent or SR-IOV https://msdn.microsoft.com/en-us/library/windows/hardware/hh440148(v=vs.85).aspx that almost bypasses the virtual switch completely to save the precious CPU/memory resources on the host OS. - Virtual Switch is built on top of NDIS library. So we must get acquainted with the NDIS before digging into the virtual switch operation. Here is a great post about debugging NDIS https://blogs.msdn.com/b/ndis/archive/2010/03/05/getting-started-with-ndiskd.aspx. In addition, we also uses quite a few native Windows Debugger commands that facilitate the debugging purpose. If you are new to Windows kernel debugging, check out Ilias’s thorough tutorial https://blogs.msdn.com/iliast/archive/2006/12/10/windbg-tutorials.aspx. You should follow that tutorial to get your kernel debugger attached to another computer. While you can use either Windbg.exe or Kd.exe, I highly suggest using Windbg
Okay, let’s unfold the first post. When you do connect to a Hyper-V host via live kernel debugging or open a memory dump via Windbg for debugging, you can double-check that the NDIS symbols are correctly loaded. Use .reload /f vmswitch.sys to force the debugger to download the vmswitch.pdb symbol file. Then use the !lmi vmswitch and look for the magic phrase “Symbols loaded successfully”. If everything went according to plan, you should get output similar to this:
26: kd> .sympath https://msdl.microsoft.com/download/symbols
Symbol search path is: https://msdl.microsoft.com/download/symbols
Expanded Symbol search path is: https://msdl.microsoft.com/download/symbols
************* Symbol Path validation summary **************
Response Time (ms) Location
Deferred https://msdl.microsoft.com/download/symbols
26: kd> .reload /f vmswitch.sys
26: kd> !lmi vmswitch
Loaded Module Info: [vmswitch]
Module: vmswitch
Base Address: fffff8007607c000
Image Name: vmswitch.sys
Machine Type: 34404 (X64)
Time Stamp: 53905875 Thu Jun 05 19:45:57 2014
Size: ae000
CheckSum: af005
Characteristics: 22
Debug Data Dirs: Type Size VA Pointer
CODEVIEW 25, 6ae34, 6a034 RSDS - GUID: {5C41936C-CB8A-4FDD-A551-AAB433B8A341}
Age: 1, Pdb: vmswitch.pdb
CLSID 4, 6ae70, 6a070 [Data not mapped]
Image Type: MEMORY - Image read successfully from loaded memory.
Symbol Type: PDB - Symbols loaded successfully from image header.
C:\DevKit\wdk81\Debuggers\x64\sym\vmswitch.pdb\5C41936CCB8A4FDDA551AAB433B8A3411\vmswitch.pdb
Load Report: public symbols , not source indexed
C:\DevKit\wdk81\Debuggers\x64\sym\vmswitch.pdb\5C41936CCB8A4FDDA551AAB433B8A3411\vmswitch.pdb
Now that your debugger is set up correctly, we can start using nvkd. The first command we’ll try is !nvkd.vswitch
2: kd> !nvkd.help
Commands for C:\DevKit\wdk81\Debuggers\x64\WINXP\nvkd.dll:
!help - Displays information on available extension commands
!proc - Displays information about a given processor
!qosline - Displays information about a given QoS Line
!vmq - Displays information about a given VMQ (Virtual Machine Queue)
!vnbl - Displays Extensible Virtual Switch specific data of an NBL
!vnic - Displays information about one or more vnic(s)
!vport - Displays information about the given vport
!vswitch - Displays information about one of more vswitch(es) and child
vports
Then you can run command, !nvkd.vswitch to list out all virtual switches running on the Hyper-V host. It is what is displayed in debugger.
2: kd> !nvkd.vswitch
Type Address Flags FriendlyName
VSWITCH fffffa80035d2000 () External Demo
VSWITCH fffffa8009a32000 () Internal Demo
VSWITCH fffffa8007c4c000 () Private Demo
It is how it looks like in Hyper-V virtual switch management console:
So the debug extension !nvkd.vswitch depicts the virtual switches on the Hyper-V host and you can drill down with the virtual switch object address as below:
2: kd> !nvkd.vswitch fffffa80035d2000
VSWITCH fffffa80035d2000
Name: AC2DA35C-661A-40AE-82F2-1F62F70DBB2A
Friendly Name: External Demo
RefCount: 9
Extensibility
ProtocolState: 3 (VmsExtPtStateRunning)
MiniportState: 3 (VmsExtMpStateRunning)
DeviceId: {95141247-9BD1-4449-9223-C402789D94DD}
Miniport: fffffa8003fb01a0 !ndiskd.miniport fffffa8003fb01a0
Protocol Open: fffffa80043af010 !ndiskd.mopen fffffa80043af010
IovPreferred: 0
vPort List: Count (6)
Type Address Id Type IsValidation MonitorMode FriendlyName
VPORT fffffa8003d80000 0x0000 Default N None Internal
VPORT fffffa8003d7c000 0x0001 Internal N None External Demo
VPORT fffffa8003d7a000 0x0002 External N None Realtek PCI GBE Family Controller - Virtual Switch_External
VPORT fffffa8008b35000 0x0003 Synthetic Y None Dynamic Ethernet Switch Port
VPORT fffffa800416e000 0x0005 Synthetic N None Dynamic Ethernet Switch Port
VPORT fffffa800808c000 0x0004 Synthetic Y None Dynamic Ethernet Switch Port
2: kd> !nvkd.vswitch fffffa8009a32000
VSWITCH fffffa8009a32000
Name: E0BB29CA-1F43-4CAC-8025-BDFDA3FD3825
Friendly Name: Internal Demo
RefCount: 5
Extensibility
ProtocolState: 3 (VmsExtPtStateRunning)
MiniportState: 3 (VmsExtMpStateRunning)
DeviceId: {4E700BA5-09D3-41D3-8EA3-B2FCB3AC0741}
Miniport: fffffa80099b81a0 !ndiskd.miniport fffffa80099b81a0
Protocol Open: fffffa80091ec7d0 !ndiskd.mopen fffffa80091ec7d0
IovPreferred: 0
vPort List: Count (2)
Type Address Id Type IsValidation MonitorMode FriendlyName
VPORT fffffa80098cb000 0x0000 Default N None Internal Demo
VPORT fffffa8009542000 0x0001 Internal N None Internal Demo
2: kd> !nvkd.vswitch fffffa8007c4c000
VSWITCH fffffa8007c4c000
Name: D48F7DC3-D15D-4A3E-BF98-7E46FB8A1E4D
Friendly Name: Private Demo
RefCount: 4
Extensibility
ProtocolState: 3 (VmsExtPtStateRunning)
MiniportState: 3 (VmsExtMpStateRunning)
DeviceId: {C264D298-D597-4BEE-BB6E-3DF414901E6B}
Miniport: fffffa80089571a0 !ndiskd.miniport fffffa80089571a0
Protocol Open: fffffa8008483330 !ndiskd.mopen fffffa8008483330
IovPreferred: 0
vPort List: Count (1)
Type Address Id Type IsValidation MonitorMode FriendlyName
VPORT fffffa8008944000 0x0000 Default N None Private Demo
The beauty of debugger extension is it provides a neat and coherent view of all important attributes of virtual switch. This is just the beginning of our journey, Please stay tuned for more walk-through in next posts.
[Update] Unfortunately, vmswitch.sys public symbol doesn't include the necessary information for public debugger to work with. nvkd.dll will work partially to print out the vmswitch data now.