How to Troubleshoot Slow BITS Performance, Hosts 'Not Responding' and 'Needs Attention' Communication Issues
You find yourself with one of the following situations
1. Deployment or migration of data utilizing LAN (not SAN Transfer) is very slow in general
2. BITS is slow in one direction only: Fast from Host01 to Host02, but slow from Host02 to Host01 for example
3. Host reports in SCVMM Admin Console "Host not responding" or "Needs attention"
4. Under 'Status' of the Host the following may show "Not Responding"
- Connection status
- Agent Status
All of these issues can be related to networking components. Three common tasks can be performed to address this
1. Test for BITS errors
2. Verify GPO settings
3. Correct vendor specific and Windows system settings for ‘Task Offloading’ (This item will likely have the greatest effect)
Additional Items that just make sense
Verify all network settings on all involved SCVMM and Hyper-V systems. Set the network speed on the NICs and switches. Do not leave as auto negotiate. Also, remove all NIC teaming software. Network Card drivers currently do not always play nice with virtualization technologies. (Uninstall the teaming software. Disabling will not prevent drivers from loading at startup). If you cannot remove your teaming software at this time, at least make sure the namespace is available by typing ‘wmic nic’ at an elevated command prompt. If you get a screen full of data, you may be ok. If you get any kind of error, yank teaming software now and reboot.
1 - BITS Errors
Determine if BITS (Background Intelligent Transfer Service) is encountering errors. Attempt a P2V/Migration/Deployment transfer with SCVMM. While the transfer is occurring between servers, run the following commands on both servers:
Bitsadmin /list /allusers /verbose >bitsadmin.txt
Example BITS output below shows how long the transfer lasted (MODIFICATION TIME minus CREATED TIME). Retry delay shows that there were no errors. This is a clean run. If there are errors review the system and application event logs for errors that are BITS related. Involve your network team as well for a trace.
Example output…
BITSADMIN version 1.0 [ 5.1.2600.0 ]
BITS administration utility.
(C) Copyright 2000-2001 Microsoft Corp.
GUID: {BA568D81-B234-47F2-9BA3-686743F89794} DISPLAY:
https://cotoso.com:443/8272f226d8b8cd818406b28de9b6ad0f012b9a55/C:A5cProgramDataA5cM
icrosoftA5cWindowsA5cHyperVA5calbtest1A5cvmguest.iso/E:A5cVirtualMachinesA5calbtest
1A5cvmguest.iso
TYPE: UNKNOWN STATE: TRANSFERRED OWNER: NT AUTHORITY\SYSTEM
PRIORITY: FOREGROUND FILES: 1 / 1 BYTES: 42948608 / 42948608
CREATION TIME: 12/3/2008 9:04:48 AM MODIFICATION TIME: 12/3/2008 9:16:44 AM
COMPLETION TIME: 12/3/2008 9:16:44 AM
NOTIFY INTERFACE: UNREGISTERED NOTIFICATION FLAGS: 3
RETRY DELAY: 60 NO PROGRESS TIMEOUT: 0 ERROR COUNT: 0
PROXY USAGE: NO_PROXY PROXY LIST: NULL PROXY BYPASS LIST: NULL
DESCRIPTION: VMM LAN Transfer Job
JOB FILES:
42948608 / 42948608 WORKING
https://contoso.com:443/8272f226d8b8cd818406b28de9b6ad0f012b9a55/C:A5cProgramDataA5c
MicrosoftA5cWindowsA5cHyperVA5calbtest1A5cvmguest.iso/E:A5cVirtualMachinesA5calbtes
t1A5cvmguest.iso -> E:\VirtualMachines\albtest1\vmguest.iso
2 - Verify GPO settings
By default, BITS transfers are wide open, not throttled down. Follow the MDSN article and look for the following settings in case they have been changed from default values.
- Any setting with 'Peercache' in the name is not relevant
- The rest of the settings should be set to 'Not Configured' or 'Disabled' to ensure that BITS is not being throttled at all. 'Maximum network bandwidth for BITS background transfers' is most important to be set to 'Not Configured' or 'Disabled'
BITS uses the following Group Policies to configure BITS
https://msdn.microsoft.com/en-us/library/aa362844(VS.85).aspx
The group policies for BITS are located in the registry here…
HKLM\Software\Policies\Microsoft\Windows\BITS
(Note that only those policies that are configured are listed in the registry.)
REMEMBER: Run ‘gpupdate /force’ following any changes from an elevated command prompt
3 - Disable Task Offloading
This work can be done easily on a non-Core server by disabling all properties on the 'Advanced' tab of all NICs. You should then follow the manual directions below to verify the rest of the Offload related settings for the NICs, and to disable this feature in Windows.
If working with a core server you may want to connect via remote registry and make these changes.
This MDSN article lists all values available.
Using Registry Values to Enable and Disable Task Offloading
https://msdn.microsoft.com/en-us/library/aa938424.aspx
All values should be set to '0' EXCEPT for the last one which is related to IPSEC, and thus Kerberos. Leave this one alone. 'HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Ipsec\EnabledOffload'
A. Locate all physical NICs in the registry under:
'HKLM\System\CurrentControlSet\Control\Class\{4D36E972-E325-11CE-BFC1-08002BE10318}'
B. There will be folders with four digits, starting with '0000'. Locate the ones that show the physical NIC names on the right. Look for the 'DriverDesc' value on the right; this should have the name of a NIC such as 'HP NC360T PCIe Gigabit Server Adapter.' For each of these, make the changes below.
C. Disable All Vendor specific Offloading
Set values for any entries below that include the word 'Offload' to '0' (Disabled)
*FlowControl
No description available
*IPChecksumOffloadIPv4
Describes whether the device enabled or disabled the calculation of IPv4
checksums.
*TCPChecksumOffloadIPv4
Describes whether the device enabled or disabled the calculation of TCP Checksum
over IPv4 packets.
*TCPChecksumOffloadIPv6
Describes whether the device enabled or disabled the calculation of TCP checksum
over IPv6 packets.
*UDPChecksumOffloadIPv4
Describes whether the device enabled or disabled the calculation of UDP Checksum
over IPv4 packets.
*UDPChecksumOffloadIPv6
Describes whether the device enabled or disabled the calculation of UDP Checksum
over IPv6 packets.
*LsoV1IPv4
Describes whether the device enabled or disabled the segmentation of large TCP
packets over IPv4 for large send offload version 1 (LSOv1).
*LsoV2IPv4
Describes whether the device enabled or disabled the segmentation of large TCP
packets over IPv4 for large send offload version 2 (LSOv2).
*LsoV2IPv6
Describes whether the device enabled or disabled the segmentation of large TCP
packets over IPv6 for large send offload version 2 (LSOv2).
*IPsecOffloadV1IPv4
Describes whether the device enabled or disabled the calculation of IPsec headers
over IPv4.
*IPsecOffloadV2
Describes whether the device enabled or disabled IPsec offload version 2
(IPsecOV2). IPsecOV2 provides support for additional crypto-algorithms, IPv6, and
co-existence with large send offload version 2 (LSOv2).
*IPsecOffloadV2IPv4
Describes whether the device enabled or disabled IPsecOV2 for IPv4 only.
*RSS
Receive side scaling
*TCPUDPChecksumOffloadIPv4
Describes whether the device enabled or disabled the calculation of TCP or UDP
checksum over IPv4.
*TCPUDPChecksumOffloadIPv6
Describes whether the device enabled or disabled the calculation of TCP or UDP
checksum over IPv6.
D. Disable Offloading in Windows.
Use the following registry values to enable or disable task offloading for the TCP/IP protocol:
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\TCPIP\Parameters\DisableTaskOffload
Setting this DWORD value to ‘1’ disables all of the task offloads from the TCP/IP transport. Setting this value to zero enables all of the task offloads.
Related Links
Network Issues with Windows Server 2008 RDP and VS/Hyper-V on Dell Servers
https://www.petri.co.il/network-issues-with-windows-server-2008-rdp-on-dell-servers.htm
Technorati Tags: microsoft,scvmm,hyper-v,virtual,virtualization,windows
Comments
- Anonymous
January 01, 2003
thank you