Jaa

TCP Offloading/Chimney & RSS…What is it and should I disable it?

  • Artikkeli

Having decided to start this blog to convey my experience with network analysis and troubleshooting, one subject instantly sprang to mind for my first post, TCP Chimney/Offloading.

I get asked about this so often I have a ready email of advice around what it is, and what I (note not Microsoft, although I think our official recommendation would likely mirror mine) recommend to customers around the use of it. My advice is based on thousands of customer cases I've handled over the years around this feature. I've therefore compiled what's hopefully a one-stop shop for all your TCP offloading needs. Apologies in advance if it's a bit wordy, but I've tried to convey everything I can around the subject for you.

So why do I get asked about it all the time? Well, let's start with what it is and what it does.

 

What is TCP Offloading/Chimney?

Starting when Windows Server 2003 SP1 was the current server OS, Microsoft released the Scalable Networking Pack https://support.microsoft.com/kb/912222/en-us

This turned on in the OS, three distinct things:

 

1.) RSS

Where multiple CPUs reside in a single computer, the Windows networking stack limits "receive" protocol processing to a single CPU. In essence, the old design of dealing with all incoming network traffic on a single processor core was starting to cause a bottleneck on newer multiprocessor systems. RSS resolves this issue by enabling the packets that are received from a network adapter to be balanced across multiple CPUs. In essence with RSS on, each incoming TCP connection is load balanced over the available cores, spreading the load and preventing a bottleneck from occurring. This is becoming a necessity as servers have to handle increasingly high loads of network traffic.

 

2.) TCP Chimney (sometimes referred to as TCP Offloading)

This feature is designed to take processing of the network such as packet segmentation and reassembly processing tasks, from a computer's CPU to a network adapter that supports TCP Chimney Offload. This has the effect of reducing the workload on the host CPU and moving it to the NIC, allowing both the Host OS to perform quicker and also speed up the processing of network traffic.

 

3.) Network Direct Memory Access (NetDMA)

https://technet.microsoft.com/en-us/library/gg162716(WS.10).aspx

The NetDMA interface provides generic access to direct memory access (DMA) engines that can perform memory-to-memory data transfers with little CPU involvement. Again, this is designed to take work away from the CPU by allowing the NIC to move data from receive buffers without using the CPU as much.

Why would I want to disable it then?

All these features sound brilliant, and only enabled with the installation of the Scalable Networking Pack, so why would you want to disable it?

Well, with the release of Service Pack 2 for Windows Server 2003, Microsoft decided to include this scalable networking pack and thus turn the features on. If a server has a NIC which supports these features, and it's enabled in the NIC properties (more on this later) then we'll use them. The problem with this approach was, that many NICs reported to the OS that they supported these features, which they indeed did, but many didn't perform these functions very well at all in reality. With 2003 this was an all or nothing effort, if offloading was turned on, we offloaded all TCP connections on a supported NIC, regardless of whether it would benefit or not.

We had a great number of issues through Microsoft support over the course of the next year or so which were caused by drivers misbehaving with these features which had a knock on effect on the network traffic. This in turn caused a wide range of weird and wonderful symptoms seen across the board from the Exchange team to SQL to BizTalk to IIS to ISA. Most of these landed on the lap of my colleagues and I in the Networking support team and as such we've probably seen issues numbering in the thousands caused by these features, where turning them off resolved the problem, so much so that turning off Offloading/RSS became an almost standard troubleshooting step with 2003 cases.

Most NIC vendors have released numerous updates over the years to resolve these issues as have Microsoft to improve the feature in the OS, however my general advice with Server 2003 is to disable these features.

In essence, with server 2003 this feature tends to cause more problems than it solves so I tend to recommend it is turned off. It doesn't really become effective until you get high speed (10gbps) networks and low latency and, as mentioned, many older NIC drivers don't implement this well and thus we get a lot of issues. If enabled, the bare minimum recommendation would be that the NIC and teaming drivers and firmware are the latest available and https://support.microsoft.com/kb/950224/en-us is applied.

 

How do I turn off these features in Server 2003?

If the features aren't needed then they should be turned off, we can do this in one of two ways.

The safest way is to do this in the registry by using method 3 in https://support.microsoft.com/kb/948496/en-us which sets the registry to configure these features to be off.

 

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

 

  • Right-click EnableTCPChimney, and then click Modify.
  • In the Value data box, type 0, and then click OK.
  • Right-click EnableRSS, and then click Modify.
  • In the Value data box, type 0, and then click OK.
  • Right-click EnableTCPA, and then click Modify.
  • In the Value data box, type 0, and then click OK.
  • Exit Registry Editor, and then restart the computer.

 

Alternatively you can disable the features in the NIC properties of most NICs, however the naming convention and exposure of these settings varies from NIC to NIC and also from driver to driver.

 

If the feature is disabled on either the NIC properties, or the registry, it's off, regardless if the other is on. This is why I recommend you use the registry as it won't be affected by driver updates etc and is much easier to control centrally.

 

So what about newer OS versions?

The first point of note is that Microsoft made a lot of effort in the newer OS versions to ensure the drivers were up to the job, and also a lot of improvements in the implementation of the features and TCP stack in the OS which makes the enabling of the features much safer in post 2003 OS versions.

 

Server 2008

In Windows Server 2008 (not R2) Offloading is turned off by default anyway. https://support.microsoft.com/kb/951037/en-us

You can enable it using the NIC properties or by using Netsh which is outlined in the link above. The offloading capabilities are more granular in 2008 than they were in 2003, we offload suitable network connections.

As before, it's always wise if using this feature that the latest NIC drivers and firmware are installed to ensure the NIC manufacturers latest updates are in place.

Ensure you have https://support.microsoft.com/kb/976035/en-us installed on top of SP2 to prevent an unexpected restart scenaro.

If the above steps are done, in my experience it's very safe to turn the feature on in 2008 if you feel it is needed.

To check the state of offloading you can run the following steps:

 

Run Netstat –t in a command prompt an you'll get the following output:

 

Active Connections

 

Proto Local Address Foreign Address State Offload State

TCP 127.0.0.1:52613 computer_name:52614 ESTABLISHED InHost

TCP 192.168.1.103:52614 computer_name:52613 ESTABLISHED Offloaded

 

InHost shows the connection is not offloaded and thus handled by the OS, Offloaded mean exactly that.

For those of you looking in memory dumps and wondering if these features are in use, you should be able to dump the registry keys used to set this by running x tcpip!*disable* in windbg for server 2003.

In this example, both RSS and TCP Chimney is disabled.

x tcpip!*disable*

b8f1a0d4 tcpip!DisableRSS = 1

b8f1a360 tcpip!DisableUserTOSSetting = 1

b8f1df34 tcpip!DisableMediaSenseEventLog = 0

b8f1a0d0 tcpip!DisableTCPChimney = 1

b8f1ae54 tcpip!DisableTaskOffload = 0

b8f1cdc0 tcpip!DisableLargeSendOffload = 0

b8f1a0b4 tcpip!DisableIPSourceRouting = 2

b8f1ae4c tcpip!DisableMediaSense = 0

b8f1a0ec tcpip!DisableUserTOS = 1

b8f01ff3 tcpip!DisableRouter (void)

b8f0c3b0 tcpip!IPDisableMediaSenseRequest (struct _IRP *, struct _IO_STACK_LOCATION *)

b8f106d6 tcpip!OlmDisableOffloadOnInterface (unsigned int)

b8f04d4b tcpip!IPDisableChimneyOffload (struct _IRP *, struct _IO_STACK_LOCATION *)

b8f048bf tcpip!IPDisableSniffer (struct _UNICODE_STRING *)

 

As these structures don't exist in 2008 + you'll need to use another command which im currently trying to confirm what the best method is and I'll update the blog with the info.

 

Server 2008 R2

With server 2008 R2 this feature is much more intelligent, it'll only offload when the conditions are right..as per https://technet.microsoft.com/en-us/library/gg162709(WS.10).aspx

Automatic. In automatic mode, TCP Chimney Offload considers offloading the processing for a connection only if the following criteria are met: the connection is established through a 10 Gbps Ethernet adapter, the mean round trip link latency is less than 20 milliseconds, and at least 130 KB of data has been exchanged over the connection. In automatic mode, the TCP receive window is set to 16 MB. Because the Windows stack has performance optimizations not found in Chimney-capable network adapters, automatic mode restricts offloads only to those connections that might receive the most benefit from it.

This is the default setting and I'd advise it's left as default. As always, ensure the latest NIC drivers/Firmware is installed to remove the risk of any known issues but in my experience taken from many thousands of customers, this feature is a real benefit to the OS. In fact I've seen multiple customers who have gotten into the habit of disabling these features in their OS build following the issues they experienced with Server 2003. I've been called out to look at performance issues and when we've re-enabled the features we notice a massive performance improvement.

If you are getting problems which are resolved by turning off TOE in 2008 R2, my first step would be to update the NIC driver and firmware as there are almost always updates for the NICs which resolve the majority of offloading issues I encounter.

If the problem persists, turning off Offloading is the wrong thing to do, raise a case with Microsoft and we'll help you get to the bottom of it, by having a policy of disabling these features, you are effectively restricting your Windows platform's network performance for the sake of one or two issues which could be investigated and resolved.

The performance improvement on certain connections is enormous and shouldn't be thrown away due to habit (i.e. the 2003 behaviour) or a few issues which haven't been fully investigated, quick fixes will eventually come round and bite you, in my personal experience.

To manage the settings in 2008 R2 the following KB gives more information on the Netsh commands available.

https://technet.microsoft.com/en-us/library/gg162682(v=ws.10).aspx

It's also advised to install https://support.microsoft.com/kb/2477730/en-usto resolve an issue with offloading in 2008 R2, this is non urgent so could be planned into your next change window.

Server 2012

Offloading in Windows Server 2012 works much as it does in server 2008 R2 so the same advice applies. RSS however becomes more important in this OS due to the fact SMB Multichannel relies on it.

https://support.microsoft.com/kb/2846837/en-us is a recommended hotfix for RSS in server 2012.

Additional points of note around offloading:

Large send offload and checksum offload

I've seen many references on the internet pointing to things around TCP task offloading, such as Checksum offloading and Large Send offload being related to TCP chimney. Its important to note, these are not related to the TCP Chinmey/offload described above. Checksum offload is where we allow the NIC to set the checksum on a packet when it leaves the machine (which is why Netmon and Wireshark often show "incorrect checksum" on packets as the driver which captures them sits above the NIC where the checksum is set). Large Send offload (LSO) allows the Application layer to dump down a packet which would be too big for transmission and allows the NIC to chop it up into transmittable sizes (which is why you can see packets > 1460 bytes of payload in Netmon/Wireshark).

These can be set in the NIC properties but are generally very very safe to leave on. You may want to disable LSO if you're sniffing traffic as you wont be seeing the packets as they are transmitted on the wire.

 

 

Network tracing offloaded connections:

Another reason you may want to disable TCP offloading is if you want to take a network trace. Both Netmon's filter driver and Wireshark's will show you only the three way handshake and the session tear down if offloading is being used. This is due to where the drivers sit, when offloading is used, the data bypasses these drivers so you'll only see the part of the session the OS is responsible for, the session setup and tear down.

https://blogs.technet.com/b/messageanalyzer/archive/2012/09/17/meet-the-successor-to-microsoft-network-monitor.aspx is a new tool from Microsoft which allows us to trace at different layers other than NDIS (where netmon sits) and thus may allow you to work round this issue depending on the scenario.

So for a short summary my recommendations for Offloading and RSS are:

Server OS version

RSS/Chimney On by default

Recommended setting

Methods to disable

Additional Recommendations

2003 SP2

Yes

Turn off unless needed

NIC properties or registry

Update NIC drivers and apply

https://support.microsoft.com/kb/950224/en-us

2008

No

Turn on only if needed

Nic properties or Netsh

Update NIC drivers and apply

https://support.microsoft.com/kb/979614/en-us

https://support.microsoft.com/kb/967224/en-us

if enabled

2008 R2

Yes (Only offloads suitable connections)

Leave enabled

NIC properties or Netsh

Update NIC drivers and apply

https://support.microsoft.com/kb/2958399

https://support.microsoft.com/kb/2511305

2012 inc R2

Yes (Only offloads suitable connections)

Leave enabled

Nic properties or Netsh

Update NIC drivers and apply

https://support.microsoft.com/kb/2885978

Please note, the hotfixes may seem irrelevant but are the latest versions (as of 19 August 2014) of the relevant binaries which contain the code for handling RSS and Offloading and thus contain any hotfixes to this date which may help performance & functionality in this area.

  • Server 2003: Turn it off unless absolutely needed
  • Server 2008: Off by default, Turn on if needed after a NIC driver update and Windows hotfix.
  • Server 2008 R2/2012: On, Automatic mode by default, leave as default and update NIC drivers if possible. Install hotfix on 2008 R2 in next change window.

 

*Update.

 

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2055853 is a recommended fix if you're using RSS and VMware.

 

Hopefully this post has provided you with all you need to know around TCP offloading/RSS and has encouraged you to keep it turned on in newer OS versions.

Comments

  • Anonymous
    January 01, 2003
    The comment has been removed

  • Anonymous
    January 01, 2003
    The comment has been removed

  • Anonymous
    January 01, 2003
    Thanks for the great article and recommendations. It's the best one I've found on the subject to date. I was curious however regarding recommendations for NetDMA. You only mention it briefly but I see many less in-depth articles on the net saying it should be turned it off (usually in combination with Chimney and RSS). The Technet description also states "You cannot use TCP Chimney Offload and NetDMA together". Can you give us further insight and/or recommendation on this subject?

  • Anonymous
    January 01, 2003
    Thank you

  • Anonymous
    January 01, 2003
    The comment has been removed

  • Anonymous
    January 01, 2003
    Hi Craig, you're correct in that key can disable task offload features such as IPSEC offload, checksum offload and large send offload. Although in newer OS versions, (Vista + if memory serves) that key is deprecated by 'netsh int ip set global taskoffload=disabled' and check with 'netsh in ip show offload a reboot is normally required to bring this into effect.
    ' I would strongly advise against disabling these settings for anything other than troubleshooting as it's likely to have an impact on performance. As this is normally done on a single server/client basis, I normally recommend it is done by the NIC driver so we can disable individual elements for testing.

  • Anonymous
    January 01, 2003
    Great post.

  • Anonymous
    January 29, 2014
    Have you any experience with OS streaming products like Citrix Provisioning server? Citrix suggests disabling TCP offloading in http://support.citrix.com/article/ctx127549. Any comments on using offloading in this scenario? Thanks.

  • Anonymous
    May 17, 2014
    Excellent information keep up your good work thanks.

  • Anonymous
    September 01, 2014
    Great article Paul, was browsing the net and stumbled upon this treasure. Everything in one place and on one page...nothing better than that!!!

  • Anonymous
    November 15, 2014
    DMA and dca aren't supported after w8.this create a huge issue since ram and had drive were always the bottleneck to begin with,could ms change MSI or msix value from 1 per socket to one per core.also,what will happen with xbox one (since it was built from the ground up for dca,DMA .

  • Anonymous
    March 09, 2015
    The comment has been removed

  • Anonymous
    March 11, 2015
    Thanks for a great post. Really helpful. Where does the term 'Chimney' come from and what does it relate to?

  • Anonymous
    March 12, 2015
    The comment has been removed

  • Anonymous
    November 12, 2015
    Thank you for this in-depth article. You have hotfix 2885978 listed for Server 2012 in the chart, but you mention 2846837 for RSS earlier in the article. Should both be applied? Neither hotfix mentions Server 2012 on the platform list on the Microsoft download page.

  • Anonymous
    November 12, 2015
    (I just realized I can log in here.) For a little more background, in September we installed new servers according to the recommendations of a vendor whose software we are configuring (JustWare). We are getting the following error about once every 20 to 60 minutes of use on the client end. The program will work, we just have to run the command again when we get the error. The software also runs slowly and we have frequent "Not Responding" delays at the top of the title bar. Here's the error: Server was unable to process request. ---> A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - The semaphore timeout period has expired.) ---> The semaphore timeout period has expired.

    Thanks again.

  • Anonymous
    December 03, 2015
    RESOLVED! If anyone else happens to have the same problem we were getting on our server, the fix for us was to set up a link aggregate (LAG) on our switch. On our Meraki (Cisco) switch, it was as easy as going to the "Switch ports", check the two ports that were plugged into our JustWare server, and then click Aggregate. https://docs.meraki.com/display/MS/Switch+Ports#SwitchPorts-LinkAggregation