Why doesn't Windows 2008 server negotiate TCP MSS smaller than 536 bytes?
Hi,
In today's blog, I'll talk about an MTU issue that occurs on Windows Vista onwards (Vista/7/2008/2008 R2).
One of our customers reported that their SMTP server (running on Windows 2008) was failing to send e-mails to certain remote SMTP servers because e-mail delivery was disrupted at transport layer.
After analyzing the network trace collected on the source Windows 2008 Server, we found out that the remote system was offering a TCP MSS size of 512 bytes and Windows 2008 server kept sending the data packets with an MSS size of 536 bytes. As a result, those packets weren't succesfully delivered to the remote system. You can find more details about the problem and root cause below:
Note: IP addresses and mail server names are deliberately changed.
Source SMTP server: 10.1.1.1 - mailgateway.contoso.com
Target SMTP server: 10.1.1.5 - mailgateway2.contoso.com
a) Source SMTP server establishes TCP 3-way handshake with the target SMTP server. Source server suggests an MSS size of 1460 bytes and the target suggests an MSS size of 512 bytes:
No. Time Source Destination Protocol Info
1 0.000000 10.1.1.1 10.1.1.5 TCP 28474 > 25 [SYN] Seq=0 Win=8192 Len=0 MSS=1460 WS=8
2 0.022001 10.1.1.5 10.1.1.1 TCP 25 > 28474 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=512
3 0.000000 10.1.1.1 10.1.1.5 TCP 28474 > 25 [ACK] Seq=1 Ack=1 Win=65392 Len=0
b) Then data starts flowing. Under normal circumstances, the minimum of MSS will be selected as the MSS of the given TCP session by both parties and it will be used throughout the session.
No. Time Source Destination Protocol Info
4 0.075005 10.1.1.5 10.1.1.1 SMTP S: 220 mailgateway2.contoso.com ESMTP Tue, 20 Apr 2010 15:18:42 +0200
5 0.001000 10.1.1.1 10.1.1.5 SMTP C: EHLO Mailgateway.contoso.com
6 0.021001 10.1.1.5 10.1.1.1 SMTP S: 250-mailgateway2.contoso.com Hello Mailgateway.contoso.com [10.1.1.1] | 250-SIZE 26214400 | 250-PIPELINING | 250 HELP
7 0.001000 10.1.1.1 10.1.1.5 SMTP C: MAIL FROM:<postmaster@contoso.com> SIZE=2616 | RCPT TO:<test@test.abc.com>
8 0.183011 10.1.1.5 10.1.1.1 SMTP S: 250 OK | 250 Accepted
9 0.000000 10.1.1.1 10.1.1.5 SMTP C: DATA
10 0.022001 10.1.1.5 10.1.1.1 SMTP S: 354 Enter message, ending with "." on a line by itself
c) Even though an MSS size of 512 should be commonly agreed by both parties, Windows 2008 server doesn't seem to be using that value and keeps sending data with an MSS of 536 bytes:
No. Time Source Destination Protocol Info
11 0.294017 10.1.1.1 10.1.1.5 SMTP C: Message Body, 536 bytes
d) Most likely the TCP segment with 536 bytes of data doesn't arrive at the target server and we don't get a TCP ACK back as a result so we start TCP packet retransmissions:
No. Time Source Destination Protocol Info
12 0.600034 10.1.1.1 10.1.1.5 SMTP [TCP Retransmission] C: Message Body, 536 bytes
13 0.190011 10.1.1.5 10.1.1.1 SMTP [TCP Retransmission] S: 354 Enter message, ending with "." on a line by itself
14 0.000000 10.1.1.1 10.1.1.5 TCP [TCP Dup ACK 12#1] 28474 > 25 [ACK] Seq=649 Ack=269 Win=65124 Len=0
15 1.010058 10.1.1.1 10.1.1.5 SMTP [TCP Retransmission] C: Message Body, 536 bytes
16 2.400137 10.1.1.1 10.1.1.5 SMTP [TCP Retransmission] C: Message Body, 536 bytes
17 4.800274 10.1.1.1 10.1.1.5 SMTP [TCP Retransmission] C: Message Body, 536 bytes
e) Finally the source server closes the TCP session as it fails to successfully deliver the 536 bytes TCP segment to the target system:
No. Time Source Destination Protocol Info
18 9.600550 10.1.1.1 10.1.1.5 TCP 28474 > 25 [RST, ACK] Seq=649 Ack=269 Win=0 Len=0
The same problem doesn't happen if the source server is a Windows 2003 server.
After explaining the problem, now let's try to understand the root cause:
This issue stems from the fact that Windows Vista onwards systems don't accept an MTU size lower than 576 bytes:
TCP/IP Registry Values for Microsoft Windows Vista and Windows Server 2008
https://www.microsoft.com/downloads/details.aspx?familyid=12AC9780-17B5-480C-AEF7-5C0BDE9060B0&displaylang=en
MTU
Key: Tcpip\Parameters\Interfaces\interfaceGUID
Value Type: REG_DWORD—number
Valid Range: From 576 to the MTU of the underlying network
Default: 0xFFFFFFFF
Description: This value overrides the default Maximum Transmission Unit (MTU) for a network interface. The MTU is the maximum IP packet size, in bytes, that can be transmitted over the underlying network. For values larger than the default for the underlying network, the network default MTU is used. For values smaller than 576, the MTU of 576 is used. This setting only applies to IPv4.
Note: Windows Vista TCP/IP uses path MTU (PMTU) detection by default and queries the network adapter driver to find out what local MTU is supported. Altering the MTU value is typically not necessary and might result in reduced performance.
Since minimum MTU that could be used by a Window Vista onwards system is 576 bytes, a TCP MSS (maximum segment size) should be 536 bytes at miminum so that's why Windows 2008 source server tries to send TCP segments with 536 bytes of data. TCP MSS value is calculated as follows:
TCP MSS = IP MTU - IP header size (20 bytes by default) - TCP header size (20 bytes by default)
Hope this helps
Thanks,
Murat
Comments
Anonymous
January 01, 2003
Hi James, Thanks for your feedback. Unfortunately there's no solution to this if the source server is Windows 2008 onwards. Such systems don't use any MTU values smaller than 576 bytes and this is hardcoded: MTU Key: TcpipParametersInterfacesinterfaceGUID Value Type: REG_DWORD—number Valid Range: From 576 to the MTU of the underlying network Default: 0xFFFFFFFF Description: This value overrides the default Maximum Transmission Unit (MTU) for a network interface. The MTU is the maximum IP packet size, in bytes, that can be transmitted over the underlying network. For values larger than the default for the underlying network, the network default MTU is used. For values smaller than 576, the MTU of 576 is used. This setting only applies to IPv4. The best way forward would be to change the MTU size at the target system (provided that you have control over the target). If the target system is a public system, the best option would be to use Windows 2003 server where a smaller MTU size could be configured. (Please note that Windows 2003 also doesn't accept an MTU size smaller than 576 bytes if it's dynamically learnt via ICMP but at least you have the option to manually configure the MTU on the interface to a value smaller than 576 bytes). You can see the below links for further information: blogs.technet.com/.../do-you-still-set-enablepmtudiscovery-to-0.aspx support.microsoft.com/.../898060 -> It's allowed to configuer the MTU as low as 68 bytes on Windows 2003: (From KB898060) Key: HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParametersInterfacesID for Adapter Value Type: REG_DWORD Number Valid Range: 68 to the MTU of the underlying network Default: 0xFFFFFFFF In short, there's no solution to this on Vista/7/2008/2008 R2 if you don't have a chance to make any modifications at the target system. Thanks, MuratAnonymous
January 01, 2003
Hi Murat - Thanks for this very detailed explanation! I am experiencing a very similar issue, but with web traffic instead of SMTP traffic. Were you able to resolve this at all? Thanks, JamesAnonymous
March 27, 2012
Why are the packets dropped? Who's dropping them in your case? We have the same problem since we migrated to W2k8, but we have no control over the end devices. It works in a controlled environment, but not in production, I suspect that some firewall is dropping them because of mss mistmatch.Anonymous
March 28, 2012
Hi again, In our case, it was a policy in a load balancer that caused the packets to be droppedAnonymous
April 03, 2012
classic issue , the loadbalancer is probably filtering icmp type 3 code 4 , anyone expericed this issue on blue coat caches ?Anonymous
August 14, 2014
The RFCs defines the minimum segment size.
This may be of assistance. http://tools.ietf.org/html/rfc879