Winsock Connect and TCP SYN Attack Protection

TCP SYN attack protection has been available on Windows platforms since Windows 2000, but by default is turned off. This has changed for the Windows 2003 SP1 release as SP1 enables SYN attack protection by default (which is a good thing). However, once enabled it affects how TCP connections are accepted. Specifically it accepts how Winsock enforces the backlog parameter for TCP listening sockets.  Without SYN attack protection enabled, the TCP stack indicates an incoming connection to Winsock before the connection request is acknowleged (i.e. when the first SYN is received). At this point, Winsock can indicate to the TCP stack to reject the connection of the TCP backlog is exceeded. With SYN attack protection enabled, the TCP stack performs a full 3-way handshake and then indicates the connection to Winsock (assuming the stack does not detect a SYN attack). Once Winsock is notified of the connection it checks if the backlog has been exceeded and if it has, the connection is reset.

What this means for Winsock clients connecting to a SYN attack protected server is the initial connection request may succeed even though the TCP backlog has been exceeded, but the connection has been aborted and the very next Winsock call made by the client socket will fail with WSAECONNREST. This contrasts with the case where SYN attack protection is disabled and the connection request that exceeds the backlog will fail with WSAECONNREFUSED.

Applications should not rely on the fact that if the backlog is exceeded a connection request will fail with WSAECONNREFUSED. After all, the MSDN documentation for Winsock does state that the maximum backlog value is provider dependent. Robust applications should be preparted to handle failures at any point in the socket lifetime.