Choosing the IPSec Protocol

Applies To: Windows Server 2003, Windows Server 2003 R2, Windows Server 2003 with SP1, Windows Server 2003 with SP2

Windows Server 2003 IPSec protects data sent across the network by encapsulating the original payload with either an IPSec header (ESP or AH) for transport mode or an IPSec header and an additional IP header for tunnel mode. An IPSec protocol is often used to convert many different TCP and UDP ports used by applications and system services into one stream of secure traffic for the purpose of traversing an unsecured network. IPSec policy filters can control what protocols and ports are protected by IPSec.

Depending on which protocol is used, the entire original packet can be encrypted, encapsulated, or both. There are two IPSec protocols, Authentication Header (AH) and Encapsulating Security Payload (ESP).

AH

AH provides data origin authentication, data integrity, and anti-replay protection for the entire packet (both the IP header and the data payload carried in the packet), except for the fields in the IP header that are allowed to change in transit. It does not provide data confidentiality, which means that it does not encrypt the data. The data is readable, but protected from modification.

ESP

ESP provides data origin authentication, data integrity, anti-replay protection, and the option of confidentiality for the IP payload only. ESP in transport mode does not protect the entire packet with a cryptographic checksum. The IP header is not protected. ESP in tunnel mode encapsulates and secures the entire original packet, including the IP header (but not the outer IP header). Windows Server 2003 supports IPSec NAT-T, and ESP is the only protocol that can be used to traverse network address translators. The IKE protocol automatically detects the presence of a network address translator and proposes UDP-ESP encapsulation, rather than ESP, to allow IPSec traffic to pass through the network address translator. If only AH is specified as a security method in the filter action, the security negotiation will fail. ESP supports the option of null encryption when network address translator traversal, and not encryption, is needed.

Use Table 6.7 to choose which protocol to use.

Table 6.7   Choosing IPSec Protocol Types

Protocol	Requirement	Usage
AH	The data and the header need to be protected from modification and authenticated, but remain readable.	Use for data integrity in situations where data is not secret but must be authenticated — for example, where access is enforced by IPSec to trusted computers only, or where network intrusion detection, QoS, or firewall filtering requires traffic inspection.
ESP	Only the data needs to be protected by encryption so it is unreadable, but the IP addressing can be left unprotected.	Use when data must be kept secret, such as file sharing, database traffic, RADIUS protocol data, or internal Web applications that have not been adequately secured by SSL.
Both AH and ESP	The header and data, respectively, need to be protected while data is encrypted.	Use for the highest security. However, there are very few circumstances in which the packet must be so strongly protected. When possible, use ESP alone instead.

Using both AH and ESP is the only way to both protect the IP header and encrypt the data. However, this level of protection is rarely used because of the increased overhead that AH would incur for packets that are already adequately protected by ESP. ESP protects everything but the IP header, and modifying the IP header does not provide a valuable target for attackers. Generally, the only valuable information in the header is the addresses, and these cannot be spoofed effectively because ESP guarantees data origin authentication for the packets. In addition, some IPSec hardware offload adapters do not support the use of AH and ESP on the same packet. Hardware offload network adapters can accelerate IPSec processing by performing hardware offload of IPSec cryptographic functions. If you are using such offload adapters, determine the protocol support that they provide before selecting an IPSec protocol to use.