Tunnel mode

Applies To: Windows Server 2003, Windows Server 2003 R2, Windows Server 2003 with SP1, Windows Server 2003 with SP2

Tunnel mode

When IPSec tunnel mode is used, IPSec encrypts the IP header and the payload, whereas transport mode only encrypts the IP payload. Tunnel mode provides the protection of an entire IP packet by treating it as an AH or ESP payload. With tunnel mode, an entire IP packet is encapsulated with an AH or ESP header and an additional IP header. The IP addresses of the outer IP header are the tunnel endpoints, and the IP addresses of the encapsulated IP header are the ultimate source and destination addresses.

IPSec tunnel mode is useful for protecting traffic between different networks, when traffic must pass through an intermediate, untrusted network. Tunnel mode is primarily used for interoperability with gateways, or end-systems that do not support L2TP/IPSec or PPTP connections. You can use tunnel mode in the following configurations:

• Gateway-to-gateway

• Server-to-gateway

• Server-to-server

AH tunnel mode

As shown in the following illustration, AH tunnel mode encapsulates an IP packet with an AH and IP header and signs the entire packet for integrity and authentication.

ESP tunnel mode

As shown in the following illustration, ESP tunnel mode encapsulates an IP packet with both an ESP and IP header and an ESP authentication trailer.

The signed portion of the packet indicates where the packet has been signed for integrity and authentication. The encrypted portion of the packet indicates what information is protected with confidentiality.

Because a new header for tunneling is added to the packet, everything that comes after the ESP header is signed (except for the ESP authentication trailer) because it is now encapsulated in the tunneled packet. The original header is placed after the ESP header. The entire packet is appended with an ESP trailer before encryption occurs. Everything that follows the ESP header, except for the ESP authentication trailer, is encrypted. This includes the original header which is now considered to be part of the data portion of the packet.

The entire ESP payload is then encapsulated within the new tunnel header, which is not encrypted. The information in the new tunnel header is used only to route the packet from origin to tunnel endpoint.

If the packet is being sent across a public network, it is routed to the IP address of the gateway for the receiving intranet. The gateway decrypts the packet, discards the ESP header, and uses the original IP header to route the packet to the intranet computer.

ESP and AH can be combined when tunneling, providing both confidentiality for the tunneled IP packet and integrity and authentication for the entire packet.

Using IPSec tunnels

IPSec tunnels provide security for IP traffic only. The tunnel is configured to protect traffic between either two IP addresses or two IP subnets. If the tunnel is used between two computers instead of two gateways, the IP address outside the AH or ESP payload is the same as the IP address inside the AH or ESP payload. In Windows XP and the Windows Server 2003 family, IPSec does not support protocol-specific or port-specific tunnels. You can configure tunnels by using the IP Security Policy Management and Group Policy consoles to configure and enable two rules:

1. A rule for the outbound traffic for the tunnel.

   The rule for the outbound traffic is configured with a filter list that describes the traffic to be sent through the tunnel and a tunnel endpoint of an IP address configured on the IPSec tunnel peer (the computer or router on the other side of the tunnel).

2. A rule for the inbound traffic for the tunnel.

   The rule for the inbound traffic is configured with a filter list that describes the traffic to be received through the tunnel and a tunnel endpoint of a local IP address (the computer or router on the local side of the tunnel).

Additionally, filter actions, authentication methods, and other settings need to be specified for each rule.

For conceptual information about IPSec policy tunnel settings, see Tunnel endpoint. For information about configuring an IPSec tunnel, see Specify an IPSec tunnel. For information about how tunneling is used for virtual private networking, see Virtual private networking with IPSec.