What Is IPSec?

Applies To: Windows Server 2003, Windows Server 2003 R2, Windows Server 2003 with SP1, Windows Server 2003 with SP2

What Is IPSec?

In This Subject

• IPSec Scenarios

• IPSec Dependencies

• IPSec and ICF

• Related Information

Internet Protocol security (IPSec) is a framework of open standards for helping to ensure private, secure communications over Internet Protocol (IP) networks through the use of cryptographic security services. IPSec supports network-level data integrity, data confidentiality, data origin authentication, and replay protection. Because IPSec is integrated at the Internet layer (layer 3), it provides security for almost all protocols in the TCP/IP suite, and because IPSec is applied transparently to applications, there is no need to configure separate security for each application that uses TCP/IP.

IPSec helps provide defense-in-depth against:

• Network-based attacks from untrusted computers, attacks that can result in the denial-of-service of applications, services, or the network

• Data corruption

• Data theft

• User-credential theft

• Administrative control of servers, other computers, and the network.

You can use IPSec to defend against network-based attacks through a combination of host-based IPSec packet filtering and the enforcement of trusted communications.

IPSec is integrated with the Windows Server 2003 operating system and it can use the Active Directory directory service as a trust model. You can use Group Policy to configure Active Directory domains, sites, and organizational units (OUs), and then assign IPSec policies as required to Group Policy objects (GPOs). In this way, IPSec policies can be implemented to meet the security requirements of many different types of organizations.

This section describes the solution that IPSec is intended to provide by providing information about core IPSec scenarios, IPSec dependencies, and related technologies.

The following figure shows an Active Directory-based IPSec policy being distributed to two IPSec peers and IPSec-protected communications being established between those two peers.

Two IPSec Peers Using Active Directory-based IPSec Policy

The MicrosoftWindows implementation of IPSec is based on standards developed by the Internet Engineering Task Force (IETF) IPSec working group. For a list of relevant IPSec RFCs, see the “Related Information” section later in this subject.

IPSec Scenarios

IPSec is a general-purpose security technology that can be used to help secure network traffic in many scenarios. However, you must balance the need for security with the complexity of configuring IPSec policies. Additionally, due to a lack of suitable standards, IPSec is not appropriate for some types of connectivity. This section describes IPSec scenarios that are recommended, IPSec scenarios that are not recommended, and IPSec scenarios that require special consideration.

Recommended Scenarios for IPSec

IPSec is recommended for the following scenarios:

• Packet filtering

• End-to-end security between specific hosts

• End-to-end traffic through a Microsoft Internet Security and Acceleration (ISA) Server-secured network address translator

• Secure server

• Layer Two Tunneling Protocol (L2TP) over IPSec (L2TP/IPSec) for remote access and site-to-site virtual private network (VPN) connections

• Site-to-site IPSec tunneling with non-Microsoft IPSec gateways

Packet Filtering

IPSec can perform host-based packet filtering to provide limited firewall capabilities for end systems. You can configure IPSec to permit or block specific types of unicast IP traffic based on source and destination address combinations and specific protocols and specific ports. For example, nearly all the systems illustrated in the following figure can benefit from packet filtering to restrict communication to only specific addresses and ports. You can strengthen security by using IPSec packet filtering to control exactly the type of communication that is allowed between systems.

Filtering Packets by Using IPSec

As illustrated in this figure:

• The internal network domain administrator can assign an Active Directory-based IPSec policy (a collection of security settings that determines IPSec behavior) to block all traffic from the perimeter network (also known as a demilitarized zone [DMZ], demilitarized zone, or screened subnet).

• The perimeter network domain administrator can assign an Active Directory-based IPSec policy to block all traffic to the internal network.

• The administrator of the computer running Microsoft SQL Server on the internal network can create an exception in the Active Directory-based IPSec policy to permit structured query language (SQL) protocol traffic to the Web application server on the perimeter network.

• The administrator of the Web application server on the perimeter network can create an exception in the Active Directory-based policy to permit SQL traffic to the computer running SQL Server on the internal network.

• The administrator of the Web application server on the perimeter network can also block all traffic from the Internet, except requests to TCP port 80 for the HyperText Transfer Protocol (HTTP) and TCP port 443 for HTTPS (HTTP over Secure Sockets Layer/Transport Layer Protocol [SSL/TLS]), which are used by Web services. This provides additional security for traffic allowed from the Internet in case the firewall was misconfigured or compromised by an attacker.

• The domain administrator can block all traffic to the management computer, but allow traffic to the perimeter network.

You can also use IPSec with the IP packet-filtering capability or NAT/Basic Firewall component of the Routing and Remote Access service to permit or block inbound or outbound traffic, or you can use IPSec with the Internet Connection Firewall (ICF) component of Network Connections, which provides stateful packet filtering. However, to ensure proper Internet Key Exchange (IKE) management of IPSec security associations (SAs), you must configure ICF to permit UDP port 500 and port 4500 traffic needed for IKE messages.

End-to-End Security Between Specific Hosts

IPSec establishes trust and security from a unicast source IP address to a unicast destination IP address (end-to-end). For example, IPSec can help secure traffic between Web servers and database servers or domain controllers in different sites. As shown in the following figure, only the sending and receiving computers need to be aware of IPSec. Each computer handles security at its respective end and assumes that the medium over which the communication takes place is not secure. The two computers can be located near each other, as on a single network segment, or across the Internet. Computers or network elements that route data from source to destination are not required to support IPSec.

Securing Communications Between a Client and a Server by Using IPSec

The following figure shows domain controllers in two forests that are deployed on opposite sides of a firewall. In addition to using IPSec to help secure all traffic between domain controllers in separate forests, as shown in the figure, you can use IPSec to help secure all traffic between two domain controllers in the same domain and between domain controllers in parent and child domains.

Securing Communications Between Two Domain Controllers in Different Forests by Using IPSec

End-to-End Traffic Through an ISA-Secured Network Address Translator

Windows Server 2003 supports IPSec NAT Traversal (NAT-T). IPSec NAT-T allows traffic to be secured by IPSec and also to be translated by a network address translator. For example, you can use IPSec transport mode to help secure host-to-host traffic through a computer that is running ISA Server and that is functioning as a network address translator if ISA (or any other NAT device) does not need to inspect the traffic between the two hosts. IPSec transport mode is used to protect traffic between hosts and it can provide security between computers that are on the same local area network (LAN) or connected by private wide area network (WAN) links. In the following figure, a computer running Windows Server 2003 and Microsoft Internet Security and Acceleration (ISA) Server is functioning as a network address translator. The IPSec policy on Server A is configured to secure traffic to the IP address of Server B, while the IPSec policy on Server B is configured to secure traffic to the external IP address of the computer running ISA Server.

Securing Communications Through an ISA-Secured NAT by Using IPSec NAT-T

Secure Server

You can require IPSec protection for all client computers that access a server. In addition, you can set restrictions on which computers are allowed to connect to a server running Windows Server 2003. The following figure shows IPSec in transport mode securing a line of business (LOB) application server.

Securing an Application Server by Using IPSec

In this scenario, an application server in an internal corporate network must communicate with clients running Windows 2000 or Windows XP Professional; a Windows Internet Name Service (WINS) server, Domain Name System (DNS) server, and Dynamic Host Configuration Protocol (DHCP) server; Active Directory domain controllers; and a non-Microsoft data backup server. The users on the client computers are company employees who access the application server to view their personal payroll information and performance review scores. Because the traffic between the clients and the application server involves highly sensitive data, and because the server should only communicate with other domain members, the network administrator uses an IPSec policy that requires ESP encryption and communication only with trusted computers in the Active Directory domain.

Other traffic is permitted as follows:

• Traffic between the WINS server, DNS server, DHCP server, and the application server is permitted because WINS servers, DNS servers, and DHCP servers must typically communicate with computers that run on a wide range of operating systems, some of which might not support IPSec.

• Traffic between Active Directory domain controllers and the application server is permitted, because using IPSec to secure communication between domain members and their domain controllers is not a recommended usage.

• Traffic between the non-Microsoft data backup server and the application server is permitted because the non-Microsoft backup server does not support IPSec.

L2TP/IPSec for Remote Access and Site-to-Site VPN Connections

You can use L2TP/IPSec for all VPN scenarios. This does not require the configuration and deployment of IPSec policies. Two common scenarios for L2TP/IPSec are securing communications between remote access clients and the corporate network across the Internet and securing communications between branch offices.

Note

• Windows IPSec supports both IPSec transport mode and tunnel mode. Although VPN connections are commonly referred to as “tunnels,” IPSec transport mode is used for L2TP/IPSec VPN connections. IPSec tunnel mode is most commonly used to help protect site-to-site traffic between networks, such as site-to-site networking through the Internet.

L2TP/IPSec for remote access connections

A common requirement for organizations is to secure communications between remote access clients and the corporate network across the Internet. Such a client might be a sales consultant who spends most of the time traveling, or an employee working from a home office. In the following figure, the remote gateway is a server that provides edge security for the corporate intranet. The remote client represents a roaming user who requires regular access to network resources and information. An ISP is used as an example to demonstrate the path of communication when the client uses an ISP to access the Internet. L2TP/IPSec provides a simple, efficient way to build a VPN tunnel and help protect the data across the Internet.

Securing Remote Access Clients by Using L2TP/IPSec

L2TP/IPSec for site-to-site VPN connections

A large corporation often has multiple sites that require communication — for example, a corporate office in New York and a sales office in Washington. In this case, L2TP/IPSec provides the VPN connection and helps protect the data between the sites. In the following figure, the router running Windows Server 2003 provides edge security. The routers might have a leased line, dial-up, or other type of Internet connection. The L2TP/IPSec VPN tunnel runs between the routers only and provides protected communication across the Internet.

Establishing an L2TP/IPSec VPN Tunnel Between Sites

Site-to-Site IPSec Tunneling with Non-Microsoft Gateways

For interoperability with gateways or end systems that do not support L2TP/IPSec or Point-to-Point Tunneling Protocol (PPTP) VPN site-to-site connections, you can use IPSec in tunnel mode. When IPSec tunnel mode is used, the sending gateway encapsulates the entire IP datagram by creating a new IP packet that is then protected by one of the IPSec protocols. The following figure illustrates site-to-site IPSec tunneling.

Establishing an IPSec Gateway-to-Gateway Tunnel Between Sites

In this figure, traffic is being sent between a client computer in a vendor site (Site A) and a File Transfer Protocol (FTP) server at the corporate headquarters site (Site B). Although an FTP server is used for this scenario, the traffic can be any unicast IP traffic. The vendor uses a non-Microsoft IPSec-enabled gateway, while corporate headquarters uses a gateway running Windows Server 2003. An IPSec tunnel is used to secure traffic between the non-Microsoft gateway and the gateway running Windows Server 2003.

Scenarios for Which IPSec Is Not Recommended

IPSec policies can be quite complex to configure and manage. Additionally, IPSec can incur performance overhead to establish and maintain secure connections, and it can incur network latency. In some deployment scenarios, the lack of standard methods for user authentication and address assignment make IPSec an unsuitable choice. Because IPSec depends on IP addresses for establishing secure connections, you cannot specify dynamic IP addresses. It is often necessary for a server to have a static IP address in IPSec policy filters. In large network deployments and in some mobile user cases, using dynamic IP addresses at both ends of the connection can increase the complexity of IPSec policy design. For these reasons, IPSec is not recommended for the following scenarios:

• Securing communication between domain members and their domain controllers

• Securing all traffic in a network

• Securing traffic for remote access VPN connections using IPSec tunnel mode

Securing Communication Between Domain Members and their Domain Controllers

Using IPSec to help secure traffic between domain members (either clients or servers) and their domain controllers is not recommended because:

• If domain members were to use IPSec-secured communication with domain controllers, increased latency might occur, causing authentication and the process of locating a domain controller to fail.

• Complex IPSec policy configuration and management is required.

• Increased load is placed on the domain controller CPU to maintain SAs with all domain members. Depending on the number of domain members in the domain controller’s domain, such a load might overburden the domain controller.

Securing All Traffic in a Network

In addition to reduced network performance, using IPSec to help secure all traffic in a network is not recommended because:

• IPSec cannot secure multicast and broadcast traffic.

• Traffic from real-time communications, applications that require Internet Control Message Protocol (ICMP), and peer-to-peer applications might be incompatible with IPSec.

• Network management functions that must inspect the TCP, UDP, and other protocol headers are less effective, or cannot function at all, due to IPSec encapsulation or encryption of IP payloads.

Securing Traffic for Remote Access VPN Connections by Using IPSec Tunnel Mode

IPSec tunnel mode is not a recommended technology for remote access VPN connections, because there are no standard methods for user authentication, IP address assignment, and name server address assignment. Using IPSec tunnel mode for gateway-to-gateway VPN connections is possible using computers running Windows Server 2003. But because the IPSec tunnel is not represented as a logical interface over which packets can be forwarded and received, routes cannot be assigned to use the IPSec tunnel and routing protocols do not operate over IPSec tunnels. Therefore, the use of IPSec tunnel mode is only recommended as a VPN solution for site-to-site VPN connections in which one end of the tunnel is a non-Microsoft VPN server or security gateway that does not support L2TP/IPSec. Instead, use L2TP/IPSec or PPTP for remote access VPN connections.

IPSec Uses That Require Special Considerations

The following scenarios merit special consideration, because they introduce an additional level of complexity for IPSec policy configuration and management:

• Securing traffic over IEEE 802.11wireless networks

• Securing traffic in home networking scenarios

• Securing traffic in environments that use dynamic IP addresses

Securing Traffic Sent over 802.11 Networks

You can use IPSec transport mode to protect traffic sent over 802.11 wireless networks. However, IPSec is not the recommended solution for providing security for corporate 802.11 wireless LAN networks. Instead, it is recommended that you use either 802.11 Wired Equivalent Privacy (WEP) encryption or Wi-Fi Protected Access (WPA) and IEEE 802.1X authentication.

To use IPSec to help secure traffic sent over 802.11 networks, you must ensure that client computers and servers support IPSec. Configuration management and trust are also required on client computers and servers when IPSec is used. Because many computers on a network do not support IPSec or are not managed, it is not appropriate to use IPSec alone to protect all 802.11 corporate wireless LAN traffic.

Securing Traffic in Home Networking Scenarios

Although IPSec is not optimized for use in general home networking scenarios, when network security administrators deploy IPSec with appropriate scripts and support tools, it can be used effectively on home computers for specific scenarios.

IPSec can be used to connect home computers to a corporate intranet for remote access. Network security administrators can use scripts and support tools to deploy IPSec on the home computers of employees who require secure connectivity to the corporate network. For example, an administrator can use a Connection Manager profile to deploy an L2TP/IPSec-based VPN connection on home computers. Employees can then establish IPSec-secured connections across the Internet to the corporate network by using the VPN client built-in to Network Connections.

Note

• In some cases, non-Microsoft VPN or firewall clients might disable the IPSec service, which is required for IPSec to function. If you encounter this problem, it is recommended that you contact the VPN or firewall vendor.

IPSec is not recommended for end users in general home networking scenarios for the following reasons:

• The IPSec policy configuration user interface (IP Security Policy Management) is intended for professional network security administrators, rather than for end users. Improper policy configuration can result in blocked communications, and if problems occur, built-in support tools are not yet available to aid end users in troubleshooting.

• Some home networking applications use broadcast and multicast traffic, for which IPSec cannot negotiate security.

• Many home networking scenarios use a wide range of dynamic IP addresses.

• Many home networking scenarios involve the use of a network address translator. To use IPSec across a NAT, both IPSec peers must support IPSec NAT-T.

Securing Traffic in Environments That Use Dynamic IP Addresses

IPSec depends on IP addresses for establishing secure connections, and it is often necessary for a server to have a static IP address in IPSec policy filters. In large network deployments and in some mobile user cases, using dynamic IP addresses at both ends of the connection can increase the complexity of IPSec policy design.

IPSec Dependencies

There is no single optimal environment for IPSec. However, there are dependencies that are critical to the successful deployment of IPSec. This section describes how the following two IPSec dependencies affect the deployment of IPSec:

• Active Directory (if your deployment requires the use of Active Directory-based IPSec policies, rather than local IPSec policies)

• Successful mutual authentication

Active Directory

For organizations with large numbers of computers that must be managed in a consistent way, it is best to distribute IPSec policies by using Group Policy to configure Active Directory domains, sites, and organizational units (OUs), and then assigning IPSec policies as required to Group Policy objects (GPOs). Although you can assign local IPSec policies to computers that are not members of a trusted domain, distributing IPSec policies and managing IPSec policy configuration and trust relationships is much more time-consuming for computers that are not members of a trusted domain.

If you do use Active Directory-based IPSec policies, IPSec policy design and management must take into account the delays that result from the replication of Group Policy data from domain controllers to domain members. Often, the first step in troubleshooting a problem with IPSec connectivity is to determine whether the computer in question has the most current Group Policy assignment. To do this, you must be a member of the local Administrators group on the computer for which troubleshooting is being performed.

Successful Mutual Authentication

For IPSec-secured communications to be established, there must be successful mutual authentication between IPSec peers. IPSec requires the use of one of the following authentication methods: Kerberos version 5, an X.509 version 3 computer certificate issued by a public key infrastructure (PKI), or a preshared key. The two IPSec peers must use at least one common authentication method or communication will fail. Make sure that you choose an authentication method that is appropriate for your environment.

When you deploy IPSec to negotiate security for upper-layer protocols such as TCP connections, failures to communicate are often caused by the failure of IPSec to mutually authenticate the two communication endpoints. Authentication might succeed for some computers and fail for others due to issues within the authentication system itself (typically, Kerberos or public key certificates, rather than preshared keys, because preshared keys are not recommended). For these reasons, it is important to evaluate how the dependency of IPSec connectivity on authentication affects your environment and support practices. Additional training is recommended so that administrators can quickly determine whether a connectivity problem is caused by an IPSec authentication failure and understand how to investigate the authentication system.

IPSec and ICF

IPSec is similar to the ICF feature of Network Connections. However, there are important differences between these two technologies as well. It is important to understand the similarities and differences, so that you can deploy IPSec where it is truly needed and obtain the maximum benefits from the security that IPSec provides. This section describes similarities and differences between IPSec and ICF.

Microsoft ICF is a locally managed, stateful host firewall that, by default, discards all incoming packets except those sent in response to packets sent by the host. One primary difference between IPSec and ICF is that IPSec provides complex static filtering based on IP addresses, while ICF provides stateful filtering for all addresses on a network interface. For example, when you use IPSec, you can configure a filter to block all inbound traffic that is sent to a specific IP address over a specific protocol and port (for example, “Block all inbound traffic from the Internet to TCP port 135”). You can also configure exemptions to permit specific types of traffic from specific source IP addresses. When you use ICF, you can configure exemptions to permit traffic based solely on the port, regardless of the source IP address.

It is recommended that you use ICF when you want a firewall for a network interface that can be accessed through the Internet. It is recommended that you use IPSec when you want to secure traffic over upper-layer protocols or when you need to allow access only to a group of trusted computers. Note that it is easier to configure ICF to permit traffic over a certain port than it is to configure an IPSec policy.

IPSec is not a full-featured host firewall. However, it does provide the ability to centrally manage policies that can permit, block, or secure unicast IP traffic based on specific addresses, protocols, and ports. Some of the functions found in standard firewalls that IPSec does not provide include stateful inspection, application protocol awareness, intrusion detection, and packet logging. Although IPSec lacks some features of firewalls, the packet blocking and filtering it provides can be effective in helping to limit the spread of viruses and thwart specific attacks known to use specific ports. You can also use IPSec to prevent specific applications and services from being used on the network.

Related Information

The following resources contain additional information that is relevant to this section.

• Active Directory Collection

• How IPSec Policy Extension Works

• How TCP/IP Works

• How VPN Works

• How 802.11 Wireless Works

The following RFCs and Internet Drafts are relevant to IPSec. To find the RFCs, type the appropriate RFC number in the IETF RFC Database. To find the Internet Drafts, type the appropriate keyword in the IETF Internet Drafts database.

• RFC 2401: Security Architecture for the Internet Protocol

• RFC 2402: IP Authentication Header

• RFC 2406: IP Encapsulating Security Payload (ESP)

• RFC 2408: Internet Security Association and Key Management Protocol (ISAKMP)

• RFC 2409: The Internet Key Exchange (IKE)

• UDP Encapsulation of IPSec Packets (draft-ietf-ipsec-udp-encaps-02.txt)

• Negotiation of NAT-Traversal in the IKE (draft-ietf-ipsec-nat-t-ike-02.txt)