Share via

The Cable Guy - August 2001

  • Article

Layer Two Tunneling Protocol in Windows 2000

TechNet's The Cable Guy

By The Cable Guy

Layer Two Tunneling Protocol (L2TP) is a combination of the Point-to-Point Tunneling Protocol (PPTP), a technology developed by Microsoft and other companies in the PPTP Forum, and Layer 2 Forwarding (L2F), a technology proposed by Cisco Systems, Inc. Rather than maintaining two incompatible tunneling protocols that compete in the marketplace, the Internet Engineering Task Force (IETF) mandated that the two technologies be combined into a single tunneling protocol that represents the best features of PPTP and L2F. L2TP is documented in RFC 2661.

L2TP encapsulates PPP frames that are sent over IP, X.25, Frame Relay, and ATM networks. When sent over an IP network, L2TP frames are encapsulated as User Datagram Protocol (UDP) messages, as shown in the following illustration.

If your browser does not support inline frames, click here to view on a separate page.

L2TP frames include L2TP connection maintenance messages and tunneled data. L2TP connection maintenance messages include only the L2TP header. L2TP tunneled data includes a PPP header and a PPP payload. The PPP payload can be encrypted or compressed (or both) using standard PPP encryption and compression methods.

For Microsoft operating systems, L2TP connections do not negotiate the use of PPP encryption through Microsoft Point-to-Point Encryption (MPPE). Instead, encryption is provided through the use of the Internet Protocol security (IPSec) Encapsulating Security Payload (ESP) header and trailer. The following illustration shows the result of using ESP on an IP datagram that contains an L2TP frame.

If your browser does not support inline frames, click here to view on a separate page.

The combination of L2TP for encapsulation and IPSec for encryption is known as L2TP/IPSec.

L2TP vs. PPTP

One of the choices to make when deploying Windows 2000-based VPNs is whether to use L2TP/IPSec or PPTP. Windows XP VPN client computers and Windows 2000 VPN client and server computers support both L2TP/IPSec and PPTP by default. However, you still must decide whether one or both are supported on your network.

L2TP/IPSec and PPTP are similar in the following ways:

  • They provide a logical transport mechanism to send PPP payloads.
  • They provide tunneling or encapsulation so that PPP payloads based on any protocol can be sent across an IP network.
  • They rely on the PPP connection process to perform user authentication and protocol configuration.

L2TP/IPSec and PPTP are different in the following ways:

  • With PPTP, data encryption begins after the PPP connection process (and, therefore, PPP authentication) is completed. With L2TP/IPSec, data encryption begins before the PPP connection process.
  • PPTP connections use MPPE, a stream cipher that is based on the Rivest-Shamir-Aldeman RC-4 encryption algorithm and provides 40, 56, or 128-bit encryption keys. Stream ciphers encrypt data as a bit stream. L2TP/IPSec connections use the Data Encryption Standard (DES), which is a block cipher that uses either a 56-bit key for DES or three 56-bit keys for 3-DES. Block ciphers encrypt data in discrete blocks (64-bit blocks, in the case of DES).
  • PPTP connections require only user-level authentication through a PPP-based authentication protocol. L2TP/IPSec connections require the same user-level authentication and, in addition, computer-level authentication through a computer certificate.

Advantages of L2TP

The following are the advantages of using L2TP in Windows 2000:

  • IPSec provides per-packet data origin authentication (proof that the data was sent by the authorized user), data integrity (proof that the data was not modified in transit), replay protection (prevention from resending a stream of captured packets), and data confidentiality (prevention from interpreting captured packets without the encryption key). By contrast, PPTP provides only per-packet data confidentiality.
  • L2TP/IPSec connections provide stronger authentication by requiring both computer-level authentication through certificates and user-level authentication through a PPP authentication protocol.
  • PPP packets exchanged during user-level authentication are never sent in an unencrypted form because the PPP connection process for L2TP/IPSec occurs after the IPSec security associations (SAs) are established. If intercepted, the PPP authentication exchange for some types of PPP authentication protocols can be used to perform offline dictionary attacks and determine user passwords. By encrypting the PPP authentication exchange, offline dictionary attacks are only possible after the encrypted packets have been successfully decrypted.

Because L2TP/IPSec requires the high security features of IPSec:

  • L2TP/IPSec requires a certificate infrastructure for issuing computer certificates to the VPN server computer and all VPN client computers. Computer certificates are needed to negotiate the IPSec security association. By contrast, PPTP can use password-based authentication protocols and does not require an installed certificate.
  • Windows 2000 and Windows XP VPN clients cannot be placed behind a network address translator (NAT) unless the L2TP/IPSec NAT-T Update for Windows XP and Windows 2000 is installed on the client. By contrast, PPTP traffic is supported by most NATs. IPSec NAT Traversal (NAT-T) is a new set of standards for sending IPSec traffic across a NAT and is supported by Microsoft L2TP/IPSec VPN Client, the Windows Server 2003 family, and L2TP/IPSec NAT-T Update for Windows XP and Windows 2000.

Notes

Although it is possible to configure Windows XP, Windows 2000, and Microsoft L2TP/IPSec VPN Client computers for pre-shared key IPSec authentication, it is not recommended. To configure computers running Windows XP for pre-shared key authentication, see Windows XP online Help. To configure computers running Windows 2000 for pre-shared key authentication, see either the "Virtual Private Networking" chapter of the Windows 2000 Server Resource Kit Internetworking Guide or Microsoft Knowledge Base article Q240262. To configure Microsoft L2TP/IPSec VPN Client computers for pre-shared key authentication, see Microsoft L2TP/IPSec VPN Client Help.

L2TP/IPSec does not obsolete the use of PPTP. L2TP/IPSec is a more secure VPN technology than PPTP. However, PPTP is still widely used and, when using the Microsoft Challenge-Handshake Authentication Protocol version 2 (MS-CHAP v2) authentication protocol and strong passwords, provides enough security for many situations.

Deploying Computer Certificates

When you configure Windows 2000 IPSec policy for certificate authentication, you can manually specify the list of root certification authorities (CAs) from which the IPSec peer will accept certificates. IPSec policy for L2TP connections does not have to be manually configured. It is automatically configured when you either initiate a VPN connection on the VPN client or start the Routing and Remote Access service on the VPN server.

However, for the IPSec policy that is automatically created for L2TP traffic, the list of CAs is not configurable. Instead, each L2TP/IPSec peer sends a list of root CAs to its IPSec peer (from which it accepts a certificate for authentication) that corresponds to the root CAs of installed computer certificates in its local machine store. For example, if Computer A was issued computer certificates by root CAs CertAuth1 and CertAuth2, it notifies its IPSec peer during main mode negotiation that it will accept certificates for authentication from only CertAuth1 and CertAuth2. If the IPSec peer, Computer B, does not have a valid computer certificate issued from either CertAuth1 or CertAuth2, IPSec security negotiation fails.

Ensure one of the following before attempting an L2TP/IPSec connection:

  1. Both the VPN client and VPN server were issued computer certificates from the same CA.
  2. Both the VPN client and VPN server were issued computer certificates from CAs that follow a certificate chain up to the same root CA.

In general, the VPN client must have a valid computer certificate installed that was issued by a CA that follows a valid certificate chain (from the issuing CA) up to a root CA that the VPN server trusts. Additionally, the VPN server must have a valid computer certificate installed that was issued by a CA that follows a valid certificate chain (from the issuing CA) up to a root CA that the VPN client trusts.

A single CA commonly issues computer certificates to all computers in an organization. Because of this, all computers within the organization have computer certificates from a single CA and request certificates for authentication from the same single CA.

Setting up a CA and configuring auto-enrollment

To install a computer certificate, a certification authority (CA) must be present to issue certificates. You can use the Certificate Service for Windows 2000 or a third-party certificate server.

After the Windows 2000 CA is configured, you can install a computer certificate in one of two ways:

  1. By configuring the automatic allocation of computer certificates to computers in a Windows 2000 domain.

    This method allows a single point of configuration for the entire domain. All members of the domain automatically receive the computer certificate through Group Policy during the logon process.

  2. By using Certificate Manager to obtain a computer certificate.

This method requires that each computer separately request a computer certificate from the CA.

Based on the certificate policies in your organization, you need to perform only one of these two allocations.

To configure a single CA as an enterprise root CA, perform the following steps:

  1. If necessary, promote the computer that will be a CA to domain controller. For more information, see the topic titled "Checklist: Installing a domain controller" in Windows 2000 Server online Help.
  2. Install the Windows 2000 Certificate Services component as an enterprise root CA. For more information, see the topic titled "Install an enterprise root certification authority" in Windows 2000 Server online Help.

To configure a Windows 2000 domain to automatically issue computer certificates to all domain members, see the topic titled "Configure automatic certificate allocation from an enterprise CA" in Windows 2000 Server online Help.

For a third-party certificate server, consult the documentation for the certificate server software for information on how to configure it for a Windows 2000 domain environment.

Obtaining a computer certificate

To obtain a computer certificate for a computer running Windows 2000 that is a member of the domain for which auto-enrollment is configured, restart the computer or type secedit /refreshpolicy machine_policy at a command prompt. For a computer running Windows XP, restart the computer or type gpupdate /target:computer at a command prompt.

To manually request a computer certificate, use the Certificate Manager snap-in. For more information, see the topics titled "Manage certificates for a computer" and "Request a certificate" in Windows 2000 Server online Help.

To obtain a certificate from a third-party certificate server, consult the documentation for the certificate server software for information on how to configure the server to issue certificates that can be imported into the machine store of a computer running Windows XP or Windows 2000.

To obtain a computer certificate for a computer using Microsoft L2TP/IPSec VPN Client, see Administrator's Guide to Microsoft L2TP/IPSec VPN Client.

Phases of an L2TP Connection

In order for an L2TP/IPSec connection attempt to be successful, it must successfully establish the following:

  1. IPSec SAs for L2TP traffic
  2. An L2TP connection
  3. A PPP connection

IPSec SA negotiation

IPSec SA negotiation for L2TP traffic consists of:

  1. Main mode negotiation

    The creation of a main mode SA by exchanging encryption key derivation information, determining the security of future main mode packets, and authenticating the IPSec peers.

  2. Quick mode negotiation

    The creation of a quick mode SA by determining the security of the data sent between the IPSec peers.

Because the IPSec policy and filter settings for L2TP data are automatically configured, the inability to establish an IPSec SA is most likely caused by the failure of the main mode SA negotiation because of misconfigured or missing computer certificates. Due to the secure nature of IPSec, the failure of the negotiation is not reported to the user.

Use the following troubleshooting tools to determine the cause of the main mode negotiation failure:

  1. Audit logging

    Audit logging is the logging of events that correspond to SA negotiation successes or failures in the Security log, which can be viewed with the Event Viewer snap-in. To enable audit logging, enable success and failure auditing for the Audit logon events audit policy for your domain or local computer group policy (available from Computer Configuration\Windows Settings\Security Settings\Local Policies\Audit Policy).

  2. Oakley logging

    Oakley logging is the logging of details about the main mode and quick mode SA negotiation process to the file SystemRoot\Debug\Oakley.log. To enable Oakley logging, set the HKEY_LOCAL_MACHINE \System \CurrentControlSet \Services \PolicyAgent\Oakley\EnableLogging registry setting to 1. Next, either restart the VPN server computer or run the following commands at the command prompt:

    net stop remoteaccess

    net stop policyagent

    net start policyagent

    net start remoteaccess

L2TP connection negotiation

After the main mode and quick mode SAs are established, the VPN client and VPN server exchange a series of L2TP messages that are used to create an L2TP control connection and session. For more information about this process, see Section 5.0 of RFC 2661. The following misconfigurations of the WAN Miniport (L2TP) device of the Routing and Remote Access service are most often the reason for a failure to create the L2TP control connection or session:

  • The WAN Miniport (L2TP) device is configured with too few ports. You have either five or 128 L2TP ports available by default, depending on your choices in the Routing and Remote Access Server Setup Wizard. Verify that you have enough L2TP ports for the maximum number of simultaneous L2TP connections to the VPN server.
  • The WAN Miniport (L2TP) device is not enabled to allow inbound remote access connections. Again, this is the result of choices made in the Routing and Remote Access Server Setup Wizard. Verify that the WAN Miniport (L2TP) device is configured to allow inbound remote access connections.

Both the number of ports and the enabling of inbound remote access connections for the WAN Miniport (L2TP) device are configured from the properties of the Ports object in the Routing and Remote Access snap-in.

For more information about the choices in the Routing and Remote Access Server Setup Wizard and their results, see Configuring the Routing and Remote Access Service in Windows 2000.

PPP connection negotiation

PPP connection negotiation for L2TP connections consists of the following phases:

  1. Negotiation of the PPP link through the Link Control Protocol (LCP)
  2. Authentication of the user attempting to connect
  3. Negotiation and configuration of LAN protocols, such as TCP/IP, using PPP network control protocols (NCPs)

Troubleshooting the PPP portion of the connection attempt involves its authentication and authorization, and depends on the dial-in properties of the user account, the domain and VPN server configurations, and remote access policies.

One tool that is very useful for troubleshooting PPP connections is PPP logging. PPP logging records the series of programming functions and PPP control messages during a PPP connection attempt. To enable PPP logging, select the Enable Point-to-Point Protocol (PPP) logging option on the PPP tab on the properties of a VPN server in the Routing and Remote Access snap-in.

For more information about troubleshooting VPN connections, see Troubleshooting VPNs in the Windows 2000 Server Resource Kit.

For More Information

For more information about Microsoft support for VPNs, consult the following resources:

For a list of all The Cable Guy articles, click here.