VPN types: Protocols and network topologies of IPsec VPNs

IPsec VPNs use a number of different security protocols. Learn these VPN types, their network topologies, and the services they provide.

The IPsec suite was developed to address some the fundamental security flaws IPv4. In order to address these problems,...

four services were provided: data transmission encryption, data integrity validation, data source authentication and data state integrity. In order to provide these services, a number protocols had to be to IPsec VPNs. In this VPN tutorial, you will learn about the protocols that make IPsec secure and the network topologies IPsec VPNs. You can also navigate the table contents to read other sections the VPN tutorial.

Table contents

An to IPsec VPNs

The IPsec VPN framework is a suite IETF standards that delivers secure transmission data over unsecured networks, like the Internet. IPsec VPNs provide protocols to secure communications at the Network Layer along with a mechanism for exchanging identity and security protocol management information. The IPsec suite was developed to address some the fundamental security flaws IPv4.

To address these vulnerabilities, the IETF has developed different protocol standard definitions. These standards provide four basic services:

  • Data transmission encryption: The originating host can encrypt packets prior to transmission.
  • Data integrity validation: The receiving host can authenticate each packet sent to ensure the original data that was transmitted was received.
  • Data source authentication: The originating host can mark packets, so the receiver can authenticate them.
  • Data state integrity: The originating and receiving hosts can mark packets, so any re-transmission the data stream can be detected and rejected (this is known as anti-replay).

IPsec VPNs use a number of different security protocols to provide these services. From a lower level, these protocols can be broken down into two different camps: packet protocols and service protocols. The packet protocols are used to provide data security services. There are two IPsec packet protocols: Authentication Header (AH) and Encapsulating Security Payload (ESP). There are a number of service protocols, but the primary one is the Internet Key Exchange protocol (IKE).

Below is a quick overview of the protocols commonly used in IPsec VPN implementations:

Authentication Header: AH, defined in IETF RFC 2402, supports IPsec data validation, authentication and integrity services. It does not support data encryption. AH is typically implemented by itself, but can be implemented alongside ESP. AH is used when we only need to ensure with whom we are exchanging data.

Encapsulating Security Payload: ESP, defined in IETF RFC 2406, supports IPsec data encryption, validation, authentication and integrity services. ESP can be implemented alone or with AH. While the AH header is pre-pended to the data payload portion of the IP packet, ESP encapsulates the entire data portion of the IP packet with a header and trailer.

Internet Security Association and Key Management Protocol (ISAKMP): These provide the framework and processes for implementing IPsec VPN service negotiation. ISAKMP is defined in IETF RFC 2408. IKE is defined in IETF RFC 2409. ISAKMP defines the schemes, syntax and procedures for creating and deleting authentication keys and security associations (SAs). IPsec peers use SAs to keep track of the different aspects of the security service policies negotiated between different IPsec peers.

Internet Key Exchange: IKE is a hybrid of the Oakley key determination protocol and SKEME key exchange protocol. The IKE protocol manages the IPsec security associations within the ISAKMP of IPsec VPN peers. IKE is a protocol available to ISAKMP; but they are not the same thing. IKE is the mechanism that establishes the IPsec connection between IPsec peers.

This article excerpt was adapted from IPsec protocol details for implementing VPNs, by Michael J. Martin.

Site-to-site VPN configuration

site-to-site VPN diagram
Enlarge site-to-site VPN diagram.

In the site-to-site VPN configuration above, each node is connected to a discrete network, separated by other unsecured or public networks. Depending on the security requirements for these network segments, it could be the case that end nodes on the networks are not able to exchange data unless the VPN is in place. This type of VPN configuration is known as a closed site-to-site network topology. Alternatively, the end nodes connected to the segments could have the ability to freely exchange data, utilizing other networks to relay the data back and forth. This data exchange, however, is unsecured. In this kind of network environment, IPsec VPNs can be employed to secure some or all of these data exchanges. This type of VPN configuration is known as an open site-to-site network design. The key point is that in either case, IPsec VPNs are implemented using gateways that secure the data exchanges. And, more importantly, the securing of the data exchanges is done without any knowledge of the end nodes connected to the networks being secured.

This section was excerpted from IPsec VPN connection models: Site-to-site and client-to-site, by Michael J. Martin.

→ For more information, view these VPN and remote access security best practices.

Client-to-site VPN configuration

Client-to-site VPN diagram
Enlarge client-to-site VPN diagram.

The models open and closed hold true in the case of a client-to site topology as well as a site-to-site topology. Connectivity between nodes separated by (or adjacent to) the IPsec gateway may or may not be restricted. In an open client-to-site topology, the network path between the end node and the IPsec gateway is secured. In a closed client-to-site topology, the path between the end node and gateway is secure. But data exchanges between the client node and nodes adjacent to (i.e., behind) the IPsec gateway is only possible if a connection to the IPsec gateway exists.

In both topologies, the relationship between the client node and the IPsec gateway is architecturally similar to a traditional PSTN  (public switched telephone networks) remote-access dial network. The end node establishes a connection to the gateway and the two communicate as IPsec peers. Additionally, the gateway provides the end node an IP identity that gives the client node IP network access to other end nodes directly connected (via VPN) and adjacent to the IPsec gateway. The communications between the client end node and the gateway is secured with IPsec. Communications between the client end node and other end nodes adjacent to the IPsec gateway, however, are not secured.

This was excerpted from IPsec VPN connection models: Site-to-site and client-to-site, by Michael J. Martin.

→ Continue this VPN tutorial to learn about the different types SSL VPNs.

This was last published in December 2010

Dig Deeper on WAN technologies and services