What is IPSec?

IPSec (Internet Protocol Security) is a set of communication rules or protocols for setting up secure connections over a network. Internet Protocol (IP) is the common standard that determines how data packets travel over the Internet or any IP network. IPSec is a security layer that adds encryption and authentication to make the protocol more secure. For example, it scrambles the data at its source and unscrambles it at its destination. It also authenticates the source of the data.

The IP Security Working Group, created by the Internet Engineering Task Force (IETF), developed IPSec in the 1990s to ensure data confidentiality, integrity, and authenticity when accessing public networks. By default, data sent on a network is unsecured. IPSec provides encryption so that only the rightful recipient can decrypt the data. For example, users can connect over the internet with an IPSec virtual private network (VPN) to access private company files remotely and securely. The IPSec protocol encrypts sensitive information to prevent unwanted monitoring. The server can also verify that the received data packets are authorized.

IPsec can perform the following:

• Provide router security when sending data across the public internet
• Encrypt IP packets at the network layer
• Authenticate data packets quickly if the data originates from a known sender
• Protect network data by setting up encrypted circuits, called IPsec tunnels, that encrypt all data sent between two endpoints

Organizations use IPSec to protect against replay attacks. A replay attack is a type of man-in-the-middle attack. In a replay attack, an intermediary computer intercepts and sends data that has already been received by the destination computer. This can overwhelm the recipient's computing resources. The IPSec protocol assigns a sequential number to each data packet and performs checks to detect signs of duplicate packets.

IPSec encryption is a software function that scrambles data to protect its content from unauthorized parties. Data is encrypted by an encryption key, and a decryption key is needed to unscramble the information. IPSec supports various types of encryption, including AES and older algorithms like 3DES.

IPSec uses asymmetric and symmetric encryption to provide speed and security during data transfer. In asymmetric encryption, the encryption key is public while the decryption key is kept private. Symmetric encryption uses the same shared key for encrypting and decrypting data in IP network communications. IPSec establishes a secure connection with asymmetric encryption and switches to symmetric encryption to speed up data transfer.

IPSec allows computers to exchange data over a secure tunnel using a suite of protocols, algorithms, and key management methods. It operates at the network layer of the OSI model, using multiple protocols, including an initial handshake process before data transmission. The OSI model defines the layered structure of how computers exchange data on a network. Before exchanging data, both sender and receiver perform mathematical exchanges to generate shared cryptographic keys.

When a computer sends data, the network stack breaks the information into smaller data packets. Then, it appends headers and trailers to the payloads that consist of encryption and authentication information.

• Encryption prevents unauthorized parties from reading the underlying information.
• Authentication helps the receiver accurately identify the packet's origin and ensure it has not been altered.

Here are the steps for computers to exchange data with the IPSec protocol:

1. The sender computer determines if the data transmission requires IPSec protection by verifying against its security policy. If it does, the computer initiates secure IPSec transmission with the recipient computer.
2. Both computers negotiate the requirements to establish a secure connection. This includes mutually agreeing on the encryption, authentication, and other security association (SA) parameters.
3. The computer sends and receives encrypted data, validating that it came from trusted sources. It performs checks to ensure the underlying content is reliable.
4. Once the transmission is complete or the session has timed out, the computer ends the IPSec connection.

IPSec protocols send data packets securely. A data packet is a specific structure that formats and prepares information for network transmission. It consists of a header, payload, and trailer.

• A header is a preceding section that contains instructional information for routing the data packet to the correct destination.
• Payload is a term that describes the actual information contained within a data packet.
• The trailer is additional data appended to the tail of the payload to indicate the end of the data packet.

Here are some IPSec protocols.

Authentication header (AH) protocol

The authentication header (AH) protocol adds a header that contains sender authentication data and protects the packet contents from modification by unauthorized parties. It alerts the recipient of possible manipulations of the original data packet. When receiving the data packet, the computer compares the cryptographic hash calculation from the payload with the security authentication header to ensure both values match. A cryptographic hash is a mathematical function that summarizes data into a unique value.

Encapsulating Security Payload (ESP) protocol

Depending on the selected IPSec mode, the encapsulating security payload (ESP) protocol performs encryption on the original IP packet or only the payload. ESP adds a header and trailer to the data packet upon encryption.

Internet Key Exchange (IKE) protocol

Internet Key Exchange (IKE) is a protocol that establishes secure network communication between two devices on the Internet. Both devices set up an IPSec security association (SA) using the Internet Security Association and Key Management Protocol (ISAKMP), which involves negotiating encryption keys and algorithms to transmit and receive subsequent data packets. Modern IPSec protocols use IKEv2, which simplifies key exchange and provides better support for mobile devices.

IPSec operates in two different modes with different degrees of protection.

Tunnel mode

The IPSec tunnel mode is suitable for transferring sensitive data on public networks as it enhances data protection from unauthorized parties. The computer encrypts all data, including the payload and original IP header, and appends a new header to it.

Transport mode

IPSec transport mode encrypts only the data packet's payload and leaves the IP header in its original form. The unencrypted packet header allows routers to identify the destination address of each data packet. Therefore, IPSec transport is popular for a closed and trusted network, such as providing secure communications between two computers.

VPN, or virtual private network, is a networking technology that allows users to access remote resources privately and securely over a public network. An IPSec VPN is a VPN type that uses the IPSec protocol suite to create encrypted tunnels on the Internet. It provides tunnel end-to-end encryption, which means data is scrambled at the computer and unscrambled at the receiving server.

SSL VPN

SSL stands for Secure Sockets Layer. It is a security protocol that helps protect web traffic. An SSL VPN is typically a browser-based network security service that uses the built-in SSL protocol to encrypt and safeguard network communication. Modern SSL VPN has replaced the SSL protocol with its successor, Transport Layer Security (TLS), and the SSL VPN name persists.

What is the difference between IPSec VPN and SSL VPN?

Both IPSec and SSL VPN protocols work on different layers of the Open Systems Interconnection (OSI) model. IPSec VPNs work at the network layer in the middle of the OSI model. Meanwhile, SSL VPNs encrypt data at the session and presentation layers.

SSL VPN secures HTTP web traffic and other traffic with TLS, while IPSec VPNs encrypt all IP traffic. Organizations often use IPSec VPNs for site-to-site access, while an SSL VPN may be more suitable for accessing specific services. You can connect to an SSL VPN from a web browser, but you must configure your operating system's settings or install separate software to use IPSec VPNs.

IPSec secures data exchange for network devices with IP encryption and authentication. However, technical teams face several challenges when configuring and operationalizing the IPSec protocol. Here are some common considerations for technical implementation:

• Packet fragmentation. IPSec adds headers to the payload, which might cause the entire data packet to exceed the network's maximum segment size (MSS) and maximum transmission unit (MTU). Packets that exceed MSS are broken into several fragments, while packets exceeding MTU are dropped from transmission.
• Configuration complexity. Organizations require skilled technical teams to configure various parameters that IPSec relies on. This includes the encryption algorithms, key management, and IPSec protocols.
• NAT traversal. Network address translation (NAT) is a method that modern routers use to manage network traffic between multiple private IP addresses and a single public IP address. IPSec generally transports information with ESP, which is blocked by NAT. To enable NAT traversal, IT teams must encapsulate the IPsec packet in UDP on port 4500.
• Performance overhead. IPSec encrypts the payload or entire IP packets, which can consume processing power, increase latency, and degrade network performance. Organizations can use Amazon EC2 X1e instances, which feature modern processors that support AES-NI, to reduce this burden.

AWS Site-to-Site VPN is a fully managed service that creates a secure connection between your data center or branch office and your AWS resources using IPSec tunnels. When using Site-to-Site VPN, you can connect to both your Amazon Virtual Private Cloud (VPC) and your on-site AWS Transit Gateway, and two tunnels per connection are used for increased redundancy.

AWS Site-to-Site VPN brings many benefits, such as:

• Visibility into local and remote network health with performance monitoring
• Secure and easy migration of local applications to the AWS cloud
• Improved application performance when integrated with AWS Global Accelerator

Get started with AWS VPN by signing up for a free AWS account today.