IP traffic are inherently insecure therefore a way to protect sensitive information that it carries needs to be incorporated. This is where IPSec comes into the picture. In its simplest form, the IPSec provides security services on top of a typical IP packet. This services may include replay protection, integrity, data origin authentication and confidentiality thru the use of cryptography.
It’s implementations operates in three different configurations:
1. Host to Host (HH)
2. Host to Gateway (HG)
3. Gateway to Gateway (GG)
There are known two modes of operation:
1. Transport mode
2. Tunnel mode
The transport mode is use for HH and HG communication where as the tunnel mode is use for GG.
Common to both modes of operations are two protocols use to provide the protection or security services. These are the:
1. Authentication Header (AH)
2. Encapsulating Security Payload (ESP)
The AH provides integrity only while ESP provides integrity and confidentiality. You can think of it such as AH use cryptographic hashing to provides integrity while ESP use hashing and encryption to provide integrity and confidentiality. A question would be, why select AH if you can use ESP to do both? Well, the options are there for different implementations. One may prefer to use AH only if confidentiality is not a concern thus removing a bit of overhead.
Typical IP packet includes
[ IP header] [Data]
In Transport mode, the AH/ESP protocols are included to provide integrity to data and some portion of the header
[IP header] [AH/ESP] [Data]
In Tunnel mode, the AH/ESP protocols are included to provide integrity and confidentiality to both IP header and Data. Obviously, you cannot encrypt the IP header because gateways on both sides needs this information to route the packets. Therefore, a new IP header is created.
[New IP header] [AH/ESP] [ Original IP header] [Data]
The new IP header includes both addresses of the communicating gateways but not the hosts behind those gateways. If a sniffer is placed between the two gateways to capture the packets, only the gateway information is revealed thus providing protection between the actual host that is talking. However , it is important to note that traffic between the host and it’s respective gateway are not protected .
If you like to deep dive to IPSec, best to look into RFC 4301, 4302 and 4303.