Troubleshooting IPSec
Tom Lancasater
Configuring a VPN is a fairly complex task, and configuring ISAKMP (Internet Security Association and Key Management Protocol) and IPSec (IP Security Protocol) can be especially challenging. Odds are good that the configuration may not be quite right on your first attempt. This is a brief summary of the best practices for problem determination.
The first thing you should do is eliminate the easy stuff. Remove your security policies and verify that you have basic IP connectivity between your tunnel endpoints using PING.
Next verify that no device in the path is blocking your traffic. In this case, that would be UDP 500 and IP protocols 50 and 51, for ISAKMP, ESP and AH respectively. Also make sure NAT isn't in the path, either.
Next, break your security policies into three major sections:
ISAKMP negotiation
ISAKMP, when you use it, will negotiate a connection with the peer before IPSec begins its negotiation, so check to make sure the ISAKMP negotiation is successful. The tools used to verify this process will be different for each vendor, but generally, ISAKMP should recognize its peer and exchange several packets (including transforms and keys).
IPSec negotiation
Once ISAKMP is finished, IPSec will attempt to negotiate its SA (Security Association). This process is similar to ISAKMP negotiation. If your vendor's equipment is capable of showing the details of this process, check to make sure its SA is using the correct source and destination IP addresses, the proper encryption protocols, and be sure that the SA lifetimes match. The SA lifetime can be expressed in either time or the amount of traffic passed.
Traffic classification
Finally, in your configuration, you typically specify what traffic to encrypt. Make sure this is correct and that any routing you might be doing through the encrypted tunnel is correct.
Also, be careful to include your testing tools in this configuration. For instance, if you want to test connectivity through your encrypted tunnel using PING, you need to let the system know to encrypt the ICMP traffic.
Thomas Alexander Lancaster IV is a consultant and author with over ten years experience in the networking industry, focused on Internet infrastructure.
