What are the algorithms used in intrusion prevention systems (IPS) and what has been the success of these systems?
IPS systems are becoming a required component of networks in much the way that firewalls are. An IPS is one means of adding an additional layer of protection. Most IPS/IDS's work by means of signature or anomaly detection. Both types of intrusion engines have the ability to do some amount of protocol decoding.
Anomaly detection systems require the administrator to place the device in a non-blocking mode so that it can learn what constitutes normal activity. Anomaly detection is good at spotting behavior that is greatly different from normal activity. As an example if a group of users that only log in during the day suddenly start trying to log in at 1 a.m., the system can trigger an event.
On the opposite end of the scale there is signature matching. Signature matching systems rely on a database of known attacks. While it may not be possible to test all of the signatures in the vendor's database you should initially test the device by running your own traffic through the unit to examine the effects. The signatures are usually given a number or name so that the administrator can easily identify suspicious events. These signatures can spot fragmented IP packets, streams of SYN packets (DoS), viruses, worms, or even malformed ICMP packets.
Somewhere in the middle of the spectrum of anomaly detecting and signature detection is protocol decoding. Protocol decoding alludes to the ability to reassemble packets and look at higher layer activity. If the system knows normal activity it can easily pick out abnormal protocol and application events. Protocol decoding systems have the ability to maintain state. As an example, DNS is a two step process therefore if a number of DNS responses occur without a DNS request the system can flag that activity as cache poisoning.
Cisco has a good white paper on the subject called The Science of IDS Attack Identification.
This was first published in April 2007