Problem solve Get help with specific problems with your technologies, process and projects.

Anomaly-based intrusion protection configuration and installation

Network behavior analysis may be the answer to preventing zero-day attacks. You must install an anomaly-based intrusion protection system (IPS) or intrusion detection system (IDS). This article explains step-by-step measures to configure your enterprise network anomaly detection system.

Anomaly-based intrusion protection devices operate by detecting network activity that is out of the ordinary and...

unexpected, such as zero-day hacker attacks. Installing and configuring a system that will recognize unexpected activity requires an understanding of the activity that is expected.

Monitoring the network for a few hours is not sufficient. Patterns of activity change over the course of a day and at different times of the month. Sample expected behavior from normal day-to-day operations and any end-of-month or end-of-year activities. An accurate understanding of behavior requires analysis of each application during these periods.

To install anomaly-based intrusion protection, analyze network applications

The first step is to determine which applications run on the network. While it may seem that this step is unnecessary since the inventory of applications should already be up to date, that is not always true. Applications may have been running for years without any need for upgrade or support and may have been forgotten. A detailed inventory may never have been created or may not have been kept up to date. In any case, now is the time to create the inventory or update it.

Creating a profile of expected activity for each application is the next step. An accurate, detailed profile is based on an understanding of what the program does. For example, an application that processes customer credit and checks each time a purchase is made will deliver a single customer record for each transaction, while a program that analyzes monthly patterns of purchases is expected to return much larger blocks of data. An end-of-month accounting application will typically not be accessed mid-month.

The profile should include a listing of the other systems and applications with which the application communicates. If user workstations connect with the application, document exactly which users and which workstations legitimately access the application.

After you create or update the network application profile, review the expected transaction rates. The application accessing customer records when a purchase is made will normally be executed at the rate of customer transactions. An attack may generate a rapid sequence of transactions. Each transaction may access just a single customer record, but the rate of transactions may indicate that an attack is under way.

Keeping the profiles up to date is a time-consuming task. Any change requires an update. Any time an application is added, an existing application is modified, new equipment is added, the network is modified, or transaction rates change to a significant degree, the change must be reflected in the profiles.

Network behavior analysis (NBA) tip series
Prevent hackers attacks with NBA
Anomaly-based intrusion protection configuration

Simplify application profile updating by segmenting the network

Configuring all anomaly-based intrusion protection devices with profiles of all the applications is difficult. Doing so requires updating each device every time any application changes. The task can be made easier by grouping applications on the network so a single intrusion prevention device monitors network activity for a single application or small set of applications.

If multiple instances of an application are run, they should all be grouped on a single physical network link, subnet or virtual LAN (VLAN). In many cases, applications that interact intensively with each other should be grouped together. The intrusion-prevention device on that subnet or VLAN will be configured to recognize only the patterns of behavior expected for the single application or group of applications. Updating the configuration for that device need be done only when a change is made on that small set of applications. Responsibility for maintaining the configuration can be assigned to staff members responsible for the set of applications instead of requiring a central group to be responsible for monitoring all application changes and maintaining all configurations.

Virtualization would appear to make segmenting the network more difficult. Virtual machines (VMs) move from physical server to physical server as load increases or decreases or systems are taken down for maintenance. Grouping applications on a VLAN eliminates the difficulty. VMs maintain the same VLAN membership as they move. All that is required is to configure all of the switches for all of the VLANs in use.

Once installed and configured, anomaly-based intrusion protection is quite effective. But no technology is perfect. A cleverly constructed attack could remain within expected network behavior. False positives are possible. A sudden increase in sales may trigger a level of activity that appears to be an attack. Part 3 of this series examines integrating anomaly-based protection with other technologies.

About the author:
David B. Jacobs of The Jacobs Group has more than 20 years of networking industry experience. He has managed leading-edge software development projects and consulted to Fortune 500 companies as well as software startups.

This was last published in October 2009

Dig Deeper on Network Security Monitoring and Analysis

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.