Advanced threat protection: Behavior-profiling network communications

Advanced threat protection specializes in detecting advanced and persistent malware inside networks.

Advanced threat protection technology uses behavior profiling to study enterprise network communications and detect sophisticated cyberattacks.

Malicious hackers have become so skilled at evading traditional signature-based network security that it's critical to detect and stop advanced malware as quickly as possible to minimize the risks of a full-blown breach.

Vendors such as FireEye, Damballa and RSA NetWitness have developed next-generation security technologies that profile normal enterprise network communications versus abnormal communications to detect the presence of advanced malware and infections.

What is an advanced threat?

Imagine that a criminal has a key to your house and can go inside without your knowledge. Advanced threats target enterprises in a similar way, in the sense that they're infections inside a network. And the attacker's goal is to steal data and evade detection.

How can malware use your network?

For more information about advanced threat protection, read this related story.

"The real risk to an enterprise is that an attacker outside the network is able to control infected devices in your network -- without your knowledge," said Stephen Newman, vice president of products at Damballa. "That's the general concept of the advanced threat, but there are varying degrees of who the attacker in control of the device might be and what they may want to do with their access to your data and network."

Malware can be slipped in using many techniques, but attackers generally target the path of least resistance. All those ridiculous emails you get, especially at work, with horrible misspellings and Viagra somewhere in the subject line are targeted malware intended to fly under the radar and get you to click on a link.

"As firewalls and IPS were made stronger in the network, bad guys kept coming up with more ways to evade detection," said Greg Young, Gartner research VP and lead analyst for network security. "Firewalls and IPS have to run at wire speed and can't inspect executables or related content, so occasionally that kind of malware slips through."

How can advanced threat protection find malware?

Advanced malware can give itself away, because it must use the network to communicate with its command and control system.

"Malware shows weird behavior by sniffing around on the network," explained Rob Rachwald, senior director of research at FireEye Inc. "That's not normal network behavior. Malware also needs initiate and perform a 'callback' to the attacker -- essentially picking up the phone from inside a trusted network and establishing a connection by dialing out."

Studying network communications can reveal unusual behavior and infected devices as they're acting. "You can discover a lot of these hidden infections in your network based on their network communications," said Newman. "We can zero in and pinpoint infected devices -- even though we may never see the malware or the infection vector."

Many enterprises realized that their security solutions today, which are predominantly prevention-based, aren't foolproof. "Infections are going to happen. So it helps to have techniques and automated solutions that can unearth and discover infections hidden inside their networks," said Newman.

By reducing the time between infection and detection, you can greatly reduce the odds of a full-blown security breach.

Advanced threat protection: Behavioral analysis tools study network activity

Since many forms of malware exhibit distinctly nonhuman behavior on the network, behavior analysis tools are essential for advanced threat protection.

From an analytic perspective, it's extremely important to look beyond signatures to behavior. "Signatures only give you insight into a slice of time, whereas behaviors provide much more in the way of nuance associated with an attack vector," explained Will Gragido, senior manager, RSA FirstWatch advanced research intelligence at RSA NetWitness. "We spend a lot of time conducting analysis on attackers, as well as on malicious code and content. We're zeroing in on their behavioral patterns as well as what they're actually doing."

When a human accesses a database, they do it at human scale with a mouse and keyboard. If malware is doing accessing the database, it's very automated, fast, with high-volume click rates. "Nothing about it looks or smells human," said FireEye's Rachwald. "But we're also starting to see attackers create malware that acts more human to evade detection."

Once suspicious activity is identified, it's critical to look at where the malware is communicating to, the content of the communications and the overall behavior of the device's communication.

Then a virtual version of the laptop or device suspected of having an infection can be created to check the behavior of emails and other files.

"We can take a copy of the file and run it in our dynamic analysis engines, in our 'sandbox' virtual environments to capture all of the network communications," explained Damballa's Newman. "We buffer and store all the communications the device is making. If there's a statistical match between the network commun

ications when we ran it in the sandbox and on the device it was headed toward, we can determine whether that device is infected or not."

At the end of the day, a human element plays the strongest role in the process of advanced threat protection. Enterprises need security analysts equipped with advanced tools and training to identify threats the technology flags, according to RSA's Gragido.

Extra layer of security

Although everyone is subject to an attack, not everyone can afford an extra security product or spare the people needed to manage it.

"To date, we've seen the most security aggressive companies with lots of staff, who are forward-leaning on managing their security, pursue this technology," Young said. "But many enterprises can't afford extra spending in security right now or have someone spend extra hours doing the monitoring and analysis. It's a great technology and if you have the resources, investigate it."

Many enterprises, 25%, have a firewall only and don't have an IPS, according to Gartner data. "Before the really advanced technology, enterprises need to ensure they're able to deal with the bulk of attacks first by using a firewall and IPS," Young said.

Enterprises must also instrument their advanced threat protection systems properly.

"Advanced threat protection can only protect the network connections it's attached to. While the technology can be put in-line, enterprises tend to use it on a SPAN port on a switch to mirror traffic coming from the public Internet to the device," said John Kindervag, Forrester principal analyst serving security and risk professionals. "But malware can also come through other connections, such as VPN, wireless and WAN. Protecting all these points of ingress can be a challenge for enterprises."

This was last published in April 2013

Dig Deeper on Network Security Monitoring and Analysis