kentoh - Fotolia
Published: 02 Nov 2015
Amy Arnold sighs as she thinks about the uphill battle facing network engineers when it comes to security.
Arnold, a senior network engineer who works in the public sector in Texas, recently read about a vulnerability that was uncovered in a storage vendor's wireless external hard drives. The manufacturer had unknowingly provided access to Telnet services by leaving a default password hardcoded into the firmware -- enabling anyone to log into the device and access its contents. From Arnold's perspective, it's a perfect example of what happens when vendors treat network security as an afterthought.
"Why is that still a thing? Why are we still falling for that?" she asks. "One of the [other] things I find challenging is educating users, because you need your users to be thinking securely as well. You need to be teaching them, 'Don't click on every attachment. Don't click on every link on a webpage.'"
Once upon a time, another question network engineers like Arnold would have been asking themselves was, "Why is it my job to worry about that?" But amid the constant barrage of breaches that have made stolen data -- be it credit card numbers or the names of cheating spouses -- into a lucrative business for cybercriminals, many network engineers say they feel a greater responsibility to become more security savvy and seek out network security training.
"Passing the buck is not really an acceptable answer because it falls to everyone -- networking, database people, application writers, Web interface developers and all of that. But, in addition, from a network perspective specifically, you need to know what's going on in your network," Arnold says. "When headlines of you getting hacked come up, you don't want to be sitting there and going, ‘Well, I thought it was someone else's problem.'"
Jess Probasco, a senior network engineer at American Public Media in St. Paul, Minn., agrees that networking pros -- or any other role within IT -- can no longer afford to distance themselves from security in today's threat climate.
"Even if it's not your primary role, everybody gets pulled in when something happens," Probasco says. "So when something goes wrong, you want to be able to say, ‘Yes, we've been doing what we can to mitigate these things, and because of that, only this happened.' Not, ‘We left a gaping hole.'"
Most network engineers know their way around a firewall. Yet those who focus primarily on routing and switching -- and have limited network security training -- will likely encounter a learning curve as they get deeper into security, says John Pescatore, director of the SANS Institute, a cybersecurity training and certification organization in Bethesda, Md.
"They're not thinking, ‘Oh, gee, there is some new version of Cisco IOS out there that fixes a vulnerability. We need to take the network down and get everyone up on the latest release of IOS.' They're more in the mindset of, ‘Hey, the network needs to be up 100% of the time. We would never take it down,'" Pescatore says. "[But] the attacks have become more dangerous than the patching downtime."
Does networking prepare you for security?
It's often said that security is more art than science. Networking, on the other hand, is most often compared to plumbing. Security is typically regarded as a craft that is cultivated rather than a technology that is studied. Contrast that to the complexity and precision of packets, ports and protocols in networking.
"With the network, it's all about how we get from point A to point B. And with security, it's more a matter of policies," says American Public Media's Probasco. "It's not so much like you can just pick security up. It's a lot more about long-term audits, watching logs and piecing things together. The challenging part is trying to dedicate the time to learn all the different pieces."
Those pieces go beyond simply shutting down stray ports, he adds. Today's threats have placed an emphasis on next-generation firewalls, multi-factor authentication, insider threats and intrusion detection, says Probasco, who is actively trying to learn more about all of those areas.
Being fluent in security has become a critical skill set as a growing number of unconventional endpoints -- from temperature sensors to personal iPads -- are popping up on corporate networks, thanks to initiatives like BYOD and the Internet of Things.
Nearly one in three enterprises said they had a "problematic" shortage of network security skills in 2014, according to a report by Enterprise Strategy Group, which surveyed 601 IT pros in North America and Western Europe.
"We've got all sorts of devices now that want to be on our network and leverage network resources. How do you keep up with all of that? How do you set policy for all of that? It's a little overwhelming," Arnold acknowledges. "Security has always been a piece of networking, but now I think that's why you see such a need specifically for security engineers. Keeping up with all these things is really difficult. If you can't focus on it full time, it's hard."
Hard doesn't mean impossible, assures Pescatore, but getting more hands-on with security will require many network engineers to approach their jobs with a different mindset.
"It's not necessarily difficult, but it does require understanding how the bad guys think," Pescatore says. "If I'm a good network engineer, I want to make traffic flow as quickly as possible -- low latency, low jitter -- so the good guys can communicate over the network. I'm not necessarily thinking about what some bad guy would do to disrupt this."
But once that mental hurdle has been cleared, a background in networking can actually be a powerful advantage in the world of security.
"Security guys running firewalls don't necessarily understand all the network protocols as well as the network guys do," Pescatore says. "Network guys that get trained on perimeter security, boundary security, firewall operations -- they're actually doing some of the best defense work out there. They're often the ones who recognize, just from looking at firewall logs or odd occurrences, that, ‘Hey, there's something going on that we need to drill down and look at.'"
As she tries to learn more about the latest attack vectors and best practices in network security, Arnold says her networking knowhow has helped provide additional perspective for tasks like reviewing packet captures in Wireshark.
"I'm finding that having a route-switch background is really good because you understand what [an attacker] is trying to do on the network," she says. "I might not understand all of the detail about a database exploit because I don't talk database every day, but I can get a sense of, ‘OK, here is what we can do to protect ourselves from this. Here's how that's working. Here's what they leveraged.'"
Being familiar with the ins and outs of the protocol stack makes it easier to understand, anticipate and respond to a potential attack, says João Taveira, an engineer at Fastly, a content delivery network startup based in San Francisco.
"Most network engineers have a tremendous amount of insight into what happens between the layers," says Taveira, who develops the network software that orchestrates Fastly's bare-metal switches. "[Understanding] stuff like NTP reflection attacks is not something that comes naturally to most people. Because to most developers, NTP is just a service. It's something you interact with as a black box. You don't really see the protocol defects quite as easily as a network engineer would."
That depth of understanding is especially vital in the midst of a breach, he explains.
"If you are operationally responsible for a network, you will be the first person to deal with the fallout of an attack," Taveira says. "Being able to identify or immediately be responsive toward an attack is important. You can only really do that if you are able to detect the symptoms correctly."
At American Public Media, Probasco has also found that his networking background gives him a stronger foundation for learning more about security.
"I've been able to jump in a lot quicker and get more focused on what we can do to prevent [attacks], and spend less time trying to understand the concepts and theories behind how a DNS request gets intercepted and redirected to a malware site," he says.
Top skills to focus on
Like networking, security encompasses a broad and multifaceted range of technologies, disciplines and domains. Narrowing down which areas of network security training to focus on can seem challenging, but the SANS Institute's Pescatore says network engineers can benefit from three skill sets: penetration testing, understanding the role of DNS in security and network security forensics.
"There's lots of demand for companies to do penetration testing against their own networks -- to think like a bad guy and find the places the bad guy could break in, and then close the holes. Pen testing, done right, requires really good network knowledge," Pescatore says.
As for DNS, Pescatore says it's worth brushing up not only on how DNS can be compromised but also on how it can be used to bolster an enterprise's defenses.
"When one of these advanced targeted threats does get installed on some machine on the inside, typically the first thing it has to do is call out to the command and control center," he says. "There are techniques called DNS sinkholing, which basically says that if anything on the inside tries to resolve any of these addresses or domains, don't let them. They are known to be bad. It's a blacklist approach."
Another approach is to look at patterns of DNS access and look for anomalies, such as a server that never tries to resolve an external address and suddenly attempting to, Pescatore says.
Preparing for today's threats
In terms of network security training opportunities, vendor-specific programs like those from Cisco or Juniper are typically the most helpful for networking pros who manage a single-vendor environment, Pescatore explains. Engineers who work in multi-vendor environments are better off pursuing third-party training programs, he says.
Arnold says she is considering formal network security training programs or pursuing a certification like a CISSP, but not because she wants to add another abbreviation to her business card.
Amy Arnoldsenior network engineer
"Certifications are not the end all, be all. They don't make you a better engineer. But for me, they do make me study, because I can't be one of those [people] who goes in and just tries to pass a test," she says.
"I think security is one of those things that lends itself significantly to being aware, doing a lot of reading and following people in the space," Arnold adds. "Yes, you can be certified in security -- and lots of people are -- but if you're more like me, a network engineer doing security as well, then it's hard to really focus on this. But that's not an excuse. You can't just be like, ‘Well, I just won't care about security then.'"
Even though security has always been and continues to be a fast-moving target, American Public Media's Probasco says he's committed to learn as much as he can in light of the ongoing attacks over the past few years.
"It's definitely a lot of after-hours reading and studying to try and figure this out," he says. "It's hard to always pull away from your main job focus, but it's something I feel is important to keep up on, try to learn more about and seek out training for."
Both Arnold and Probasco say they find today's threat landscape more motivational than anxiety-producing, however.
"Every time you hear about [a data breach] in the news, you think this happened to ‘them,'" Probasco says. "But we should be thinking, ‘Is this going to happen to us? What have we done to address this same type of issue?'"
Amid constant threat of network breaches, visibility is key
Network security challenges, by industry
How to use DNS monitoring to detect network breaches