Hospital gains network visibility by convincing vendors to collaborate |
 |
By Shamus McGillicuddy, News Editor
30 Oct 2008 | SearchNetworking.com |
|
|
When Concord Hospital began building two new redundant data centers connected via a multi-link trunking technology five years ago, it was confronted with a network monitoring and analysis challenge.
The core Nortel 8600 10 GbE switches in each of the two data centers were going to be connected to the New Hampshire hospital's campus network via Nortel's Split Multi-Link Trunking (SMLT) technology. Any host device could connect to either core switch at any time, which meant there was no longer a single chokepoint in the network. The IT organization could no longer place a monitoring device on a core switch and be assured that it had full visibility into the network, according to Mark Starry, manager of enterprise architecture and security.
"Once [the data centers] were built, how do I get a handle on the traffic now?" Starry said. "If you have two 10 Gb links going to a huge PACS [Picture Archiving and Communication System] area, and one link gets busy, it may send the next guy who comes on to the other data center through the interlink switch trunk. There's no way to predict where your traffic is going to go," he said.
"In a clustered switch approach, it's not predictable," he said. "We're getting all these calls that this is slow and that is slow. Where do we start? To troubleshoot a network problem, you need both ends of things. You need to see not only the person who is transmitting something. You need to see the response. And if you don't hear a response back, how do you know where it was?"
At a conference, Starry encountered StealthWatch, a network behavioral analysis tool from Lancope, which could collect NetFlow data from anywhere in a network, regardless of how packets were routed through the two data centers. Unfortunately, at the time, Lancope supported only Cisco's proprietary form of the NetFlow protocol. Concord Hospital was mainly a Nortel network, and it wasn't inclined to rip and replace with Cisco.
"So I kind of talked to Lancope and talked to Nortel," Starry said. He encouraged the two vendors to work together to make StealthWatch run on IP-FIX (IP Flow Information Export), a nonproprietary, standardized version of NetFlow that Nortel was using on its newer switches. To convince Nortel to make its source code available to Lancope, Starry said, "I'm going to retrofit my network, spend all this money to rip out my [switch] blades and put in your new blades to support NetFlow, and I'm not going to do that unless you work with Lancope to get it working."
Nortel and Lancope agreed to cooperate to make StealthWatch run on Nortel's network technology. By the time Concord Hospital's new data centers were up and running, the product was ready to go. Starry installed the StealthWatch collector and a console into the network and had the visibility he was looking for. His staff instantly started solving network problems.
"Both my security team and my network team use it," he said. "Once something changes, we know about it. We get alerted. 'This PC usually only transfers 10 MB a day. Did you know it just sent 50 GB to the Internet?' That's just not right. We can actually see all the data flowing through our entire network and make decisions between what's good and what's bad."
Many users had been complaining that their PCs were running slowly on the network. With StealthWatch, Starry learned that hundreds of PCs were sending traffic to Slovakia. Instantly, he assumed he had a botnet problem. StealthWatch revealed something completely different.
"It turns out it was just some systems that were misconfigured," he said. "They were supposed to have RFC-type internal addresses, like 192.168. Instead, someone had typed 192.198 in there, which is a real Internet-routable address. And then they had imaged 500 machines with that. So they were going out to get their SLP scope from Slovakia instead of trying to get it here, and you're wondering why your PC is so slow. Then they went to the failover SLP that was in the hospital, and they finally connected."
Starry said the new visibility into his network will also help him with capacity planning. Most of his network closets have gigabit uplinks, but the hospital is moving to 10 Gb. StealthWatch will help him determine which closets are priorities. "Ten gigabit isn't cheap," he said. "So we're trying to figure out which links are being used most."
Let us know what you think about the story; email: Shamus McGillicuddy, News Editor
|
|
|
|
