A network engineer at a Washington medical center uses network performance management tools to keep critical applications available and to drive down network costs.
Don Lester, senior network engineer with Wenatchee Valley Medical Center in Wenatchee, Wash., said his medical center and its branch clinics have two critical applications that he worries about: a patient scheduling application and an electronic medical records system.
"If the scheduling system went down, it [would be] inconvenient and receptionists would have to backfill data later," Lester said. "If the medical record system goes down, that affects patient care because doctors won't be able to do things they might normally want to do to help patients.
"The medical records application, with medical imaging and everything else on it, is a lot more intensive and it is higher bandwidth," he said. "And it's time intensive. Doctors don't want to click on something to show a patient and then wait 15 seconds for it to come up. It's the interface into everything for physicians, so if it's down for some reason, it's a very big deal."
Putting out two fires at once
One morning last month, Lester looked at the dashboard on his application performance monitoring tool, SuperAgent from NetQoS, and saw that the performance of the medical records application was suffering badly at two branch clinics.
"In any network infrastructure, when you have two incidents that start at the same time, your natural instinct is to start looking at points in common for the two sites because that's probably the source of the [problem]," he said.
But in this case, searching for a common cause would have been a waste of Lester's time. He learned this quickly when he called up another tool from NetQoS.
"I popped up Reporter-Analyzer first because it's one of the quickest ways for me to look at what's going to a location," he said. "I was able to see that there was a whole bunch of traffic going to one of the locations from the workstation patching server."
Through some quick investigation, Lester was able to learn that the medical center's PC technicians were supposed to push out a patch in the middle of the night using their ability to turn on machines remotely for update. But something had gone wrong, and the PCs had never been turned on. Instead, the patching server waited until morning, when users started turning on their computers. The computers started pulling patches, which slowed down the WAN link at the remote location.
The problem at the second location was completely unrelated.
"There were no signs of anything," Lester said. "The link didn't have any significant traffic at all."
He pulled up another NetQoS tool -- SuperAgent, which analyzes TCP transaction -- and saw a high level of retransmission delay occurring at the second WAN link.
"So there was a problem with that circuit," he said. "We had to work with a third party who managed the circuit because it's not something we have a lot of eyes into. We told them to fix it. And we were able to use the same tool to tell when, in fact, they had fixed it and if it was as good as it was before the malfunction.
Watching systems at night
In addition to the monitoring tools from NetQoS, Lester uses Whatsup Gold from IPswitch.
"It's a very simple thing to set up that gives you a really quick and dirty up-down status on something," he said. "We use it for after hours, when only an operator is in the operations center. When something goes down on the network, the map on the central monitor goes red. The operator is supposed to call somebody to come in and fix it."
Reducing costs through visibility
As well as maintaining critical application performance, Lester's NetQoS tools control network costs.
"It isn't unusual for us to be negotiating a [WAN] contract based on a tier of service that is defined by the average amount of bandwidth consumed over a given period of time," he said. "There have been times when the vendor in question will inadvertently overstate our usage. When that happens, we will produce usage graphs over whatever time period is appropriate, detailing our actual usage -- and ultimately reclassify our tier level and negotiate a lower price."
Lester said a more common issue that comes up is dealing with application vendors. Often, when there is a performance problem with an application, the vendors will take a number of standard corrective actions to try to solve the problem. If none of these actions works, the application vendor will often blame the network and claim that the medical center needs more bandwidth.
"We are able to tell rather easily using NetQoS tools whether or not we really have a bandwidth issue and can then share that information in a concise graphical format," Lester said. "This not only saves us the cost of unnecessarily upgrading circuits, but it also results in better service to the end user since this re-engages the vendor. The troubleshooting escalates, and the root cause is isolated and repaired."
Let us know what you think about the story; email: Shamus McGillicuddy, News Editor