Proper use of network visibility can help IT managers head off problems before they hit an enterprise's front lines, or can at least help resolve problems quickly. But deciding what kind of visibility is needed can be half the challenge.
"The bottom line is that IT shops need to know when quality is degrading -- and know before the end user -- and they need to have a pretty good idea why," said Jim Metzler, vice president of Sanibel, Fla.-based consultancy Ashton, Metzler & Associates.
No single approach will fit all situations because requirements vary so widely, depending on which applications are in use and which services are considered critical assets, Metzler said. Controlling jitter, latency and packet loss are all important, but their relative importance depends on a network's utilization. Quality VoIP requires very low latency but is more forgiving of packet loss, while database systems may require the opposite.
The need to log historical network data also varies, involving a tough balancing act.
"Once you start throwing away data, you can't do a root-cause analysis," Metzler said. "If you keep data for months, you're going to have huge disk arrays filled up." He suggested that two weeks of detailed network information could be counted on to examine most complaints.
Before installing a network analysis appliance, Bernalillo County's IT department had difficulty just keeping up with troubleshooting problems, according to Michael Gruen, IT project manager for the New Mexico county, which includes Albuquerque.
"We had no real way of determining what our network performance was historically," Gruen said. "It was really a pain to do analysis of long-term trends and even short-term hiccups."
After deciding about a year ago on an appliance manufactured by Xangati, Gruen has lowered trouble ticket response times and headed off some larger issues, previously unseen, before users have detected a problem. This visibility has been particularly important while the county migrates to a 100% VoIP environment. Once in production, the VoIP system saw call quality degrade during data backups that were scheduled for the middle of the day. Switching the timing of the backups was a simple procedure, but one the IT department might not have known to make without the increased visibility the network monitoring provided them.
Cultural shifts are key
Fully benefiting from increased network visibility also requires something of a culture shift. The information is of little use if the same tactics – waiting for user complaints and addressing those symptoms – are followed. That shift does not always come naturally for networking pros.
"It's been slow to be adopted," Gruen said. "Some of our systems people tend to forget the Xangati machine is there. Some of us have adopted it wholeheartedly." Part of the problem might be a bit of pride in the old way of doing things, he said, but those who pair trouble tickets with Xangati's network analysis could often halve the time to problem resolution.
Metzler stressed that it is important for organizations to develop a culture of looking beyond immediate user complaints to analyzing underlying root causes and potential trouble spots down the road.
"The great majority of times, when an application is degrading, it's noticed by the end user and not by IT," he said. "There's this huge exposure in terms of a lack of knowledge of what's going on, and it makes [IT departments] look like bumbling idiots."
A properly utilized network visibility tool can fill that knowledge gap and head off issues before the angry calls flood in.
So what to look for? Metzler suggested several categories to keep under close watch. One is recreational traffic, which may or may not be having a serious impact on quality of service. Collecting data on the amount of bandwidth Internet radio, YouTube, or video games can provide some insight into what is acceptable use and what should be banned. It could even lead to surprising conclusions that have ramifications beyond the network.
"I've come across situations where managers are playing Doom in the afternoon," Metzler said. "That's not just a bandwidth cost but also a productivity cost."
For Gruen, one surprise was the bandwidth drain that desktop-sharing software caused, leading him to evaluate a more stringent policy on its use.
Network managers should also decide on the five or six most business-critical applications and carefully monitor their performance, Metzler said.
"A good-sized organization might have … 400 applications," he noted. "You can't watch them all." But almost as important as what you monitor is how you monitor it. It is important to focus on real application performance metrics, he said, not just rely on the belief that if network performance is OK, then the applications are running fine.
Finally, it is just as important to make sure that the right information is presented in the right way. Many new applications and appliances come with detailed graphs or Web 2.0-inspired interfaces, perfect for those in other divisions or CIOs needing a quick overview but poor replacements of real data for those in charge of the nuts and bolts.
"If I'm working in the NOC, I'm not going to talk to the CIO very often," Metzler said. "The graphical interfaces can be very good, but I want them to tell me nerdy things like delay and jitter and packet loss."