Over the years, most annoying network routing issues have been resolved with one of two ways: either the multivendor...
adoption of standard technologies built into our gear or the deployment of clever third-party software.
For example, if you know your spanning tree network diameter off the top of your head or can't imagine life without nProbe or Wireshark, you're taking advantage of proven solutions to issues that, in the old days, took hours to troubleshoot. IEEE 802.1D ensures loop-free connectivity, and Wireshark can untangle just about any packet in a few clicks. Even with so many mature network management technologies and tools, however, there are a few network routing issues that, as administrators, we end up debugging with secure shell more than we should. Untangling routing is a good case in point.
The network hitchhiker's towel: The console cable
Maybe we don't keep light-blue serial cables mounted to the wall like fire extinguishers anymore, but every administrator knows where to find one in hurry. Resolving routing issues involves repeatedly showing status tables via the command line interface, and every now and then you'll still hike to a rack. You may even end up connecting console sessions on several devices to trace along routes. But these solutions may not address the real challenge of route debugging. Routes tend to cause trouble over time -- often intermittently -- and the most effective way to find the root cause is to catch breaks in real time.
Even if you've configured good syslog alerts, many route changes only happen once every few days, making them tricky to catch. And users aren't the best performance agents for reporting route failovers either. They just grumble to each other when pages load a little more slowly or as chatty apps bog down when latency shoots up. (Although there are some VoIP customers who have an unnatural desire to provide actual subjective mean opinion score reporting when your paths get jittery or drop packets.) So the question becomes, "How do you regularly poll your route tables and receive alerts when network routing issues occur?"
Under the influence while routing
The actual routing on your network is the sum of a number of factors, all interacting generally -- but not always -- in obvious ways. Routing information protocol, open shortest path first, border gateway protocol, enhanced interior gateway routing protocol and static configurations each exert its own influence on resulting routes. In essence, together they represent a virtual configuration. But the consequential "physical" routes are only truly available for confirmation in one place: the routing table. But who has time to preemptively keep an eye on route tables from all devices (and why would you want to)?
First, every time a route flaps it can force a recalculation of network topology by all participating routers and thus flood the network with update packets. Not good. Second, route flapping is usually detected after the fact, and you'll spend considerable effort reading router tea leaves and determining what happened hours ago. In some cases, route changes might simply slow traffic as it falls back to a less optimal route. But other times those changes may sever a link, disrupting critical user services.
Read more of Patrick's advice
- Network engineer as mobile device manager: Tracking users and devices
- Getting network security administration back in sync
- Have IP address management issues? Think DNS, DHCP, IPAM trifecta
Fortunately, many network performance monitoring solutions offer easy access to the route tables in the devices they monitor. Because they're aware of changes, you can receive alerts, run reports or directly view routes tables without getting out your console cable. Combined with network topology discovery, you can walk the full traffic route from endpoint to endpoint, examining the route details in context with how interfaces are connected. Layer 1 is still the root of all evil and tracking a broken packet flow from a server to a port with an extinguished link light should not require a hike away from the administrator's desk. Route change history reports and flapping route advisory views usually come along for free.
Sure, sending syslog messages on route changes is handy, but you'll still have to watch your logs to catch the sometimes-elusive changes as they happen. To enjoy your weekends more, spend a little time configuring your route monitoring tools to send intelligent alerts on changes in real time and to keep track of their change history. You'll troubleshoot faster and give your console cable a well-earned break.
About the author:
Patrick Hubbard is a head geek and senior technical product marketing manager at SolarWinds. With 20 years of technical expertise and IT customer perspective, his networking management experience includes work with campus, data center, storage networks, VoIP and virtualization with a focus on application and service delivery in both Fortune 500 companies and startups in high tech, transportation, financial services and telecom industries. He can be reached at Patrick.Hubbard@solarwinds.com.