Problem solve Get help with specific problems with your technologies, process and projects.

Invisible traffic that steals bandwidth

If you network is busy though you know it is empty you may be experiencing "phantom traffic." Here Loki Jorgeson takes the mystery out of of the sources of traffic-like behavior.

Your application may be acting like the network is busy though you know it is empty. Response is slow, the updates are jerky, there are episodes of momentary hang-up -- and yet the counters at the switch show few packets are in transit. In that case, your network may be experiencing "phantom traffic." Sound mysterious?

Well it isn't. It is a form of "apparent traffic" in the behavioral sense -- packets are being spaced and delayed as if there is competing cross traffic. As overviewed in What Ping doesn't tell you, packet behaviors can provide unique insight into the nature of the network. And while actual cross-traffic has a very specific effect on packet flow, there are a variety of other common network phenomena that may have a similar effect.

Congestion effects are considered in the design of most IP network devices and applications. The usual consequence is some delay variation (or jitter) in the packet arrival times. In extreme cases where the specifications of the network design are being exceeded, there can be packet loss. Packet loss (or extreme jitter) is usually interpreted by TCP to indicate congestion and causes the sending application to reduce its traffic load. In this way, TCP accomplishes "fair use" of the network between applications.

Whenever loss or extreme jitter is caused by phenomena other than traffic, application performance may be adversely affected, although for reasons that are not part of the network design. And even when jitter is reasonable and there is no loss, applications are effectively behaving as if they are competing with cross-traffic.

The result is often stranded bandwidth -- bandwidth you paid for but are not able to use due to "phantom traffic" effects. So what are the sources of traffic-like behavior? Most of them are probably familiar to you.

Stuttering NICs
A particular combination of NIC and driver may result in packet stutter – the inability to put packets to the wire quickly, cleanly and back-to-back. Packets leave the NIC with gaps between them even though the NIC is intending to send at full rate. Outdated or defective drivers are the likeliest culprit and often can simply be upgraded. For some real-world examples, review the NIC column.

In this case, the transmitting host simply cannot see the full capacity of the link -- and if packet traces are performed, it often is the case that the packets are being transmitted very irregularly. This variation in packet transmission means that packets arrive at their destination with significant jitter. Sometimes this doesn't significantly affect an application's performance -- but, to real-time applications like video and voice or demanding applications like backup, it can seriously degrade quality.

This is the most typical instance of phantom traffic and, thankfully, the easiest to resolve. The biggest challenge is detecting and diagnosing the packet jitter associated with a bad NIC/driver.

Problem routers
Routers are designed to pass packets quickly and efficiently. Any time they are not doing that, they can be assumed to be having problems. Overloaded routers are one such example. Although it usually becomes apparent when an outdated router simply needs to be replaced, there are cases that are not so obvious.

A classic example is when a router has been temporarily assigned filtering duties, possibly as a response to a DOS -- or other network-based attack. Some router designs can handle this well. Others cannot. When they take on these functions, the router can become overburdened. Packets may continue to be passed relatively quickly but the router's overall capacity will be reduced. Once traffic reaches critical levels, well below its original design threshold, delay variation increases and packets are forwarded with additional jitter

An analysis of a router's response on the slow-path will often give some indication of loading under these conditions. While the ASICs continue forward effectively, the CPU comes under load first and will inject jitter into any slow-path response, well ahead of any jitter experienced by traffic.

Another rarer source of router jitter can come from poor design where oversize buffers are configured. As traffic mounts, buffers hold packets longer and longer (instead of dropping them) and consequently packet arrival jitter can become quite extreme.

TCP congestion avoidance
As mentioned, TCP typically interprets loss as congestion and reduces its transmission levels. While this is intentional, there are cases where persistent low-level loss, particularly on paths with large propagation delays, can cause TCP to remain throttled down, even though there is very little traffic. To the TCP-based application attempting to use the network path, it will never be able to fully utilize its share of the network bandwidth.

This may not seem to qualify as "phantom traffic" insofar as the packet delay variation remains minimal, but from the application's point of view, it appears as reduced capacity as it cannot access it. And since the limiting loss is interpreted as congestion, traffic is apparently the cause.

Layer 2 retransmission
WLAN media is well-known to be lossy. That loss is often detected at Layer 2 and may result in retransmission. In many cases, the retransmission may be attempted many times before Layer 3 experiences loss or the packet finally is transmitted. The hidden cost of retransmission is an acquired packet jitter – each retransmission adds a finite amount of time to the overall latency.

In cases where retransmission is almost always successful, the receiving application may still experience significant jitter. Once again, this packet delay variation is essentially equivalent to the effect of cross traffic. Capacity is lost inevitably to the variable quality of the medium.

Traffic and apparent traffic
Your network may have a significant amount of stranded bandwidth -- "stranded" in the sense that some applications may not have access to the full capacity they need. Further, other applications may suffer in the presence of traffic-like packet jitter (such as VoIP). If you really do have congestion problems, you need to consider adding bandwidth or reducing traffic levels. But if you simply have annoying performance degradation issues causing traffic-like conditions, you want to find them and resolve them and get your stolen bandwidth back.

Naturally, identifying the sources of packet jitter is the hard part. Despite offering rough statistics on RTT variation, Ping can't tell you enough to determine the fact of traffic-like jitter, never mind where it originates and whether it is real traffic. However, VoIP will certain tell you when you have a problem and some of the new diagnostic offerings for voice networks have the right tools for the job.

NetworkingChief Scientist for Apparent Networks, Loki Jorgenson, PhD, has been active in computation, physics and mathematics, scientific visualization, and simulation for over 18 years. Trained in computational physics at Queen's and McGill universities, he has published in areas as diverse as philosophy, graphics, educational technologies, statistical mechanics, logic and number theory. Also, he acts as Adjunct Professor of Mathematics at Simon Fraser University where he co-founded the Center for Experimental and Constructive Mathematics (CECM). He has headed research in numerous academic projects from high-performance computing to digital publishing, working closely with private sector partners and government. At Apparent Networks Inc., Jorgenson leads network research in high performance, wireless, VoIP and other application performance, typically through practical collaboration with academic organizations and other thought leaders such as BCnet, Texas A&M, CANARIE, and Internet2.
This was last published in May 2006

Dig Deeper on Network Infrastructure

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.