Jumbo packets can make a big difference on high performance networks. Bigger packets mean bigger payloads and fewer packet headers. And that translates to faster networks -- much faster. Contrary to my last column about bandwidth, this is one area where more is definitely better.
In the history of Ethernet, IP packets have been limited in size to be a maximum of 1500 bytes. The 10 and 100 Mbps Ethernet standards define that to be largest possible packet size. Gigabit Ethernet dispensed with that limit and the standard allows arbitrarily large packets (at least up to 12,144 bytes – thereafter, CRC-32 begins to break down).
A "jumbo" packet is simply any packet larger than 1,500 bytes. There is no standard jumbo size but the industry is gravitating toward supporting sizes around 9,000 bytes (or slightly larger than 8 * 1,024). Most GigE equipment supports some form of jumbo but you have to read the fine print carefully to avoid disappointment.
So why does jumbo make a difference? Networks are rapidly outstripping CPUs. Or to be more specific, networks are outstripping NICs, drivers, and the busses they are attached to. Performance bottlenecks are increasingly found at the end-host rather than in the mid-path. Packet header generation can be costly and adversely affects throughput. Research has shown that typical workstations can't fill the pipe at GigE using 1,500 bytes, often limited to just a few 100 Mbps. Wizards can achieve much more, for example by clever tweaking of the TCP stack. But, in many cases, simply switching to jumbo 9,000 byte packets can as much as double the achievable throughput.
Matt Mathis, a network research scientist for Internet2 and the Pittsburgh Supercomputing Center, has made the case that maximum packet size not only needs to increase to 9,000 bytes, but for reasons of future network performance needs to keep on increasing. His view is that the minimum time between packet headers needs to be kept approximately constant so that CPU and NICs don't have to work harder generating packets as the line rates increase. Larger payloads increase the separation between headers. So, for example, packet headers at 10 Mbps are separated by about 1,200 μsec -- to keep that separation at one Gbps, the packets would have to be roughly 100 KBs!
So taking a small step toward larger packets can mean immediate benefits today -- but it is only the beginning. Networks will only get faster. And the bottlenecks due to packet size will only get tighter. It's time to take size seriously.
I say, "Make mine jumbo."
Chief Scientist for Apparent Networks, Loki Jorgenson, PhD, has been active in computation, physics and mathematics, scientific visualization, and simulation for over 18 years. Trained in computational physics at Queen's and McGill universities, he has published in areas as diverse as philosophy, graphics, educational technologies, statistical mechanics, logic and number theory. Also, he acts as Adjunct Professor of Mathematics at Simon Fraser University where he co-founded the Center for Experimental and Constructive Mathematics (CECM). He has headed research in numerous academic projects from high-performance computing to digital publishing, working closely with private sector partners and government. At Apparent Networks Inc., Jorgenson leads network research in high performance, wireless, VoIP and other application performance, typically through practical collaboration with academic organizations and other thought leaders such as BCnet, Texas A&M, CANARIE, and Internet2. www.apparentnetworks.com