Most logical architectures for routing and switching are based around a system whereby three sets of functions are abstracted logically from one another. A common one is Core, Distribution and Access. These are often thought of as network layers.
For a quick refresher, in this system, the Access layer is responsible for connecting devices to the network. Its defining characteristics generally revolve around either high port density or the ability to overcome physical "last mile" type challenges, like wireless 802.11, or remote access via modems or VPN.
The Distribution layer is where policies are applied. It's where access-lists, or QoS, and CPU-intensive routing decisions should occur (as opposed to just a default route or default gateway). Distribution layer designs usually focus on aggregating Access devices into boxes with significant processing resources so that policies can be applied.
Finally, the Core is the "backbone." Its job is simply to move packets from point A to point B as fast as possible and with the least possible manipulation.
This academic model is probably familiar to most SearchNetworking readers, but in practice, there is much debate about how to translate these logical roles and responsibilities into physical boxes. For instance, in your network, when does it make sense to collapse the Distribution and Access functions into the same box, while the Core is a separate box? Or vice versa, when would it make sense to collapse the Core and Distribution into one box, while leaving the Access layer separate? Or perhaps all three deserve their own boxes? Or all three could be implemented on the same box.
The answer is "it depends" on what you need to do. That is, what are your requirements?
When you start to design a network, you should get a list of requirements which will include such things as "availability", and "security" and of course, a budget. As an example, your network may require that certain servers always be able to communicate with each other. If they share Access equipment, then having that hardware separated from the Distribution layer, means that you can do maintenance on the Core and Distribution boxes without disrupting these servers. If your Access and Distribution are combined into a single switch, you can't make changes without a fuss.
Conversely, you need to compare the amount of data you plan to transport across your backbone with the types of policies you plan to implement. If your plan includes few access-control lists, and no traffic-specific routing decisions (e.g. to give preferential treatment to something like VoIP), and you don't have a lot of data, then you can save some money by combining layers into a single device (usually at least two for redundancy, of course). But if you have a large amount of data to transport, or complex policies to implement, then it may be worth the dollars to separate those features.
However, you should realize that this decision is rarely based on hardware constraints, because most modern network platforms are capable of providing all three layers, at very high performance. Because of this, many argue that separate hardware adds points of failure and wastes money. But, you should consider things like what administrative groups will be supporting each function, and what additional features are available. For instance, if you need to add a service like IP telephony services or intrusion detection, does your decision still make sense?
Generally speaking, separating all three layers into different hardware is the most flexible and most expensive option. The real question then is, how much is flexibility worth? I hate to invoke a phrase like "total cost of ownership," but if you look at the big picture, I think you'll find that separating these functions justifies the cost of extra hardware in most circumstances.
About the author:
Tom Lancaster, CCIE# 8829 CNX# 1105, is a consultant with 15 years experience in the networking industry, and co-author of several books on networking, most recently, CCSPTM: Secure PIX and Secure VPN Study Guide published by Sybex.