Problem solve Get help with specific problems with your technologies, process and projects.

How to build a scalable IaaS cloud network infrastructure

Attempts to scale Infrastructure as a Service cloud network infrastructure often involves workarounds, but providers should look to IP-over-IP transport and invest in design.

Building an Infrastructure as a Service (IaaS) proof-of-concept is a no-brainer these days—take a few high-end servers, cram as much memory into them as you can, deploy the virtualization platform of your choice (VMware or Xen seem to be the most popular choices), slap some management software in front of the infrastructure—be it a point solution like vCloud Director or a services-oriented one like those offered by Joyent—and finally add a bit of networking and storage to the mix.

But scaling the architecture to accommodate an Amazon-sized cloud is a totally different undertaking.

Most IaaS architects design their cloud network infrastructure based on two simple—and wrong-headed—assumptions:

  • Customers want Layer 2 broadcast domains.
  • If you want to achieve high utilization, then having unrestricted virtual machine mobility across the whole infrastructure is a must.

These assumptions unnecessarily limit your design options to stretched VLANs spanning a whole data center. When implemented with traditional data center switches (for example, Cisco Nexus 5000 or equivalent switches from other vendors), the number of VLANs supported by the switches—1,000 to 4,000, depending on the model—directly dictates the number of customers you can support.

Most IaaS architects design their networking infrastructure based on a few simple -- and wrong-headed -- assumptions.

Amazon Web Services (AWS) has already proven that the customers do not necessarily demand Layer 2 domains in cloud services. AWS doesn’t support broadcasts or IP multicasts, even with its latest set of networking features, but it still boasts tens of thousands of customers. Likewise, unrestricted mobility isn’t a must; you just have to have a large enough number of virtual machines and physical hosts in a cluster to achieve good load distribution. VMware can’t have more than 32 hosts and 3,000 virtual machines in a cluster anyway.

Even if your cloud designers insist on having end-to-end VLANs across your data center, all is not lost. After all, service providers have been building large Carrier Ethernet networks for years, and what tenants in your IaaS cloud data center need is no different from what Carrier Ethernet (or Metro LAN if you prefer a better-sounding name) customers need—a VLAN or a set of VLANs between a number of endpoints.

Once you get past the notion that you have to buy data center switches, there’s a variety of field-proven choices that you probably already use in your service provider network. You just have to check whether VLAN provisioning on the switches interoperates with your chosen virtualization platform—ideally, the switches would auto-provision VLANs on access trunks based on VMware VirtualCenter information.

You can also scale your IaaS cloud infrastructure with judicious use of Layer 3 MPLS VPN services that use VLANs in smaller clusters, or you can use an MPLS/VPN with Virtual Routing and Forwarding (VRF) and MPLS transport in the network core. After all, MPLS/VPN networks have been shown to scale to tens of thousands of customers and hundreds of thousands of routes. There’s no good reason you can’t use the same technology in your data center that you already use in your WAN.

Scaling IaaS cloud network infrastructure without workarounds

All of the solutions I’ve mentioned so far, however, are kludges and workarounds caused by the relative lack of networking focus among virtualization vendors. After all, we all know bridging does not scale. And to scale the IaaS cloud, you have to get rid of bridging. Hint: why do you think Amazon can’t offer broadcast or multicast support?

Implementing an MPLS/VPN Provider Edge (PE) router in the hypervisor would be one option, but even that doesn’t scale as far as we’d like to go. Every PE router needs a label switched path to every other PE router, which is not a big deal in service provider networks that usually have a few hundred PE routers. Large IaaS data centers could have many more servers—as much as two orders of magnitude more.

The only truly scalable solution is IP-over-IP transport since the network core provides scalable IP transport. We know how to design that. After all, we learned a bit while building the Internet, and the hypervisor hosts use the IP core to transport virtual machine data encapsulated in IP datagrams.

Sadly, vendors’ hypervisor switches are the weak link

Unfortunately, you can’t buy a hypervisor switch with these features from any virtualization or networking vendor. If you want to be in the same league as Amazon, you have to do what NASA did after the Sputnik launch: figure out what the other side did (after all, the principles are there for all to see), invest a lot in development and build your own system from scratch. Or you could decide that just good enough works for you and build your cloud network infrastructure using the same building blocks as anyone else, but then you have to ask yourself what your competitive advantage will be.

About the author:
Ivan Pepelnjak, CCIE No. 1354, is a 25-year veteran of the networking industry. He has more than 10 years of experience in designing, installing, troubleshooting and operating large service provider and enterprise WAN and LAN networks and is currently chief technology advisor at NIL Data Communications, focusing on advanced IP-based networks and Web technologies. His books include MPLS and VPN Architectures and EIGRP Network Design. Check out his IOS Hints blog, and ask him your networking questions at's Ask the Expert.

This was last published in May 2011

Dig Deeper on Telecommunication networking