This article can also be found in the Premium Editorial Download "Network Evolution: Is the network ready for the private cloud?."
Download it now to read this article plus other related content.
Private cloud computing enjoys almost as much hype as public cloud computing, but it poses just as many challenges ... especially in the network.
The idea of using server virtualization to abstract data center resources in order to provision applications and services on the fly is enticing, but the network must keep up with the rapid changes.
“From the network perspective, even moving virtual machines across a data center still takes anywhere from five minutes to two days, depending on the systems they are using,” said Andre Kindness, senior analyst with Forrester Research.
The networking team faces two considerable challenges in a private cloud environment. First, a data center needs some degree of network automation so that configuration changes happen quickly in response to new services or migrating workloads.
Second, the network team needs good virtualization monitoring capabilities that are tied into the rest of the private cloud orchestration system so the team can prevent new and migrating private cloud services from creating unexpected bottlenecks within the infrastructure.
Automation is key to network operations in the private cloud
Two years ago when Cisco addressed the private cloud with its global customer advisory board, consisting of its 30 largest customers, the members pushed back. Just a year later, these same customers had to start moving forward into the cloud mostly under pressure from their business units, which saw the potential savings. But their big concern was the network and its operational model, said Guy Daley, director of data center product management at Cisco Systems.
“Trying to coordinate configuration changes across multiple physical and virtual platforms just for virtual machine mobility, you end up with these hugely complex and riddled scripts,” said Daley.
The solution to that problem would be automation tools that function like a closed loop system so that changes up the cloud stack trigger commensurate changes in the network, such as adjustments to VLANs, ACLs and QoS settings.
The need for this solution prompted Cisco to acquire companies like LineSider Technologies and Tidal Software, which specialize in network automation to support cloud computing. LineSider’s software automates network configuration changes based on service provisioning, eliminating the need for manual intervention. Tidal’s software works to correlate business processes with virtual infrastructure provisioning and scheduling through a centralized management tool.
Meanwhile, third-party network management vendors, such as Infoblox, are rapidly moving into the automation and virtualization management field. Infoblox, an IP address management (IPAM) and network change and configuration management (NCCM) vendor, recently updated its NetMRI NCCM product to improve network automation capabilities for its customers. It added run book automation capabilities and opened up an API interface that will allow organizations to use Perl scripts to automatically tie network configuration to cloud orchestration technology.
When automation doesn’t work, old-fashioned organization might
While a plethora of automation tools emerge, enterprises are still struggling to make them work across the network. This can be especially challenging in a heterogeneous environment, said Richard Rothschild, senior director of infrastructure at Oclaro Inc., a manufacturer of optical and laser components.
Rothschild is building a private cloud across sixteen sites in the United States, Europe and Asia. The task is especially complicated by the fact that his firm is actually a collection of several companies that have been run independently.
Oclaro’s approach is to consolidate its infrastructure in a private cloud using Infoblox’ network automation tools, but Rothschild is also relying on an epic amount of organizational planning among his team to handle the change.
“You have to have a central architect who is the arbiter that resolves conflicts,” Rothschild said. “Then you have regional folks. If they want to make a change and put up a new service, assuming we’re not good enough to plan far enough ahead for it, then they can have a local guy, who is in synch with how we’re doing things globally, who can make a short-term call. He can say, ’OK, we need to get you some services up and running, which means we have to get you some IP addresses and get you on the standard configuration.’ We have a regional person who can deal with it in that time zone. And it’s his responsibility to communicate that fact to the rest of the organization.”
Where are the virtual traffic monitoring tools?
In addition to network automation, network operations must also ensure that bottlenecks don’t occur when systems administrators instantiate or migrate virtual machines in the private cloud, said Forrester’s Kindness. That will take new kinds of monitoring tools that don’t necessarily exist yet.
On the systems side of the picture, administrators watch CPU and memory usage, as well as I/O utilization on their host servers to maximize use, but they don’t have visibility further up the network.
“Maybe you move that virtual machine to a server that has plenty of resources on it, but what I’m hearing is that they have no visibility when they move stuff over,” Kindness said. “The uplink from the server to the switch might be fine, but now the link from switch to switch further into the network [is a problem].”
Yet visibility must cut across all data center resources and be accessible to every team involved in provisioning for the private cloud.
“What we’re finding is network engineers, when they are trying to diagnose a problem in a cloud service, need to understand what the components are,” said Cisco’s Daley. “What are the configurations that have been made relative to that service? That’s hard to see in cloud implementations today.”
Customers are looking for tools to help the engineers in their organizational silos understand the end-to end relationship involved in providing a service so they can diagnose a problem as rapidly as possible.
“I think they need some new tools... that could help them see the service, what’s being consumed by a tenant or line of business, [and] the relationships between the different switching layers, the load balancer, the firewall and into the virtual machine environment,” said Daley.
The lack of appropriate visibility tools is partially due to the fact that there are not many vendors tied into both the software and hardware worlds that can provide this view into east-west traffic inside the virtual infrastructure.
“A lot of vendors are tying into [VMware] vCenter, but is that the way you want to do things? Do [networking pros] really want to depend on vCenter?” Kindness said.
Throwing capacity at the problem
Until better monitoring tools for virtualization emerge, some IT shops will go the path of increasing capacity as much as possible to handle rapid provisioning. Forrest Schroth, lead data network engineer at SFN Group, a strategic workforce solutions firm, avoids bottlenecks in his highly virtualized environment by investing heavily in network hardware, particularly 10 Gigabit Ethernet server uplinks.
“Rather than granularly watch stuff or QoS every packet and get into that layer, we threw a lot of hardware at it,” he said. “We still do some of the QoS in pieces and try to make sure we don’t oversubscribe some of our vMotion. We still monitor that stuff, but one reason we’re not monitoring intensely is because we threw bigger links at it.”
This operational model won’t necessarily work when a company grows the size of its private cloud and the number of applications and services it is running, Kindness said.
“If you have only a few applications and you have a ton of 10 Gigabit uplink bandwidth, that’s no problem at all. But in situations where you have 4,000 applications running in a data center, keeping track of it all is going to be a challenge,” he explained.
Private clouds will force new approach to operations teams
Aside from having the right tools and designs for the private cloud in place, IT shops must better integrate their currently separate management teams. The networking team will no longer exist as a silo in the data center when an enterprise moves toward a private cloud model, according to Derek Masseth, senior director of client and infrastructure services for the University of Arizona. Instead it will have to work closely with the teams managing the rest of the cloud to ensure seamless operations.
“When moving to a converged network in the data center and toward the private cloud space, a new team must emerge, and it is a combination of the traditional data center tribes, but the network operations skills set absolutely must be at the table,” Masseth said.
Masseth’s organization has been building an informal cross-organizational team to manage his highly virtualized environment with about 500 virtual machines running on 24 physical servers. He expects this informal team will become a formal one on his organizational chart within a couple of years.
“We also consolidated our data center monitoring, automation and maintenance team with the network monitoring and automation team based in part on the knowledge that convergence of workload was coming,” Masseth said. “We are in the fortunate position where the data center, network operations, systems and storage teams all ultimately report to me, so I have been able to make changes that some IT departments will have more resistance to making.”
About the Author
Shamus McGillicuddy is the News Director for TechTarget Networking Media.
This was first published in August 2011