CenturionStudio.it - Fotolia
The typical network engineer may consider bare-metal switches a niche technology, but these commodity devices -- which run third-party network operating systems -- are precipitating a seismic shift in the networking industry. Many data center operators, especially in the cloud services, Web content and financial services markets, are eager to try bare-metal switching, which has the potential to transform the economics of the networking industry and also give network engineers unprecedented flexibility and agility.
Unlike traditional, vertically integrated switches from leading original equipment manufacturers (OEMs) like Cisco, bare-metal switches -- also referred to as white-box switches -- separate network hardware and software, which lowers prices and provides more flexibility in operations and network functions.
"Cost is the first thing that gets your attention [with bare-metal switches], but it's not what keeps us there," says Mike Dawson, co-founder and director of cloud architecture at Cloudapt, a public cloud provider in Indianapolis. "We're completely tied to white-box switching from here on out. We'll never do anything again that ties us to Cisco, Juniper or even Arista, which is the most open of the bunch."
Bare-metal switching is still an immature market, however. Most of the software vendors in the space are newer startups, and the hardware vendors are original design manufacturers (ODMs) whose distribution channels and enterprise support organizations in North America are limited or nonexistent.
Dell Networking changed that dynamic recently when it cracked open its data center switches and agreed to support third-party network operating systems from startups Cumulus Networks and Big Switch Networks. Dell is not a market leader in switching by anyone's reckoning, but it does have an enterprise-class supply chain and support organization that makes a bare-metal switch much more appetizing to companies that want disaggregated switch hardware and software. Ever since Dell made its bare-metal decision, the industry has been asking one question: Who's next?
"Without saying anything I can't tell you, I wouldn't be surprised [if other OEMs followed Dell's lead]," says Alex Benik, principal at Battery Ventures, a Boston-based venture capital fund that made an early investment in Cumulus Networks. "What is exciting about Dell is that it invented the modern server supply chain. I think they're the ideal partner to help reinvent and create the modern networking supply chain."
Bare-metal switch has appeal, as well as challenges
Cloudapt launched about 18 months ago with a single pod of server racks and a network of five 48-port, bare-metal switches sourced through Pica8, a software-defined networking (SDN) software startup that helps customers buy bare-metal hardware from Asian ODMs.
"We're open source guys on everything we do -- on storage, operating systems and applications. We're used to having direct access to engineers when there are problems," Dawson says. "Working with Pica8 allowed us [to apply] that same model on the network. It was the first time we had that ability. You can't get a Cisco engineer on the phone without selling your firstborn child. Working with smaller company that has roots and ties to the open source community is completely refreshing."
The pace of innovation in bare-metal switching is also faster, Dawson says, and new features don't always require new hardware.
"When we started working with Pica8, there was zero support for BGP, for instance. It's fully implemented now, and we've had a few [revisions] of code fixes on top of that. But I'm running the same ASIC and switch hardware," he says. "It's just purely a software change for Pica8, and it came much quicker than a new version of Nexus fully baked and melded together."
Still, bare-metal switching is unexplored territory for most network engineers, who are often extremely risk-averse. They have built networks with vertically integrated switches for decades. Disaggregating hardware and software presents new risks. Will everything work? Can you get enterprise-grade technical support? Can you get a replacement switch quickly? Will the startup you are working with be around in three years?
"Most network guys are rewarded based on availability, not necessarily on innovation," says Andrew Lerner, research director at Gartner. "People lose jobs when the network goes down. If you are a network guy and you take the network down, that is 10 [times worse than] if you are a server guy and just knock a server down."
Andrew Gallo, senior information systems engineer at George Washington University in Washington, D.C., agrees that bare-metal switching, as the market for it stands now, is unlikely to get much traction where he works.
"I would definitely consider [bare-metal switches], but … my organization is extremely risk-averse, so I doubt we would do it," he says. "There is value in maintaining a relationship with a name brand. The CIO is looking and saying, ‘Am I going to deploy a critical resource on a non-name-brand service?' It seems pretty obvious that is a really tough case to win."
But risk-averse engineers will warm up to bare-metal switching if more OEMs follow Dell's lead.
"From a market perspective and a revenue perspective, the overall percentage of [vendors] who do [bare-metal switching] is very small, about 3%," Lerner says. "If all of a sudden you saw Extreme get involved in this, HP get involved, even Cisco … it becomes a much more mainstream, accepted technology."
What goes into running a bare-metal network?
Google is well known for adopting a bare-metal approach to networking in its data centers. ODMs build switches to Google's specifications, and Google runs custom software on them tailored to its infrastructure needs. Facebook is pushing for the same thing through its Open Compute networking project, an effort to develop open specifications for bare-metal switches and software. It is testing Wedge, its own design for a top-of-rack switch, which it will submit to Open Compute shortly.
Facebook and Google have been able to pioneer bare-metal switching in part because they have the in-house engineering talent to build and operate such networks. Many enterprise engineers doubt they have the skills and manpower to put together bare-metal switches. They would rather stick to what they know.
"There is a big chasm between the enterprise guys who always want to have someone's neck to choke and people who are trying to build Web-scale infrastructure, who tend to be more open source and who have the attitude of doing it themselves," says Cloudapt's Dawson.
But bare-metal switching proponents say that gap is closing.
"I think there's a perception out there that is false -- that white-box switching means you do everything yourself," says Battery Ventures' Benik. "Obviously, Google and Facebook do a lot of things themselves because they have the resources to do so. The next 12 months are going to be the year of increasing market adoption, now that there are great relationships like Dell in place, which take care of a lot of the supply chain issues. The Open Compute movement for networking is definitely going to have a positive impact on adoption. It will pave the way for people to develop more commercially packaged solutions that come with enterprise support and an enterprise feature set that a more traditional networking user would be looking for."
Bare-metal switches offer new ways to do automation. Big Switch Networks and Pica8 sell licenses for switch software that works with their SDN controllers. And most network operating systems for bare-metal switches are open platforms based on Linux, which means data center operators can use Linux management tools and DevOps tools like Chef and Puppet to automate their networks and align network operations more closely with server and virtualization teams.
"When I do troubleshooting on my Pica8 switch, sometimes I drop into Linux, a lower level than the switch operating system," Dawson says. "Pica8 allows me to do that, and I'm more comfortable in Linux than I am in a Cisco switch or firewall operating system. And there are a lot more tools available to me. I can install my own software on a Cumulus or Pica8 switch and run automatic things that I never had the keys do to on a Cisco operating system."
Bare-metal switching's openness with Linux and DevOps tools make network automation much more doable, and it comes at a time when network engineers are struggling to keep up with more automated silos in data centers.
"You have all these processes spinning up new virtual machines and creating all the stuff needed to support it in an automated fashion. Then it hits networking for VLAN or firewall rules changes and it takes days because you are interfacing it with a human process that is frankly broken," says George Washington's Gallo. "Data center networking needs to be run through DevOps, or at least in a DevOps fashion. I find it harder and harder to [hire] traditional network engineers, so having some way to manage the network in an automated, multivendor fashion is critical for us."
Dawson has not found the bare-metal switches used at Cloudapt to be any more difficult to work with than traditional switches from mainstream vendors like Cisco. If a data center operator properly architects a network based on bare-metal switching, there should be no operational problems, he says. He has been able to set and forget his bare-metal network, thanks to a software overlay network based on OpenStack Neutron and Generic Routing Encapsulation.
"All the difficult networking things like customer isolation and customer-edge connectivity is all orchestrated on my servers in software," he says. "It doesn't even touch my white-box switches, which simply pass tunneled traffic around."
Learn thebasics of white-box switching
Willbare-metal switches alter networkingwith merchant silicon?
Q&A: Big Switch CEO talks aboutbare-metal switching