Manage Learn to apply best practices and optimize your operations.

40 Gigabit Ethernet in data center networks: Migration best practices

Next year 40 Gigabit Ethernet migration will begin in earnest. Learn how to introduce the technology to your data center network.

By next year, most networking vendors will be offering 40 Gigabit Ethernet (GbE) switches for enterprise data center networks.

While most enterprises are largely focused on upgrades from 1 GbE to 10 GbE, network engineers should start charting out an upgrade path to 40 Gigabit Ethernet.

“When it comes to upgrading the network, organizations only want to do it one time because it’s going to be in place for at least five years,” said Zeus Kerravala, principal analyst at ZK Research.

10 GbE servers will force adoption of 40 Gigabit Ethernet uplinks

The introduction of servers with 10 GbE network connections will force many enterprises to adopt 40 Gigabit Ethernet in the aggregation and core layers of data center networks in order to meet the overall bandwidth demands of top-of-rack servers with 10 GbE server-facing ports.

High server-to-server traffic among 10 GbE servers prompted a senior systems engineer with a fast-growing California-based jewelry manufacturer who declined to be identified, to adopt 40 GbE uplinks in his top-of-rack switches from Extreme Networks.

Anticipating 25 to 30 gigabits of server-to-server traffic in data center networks running VMware, vMotion and vSphere Fault Tolerance, the engineer designed a network that interlinked top-of-rack switches with 40 GbE uplinks.

“We’ll have multiple 10 GbE connections going into an Extreme Summit X670 switch with 40 GbE connectivity uplinking X670 switches to each other,” he said.

The Summit X670 provides up to 48 10 GbE ports and four 40 Gigabit Ethernet uplinks. According to Harpreet Chadha, vice president of product line management at Extreme, the Summit X670 Series is designed to support emerging 10 GbE enabled servers in enterprise and cloud data center networks, while offering optional future-proofing 40 GbE uplink support.

The jewelry manufacturer’s strategy follows best practice advice for 40 GbE migration offered by Mike Spanbauer, principal analyst of enterprise networking and data center technology at Current Analysis.  As projects occur and application needs dictate, network engineers must investigate, do the analysis and explore 10 GbE switches that are capable of upgrading to 40 GbE uplinks. “Particularly if a company is rolling out new equipment, if performance needs demand it and the budget allows it, there’s no reason not to explore 10 GbE/40 GbE technology,” Spanbauer said.  The ability to plug in 40GbE optics when the time comes -- to future-proof the investment -- makes sense.

Always avoid a forklift upgrade when you can

Many vendors are offering early 40 GbE-upgradable products in their latest generation of 10 GbE data center switches.

For example, the top of rack S Series S4810 10/40 GbE top-of-rack switch from Force10 Networks -- recently acquired by Dell --is available either with 6410 GbE ports or 4810 GbE with four 40 GbE ports. The company offers a 40 GbE line card for its ExaScale E600 and E1200 core switch/routers.

Cisco Systems will announce in October a new 40 Gigabit Ethernet switch that will provide an easy transition from 10 GbE, according to Omar Sultan, senior manager data center architecture at Cisco.

Extreme Networks’ new BlackDiamond modular switches are some of the earliest 40 Gigabit Ethernet-capable switches, providing an upgrade path for customers who have opted for the Summit 670 10 GbE/40 GbE platform today. The BlackDiamond X8, in customer trials this fall, offers a maximum of 192 non-blocking 40 GbE ports or 768 10 GbE non-blocking ports in an 8-slot 1/3 rack chassis.

Cost considerations for 40 Gigabit Ethernet

Two main elements drive the cost of a 40 Gigabit Ethernet port: the physical switch and the optics. Broader adoption has pushed the cost of 10 GbE switches down to $600 per port, and the cheapest available optics are around $250, for a total cost of about $850. Direct attach cables that connect to the SFP+ garage range from $100 to $300, including the terminated ends on the cable.

Since only a few vendors have made 40 GbE commercially available, the cost of 40 GbE technology is still variable. Port prices range from $1,000 to $1,600 per port. “There really isn’t an average for optics prices yet, as they aren’t shipping in significant volume,” Spanbauer said. Optics costs will settle in about 12 months, he said.

Extreme is one of the first vendors to bring 40 GbE to market. Its cheapest, short run transceiver, the 20m XSFP+, carries a $1,795 list price.  Extreme’s modular BlackDiamond switches are $4,000 per 40 GbE port, and uplink modules for Summit switches start at $750 per 40 GbE uplink port, according to Chadha.

The other nontrivial variable of the 40 GbE cost equation is cabling. Here, enterprises have two options: OM3 and OM4.

According to the Ethernet Alliance, OM3 or OM4 is optimal for both the 40 GbE and 100 GbE data center network. In a 40 GbE or 100 GbE environment, OM3 has a reach of up to 100 meters and OM4 reaches 150 meters, according to the IEE802.3ba standard.

“Everyone wants to go as long as they can, but they want it to be cost efficient for the application,” said John D’Ambrosia, chair of the Ethernet Alliance.

Finally, although 100 GbE is not expected to be within financial reach for most organizations for another three years, it’s imperative that as IT managers think about transitioning their existing network infrastructure to 40 GbE, they consider how today’s migration planning will impact future migration to 100 GbE.

Cisco’s Sultan says that his company’s switches are designed to provide an easy migration path from 10 GbE to 40 GbE to 100 GbE.  The BlackDiamond X8 is designed to support high-density 100 GbE in the future but will require customers to purchase new transceivers. Current top-of-rack and core platforms from Force10 Networks are not upgradeable to 100 GbE, according to the vendor. Expect to see 100 GbE systems introduced when market and customer requirements spur demand. 


Lynn Haber reports on business and technology from Norwell, Mass.

This was last published in October 2011

Dig Deeper on Data Center Networking