Get started Bring yourself up to speed with our introductory content.

NVGRE standard primer: More VLANs and isolated tenants in the cloud

Microsoft's NVGRE standard offers competition to VMware's VXLAN standard, promising to create more VLANs that stretch across Layers 2 and 3 for multi-tenant cloud networks.

The Network Virtualization using Generic Routing Encapsulation (NVGRE) standard proposal recently submitted to the IETF by Microsoft, Arista Networks, Intel, Dell, Hewlett-Packard, Broadcom and Emulex addresses the same challenge as the VXLAN standard recently proposed by Cisco, VMware, Citrix, Red Hat, Arista and Broadcom: Both use encapsulation strategies to create a larger number of VLANs for subnets that can extend across dispersed data centers and Layers 2 and 3. Both standards aim to enable load-balanced, multi-tenant networks that can be shared across cloud and on-premises environments.

NVGRE addresses the multi-tenant network with Tenant Network Identifier

A large cloud may support hundreds of tenants, each running multiple applications simultaneously, and traffic from one tenant must be isolated from all of the others.

Currently, VLANs are used to isolate traffic, but this approach suffers from several shortcomings. For example, there are often too few VLANs to handle too much need for segmented capacity. In a large virtual cloud environment, each application may be comprised of components executing in multiple VMs, and inter-VM traffic for each application must be isolated in a separate VLAN. The result is the following: The total number of VLANs that are needed can easily exceed the 4,094 limit defined by the IEEE 802.1Q VLAN specification.

The second problem arises when VMs move from one physical server to another. VM movement must be transparent to the executing application, so the same Layer 2 addresses must be retained. For that to happen, the network must be reconfigured to extend the VLAN to a different server; this is an error-prone process. 

NVGRE addresses these problems by specifying a 24-bit Tenant Network Identifier (TNI). Each TNI identifies a single virtual Layer 2 network allowing up to 16 million networks.

Generic Routing Encapsulation creates a virtual Layer 2 network

Generic Routing Encapsulation (GRE), a tunneling protocol defined by RFC 2784 and extended by RFC 2890, provides a method for encapsulating and sending packets to a destination across Layer 2 or Layer 3 networks. The NVGRE proposal uses GRE to create an isolated virtual Layer 2 network that may be confined to a single physical Layer 2 network or extend across subnet boundaries. 

Each TNI is associated with an individual GRE tunnel. Packets sent from a tunnel endpoint are forwarded to the other endpoints associated with the same TNI via IP multicast. Use of multicast means that the tunnel can extend across a Layer 3 network, thereby limiting broadcast traffic by splitting a large broadcast domain into multiple smaller domains.

An NVGRE endpoint receives Ethernet packets from a VM and encapsulates and sends them through the GRE tunnel. The endpoint decapsulates incoming packets, distributing them to the proper VM.  Endpoints can be located in any network component but typically will be implemented in the server hypervisor.

Each NVGRE endpoint will be assigned at least one and possibly multiple IP addresses. Each IP address is referred to as a Provider Address (PA). VM IP addresses are Customer Addresses (CA).  Assigning multiple PAs to an endpoint enables load balancing.  This proposal does not define how a PA is assigned to traffic from a specific CA or the load balancing algorithm. 

Isolating tenant data with the NVGRE standard

The NVGRE endpoint isolates individual TNIs by inserting the TNI specifier in the GRE header. The TNI identifier is placed in the key field specified by RFC 2890. The key field is defined as a 32-bit field, but the TNI occupies 24 bits with the remaining eight bits reserved.  NVGRE does not use the sequence number defined by the RFC.

The NVGRE endpoint encapsulating a packet must specify the destination address to which the packet should be sent. The current draft of the NVGRE proposal defers this issue to a future release of the document, stating that “any number of techniques can be used in the control plane to configure, discover and distribute the policy information.”

Moving entire subnets into the cloud using NVGRE

The proposal does not specify a method for assigning CA and PA addresses but states that address assignment can be done statically, dynamically or using stateless address auto-configuration. Either IPv4 or IPv6 may be used. CAs could be assigned IPv4 addresses while PAs are assigned from the IPv6 address space or vice versa.

The specification is designed to enable an enterprise to move an entire subnet into the cloud, according to the NVGRE proposal. A portion of a subnet can be moved, with the remainder hosted on the enterprise network, in which case a VPN would connect the cloud to the enterprise network.

Efficient use of the network requires spreading traffic across multiple network paths. The proposal states that techniques such as Equal Cost Multipath (ECMP) could be used with the eight-reserved bits in the GRE header key field used to support use of multiple paths.  The proposal does not include details on an algorithm to achieve balanced flows. An alternative approach to enabling ECMP is to use allocate multiple PAs in a server.  Switches can then spread traffic across routes passed on PA.

NVGRE standard vs. VXLAN standard

Clearly, both proposals address the same problem: 4094 VLANs are not sufficient to support multiple cloud tenants and applications. Other key proposal similarities include the following:

  • Introduction of a new 24-bit identifier,
  • Use of tunnels to carry packets across the network and across subnet boundaries,
  • Use of multicast to connect application components executing in VMs spread across the network.

The two proposals differ in the way destination addresses are located. The VXLAN proposal describes in detail how packets are flooded through the tunnel to locate destination endpoints. The NVGRE proposal defers the method for locating destinations to a future version of the specification.

Both proposals state that load balancing is necessary for efficient operation. The VXLAN proposal explains how random assignment of port numbers can be used to spread load. The NVGRE proposal suggests using the eight reserved bits in the GRE key field but does not provide detail on how they could be used.

In general, VXLAN is more fully specified. Cisco and VMware have announced that products utilizing the specification are now available. Future updates to the proposal will be required before NVGRE products can be released.

About the author:
David B. Jacobs of The Jacobs Group has more than 20 years of networking industry experience. He has managed leading-edge software development projects and consulted to Fortune 500 companies as well as software startups.

Next Steps

Cloud computing network primer

Multi-tenant networks for private clouds

Live migration networking in long-distance data center bridging

Data center bridging primer

In this VXLAN primer, learn how Cisco and VMware's proposed standard also creates more VLANs to multi-tenant cloud networks.

Not everyone thinks VXLAN is ready for primetime. In this Fast Packet blog, Ivan Pepelnjak points out problems with VXLAN.

This was last published in October 2011

Dig Deeper on Data Center Network Infrastructure