Markus Voegele has learned to expect the unexpected.
The phone could ring at any moment with news that he would have only a few weeks to set up wide area network (WAN) connectivity at a new branch location. It would be just another day for Voegele, a senior system and design engineer at Lufthansa Systems, a managed service provider and wholly owned subsidiary within Lufthansa Group that serves the German company's flagship airline along with more than 300 other airline clients.
"If Lufthansa decides tomorrow, for example, to fly to Anchorage, they would tell the IT department at the last stage," says Voegele, explaining that any such move would be kept secret, even internally, to reduce the chance of it leaking to competitors. "Normally at this point, there would be four to six weeks left. That's not sufficient to get an MPLS line."
Fortunately, Voegele and his team always have a backup plan. In this type of scenario, Voegele says he would order a standard Internet connection and ship a router that he preconfigured to connect to Lufthansa Systems' network using a dynamic multipoint virtual private network (DMVPN).
"As soon as this router connects to the network, it's done. We put our DMVPN on top of [the Internet connection], and the site is up and running," he says. If the location needed a more reliable private connection, Voegele would order a T1 circuit or MPLS link to be installed at a later date. And throughout Lufthansa Systems' network, Voegele uses a combination of homegrown technology and Cisco's Intelligent WAN (IWAN) platform to aggregate Internet and private WAN connections, as well as to assign dynamic policies that enable the network to automatically adjust bandwidth and make routing decisions, on a per-application basis, according to real-time conditions.
Such is life when managing a WAN. Long provisioning times, logistical puzzles and the occasional fire drill mean network engineers always need to be ready to pivot with a contingency plan. It can take anywhere from one to three months to provision a new WAN connection—even if there is an existing fiber or copper line in the building—and sometimes several weeks to upgrade an existing one.
It's a recurring source of frustration for networking professionals like Forrest Schroth, director of network engineering at Cross Country Healthcare Inc., a healthcare staffing company based in Boca Raton, Fla. Schroth says he's tired of waiting up to 90 days for telcos to install a new Ethernet circuit—never mind the fallout that typically follows after the business has confirmed where it will open a new branch.
"They finally sign a lease, they get it addressed, and they're all excited," he says. "The company will say ‘Go install your circuit.' And I've got to go back to them and tell them, ‘Hey, it's going to be two months before you'll have data there,' and they say, ‘What? That's too long.' And I say, ‘There's nothing you can do.' That is a daily conversation in my world."
While this isn't a new problem -- provisioning windows have, in fact, dropped over the years -- the long lead times are becoming more painful. As enterprises face more pressure to be responsive and agile, it's increasingly obvious that parts of the network, like the WAN, aren't keeping up with those demands.
"Five years ago, [this problem] was 10% worse than it is today. Seven or eight years ago, it was a little worse than that. But one of the reasons it's become more of an issue now is all of the other infrastructure used to be relatively slow [to provision]," says Andrew Lerner, a research director at Gartner.
"If you wanted a server, it took a lot of time to order, receive, rack, stack, power and cable it," he adds. "Now we can spin up a VM in a couple of minutes and orchestrate a bunch of stuff."
So while other components of IT infrastructure have become more programmable and allow for faster, automated provisioning, installing WAN circuits is still a painstakingly manual process, Lerner explains. Service providers are testing out software-defined networking (SDN) partly to automate most or all tasks involved in provisioning. AT&T has been particularly aggressive with its Domain 2.0 strategy, but for most carriers, change comes slowly at such a large scale.
"Carriers have built their cultures around managing hardware. When it comes to processes and tasks, they aren't necessarily quick to change models," Lerner says. "When you're dealing with a change within a carrier infrastructure, it's like steering a battleship."
Faster provisioning equals greater agility
Greg Driscoll is intimately familiar with how much trouble it can be to get a new branch office connected to the WAN.
Penn Mutual Life Insurance Co., where Driscoll works as the CIO, was planning to open a new location outside of Bellevue, Wash., in an existing office building. Driscoll had been phasing out his use of fractional T1 circuits at his branches in favor of business Internet services from cable providers like Comcast and Time Warner Cable. Paired with a VPN tunnel, basic broadband offered more bandwidth for a drastically lower price and, in most cases, faster provisioning for the Horsham, Penn.-based insurance agency.
When Driscoll reached out to a local broadband provider for a quote, he found out the provider would need to trench new fiber to service the building—a process that would take several weeks or even months—and Penn Mutual was expected to foot the bill. The estimated cost: $104,000. Driscoll just laughed.
"I said, ‘You could've told us you didn't want to do it. It would've saved all of us the trouble,'" he says.
Fortunately, Driscoll and his team had a backup plan. They ordered a 10 Mbps dedicated Ethernet connection from Level 3 Communications, which got the circuit provisioned on schedule. If it hadn't been ready in time, Driscoll says he would have purchased and configured several Cisco Meraki access points, and shipped them to employees to work from home. The devices, which are used in Penn Mutual's wireless LAN, would automatically set up a VPN at the employee's house and connect to the company's network via a standard residential Internet connection.
"It gives us a level of flexibility that we never would've had prior—not even remotely close—so our ability to be more agile and nimble in how we bring up office space is tremendous," Driscoll says. "If we can get you a broadband connection into that space, we can have you up 25 minutes later. We're not dealing with the 45-day [window] we had traditionally."
Yet most enterprises don't have a single, by-the-book contingency plan when WAN connectivity is scarce. Network engineers often need to do whatever it takes to get a branch up and running, says Gartner's Lerner.
"You get whatever bandwidth you can from whomever you can. You get people MiFi cards, you get them iPads with integrated 4G, you tell them to work from home, or you tell them to go to a nearby branch," he says. "You beg, borrow or steal bandwidth in any way, shape or form."
And while long provisioning times typically aren't the No. 1 challenge enterprises face with WAN connectivity -- cost takes that dubious honor -- it's certainly among the top five and has been an enduring problem, Lerner explains.
"It's been an issue for so long that it's a built-in expectation, and people have been programmed to accept it," he says.
Peter Clark, director of information services at Jordan's Furniture Co., a regional chain of furniture retail stores based in Taunton, Mass., is acutely aware of this as well. He also recently swapped out his T1 lines from incumbent and regional carriers for an alternative service. But instead of looking to the Internet as a primary WAN connection, Clark deployed Metro Ethernet services from Comcast Business at all of Jordan's six locations.
He made the move primarily for the bandwidth gains and cost savings, but Clark immediately noticed -- and appreciated -- how much faster the cable provider could provision or upgrade its Ethernet circuits compared to a telco's T1s. Comcast typically takes less than a month to complete its work, often beginning the week after he's submitted an order, he says. The same process with a carrier and a T1 line would often take three months to complete.
"That was brutal. At that time, though, I didn't know there was a better way and that things had evolved," Clark says. "Normally, I don't need something [provisioned] next week, but it sure is nice to have that option or the option to increase bandwidth rapidly too."
If a building already has access to Comcast's hybrid fiber coaxial network, customers can scale their bandwidth quickly because the process is software-based on the back end, according to Mike Tighe, executive director of data services at Comcast Business.
"We can just go into software and re-provision the speed of the virtual connection," Tighe says. "We can deliver [an Ethernet circuit] within 30 days, and then I can scale from 2 Mbps to 10 Mbps with just a phone call."
SDN helps make WAN changes within minutes
Software is only one step toward faster provisioning. The real goal is automation, orchestration and programmability -- that is, the ability to log into a self-service Web portal, submit several changes to an existing connection and see those changes reflected on the WAN within minutes, without any human intervention from the carrier. SDN enables that.
The good news is that the concept is not mere fantasy. AT&T is piloting this very capability in six cities -- Austin, Dallas, Houston, Los Angeles, San Antonio and San Francisco -- with its SDN-based Network on Demand service. It's available at no additional charge to eligible Switched Ethernet Service customers in those cities, according to Rupesh Chokshi, a senior director of product marketing management at AT&T. Eligibility is determined by whether the customer's building has access to AT&T's fiber network and is situated near its SDN infrastructure.
The service allows customers to increase or reduce their bandwidth, or adjust their quality of service policies, on existing circuits and see almost real-time changes, Chokshi says.
"From a customer perspective, you start from a simple click on the user portal and say, ‘I want to scale my bandwidth up from 10 meg to 50 meg,' and you press that ‘submit' button," he says. "Via APIs, it's communicating through the provisioning system stack to the SDN controller, to the network element, to the physical port on which the customer resides, and it makes that change in near real time."
The service has also dramatically sped up provisioning times for new connections, getting circuits installed within five days, Chokshi adds. The only on-premises work required is physically pulling the cable through a building.
While AT&T may be the farthest along, it isn't the only carrier looking at how to use SDN to speed up provisioning, among other things. Service providers are looking at using multimode optical networking equipment with SDN infrastructure to create "bandwidth calendaring" services, explains Akshay Sharma, a research director at Gartner who follows the service provider market. These would allow an enterprise to adjust the WAN according to day-to-day needs, enabling a network manager to add more bandwidth one day to accommodate a short-term spike and pay only for the exact amount of time the larger pipe was used.
Sharma predicts SDN-enabled WAN services like AT&T's will become more mainstream around 2017.
"All the major carriers have projects underway," he says, "and they're trying to bring innovation back."