kitson - Fotolia
In 2015, when Synchrony Financial, a $12 billion consumer financing and credit card company, spun off from parent company General Electric, IT leaders faced a challenge: untangle Synchrony Financial's core network from GE's so it could operate independently.
Ideally, Synchrony Financial's IT team would have created a duplicate of the network in an in-house lab to test all the moving parts before deploying to production. But with the fast pace of the deal -- Synchrony Financial's IPO happened in 2014 and the spin-off was complete by the end of 2015 -- there was little time to think about in-house network test labs.
"Building a physical lab would have required a lot of equipment and a lot of cabling. It would have cost way too much. And there was potential for errors because of the complexity," says Robert Morel, lead network architect at Synchrony Financial. Both GE and Synchrony Financial have multicampus, multi-data-center networks.
Instead, the Synchrony Financial team turned to Cisco Modeling Labs (CMLs), a virtual test environment for designing, simulating and testing network models. Virtual modeling environments, also available from other vendors such as Juniper and VMware, enable IT to model their existing environment, plot out changes and identify issues before they impact production traffic.
"By putting technologies under the microscope in CML, we can control risk," Morel says.
Virtual test environments are just one way that companies are avoiding the cost, complexity and, in some cases, futility of maintaining a pristine internal network test lab. Other options include third-party, independent testers or putting certain non-business-critical gear and applications directly onto production networks accompanied by heavy amounts of monitoring and alerts.
Sink or swim
Internal network test labs have diminished in popularity due to the multilayered nature of most networks and the integration with cloud and other networks that are difficult to replicate. Also, features and functionality now change so fast, keeping an in-house lab up to date requires staffing expertise and money -- two things most IT teams don't have to spare.
"Less and less customers are setting up production representative environments," says Glen Kemp, an independent consultant. "You can spend all the money in the world and lab everything up, but nothing survives the first encounter with the user environment."
Add to that most network test labs "are a facsimile of a production environment, not a one-to-one reproduction," he says.
CIO of Ohio's Portage County, Brian Kelley, says no test lab can truly represent the intricacies of today's networks, so he incrementally deploys changes in production environments flanked by intense monitoring.
"The problem today is there are so many variables you can't recreate in a laboratory situation, especially with the internet of things," Kelley says. "There are so many layered applications and devices in use it's impossible to create a pristine lab environment."
For instance, unpredictable traffic flows, such as those generated by shadow IT, would be difficult to test. "Because the network has gotten so complex, changes need to be done in a live environment," he says, pointing to a recent upgrade of the county's entire network -- routers, switches and firewalls.
Kelley's team installed the new gear site by site for the dozens of locations in the county's heterogeneous network.
"The things we learned we applied to other sites that had similar configurations," he says.
He adds that intense testing can be a burden on the eight individuals that support the county's 1,000 employees.
"We still have normal operations to support," Kelley says, calling it unrealistic to have a dedicated testing team.
The one caveat to this strategy is a critical system such as 911.
"911 is something we would want to test outside of the live environment rather than risk losing incoming emergency calls," he says. "Other things can be switched on and off if they're not working properly until you figure out the problem -- not 911."
Independent network test labs
Kevin Tolly, founder of independent testing firm The Tolly Group, says time is a big challenge for IT teams, and that's where a third-party tester could help.
"You need time to build up testing expertise. You also need time to set up a test bed/lab, validate everything is working properly, and conduct testing, analysis and reporting," he says. "It is very hard to compress this process. Months could be required, and, when you switch to a different technology to evaluate, you often have to start from square one."
In-house network test labs also require space and sophisticated testing tools -- both of which can get pricey.
Portage County, Ohio, CIO Brian Kelley
"It is one thing to deploy and manage network infrastructure; it is quite another to have the expertise to know how and what to benchmark," he says. For instance, while traditional devices such as switches and routers have Internet Engineering Task Force RFCs to guide testing, other network components such as application delivery controllers are more difficult to benchmark performance.
Joel Snyder, senior partner at Opus One, a consulting and information services firm, agrees that cost and expertise are huge barriers to the success of in-house labs. A lab, he says, has to be ready all the time due to a shift from large, dramatic network changes to small, incremental changes.
"Software changes to the network or functionality changes are more common than hardware forklifts," Snyder says. "It gets expensive to have a real test lab, and it's hard to maintain specialized knowledge." Third-party testers, he says, are able to amortize equipment across all their clients, enabling them to always have the most current gear.
Synchrony Financial's Morel says while he has seen successful third-party testing, having his team do virtual modeling has been helpful in working through new ideas.
"In CML, we can very quickly position routers and links in that type of topology and then make adjustments," he says. "It's pretty much identical to what runs on the real router."
Synchrony Financial uses a hybrid approach where real devices are connected to a simulation of multiple data centers to test the performance of routers.
"Real life is always a little more complicated, but modeling the network virtually enables us to proactively get ahead of some problems," he says. "We can catch things in the lab before they come into production."
For instance, the Synchrony Financial team recently modeled the impact of Cisco Intelligent WAN technology on the network and found that in certain failure scenarios, it would have caused outages.
Router upgrades, he says, is one area where he would still rely on a small in-house network test lab.
"CML is more about new features on hardware rather than the hardware itself," Morel says.
In his previous IT leadership positions, Dwight Gibbs, an industry veteran with more than 20 years of experience, would maintain a mirror of a production network to facilitate testing. But nowadays, Gibbs, CEO of procurement management automation startup Contraqer, says spending a couple hundred thousand dollars in capital expenditures for testing is unacceptable -- especially when viable alternatives to physical network test labs exist.
Gibbs uses the Microsoft Azure platform as a service (PaaS) to spin up a virtual clone of Contraqer's production environment, which is also in the cloud. He then develops routines to create data -- instead of using sensitive production data -- to test code changes. The clone environment does have some differences though, such as fewer, less powerful servers and the absence of some of a production environment's complexity.
"You can't replicate it exactly -- it would be difficult and expensive," he says. But using the PaaS to create a virtual test environment gets him closer to production-level more cost-effectively and efficiently than maintaining an in-house network.
To account for any gap between the virtual test bed and the realities of production traffic, he relies on a host of monitoring tools, including New Relic, a real-time application performance management tool, to quickly spot errors and anomalies such as an increase in page rendering time.
"When we deploy [changes], we are more focused on monitoring than usual for the next week to see what kind of unusual data conditions we're going to hit," Gibbs says.
Kemp says cloud-based applications let professionals get as close to production as possible without investing heavily in physical network test labs.
"You build something up and tear it down. You can run through all the iterations and sanity check designs you're proposing," he says. "And you can produce perfect documentation and avoid having outages because, in theory, you've already walked through every combination."
But, Kemp warns, no matter what testing platform IT uses, the true test still comes when the changes are in front of the user.
DIY: Construct your own virtual test lab
How to put an enterprise network to the test
Testing company Ixia wins Network Innovation Award