Rawpixel.com - Fotolia

News Stay informed about the latest enterprise technology news and product updates.

Cisco refused to participate in NSS Labs report on SD-WAN

Cisco wouldn't activate NSS Labs' Viptela license after the testing firm paid between $30,000 and $40,000 for the technology for a comparative NSS Labs report on SD-WAN products.

Cisco refused to activate the Viptela software-defined WAN product NSS Labs bought for testing, leaving the research firm with a noticeable hole in its recent comparative report on SD-WAN vendors.

Cisco did not provide a reason for refusing to activate the product NSS Labs had purchased for between $30,000 and $40,000, NSS Labs CEO Vikram Phatak said this week. "There was no reason given other than, effectively, they didn't want to be tested (for the NSS Labs report)."

Cisco's action marked the first time a vendor had refused to turn on a product NSS Labs had bought for evaluation, Phatak said. Cisco's Viptela team had initially told NSS Labs it would support the test, which led the firm to buy the product.

"That's a first for us, candidly," Phatak said. "And given Cisco's ethical rules and so on -- rules of conduct -- I'm in shock because normally, they're pretty straightforward to work with."

Cisco's response

After initially refusing to discuss the matter, Cisco released a statement that said NSS Labs's proposed testing methodology "fell well short" of reflecting "the enterprise-grade nature of customer SD-WAN requirements."

"We shared real customer deployment requirements with them, but our feedback wasn't incorporated," Cisco said. "At that point, we decided not to participate in the test."

Not participating, however, usually doesn't mean the vendor will prevent the test from going forward. Typically, when a company refuses to join a test, NSS Labs will conduct it alone.

"If someone says they don't want to be tested, we say, 'That's great, but if a product is good enough to be sold to the public, it's good enough to be tested,'" Phatak said. "We're going to buy it, and we'll report to the public."

NSS Labs wants a refund

NSS Labs wants Cisco to refund the money spent on Viptela. It is hoping it can get the money back without going to court.

"I hope it doesn't come to that," Phatak said. "We haven't talked to any lawyers. I'm assuming that we'll be able to have the conversation and get our money back."

Typically, NSS Labs buys products, and the vendors turn them on like they would for any other customer.

That's a first for us, candidly. And given Cisco's ethical rules and so on -- rules of conduct -- I'm in shock.
Vikram PhatakCEO, NSS Labs

NSS Labs noted Cisco's refusal to activate the Viptela purchase in its SD-WAN Comparative Report, which was the company's first SD-WAN test. Not having Cisco in the evaluation left out one of the largest SD-WAN vendors and a major tech company.

In the first quarter, London-based IHS Markit listed Cisco as No. 4 in the SD-WAN market, just behind Silver Peak. VMware was first with a 19% share, followed by Aryaka with 18%.

The NSS Labs report, released this month, compared the products of nine vendors, including VMware's NSX SD-WAN, formerly VeloCloud. VMware is Cisco's largest competitor.

NSS Labs had also planned to include Silver Peak in the comparison but noted it was unable to obtain the product in time for testing.

Tech companies often cite recommended ratings in NSS Labs reports in marketing materials. In April, Cisco highlighted in a blog post the organization's "recommended" rating for the Cisco Advanced Malware Protection for Endpoints product.

Based on its recent SD-WAN tests, NSS Labs recommended products from VMware, Talari Networks and Fortinet and listed products from Citrix Systems, FatPipe Networks, Forcepoint and Versa Networks as "verified." Tech buyers should consider recommended and verified products as candidates for purchase, according to NSS Labs.

The company issued "caution" ratings for Barracuda Networks and Cradlepoint, which means companies should not deploy their products without a comprehensive evaluation, NSS Labs said.

Dig Deeper on Software-defined WAN (SD-WAN)

Join the conversation


Send me notifications when other members comment.

Please create a username to comment.

Should Cisco refund the money NSS Labs spent for the Viptela SD-WAN?
Playing devil's advocate, if a product is publicly available, shouldn't buyers choose how to use it? Consumer Reports purchases products for testing all the time as a way to maintain independence. Why should tech companies be treated differently than any other business?

That’s exactly why we purchase if we need to. Public testing is not sponsored testing, we perform it at our expense with independence. Our view is if it is good enough to market and sell it is good enough to be tested. This is the first time in our experience a vendor has refused to service a customer. Imagine buying a truck and finding out that the manufacturer refused to let you drive it the next day because they thought you intended to test that the brakes worked before taking it on the highway.
Hi, Mr. Brvenik. Thanks for joining the discussion. Since NSS Labs will buy the product or get it for free from the vendor, what happens differently if it's the latter. What's the impact on the testing process? Also, how do you pick the vendor to test? Is it based on a poll of your enterprise clients or are there other criteria?
Hi Antone. We invite vendors to participate and configure the systems appropriate for our published methodology. If a vendor declines or is unresponsive we will purchase it and either configure it using their documentation in a manner recommended there and appropriate for our published methodology. If the product requires an “expert” or we suspect through testing that there is a problem with the configuration that’s not obvious through the documentation or support we will bring in a qualified consultant to configure the product. In this case we brought in an experienced consultant early on and that’s when we discovered that Cisco refused to activate the product. The testing process remains the same for all products being tested and scoring and validation is conducted the same for all products tested. The appropriate vendors are identified through analysis of analyst reports, revenue, vendor claims, primary research, etc. we will also take feedback from our enterprise interactions and customers. From there we assess appropriateness for the test and produce a selection. We also announce testing through press releases and perform outreach. We won’t prevent someone from submitting products to one of our tests and we never charge for participation in a public test.
No - don't give it back. My previous company was doing NSS testing for years and everyone should know that NSS does not buy gear - the vendor provides it. If NSS buys gear against the vendors will, it is to start stuff like this. They have done it many times before and will go whining publicly again about something else probably many times in the future. Completely irrelevant mumbo-jumbo from NSS who must be sore about something else and knowing SD-WAN like I do, I would put money on Cisco chose not to participate in the test because it sucks. It is 1.0 and doesn't even come close to testing some of the most important SD-WAN capabilities.  Stay strong Cisco - I wouldn't have run the test either. It is useless in its current form. NSS should have invested that money into developing a better test instead of trying to start a $h*t$t0*m to cover up their own ineptitude.
Actual NSS CTO here. Clearly you’ve an issue with NSS and I am okay with that. While it’s unfortunate, some products simply aren’t as good as they’d like you to believe. It is also unfortunate that a some vendors would rather make us out as the enemy than own up to their shortcomings. We get it. Bottom line is we will keep testing and reporting our findings and eventually those products will fix their issues. If anyone wants to know how things actually work start at the link below and if you have questions feel free to send me a note. https://www.nsslabs.com/security-test/nss-labs-test-policies/
My only issue with NSS Labs is the vendor bashing when things don't go your way.  Extremely unprofessional, especially when we are supposed to 'trust NSS' implicitly. As the public, we don't ask for it, care to hear about it and if you want to be respected and trusted by the industry, you should take the high road instead of reporting junk like this.  It is completely unnecessary and should remain in private channels.  I know your reports are expensive and if you aren't creating a stir up in the industry the reports don't sell, but show some respect for the industry that we all have to exist in.  I take personal issue with any CEO that randomly posts an article like this, out of the blue, to bash someone else when no one is looking or asking, is immature, unprofessional and simply adds fuel to the fire of dwindling credibility, just to sell a report.  Who cares what happens between NSS and other vendors behind the scenes. Seems it is your job to minimize the bad press versus actually creating it.  I have searched and don't see any commentary anywhere that would or should have sparked this random rant by your CEO.  In fact, the Viptela EULA is pretty clear that neither an evaluation nor actual license can be used for this type of testing without approval, so it should not have been a surprise to anyone: "Customer will not, and will not permit others to, whether directly or indirectly: publicly disseminate performance information or analysis about the Products including, without limitation benchmarking test results, " So it appears you didn't get a license allocated right in line with the EULA. Does NSS feel they are above the law?
Reporting the truth and responding to questions about it isn’t bashing and while I understand you might not like the truth, it doesn’t obviate it. I ask if you also believe that an automaker should be able to hide poor performance? You buy a supercar that claims the best gas mileage ever of 5000mpg and a 0-60 in 1.83 seconds. You take it on the highway and can’t make it go faster than 55 and can’t do it in less than 7.4 seconds downhill - without burning 250 gallons of gas per mile. You take it to the track and it’s worse because it’s level ground. Hmm? I’m pretty sure you’re well within your rights to post a review of that car with all of your results in the car and driver forums, on your blog, in the local newspaper, and any other place you feel appropriate. I know I will. You wouldn’t?
First, I wouldn't buy a supercar with those kind of stats without my own test drive and I certainly wouldn't trust a 3rd party to provide me those stats if they published the data illegally and or complained because the car maker didn't allow them to publish the data illegally.  It sounds to me that this debate continues to go back to a vendor bias - based upon the way you continue to respond i.e. " while I understand you might not like the truth, it doesn’t obviate it" So, you know the truth then?  What is it?  Are they really hiding something, or is that simply your opinion? Who said anything about hiding poor performance anyway? 

Besides the hopeful opportunity that I would get to debate the great Jason Brevnik, my initial point has nothing to do with any product performance, rather has to do with being honorable about how results are obtained and published, regardless of the vendor in question.  If this was the first time this had happened with NSS in the news, then no big deal. But it seems to be something we now expect with nearly every test released by NSS over the past year or more.  Palo Alto Networks, Crowdstrike, FireEye and others have had to take a public beating to save their reputations and to enforce their own legal responsibilities. I mean really, who is NSS to try and control the industry view?  Are your tests perfect? Can you prove that? Is there no deviation whatsoever from what is seen on an NSS test and in the real world at the millions of varying sites that house the diverse IT environments in 2018? You can't possibly cover every scenario. That said, NSS should consider abandoning the victim role and listen to their own truth as they expect others to.  You can't really expect a sane person to think that you are perfect and completely innocent in all of these scenarios.  At some point a coincidence is no longer a coincidence and a trend is born. 

At the end of the day, when things are done within legal parameters, under a non-bias and using fair reporting, then I agree with the mission-statement. However to spend time complaining when someone doesn't allow you to violate the law to build your report, there is no room to debate, especially with personal undertones.  Doesn't automatically indicate they have something to hide. Other vendors did not participate in this test, such as Riverbed, and are not receiving the public scrutiny that a particular vendor (Viptela/Cisco) did for enforcing the data contained within their EULA.  

Thanks for your opinion. I’ll remind you and everyone else that might read your commentary that NSS broke no laws. There is no bias. We perform perfectly legal and independent testing. The truth is Cisco sold us a product and then refused to honor that contract after taking payment for it. If you think otherwise (and it seems you’ve got a vested interest in thinking otherwise) please feel free to step up from anonymous commenter to official person with a complaint. I am more than happy to review all of the data and facts and communications and people that were and are involved. If you don’t know how to reach us or me please visit http://www.nsslabs.com and follow any number of paths.
Nope. No need. You know me - from our dog days at SF. Working for a partner now; a partner who knows we were both reimbursed for this. Amazing how deception and animosity can fire up such a great public debate. Thanks Jason. I will just call you.
Not following your comment. If you know me then step up and be honest about who you are. Quit hiding and represent the values we championed and lived by at SF.
Nice Try. Look in the mirror - are those values represented? I don't agree.

My comment was to set the record straight for the readers; we reimbursed you for the SD-WAN gear you bought from us (partner) and Cisco reimbursed us accordingly and enforced the EULA.  They did nothing wrong and neither did we. Everyone knows that Cisco doesn't sell direct to small companies. It is always through a Partner Channel, so your claims of purchase/refund issues directly affect our reputation as well.  

This story is fake news just to create controversy so NSS can sell some reports, bash the 'vendor of the week' by claiming deception or distrust and pretend to be the victim yet again.  There is no point to this discussion, nor the claims made by your CEO.  Bye
Yeah. Nice try. You’re definitely not the partner and you’ve not reimbursed anything. Good luck. Keep trying.
Just noticed your “I take issue with any CEO that randomly posts” claim. Seriously? You do see that this is an article written after an interview and not a blog by him, yes?
oh absolutely.  I was just trying to fire you up to inspire the debate. Worked. Surprised it took you two reads to pick up on it though. ;-P
The same can be said for Barracuda who also did not participate in test and were surprised to see their product included in report.
the test...the report...
We always invite participation in an NSS test however it’s not required and providing gear isn’t mandatory. That’s wholly separate from actually being tested. Being tested isn’t at the vendor discretion. Short version is if you are market leading or our enterprise customers request a product be tested we will test it. Barracuda was absolutely aware of the testing and declined to participate and that’s okay and happens with different tests and vendors all the time. It’s even happened with vendors on the same tests at different versions. Some times it’s just a scheduling problem for them or they are in the middle of a release etc. No problem, we purchase and test what customers are using.
How about NSS shares the feedback Cisco provided about feeling the test was lacking? Sometimes you are comparing apples to oranges especially with next gen technology such as SD-WAN.  Be more open on the feedback that was not incorporated and you might not get so much negativity in the comments. You've got a whole article here with little detail on the substance the industry cares about (why?) ....said NSS Labs's proposed testing methodology "fell well short" of reflecting "the enterprise-grade nature of customer SD-WAN requirements." "We shared real customer deployment requirements with them, but our feedback wasn't incorporated," Cisco said. "At that point, we decided not to participate in the test."
The statement was added later however that is an aside. NSS Labs' mission is to advance transparency and accountability and I am happy to share. I'm certain only some people will read the details however they are below if you are interested. It would seem to me that because we did not accept their desire to remove performance from the methodology it didn't meet their needs? Perhaps instead they wanted to see split tunneling from vEdge as a requirement? I'd assert that split tunneling (or dynamic) is core to an SD-WAN deployment and already represented in the impairment tests though not a targeted test.

Cisco provided the following feedback (their Proof of Concept plan) with our responses denoted by NSS: 
--- feedback below. Names removed and minor edits for readability. 

Here are the basic SD-WAN test plan cases that we run through for POCs that reflect the basic enterprise requirements that customers expect. Note that none of them are performance tests and there are many more tests that can be conducted.
1. System Bring-up and Baseline
 - Validate overall network health
 - Verify operational state of all control elements
 - Verify operational state of hub site vEdge routers
 - Verify routing (OSPF or BGP, connected, static and OMP)
 - Basic Connectivity Testing
NSS: Validating the sdwan setup is done before any of the tests are performed and includes many of these functions in order to proceed with any testing. If a vendor solution cannot route, it cannot be tested and is not SD-WAN.
2. Zero-touch Provisioning
 - Demonstrate Zero-Touch provisioning
 - Show template configuration of vEdge in vManage
 - Verify control-plane and data-plane are up
 - Validate reachability after the bring up
NSS: We do have test cases which will capture the operational benefits of “ZTP”. This is covered in the test methodology under the cost model (as the benefits directly drive a lower operational burden, therefore a lower TCO.)
3. Segmentation
 - Demonstrate separation exists for different VPNs
 - Demonstrate Hub-and-spoke VPN co-existing with full-mesh VPN
NSS: As depicted in the methodology, VPN’s(both full mesh and hub/spoke) are an inherent feature of an SDWAN offering and its validated during the setup.
4. Service Insertion
 - Demonstrate traffic engineering for service advertised from datacenter
NSS: Traffic engineering, steering, selection etc. is covered in wan impairment section
5. Application Aware Routing (AAR)
 - Generate packet loss and latency on the network
 - Observe voice traffic taking MPLS circuit when SLA is not met for internet
NSS: Thoroughly covered in wan impairment section 
6. DC Routing
 - Demonstrate sites in east prefer datacenter east and vice-versa for sites in west
NSS: Not a model that we cover today, but one that warrants discussion for 2.0. 
7. Enhanced data plane security        
 - Change IPSec key and observe no traffic impact
NSS: VPN authentication/encryption is not restricted to IPsec for this test and whatever vpn solution a vendor provides will be validated as part of the setup
8. High-Availability
 - Fail control components and links to assure high-availability
 - Reboot controller, branch vEdge, datacenter vEdge
 - Fail MPLS and internet links for branch and datacenter
NSS: Failover is covered in the methodology.  Specifically testing the controller or central management appliance for sdwan is not within v1 scope
9. QoS
 - Demonstrate QoS on vEdge device by generating traffic across several traffic class.
NSS: Covered in wan impairment section
10. Upgrade
 - Upgrade vEdge device from vManage
NSS: Not sure how critical this is as this ties into testing central management device. One of the vendors used in the smoke test does all config and provisioning via central management.
11. Split Tunneling
 - Demonstrate ability to do split-tunneling from a vEdge
NSS: This a possible v2 candidate but not v1
12. Deep Packet Inspection (DPI)
 - Measure identification of web applications
 - Demonstrate capability to white-list applications identified via DPI
NSS: Thoroughly covered in security section 

The bits in this discussion around whether Cisco should be refunding money back to NSS or not and under which circumstance is irrelevant IMO.

The real issue here is, if the license was indeed PURCHASED, that license must be activated. To fail to do so it is denial of lawfully-purchased right-to-use and this could very likely end in actionable legal recourse on behalf of NSS.

But to entertain Cisco's stance for a moment, if they refused to activate the license to prevent testing, claiming that the test "fell short" in standards, Cisco should disclose in detail that those "standards" are and what was left out by NSS. Only then is Cisco's stance closer to being understandable by anyone who is not a Cisco diehard fan.

please see above for what was provided. 
SD-WAN has become a 2-horse race between Cisco/Viptela and VMware/Velocloud. These two players have the largest installed base in their respective markets and have chosen to add and bundle SD-WAN functionality into their offerings. NSS is "making market" with the far followers in the segment with its head to head comparisons and giving them a marketing message to trumpet vs the big two in the category. Cisco did the smart thing. It's like a candidate in an election refusing to debate a long shot candidate. The best path to market now for the remaining vendors is OEM and/or white label deals with vendors of record that have a loyal installed base to upsell into. Landscape dominance is a fact of life in tech. Always has, always  will be. Not a "food fight" or flat market. And technical requirements are not the determinant in many settings, especially networking. Vendor trust and installed base loyalty bigger consideration in many categories. The  SD-WAN "landgrab" is over. Cisco and VMware won.
Should Cisco should return unearned fees to Viptela? If unearned fees are involved, sure refund the unearned portion. However, if Cisco doesn't agree with the NSS test method or scope or approach that will be published then Cisco's refusal to participate is their business and they have no duty or requirement to NSS except to say - "we have changed our mind."  If the Cisco license does not specifically forbid publishing test results...then NSS should do what they want. However, I am reading this as Cisco is not in agreement with the test criteria and this is the real issue -- Is the NSS test adequate and fair to the product?  I don't know.

Hi everyone, I'd like to thank the participants in this lively debate. As a point of clarification, NSS Labs did not contact me to discuss their dealings with Cisco. CEO Vikram Phatak's comments started when I asked why Cisco declined to participate in the test, as noted in the NSS Labs report. Now, to add my opinion to the discussion, independent testing of technology is an important service to tech buyers. While we can debate whether the test has shortcomings, I don't believe vendors should be allowed to prevent testing of publicly available products. How would that benefit anyone but the vendor?
How would that benefit anyone but the vendor? 
First, thanks for a lively topic and I know I'm preaching to the choir here so please bear with me.
Certainly independent testing is valuable but the performance evaluation of the test use-cases must be relevant to the intended environment to demonstrate "fitness for purpose and use".  If the test is not measuring "fitness" relative to the consumer need -- a skewed interpretation makes the consumer choice more difficult. 
So, who benefits? No one, the consumer and the vendor are both punished.

I agree. Please see the comment above containing the feedback provided and the methodology freely available on research.nsslabs.com if you think there is a mismatch there please reach out and let us know, we love to hear feedback.