It seems that the IT Industry is willing to accept that software bugs are unavoidable and that licensing agreements, along with patches, absolve vendors from any responsibility. That may be why there is so little hubbub around what I sense to be an increase in network software problems – and specifically Cisco IOS bugs.
It's not that bugs in general are a new issue. Microsoft releases between 20 to 60 patches per month for critical bugs. But with Cisco IOS software, I have noticed a significant decline in product reliability over the last two or three years, which is suspiciously the same timeframe as the company's financial problems. Maybe I am paranoid, but I have to wonder if Cisco is cutting corners on testing and validation programs in its Indian development centers.
I’ve learned that IOS software development is segmented into verticals: BGP, IP Multicast, OSPF, MPLS, etc. All of these are developed in independent teams with their own budgets and management. But there seems to be a gap in end-to-end testing. For example, I wonder if there is testing of BGP and IP Multicast integration or MPLS and OSPF integration.
Why are bugs so troubling in networking?
In an ITIL-compliant world, bugs are an identified risk and projects allocate hundreds or thousands of man hours to testing and validation in an attempt to locate product flaws. The cost of customer-driven network validation and testing has risen dramatically in the last five years. The trend is proven in the wide range of new testing products and solutions.
On one hand, this is not a bad thing as we can now build better networks. But for every bug found, the network is undermined. There is already a significant perception in IT management circles that the network is unreliable and risky. That’s why getting change windows for regular upgrades is almost impossible.
When will vendors do more?
Some people say that vendor technical support is here to fix these problems, but that's not why I pay for this service. I pay tech support for hardware failures, software upgrades and configuration support, not to receive a half-finished product.
Which leads to the question: Are vendors delivering faulty products? If customers are going to perform their own testing, locate bugs and then advise the vendors through tech support programs (paid for by the customer), then what motivates the vendor to keep software quality high?
It is true that the complexity of modern products means that some bugs or product flaws will occur. But if vendors scale back their testing programs to save money, who suffers? And who will know?