Sometimes the smallest oversight can set the wheels of panic in motion. For one inexperienced field engineer, a simple error almost drove his company to pull its product out of beta testing and put it back on the drafting board.
Every "true networking blooper" comes directly from a SearchNetworking.com user. In some cases, users do not want to be associated with the events described in our bloopers. The person who contributed this blooper preferred that we use only his first name for this story.
Siva is a chief technology officer for a software company who described an interesting experience he had during a live beta trial. Siva's company created a probe that identifies voice over IP calls from a network and sends the call data recording (CDR) to a mediation device. The installation was being done by one of Siva's employees, a field engineer (FE), in a customer's leased data center. The FE was a bit unnerved because he wasn't used to the tense atmosphere in the customer's office, nor he was used to having people constantly looking over his shoulder.
Unbeknownst to Siva and the product team, the FE missed some critical test requirements that would have ensured the proper installation and setup of the product. Later, the probe wasn't able to capture voice calls. The FE, who had spent three days setting up the system, had to call in the main support team to diagnose the problem. Everybody was quite apprehensive about the probe, fearing that there was some problem with it. Siva sent the FE out to install a patch.
The result? Still no CDRs from the probe. It was capturing all the other packets, such as ping, which obviously aggravated the team. Would they have to put a hold on their beta testing of the probe and do more analysis or possibly go back to the drawing board?
Since the customer's head honcho was complaining about the product, Siva was asked to intervene. He asked the FE to provide the gateway IP address that made the call. The CTO began to check and launched Netmeeting from his desktop with the gateway. He immediately saw the "CDR" displayed in the mediation log.
After inquiring further at the data center, he finally discovered that the network installed was on trial and didn't have any voice over IP calls going through it at all. The only call that was carried by the network was the call made for testing.
Though it was a minor error made by the FE during the setup and installation, it created turmoil for Siva and his team -- who assumed that there was a problem with their product. The episode was nerve-wracking for Siva's customer support and development groups. More than a few developers lost sleep over a simple procedural error.
The moral of this story seems to be similar to that of most of the bloopers we've received: Slow down, take your time and ensure that the tasks you're performing are done correctly -- especially those rote chores -- as they can really complicate your network if not done properly!