Bloopers make us laugh.
Think of the outtakes at the end of a TV show, or the bonus scenes on your favorite DVD. We all make mistakes in our jobs, some more painful than others. Even if you've never goofed up -- or don't want to admit it -- we bet you know someone who has. Besides being darkly humorous, first-person accounts of big networking blunders provide great "how-NOT-to" examples for our readers.
This is SearchNetworking.com's first in a series of networking blooper stories. Every article in our series is contributed by a networking professional. For obvious reasons, some contributors choose to remain anonymous. Networking pro John Dean bravely allowed us to use his name, remarking, "I stand behind everything I do good. I may as well stand behind something I did in a less-than-stellar fashion." We appreciate Dean's forthrightness.
Dean is an IT director for an engineering company that makes Internet-aware hardware, as well as programming and databases.
He claims to be known by his employer and coworkers as someone who's very careful when performing tasks such as network deployments and building servers, and he says he doesn't make mistakes often. But he points out that sometimes -- even when you do everything right -- you'll find there is something at work that you didn't even consider. This is his tale:
Dean had just purchased a new server for his Exchange needs. His current server was just a bit underpowered and outdated. Disk space wasn't an issue, so when he ordered the new server he ordered it with three 9G byte SCSI drives for the RAID 5 array, knowing that he still had about 10G bytes free on his current 18G byte drive. He thought there was no need to spend more money on additional disk space.
The glorious day arrived when Dean got his server. He announced that he planned an outage for that evening so he could do a quick, smooth swap of the servers. He built the new one as needed, set up the RAID so the three 9G byte drives would be RAID 5 -- and installed everything. "No errors, no muss, no fuss," Dean said. "Within about three hours I had the new e-mail server up and running."
Sounds too good to be true, doesn't it? Well, a few days later Dean was doing some of his daily tasks, and noticed that there weren't 10G bytes free on the drive, but more than 20G bytes! Knowing that three 9G byte drives in RAID 5 can't produce a drive that would still have 20G bytes free, he swore at himself and planned another night of redoing it, this time making sure that the system recognized a RAID 5 array. "I had an inkling that the RAID configuration program just spread the drives out with no parity, leaving me with just a large array but no redundancy," he said.
So again, Dean scheduled a night of maintenance. He went through the exact same process, but this time he watched to make sure that the drives took as a RAID 5 array. Everything was reinstalled, he moved the databases back and got the e-mail server up and running. Again, he found no problems, and everything worked beautifully. But then he looked at the drive, and guess what -- he still had more than 20G bytes of free disk space.
As Dean continues his tale, he describes going through the entire process, again… for a third time. But this time he didn't let the RAID software create automatically; he actually performed every step manually. He broke the old RAID set, selected the three drives, selected RAID 5 and had it create the set. When it was finished, he rebooted and went back into the RAID program to verify that it was RAID 5. Sure enough, it was. But this time, there were 36G bytes free. Again, Dean swore, and took a look at the front of the server where the hot swap drives were.
That's when Dean discovered his problem. Even though his purchase invoice showed he had bought a server with the three 9G byte drives he had ordered, the server actually came with 18G byte drives. Evidently, the manufacturer didn't have servers with 9G byte drives available and instead sent him a larger unit, even though his receipt showed something else.
This tale winds down with this moral: Don't automatically assume you or your machine screwed something up. When you get an unexpected result, look at every possibility. Take it from Dean: "Sometimes the worst mistake you can make is to try to fix something that wasn't broken in the first place."
We hope you enjoyed our first Networking Blooper. Stay tuned for more.
This was first published in September 2002