Undocumented and untested changes -- coupled with a touch of overconfidence -- can produce very negative results.
The person who told us about this week's networking blooper prefers to remain anonymous. So here we'll call him "Harry." Harry was working in the IT department of a large hospital. At blooper time, he was managing a help desk, supervising five system analysts.
"The network administrators had a habit of making global changes to server/router configurations without testing the changes first," he said.
The network admin at the heart of this tale had been promoted from network tech only a month or two before this particular issue surfaced. Harry described this person as a "gifted young man, but overconfident and unbloodied by real-life network issues."
One morning the help desk began receiving hundreds of calls from users who had discovered they had access to certain folders on the network that they probably should not have had access to. Somehow, every user in the enterprise had full access to information from the accounting and human resources, as well as the CEO's and company president's files. Included were financial and salary records, contracts and employee evaluations. Can you imagine the uproar from those departments? Can you hear the "Wahoo!" from the clerical staff as they read their supervisors' latest performance evaluations?
Harry walked up behind the newest and youngest network admin to check whether any network changes had been made -- and was treated to an explanation of some utilities for viewing and modifying ACLs (Access Control Lists). This admin had been experimenting with auditing permissions from the command line and had made a bigger splash than intended. It didn't take more than 10 minutes to fix the ACLs -- but that was almost 16 hours after the initial mistake had been made.
The moral of the story: The command line can get you into trouble a lot faster than the GUI, and undocumented and untested changes -- topped with a generous helping of overconfidence -- can be a recipe for trouble.