The incoming e-mail had a terse, panicked tone. The sender wondered if I’d I seen alarms for the switch stack in a particular rack because he had lost connectivity to most of his virtualized environment. I hadn't, but I committed to checking into it further while he made his way into the office. Parsing through the switch logs and reviewing the status of interfaces, I noticed that three of four uplinks were down, and found that they’d been offline since the previous evening. Many other interfaces had also changed status at that same time. Some switch ports had come up, while others had gone down.
Walking to the rack in question, an uncomfortable truth was uncovered: the rack door was open, and someone had physically rearranged the cabling. Most of the Ethernet cables that had been plugged into one switch had been randomly moved to another switch in the rack. We’d been the victims of a physical security breach.
The facility in which this occurred had appropriate security policies in place and generally accepted security mechanisms in use. Access to restricted areas inside of the building required a coded keycard, and all door accesses were logged. Policy required that racks were physically locked; only a few people were issued keys. Wall partitions extended above the ceiling plenum. Most facilities have at least this sort of basic protection in place. So what broke down?
What so often breaks down is enforcement of existing policies. Your physical security policies as written are probably sufficient to protect your facility, but are your people following the policies? Do you have a process to check for and enforce compliance? Let’s talk through some items to help protect your site from a physical security breach.
- Train staff regularly on security policies, procedures, and their logic: Employees, especially non-IT employees, prize convenience more highly than security. Regular reminders from the security team about physical security policies and the logic behind them will help develop a sense of care and concern among the employee base, and they’ll be more likely to adhere to a security policy. Having a new employee sign a form acknowledging receipt of the corporate security policy is an appropriate first step, but don’t let the education end there.
- Use video surveillance to monitor secure spaces, especially points of entry and exit: If an elaborate video monitoring system lacks appeal due to cost or complexity, consider that simple IP cameras and storage are easy to deploy and relatively inexpensive. Another common objection to video surveillance is that it violates personal privacy or sends a message of mistrust. The images that will be captured by the cameras will contain almost exclusively employees and contractors. Can’t we just leave them alone and let them do their jobs without being the proverbial Big Brother? Let me gently suggest that trust in your employees is at times misplaced. Think about this philosophically for a moment. Who has access to your physical spaces? Who has the greatest reason to be emotionally charged about your company? Who understands how your security policies are administered in practice, and therefore how to get around them? In other words, while your employees are your greatest asset, they are also a potentially great risk. A video print keeps everyone honest.
- Audit existing systems proactively before an event occurs -- not reactively after a breach
- If you use a keycard system and coded badges to allow employees into the building and secured building areas. Is the logging working? When was the last time you checked? If the logging is working, are the timestamps correct? If the timestamps seem correct, have you verified that they are adjusted for daylight savings time? How hard is it to pull a report from the system? Are all the keycards with access accounted for? Do you expire badges that haven’t been used for a certain amount of time? Have you reclaimed badges from terminated contractors or employees?
- Check your clean desk policy. Employees must lock away company confidential information at the close of the business day and securely destroy waste paper with confidential data. When’s the last time you audited an office for compliance with this policy? Random spot-checks of desks, trash cans, and lockable areas will develop a consciousness on the part of the employees about their work area. If you never check, only the most disciplined of employees will maintain a compliant clean desk.
- Check data center locks. Periodically perform a data center walk-through and verify compliance with this policy. If you find an unlocked door, try something dramatic like taking the door off its hinges and walking out of the data center with it. If you know who the offender was, leave the door at their desk; there’s nothing like a large rack door left at one’s desk to make an impression on both them and their co-workers. Often, such visual demonstrations are more effective than threatening e-mails or closed door sessions.
There are many more examples that could be cited, but the point is that you want to do preventative maintenance through auditing, not post-breach analysis through forensics.
While firewalls, stateful inspection, access lists, intrusion prevention, encryption and event correlation will help prevent the network-based attacks that make the news, they won’t keep someone from walking in the door if you’re not paying attention.
About the author: Ethan is a network engineer, blogger, and CCIE #20655. He's also a host on Packet Pushers, an independent podcast covering the data networking industry. Follow Ethan at gplus.to/ecbanks or @ecbanks on Twitter for social networking.