When an enterprise grows, the homegrown network troubleshooting methodology and management practices that network...
administrators have used for years just won't scale anymore. Brian de Loureiro, manager of systems and integration at Raleigh-Durham Airport Authority realized that quickly when the network at his airport started to expand rapidly a few years ago.
"We were kind of a small shop," de Loureiro said. "We were hoping users would call us when they were having problems, and we would investigate the network devices or the servers. We'd start to troubleshoot by logging into devices. Then we'd do the typical ping and trace route approach."
The network started to grow rapidly, especially when the airport built a second terminal and started to attach video surveillance and other airport systems into the network. It soon grew to 4,000 IP-addressed devices and 180 switches with 7,000 ports. Not only did the airport need a new network troubleshooting methodology. It also needed a new approach to IP address management
"Our staff grew from three to 19 in two and a half years, and we really needed to have a good pulse on how to monitor everything because our network almost doubled," de Loureiro said.
Shortly before he joined the organization, the airport purchased ScienceLogic's EM7, a systems, network and application management platform aimed at midsized companies. The product didn't work out.
"It just required too much care and feeding. It wasn't quite configurable," de Loureiro said. "It didn't have all the features out of the box, and it required we take almost a manual approach to configuring devices.
Knowing that he'd need to upgrade to something else, de Loureiro started looking at vendors like HP Openview and IBM Tivoli, but the price wasn't right. Then, while looking for an IP address management tool to replace the Microsoft Excel spreadsheet his organization maintained, he stumbled across SolarWinds and its free IP Address Tracker. Soon after that, he downloaded a 30-day demo version of the SolarWinds full Orion product suite.
"I downloaded [Orion] and installed it, and within a half-hour it started doing stuff that our previous network management system didn't do," de Loureiro said. "With the old system, we had issues where it wouldn't recognize a Cisco switch as a Cisco switch. We had to go in and change things manually. SolarWinds knew right away the make, model and manufacturer and had the basics of the system just by discovering that node. With the old system, it was almost 50-50 that the system didn't know what the device was, or it would recognize a voice gateway as a router or a switch as a server. It involved too much time to maintain it."
After running the demo, de Loureiro decided to abandon ScienceLogic and put Orion into production. His entire IT organization is using it, including the network administrators and the Tier 1 and Tier 2 support teams.
Dashboard with nested maps improves network troubleshooting methodology
Orion sits on a PC in the IT operations center, with a dashboard displayed on a widescreen monitor hanging from a wall, de Loureiro said. Using SolarWinds Network Atlas feature, his team built a custom map for the Orion dashboard that allows the team to pinpoint problems on the network quickly.
"The map looks kind of like a spider," he said. "It's got our environment laid out by building, and then under each building we've got a network for that building with a nested map. Then we have our systems kind of lined up on the other side of the map. So we're monitoring it all day, and if there's something red [on the dashboard], we check into it. We've got critical devices that we have set up as alerts, such as external routers, the firewall, certain ports that have critical devices on them. It's monitoring all levels of status so we can see things put into a maintenance mode that's expired."
As a result, de Loureiro's network troubleshooting methodology has been transformed.
"If we have a power outage or power blip and a device stops responding, then that building will blink red [on the dashboard]," he said. His team can click on that building and immediately see that the problem device is on a certain floor at a certain end of the building and yet all the switches near it are up.
"So we can quickly diagnose that there is something wrong with the power [on that device], and we'll go and find out why the UPS battery didn't work," de Loureiro said. "By having it all in a nested map, we know where to go right away, rather than having someone just explore out on the campus."IP address management without Excel spreadsheets
He is also using the Orion IP Address Manager (IPAM) to improve IP address management, and the Excel spreadsheet is a distant memory.
"We're managing almost 5,000 addresses in our environment now," de Loureiro said. "When a new project is coming in and we know what the IP addresses are going to be, we can go into [IPAM] and reserve them ahead of time. As the contractors come in to set it up, we can print out a list and hand it to them. We don't have to worry about somebody maintaining a list in one place while someone else is maintaining it somewhere else. Now, everyone on staff can go in and generate a scan and set up a reservation and look up an IP address."