In most IT organizations, when an application crashes or slows to a crawl, everyone assumes it's the network's fault. It's usually not true, but the persistence of an IT silo mentality places the burden of proof on the networking team. If everyone in IT had access to the same infrastructure monitoring tool, perhaps the network would stop being the scapegoat.
"We've had application issues and the thing we love [sarcasm alert] to hear is that the network is slow or 'My application doesn't work because of the firewall or load balancer,'" said Todd Marsh, senior manager of network infrastructure services at Bentley University in Waltham, Massachusetts. "From my point of view, there was always the expectation that we [the network team] would go out in solve the issue, which is a very time-consuming thing."
Marsh long believed an application performance monitoring tool that everyone could use would solve this problem and force other teams in his organization to solve problems without involving the networking group.
"We have a lot of tools that monitor what's going on in the network, but unfortunately they only make sense to network people," he said. "I've tried a lot of application monitoring software, but most of them require installing an agent. It's never made it past development because there is a fear that you're going to break something."
Then Marsh tried ExtraHop Networks' agentless, IT operational intelligence appliance. ExtraHop collects wire data from every system within an environment, from packets to logs. It decodes every wire protocol, such as HTTP, SOAP, SQL and LDAP. And it can analyze all this information, giving administrators a real-time view of the entire infrastructure in a context that any IT silo can understand.
Marsh liked the ability to give every administrator in his IT organization the credentials to log on and use the system to monitor their own environments.
"I did a demo [of ExtraHop for myself] and found a database error," he said. "I walked it down to the database people and said, 'Hey look at this. This is broken.' They said, 'This is cool, where did you get this?' Now I'm forcing the ExtraHop application on system administrators and database administrators and I'm kind of educating them on how to identify and fix their own issues, which is awesome."
With IT silo mentality gone, admins fix their own problems
Other groups within Marsh's organization are now fixing things without coming to the network team. For instance, Marsh started seeing an ActiveSync-related SMTP error in ExtraHop's console. Before looking into it, he invited a couple of administrators to a local ExtraHop training session. About a week after the class, the SMTP errors disappeared.
"I'm pretty sure that the Exchange admin who went to the class logged into ExtraHop and saw it and fixed it, which is awesome, because it required no intervention from me. I'm giving everyone who wants one a log-in. I want everybody to use ExtraHop. It makes them self-sufficient.
On another occasion, Marsh said he saw something strange had happened overnight. He looked at the dashboard display reflecting the amount of data that had passed through the network, broken down by Layer 7 protocol. He saw that traffic had shot up from multiple bytes of data to gigabytes around 3 a.m.
"That was way out of whack. I drilled down, and because of way [ExtraHop] works, I can point out which two servers are doing the transaction. It turns out it's our primary backup server and a server on the DMZ. I know that the way we do our backups: the DMZ servers are supposed to back up to a server on the DMZ and that one DMZ server does transactions with the main backup server. I walked upstairs to the system administrator that does backups and I had him log in. I walked him through exactly what I did, and showed him how I found that, the transaction, the port, the servers. And now he's doing it on his own."
It isn't easy to get everyone on board with the tool, Marsh said. But when he takes the time to walk someone through its value, they usually see the light.
"The trick has been showing them the value," Marsh said.
Erasing IT silo mentality leads to improvements in monitoring environment
Indeed, Marsh said systems and database admins are beginning to request that he expand ExtraHop's visibility so that they can see more for themselves, particularly within virtual hosts. They are working together to find blind spots in their monitoring instrumentation.
Bentley uses taps and a Net Optics network packet broker to direct packet streams to ExtraHop, but that only catches north-south traffic from physical hosts. East-west traffic inside of hypervisor hosts is currently invisible to ExtraHop.
"I was talking to one of the email administrators [about our] Cisco voice over IP system. We're about 50% through the rollout, and part of that is Unity voicemail, where you get the voicemail messages in email. There was an issue where the director of HR randomly does not get the emails. So I was working with the email admin trying to track through from the VoIP servers to the relay server. But there is that east-west conversation that happens between. That's where the holes are, because some Exchange server to Exchange server stuff I'm not seeing. So we're re-architecting that so we get a fuller, better picture."
That re-architecture will include virtual taps on hypervisor hosts, he said.
From IT silo mentality to holistic approach to infrastructure
Bentley's networking team has learned the value of understanding what goes on in other silos.
For more on this topic
Software-defined data centers will be hostile to IT silo mentality
The IT organization needs to unify and serve the business
"[ExtraHop has] forced me to learn a lot more about application and server stuff, which is good because [infrastructure operations] is not just networks and it's not just servers. It's a system," he said.
Marsh wants to put an end to the days of each functional team in IT sitting in a room and saying, "My domain is fine. You guys must be the problem." If everyone is using the same data source and looking at the same dashboards, they can resolve issues better, he said.
"It's helping me become more well-rounded … and figure out how it all works together as a system," he said.