Splunk, a powerful machine data search engine and data analyzer, is proving to be an essential network monitoring software tool. From network downtime and packet loss to malicious denial of service attacks, network managers can discover and troubleshoot events by configuring Splunk to seek and analyze the data they generate.
Splunk is often compared to Google, which indexes the Web by collecting data on every event it can track on the Internet. Likewise, Splunk builds its own repository with performance, monitoring and statistical metadata; log files; and appliance, gateway, and firewall data. Just like Google, Splunk can return search results on all this data in real time.
Data management and review is simpler
Splunk searches all data from anywhere, whether on the physical infrastructure, virtual infrastructure, or in the cloud.
It captures information that is not available through packet flow tools or Netflow, since those programs are designed to reveal network activity only between two endpoints, according to Jim Frey, research director at Enterprise Management Associates. Splunk pulls system event information from inter-related nodes, which is data that is available through Netflow only in a limited way, he said. Splunk also can receive and process Netflow records.
Splunk also collects machine data logs from all along the infrastructure, giving network administrators the visibility they need to determine customer or user activity and behavior. It can analyze transactions, locate rogue applications, and find configuration data, API data, message queues, change events and diagnostic output commands. Splunk also collects distinct log formats, many from custom applications that log information related to service problems, security threats and compliance.
As a unified system management tool Splunk has a REST-ful interface and built-in Application Programming Interfaces (APIs) that network managers can use to aggregate “tremendous amounts of data,” Frey said. Network managers need to access and analyze this data to determine how their networks are performing, one reason why Splunk is gaining popularity among networking teams.
Splunk makes network administration easier
Essentially Splunk enables network managers to make intelligent decisions, whether troubleshooting an error or looking for a suspected hack attack, said Michael Wilde, Splunk’s evangelist and director of product information.
Dig Deeper into Splunk, systems management software
Loggly CEO: Log data management takes on the original big data
Review system event logs with Splunk
Splunk gives Motorola Google-like insight into IT assets
“By displaying data from any number of locations within the system, Splunk gives them the feedback they need to discover the source of a problem. That’s why people like it, because they have access to everything,” Wilde said.
Network monitoring software like Splunk helps network administrators search for, identify and resolve network problems by harvesting log data, Frey added. In addition, administrators value log data because it is “time-series structured.” For example, a search term can be set to reflect the number of times a particular port number is logged during a 60-minute period. Splunk retrieves the data so that it can then be organized to show, for example, how many times access to that port was denied or granted in that time period.
With Splunk, network security managers can collate the information from disparate segments of the infrastructure and analyze logs that are fed into Splunk’s repository. These logs may reveal important machine use and availability patterns, or certain data flows or attack signatures that can reveal a security incident.
Splunk can also log data from inline security appliances, such as intrusion protection systems (IPS) and intrusion detection systems (IDS), to correlate network events with particular times of day, or with particular IP addresses. Splunk users can choose to index a particular log file, which creates an input to monitor that file not only for the file’s current contents, but also subsequent, automated file inputs. Similarly, Splunk receives data from locked down components like firewalls, routers, and switches, which can all be set to send data to Splunk logs.
Sometimes end users choose to use external scanners to scan for signatures, or they use Splunk’s Enterprise Security 2.0 package. It lets end users set real-time alerts, create reports, and use configuration logs to list network events. They can set alerts once and then re-use them.