Router Expert: Using Cisco IOS logging, part 1

This tip examines system message logging, Unix syslog, and Cisco's IOS logging implementation and configuration.

Using Cisco IOS logging, part 1

By Michael J. Martin

Read about Michael

A few weeks ago I received an email from a colleague, looking for an easy way to monitor whether his systems were being scanned to see if they were running SNMP. I suggested implementing a logging inbound Access Control List (ACL) and enabling system message logging, and implementing syslog trap reporting to send the logging messages to a syslog server. Once the router's logging data was on the Unix server, he could use some simple shell scripts to generate reports on the data.

In light of the recent SNMP exploit (see http://www.cert.org/advisories/CA-2002-03.html) and its broad implications, I thought this solution might be helpful to others. In part one we will review the concept of system message logging, Unix syslog (the granddaddy of logging), and Cisco's IOS logging implementation and configuration. In part two, we will work through configuring logging on a router, implementing traffic inspection logging ACL, building a Linux syslog server and implementing a Unix shell script that will generate reports from the ACL logging output.

System message logging
The system message and error reporting service is an essential component of any computer operating system. The system message service provides a means for the system and its running processes to report various types of system state information to a system operator or administrator. There are three classes of system state data: error, informational, and debug. The Cisco IOS, an advanced operating system for internetworking, provides an extensive system message and error reporting facility. In fact, the IOS has over 500 service identifiers known as "facilities" that it uses to categorize system state data for error and event message reporting. System logging data is an important resource in diagnosing problems. How an administrator chooses to implement system logging and manage logging data affects their ability to manage routers and effectively troubleshoot problems (you should also eat at least one green vegetable every day). So, take some time to develop a logging strategy so this data will be available when you need it (and eat some spinach).

Unix syslog operation
Before we look at configuring IOS logging, a review of Unix syslog is helpful. Syslog is the system error and reporting facility used by most Unix implementations. It was the inspiration and work of Eric Allman, who is more often identified for his contribution of Sendmail, to the Unix and Internet community. The idea behind syslog is a single facility to which programs write their event messages. The "syslog" package consists of three components: Syslogd, a Unix system daemon that listens to messages that are both locally generated and from remote systems (UDP port 514) if configured. Syslogd implementations for Win32 and MacOS also exist that operate as stand-alone applications. Syslog.h and associated library routines are used use to incorporate syslog-reporting functionality into developer programs. Logger is a program that allows users to implement syslog reporting into shell scripts. To categorize and prioritize messages, syslog provides two structures for sorting message data, Facility and Severity.

Under syslog, a facility relates to the source of the message. The number of available facilities varies depending on the syslog implementation. Here are the major facilities and the types of messages that are associated with them:

kern
System kernel messages
auth
Authorization and authentication messages
daemon
System service processes `
mail
Mail system messages
lpr
Print system messages
user
General system messaging (default if no facility is defined)
cron
Messages relating to jobs run by the system execute scheduler
local0 - 7
Custom message reporting

Severity relates to the importance of a message. There are eight defined severity levels (listed in descending order from highest priority to lowest):

0 - emerg
Emergency message or system failure
2 - alert
Urgent problem, requiring immediate action
3 - critical
Critical error conditions
4 - error
Standard errors
5 - warning
Warning conditions, system operation status events
6 - notice
Standard operational events, events that should be looked at
7 - info
Status messages, notification of conditional program events
8 - debug
Debugging events and trace output

Message severity levels are standard across all syslog implementations. To configure logging, "a facility" or "group of facilities" is combined with a severity level and a reporting output option or "action." Here is an example from a Unix /etc/syslogd.conf file:

kern.emerg					/dev/console
mail,user.info				/var/log/status.log

When configuring output options, it is important to understand the role severity plays in the types of facility messages that are sent to the output action. All facility messages are reported to the output action, recursively higher from the severity level. So, in the example above, only emergency messages from the kernel will be printed to the system console. But, all messages from level 0 (emergency) to level 6 (informational) reported the mail and user facility will be sent to the file /var/log/status.log. This is the default behavior for all syslog implementations. Some implementations allow for some tailoring of messages. For example, it is possible in some syslog implementations to define an output action that only has facility data at a specific severity level. So check the syslogd page of your Unix implementation.

Unix syslog output action options
For those who are not familiar with Unix, everything -- that is disk storage, tape drives, modems, video cards, CD ROM drives, directories, the mouse, etc. -- is a essentially a file in the file system. Therefore, data can be sent to it and extracted from it using the same tools used to manipulate data. What this means to us is that there is a great deal of flexibility in how syslog can relay message data. Syslog supports four reporting output actions:

  • Log to a file, which under Unix can be an output device such as the console /dev/console, a terminal (/dev/tty), a line-printer (/dev/lp) or a plain text file (/var/log/<filename>).
  • Forward the message to another syslog server on another host (@username or @ipaddress in dotted quad format).
  • Write the message to a specific user's operator window on the logging host (user,user,user).
  • Write the message to all operator windows on the logging host (*)

That's how syslog works. In a moment, we will talk about setting up syslog on a Linux system to receive IOS system messages. But first, let's look at how IOS system messaging is implemented.

IOS system message logging
As mentioned earlier, the IOS supports over 500 reporting "facilities." Some of the more common ones are:

IP
IP Protocol
OSPF
OSPF Protocol
SYS
Operating System
TAC
Terminal Access Controller Access Control System
SEC
IP Security
RSP
Route Switch Processor
IF
Interface
VIP
Versatile Interface Processor

IOS system error messages begin with a timestamp and a percent sign (%), followed by the "facility," "severity," UID (unique id), and the error message (text of varying length). Here is an example message output from a logging ACL:

Mar 14 22:57:53.425 EST: %SEC-6-IPACCESSLOGP: list internet-inbound denied udp 66.56.16.77(1029) -> 63.78.199.4(161), 1 packet

IOS debugging messages use the same "facilities" for identification. They are, however, formatted differently than error messages, beginning with a timestamp followed only by the "facility" and the debug message output. IOS error and debug messages follow the Unix syslog severity format (0 emerg to 7 debug). Messages appear in the IOS reporting output action depending on the severity level defined.

The IOS provides four output actions for viewing system event and error data. Console logging is activated by default. In its default configuration, all (severity level 7, debugging) message data is sent to the router's console port (line con0). This approach is similar to Unix, where error and event data is sent to /dev/console or /dev/tty0. To disable console logging, use the configuration mode command <no logging console>. Sending logging data only to the console port may seem odd, since most interaction is done using vty sessions, but the console port can be connected to a terminal server, which buffers the message data or a serial line printer (just like Unix) that can print out event messages. While esoteric, these two methods of data collection were quite acceptable for many years and they are secure from a networking perspective since the data is sent via serial to a locally attached display device (just remember to lock the door).

To view system messages over a vty session (line vty 0 - 4), monitor logging must be configured. To view logging data, the enable exec command <terminal monitor}> is run to activate logging output to the vty. To enable monitor logging, use the configuration command <logging monitor [severity]>. The monitor logging option is the most practical method for viewing logging events in real time. It is highly recommended that you establish two vty sessions, one for displaying event reporting data, and the other for command execution. Often, when troubleshooting or running a debugging sequence, a large amount of logging data is generated. This obscures the vty with logging output, making command entry quite difficult at times. Once terminal monitoring is enables on a vty, it cannot be disabled (unless the logging monitor service is disabled using the configuration command <no logging monitor>).

Local storage of logging messages on the router is also available via buffer logging. Since most routers do not have a hard disk, messages are saved in a DRAM buffer. While buffer logging does not directly affect the router's performance, it does consume memory. However, if your router is short on memory, you may see performance issues with processes that need memory if your logging buffer allocation is too large. To verify your router's memory configuration, use the enable exec command <show version> which will provide a variety of operational facts about your router, including the amount of DRAM allocated for packet buffers and the amount allocated for operational processes (i.e., routing tables, CEF tables, etc.) To see if you are having memory allocation issues, use the enable exec command <show memory failures alloc >. A reasonable buffer allocation is 64k; the "history" logfile is a rotating one, which overwrites the last log entry when the size limit has been reached. To configure buffered logging use the following configuration commands:

Godzilla-ABR(config)#logging buffered notice
Godzilla-ABR(config)#logging buffered 64000
Godzilla-ABR(config)#logging history size 250

The above configuration sets the buffer size at 64k and sets the history count at 250 messages. To view the buffered logging data, use the exec command <show logging>. This command performs two functions; it reports on the configuration of the router's various reporting display actions, and outputs the logging buffer history (if buffer logging is configured).

To send system messages to a remote syslog host, TRAP logging needs to be configured as a reporting output action. Remote reporting has two big advantages over local reporting.

  • History and archiving: Storing logs remotely shifts the burden of storing log output to a device with an actual file system and cheap ample storage. This provides the option to keep large-sized log files and/or the ability to archive and store log files.
  • Data Manipulation: Once the log data is on a system with tools that can manipulate it, log data can be used to generate and syndicate some very interesting and valuable reports, as you will see later with the SNMP scanner report script.

Configuring TRAP logging is a four step process:

  1. Define a syslog host using the configuration command <logging {hostname or IP address}>.
  2. Define the logging severity of the messages to be sent using the configuration command <logging trap {severity}>.
  3. Define the IP address that will be associated as the origin address of the logging messages. This is set using the configuration command <logging source-interface {interface type m/s/p}>.
  4. The final step defines the syslog "facility" that the messages are sent to on the remote syslog server. Use the configuration command <logging facility {syslog facility}>. Here is a trap reporting configuration example that uses the Loopback interface as the report origin address:

    Godzilla-ABR(config)#logging 64.128.20.12
    Godzilla-ABR(config)#logging trap informational
    Godzilla-ABR(config)#logging source-interface Loopback 0
    Godzilla-ABR(config)#logging facility local2

      Conclusion
      That covers the basics of Unix syslog and IOS logging and IOS logging configuration. In part two, we will move on to implementation and look at a practical IOS logging approach. We'll include using your routers to collect intrusion and security exploit scans from the Internet, as well as implementing syslogd on Red Hat Linux and some scripts to generate reports on the data the router collects. So stay tuned...

      Was this article helpful to you? Do you have suggestions for topics you'd like to see covered? Please e-mail us and let us know!

This was first published in April 2002

Dig deeper on Network Administration

Pro+

Features

Enjoy the benefits of Pro+ membership, learn more and join.

0 comments

Oldest 

Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to:

-ADS BY GOOGLE

SearchSDN

SearchEnterpriseWAN

SearchUnifiedCommunications

SearchMobileComputing

SearchDataCenter

SearchITChannel

Close