 | |||
|
|
||
| |||
| |||
| |||
| |
|
Home > Networking News > Network Device Management |
| |
|
|
|
|
| Networking News: |
|
||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|
|||||||||||||||||||||||||||||||||
The book from which this chapter is excerpted presents tips and tricks for four network configuration management topics. For ease of use, the questions and their solutions are divided into sections based on topic, and each question is numbered based on the topic, including Topic 1: Change Management Best Practices, Topic 2: Network Management Security, Topic 3: Network Management Troubleshooting, Topic 4: Change Management Techniques, Topic 5: Selecting and Deploying a Network Device Management Solution, and Topic 6: Enterprise Network Device Management. To download/read the eBook in its entirety, visit: http://www.alterpoint.com/ebook Topic 6: Enterprise Network Device Management
Q 6.1: Our enterprise has thousands of network devices and we’re always adding new ones. How can we ensure that a change management solution will accommodate future devices?A: A little bit of common sense and some investigation will generally provide a satisfactory answer to this question. What you need to do is evaluate each solution for its current breadth of device support, check up on how the solution is architected to accept changes, and make some educated guesses about how the solution will grow in the future.
Intended for Broad SupportOn the common sense side, look at the ties each change management solution has. Take CiscoWorks, for example, which is made by Cisco. It’s a good bet that any future Cisco equipment will be addressed by an update or add-on to CiscoWorks. It’s also a good bet that any future Nortel equipment won’t be handled at all. So, if you’re truly in an all-Cisco shop, CiscoWorks might present an acceptable path for future support.
Built for Device ExpansionOn the investigation side, find out how each solution provides for future device support. Some solutions, for example, simply ship a new version of the product (upgrading you from 2.1 to 2.2) when a sufficient number of new devices are ready to be supported. Other solutions ship “packs” that add devices to the solution’s list of supported devices. In general, the “pack” idea is preferable because the solution vendor can more easily provide support for one or two new devices rather than waiting until they’ve got enough to justify the release of a new version. Evaluate the cost of device support, too. Some vendors ship base device support with their software and charge extra to provide support for other devices. Other vendors might offer additional device support on demand as part of a product maintenance agreement. Still others might simply accept requests for device support and post popularly requested device support for free on their Web sites.
Built for Long-Term ChangeA larger, and more difficult to evaluate, question is how a change management solution will support radical new management technologies that come out in the future. Without question, every solution on the market today could be modified to support just about anything that comes along. It’s the degree of modification that you should question. For example, suppose somebody invents BTEMP, the Better Than Ever Management Protocol. Device manufacturers universally jump on the BTEMP bandwagon and build it into their latest operating system (OS) updates. Now it’s time to upgrade your management solution to take advantage of BTEMP’s better security, lower bandwidth, or whatever. Some solutions will require a complete version upgrade, meaning you’ll have to wait a while for it to be available, and then likely will have to deal with the bugs often associated with major changes in a software package. Other solutions might simply provide an add-on pack that takes advantage of BTEMP without modifying the base software. You can get a feel for a solution’s long-term change philosophy by looking at its current breadth of support. For example, a solution built entirely around Trivial File Transfer Protocol (TFTP) might not adapt well to entirely different technologies, although it could be built to easily accommodate new TFTP-based network devices. In contrast, a solution that already supports a broad variety of management technologies—TFTP, HyperText Transfer Protocol (HTTP), Simple Network Management Protocol (SNMP), or even proprietary management interfaces—is more likely to handle radical new management technologies without pain.
Evaluating Solutions’ Growth PotentialAs a part of your solution evaluation process, put together a grid that addresses future growth. Table 6.1 provides an example of such a grid.
Table 6.1: Grid to evaluate solutions’ growth potential. With a chart like this one, you can easily evaluate solutions’ growth capabilities against your specific needs and concerns. For example, if you’re in a company that only uses network devices from one vendor, than that vendor’s product (Product C in the chart that Table 6.1 shows) would be a great solution. It’s likely that the product will always parallel the vendor’s device technologies, providing a reasonable means of managing those devices well into the future.
Q 6.2: We have hundreds of network devices, so manually retrieving configurations via Trivial File Transfer Protocol just isn’t an option. What are our alternatives?A: Trivial File Transfer Protocol (TFTP) is great for small environments, but like so many manual solutions, it doesn’t scale well. My preference in a large environment is to select a network device change management solution. Such solutions usually rely heavily on TFTP and other common protocols to retrieve device configurations, but automate the entire process so that dealing with hundreds of devices doesn’t require a team of dedicated people working ’round the clock. You can, of course, stick with TFTP if you find some way to automate it. Listing 6.1 shows an example Linux script that can use TFTP to pull configurations from either Cisco or Ascend routers (this example is adapted from a more complete script that is available at http://www.securiteam.com/exploits/5RP0E000AA.html).
Listing 6.1: A sample Linux script that can use TFTP to pull configurations.
Now, if you’re perceptive, you’ll have noticed the URL to which I referenced this script: http://www.securiteam.com. This script is listed as a security exploit because, as the site states, “This allows a remote attacker fill knowledge of the router configuration, routes, etc.” And the site is absolutely correct; if your devices are accessible from the Internet, you might want to seriously consider disabling TFTP (or Simple Network Management Protocol—SNMP— which is what this script uses to force the device to send its configuration via TFTP) on them to prevent this sort of thing from happening. Fortunately, most companies’ internal routers use non-routable IP addresses (such as 192.168.0.1), meaning the routers can’t be contacted by outside attackers.
Q 6.3: We’re planning to use TACACS+ to consolidate authentication to hundreds of network devices. Is there anything that we need to be aware of?A: TACACS+ is great for consolidating authentication. In fact, that’s pretty much why it exists. However, you need to be aware of a few vulnerabilities and some limitations. A Brief History TACACS+ is a proprietary Cisco protocol, although most of its details are publicly available. TACACS+ isn’t fully backward-compatible with TACACS and XTACACS, so it’s important to distinguish which you’re using. The biggest caveat with TACACS+ is that it’s primarily supported on Cisco devices; if you assumed that TACACS+ enjoys the wide support of similar protocols such as RADIUS, be sure you check your network devices to make sure they offer TACACS+ interoperability.
How TACACS+ Works TACACS+ packets use an encryption technique to encrypt client/server messages, and the encryption key is a pre-shared secret, or password, entered on both the client and server. However, TACACS+ does not implement packet integrity checking, meaning packet capture and replay is a potential vulnerability, although the encryption limits the effectiveness of these attacks. TACACS+ also uses session numbers to track AAA communications with clients. These session numbers are nonrandom and are in fact limited to 1, 2, or 3 for authentication, authorization, and accounting. Another portion of the session ID number is determined by a cryptographic algorithm, which is also nonrandom. With a large amount of TACACS+ traffic on the network, the randomness of these session IDs decreases, potentially making it easier for an intruder to mess with packets. For example, a packet could be switched from “authorization” type to “authentication” type. However, the server would find the remainder of the packet’s structure incorrect for the new type (that structure having been based on the original type) and potentially crash, suffer a buffer overrun, or some other unexpected behavior. Note that only the body of the TACACS+ request is encrypted; the header is not, making it easy for someone with physical access to the network to capture, modify, and replay packets in an attempt to crash the server. TACACS+ implementations before version 1.78 used User Datagram Protocol (UDP) packets to carry authentication requests, which is less reliable than the Transmission Control Protocol (TCP) connections used by newer versions. Ideally, you should configure your TACACS+ implementation to use a port other than TCP 49 (the default) to help obfuscate the service and thwart potential attackers. Perhaps the biggest vulnerability in TACACS+ is the relative weakness of the overall encryption mechanism. It’s possible for someone with physical network access to capture an authentication request from a client and modify it. This request could be submitted to the server and accepted; the server’s reply would be encrypted, but because the clear text of that reply would be known, it would become much easier to break the encryption. Because the encryption is based on a shared secret that is rarely changed, breaking the encryption once would reveal the key necessary to break it from then on. Thus, a good practice is to regularly change the shared secrets used by TACACS+ clients to communicate with the server. Accounting also presents replay vulnerabilities. Because the TACACS+ accounting packets aren’t sequenced, they could be captured and replayed, essentially corrupting the TACACS+ accounting log and potentially even used as a denial-of-service (DoS) attack to prevent legitimate accounting packets from being processed-essentially disguising information. Risk Analysis
Q 6.4: How can we ensure that our devices’ software is consistent and up to date?A: Managing your devices to ensure that they always contain the most recent version of their operating system (OS) software is becoming more important. Recent virus attacks that targeted Cisco routers directly have made it apparent that network devices—which were once largely ignored by virus authors and other hackers—are now fair game. Interestingly, the very existence of a software patch for your routers makes them more vulnerable. The Blaster virus, which targeted Microsoft Windows-based computers, is an excellent example of how virus authors are taking advantage of lax patch management. A security firm notified Microsoft of a flaw in their remote procedure call (RPC) code that could allow an attacker to take complete control of a computer over TCP port 135. The notification was made quietly to Microsoft, which released a patch. The patch documentation did not describe in detail how the patch worked or what the original problem was. However, the patch contained only one replacement DLL file. The authors of the Blaster worm simply compared the new file to the old, then reverse-engineered the differences to determine what had been patched. They then wrote a worm to take advantage of the security hole on unpatched systems, knowing that most systems remain unpatched—often until the next service pack is released. The Blaster worm has no direct effect on network devices, but the fact that virus authors are now using patches to develop their viruses should be a wake-up call for everyone in the IT industry: the minute a patch is released, apply it. This uncompromising approach means that you must ensure that all your network devices have a consistent, up-to-date software configuration. Getting TFTP Ready
Next, copy your update files—the binary files provided by your device manufacturer—into the TFTP folder and give them the appropriate permissions:
Preparing Your Devices Upgrading the Firmware
The router would then display its version banner and other information. To begin the upload process, use the following commands:
Next, tell the router to save its configuration via TFTP. In the following example, the TFTP server is at 192.168.1.100:
With the configuration safely written, you can upload a new configurtion into the router:
Next, write the current flash memory out to TFTP as well:
The router will display “!” as it uploads the configuration to the server, then it will provide the following output:
Again, the router will display its configuration information, then you’ll issue the write command to reload the router:
This data is what you will see as the router reloads and restarts. The router will be in boot mode, running the boot-ROM version of its OS. The following is the simplified version of the IOS and the flash memory in read/write mode:
The important piece of the configuration information that you’re looking for is:
This data tells you that you’re running from the device’s internal memory, and you can start working with the flash memory:
The router will display “!” as it loads the configuration file. Once it is finished, it will verify the file and allow you to work with it:
With everything in place, you can reload the router. The configuration has been changed, so the system will prompt you to save the configuration; don’t bother—simply let the router restart: myrouter(boot)#reloadSystem configuration has been modified. Save? [yes/no]: no Proceed with reload? [confirm] The router will come up normally; reload the configuration that the router should be using:
Again, the version information will be displayed. You can now load the configuration file from the TFTP server:
You’ve completed the software upgrade. This sequence is fairly complex and requires that you verify that each step has completed properly. In other words, this process is not easily scripted. However, software products such as AlterPoint DeviceAuthority and Cisco CiscoWorks have the ability to run through a script like this automatically, ensuring that each step completes properly. With such software, you can upload new software to multiple devices at the same time. The benefit of using a dedicated software solution is that you don’t have to worry about individually backing up each device, running the script, and verifying that everything completes properly. The software performs these tasks for you, making it easier to manage a large number of devices. In fact, in a large enterprise with hundreds of devices, an automated software solution can be the only feasible way to keep everything updated.
Q 6.5: How can we manage device changes in real-time?A: Device change management in real-time is a tricky process. Very few devices offer any sort of events-based notification other than standard Simple Network Management Protocol (SNMP) traps sent to a management console. In other words, routers don’t typically provide details of a configuration change; instead, they tend to notify a management console via SNMP trap that someone has logged on and done something. The notifications lack the detail that a change-management solution typical requires. SNMP traps are a start—they at least provide some kind of notification that something has happened. To configure a Cisco IOS device (except some remote access devices) to send SNMP traps for most configuration-related operations, execute the following commands on the device:
Be sure to provide the correct IP address for the SNMP workstation, which, in this example, is 192.168.1.102. Shortly, we’ll explore how this process can help your configuration-management process. Another technique is to use TACACS+ accounting information, which is available on most Cisco devices (Cisco owns the TACACS+ protocol) and on some other devices; Remote Authentication Dial-In User Service (RADIUS) accounting can also be used, although it tends to provide less-specific information. The following configuration file enables accounting:
The idea for using either TACACS+ or SNMP is that if you have a change-management solution capable of supporting the functionality, the solution can either receive SNMP traps or analyze the TACACS/RADIUS log on a periodic basis (or even directly receive TACACS/RADIUS accounting information). The traps or log information don’t provide the actual change data, but they serve as a trigger to let the solution know that someone, for example, logged off of the device. This trigger tells the solution to log on via TFTP (or whatever means for which it is enabled) and retrieve the devices’ current configuration. By continually retrieving the configuration whenever someone logs off, you’re sure to have a complete record of changes that were made over time. This ability contrasts with solutions that simply retrieve the device configuration on a scheduled basis, because such devices can easily miss incremental changes performed between the retrieval intervals. Thus, there isn’t really a way to perform real-time change management; however, with an automated change-management solution capable of scanning accounting logs or receiving SNMP traps, you can enable near-real-time change management by continually downloading and storing device configurations whenever anyone performs an action that might result in a configuration change. As you’re evaluating change-management solutions, be sure to consider this capability in your features comparison. Q 6.6: What is the best way to simplify the process of reconfiguring more than one hundred network devices? A: The ability to reconfigure multiple network devices as easily as a single device—or even more easily, if possible—is important in organizations that support dozens of network devices. Some of the most basic security practices, in particular, can be more easily implemented through an automated configuration capability. Failing to change devices’ Simple Network Management Protocol (SNMP) community strings to something other than the default “public” is like failing to lock the doors on your car—eventually, someone’s going to steal it or something in it. However, many companies change those SNMP strings only once—when the device is first placed into service. As a result of the complexity and time required to change the SNMP strings on literally hundreds of routers, switches, and other devices, organizations simply don’t do it. However, the same organizations require users to change their passwords every 30 days because old passwords are more likely to be compromised. But old SNMP strings are just as likely to be compromised and are more likely to have a severe impact if they are compromised. Whether the changes are devices’ administrator passwords, access permissions, or routing tables across a group of devices, the ability to make consistent changes to a large number of devices is a very real need in large organizations. The complexity of this task requires more than simply write a script or two; you need third-party software to provide the correct combination of workflow, configuration deployment, change validation, and more. Solutions from AlterPoint, TripWire, Ecora, and Voyence can provide all or part of a mass-configuration-management solution. You’ll need to carefully evaluate these companies’ offerings to determine which solutions meet your business requirements. For example, AlterPoint’s DeviceAuthority provides an Update Module that allows you to automate the process of changing several devices’ configurations at the same time. Figure 6.1 shows the module's overview page for a password-update job. You can see that only a portion of the inventory—the list of devices that the software knows about—will be modified; the Austin Inventory selection suggests that only devices in an Austin office will be modified by this job. The job also includes a workflow, which I’ll examine in more detail in a bit. Notice that the job includes a notification capability, which will provide email notifications to selected users when the update is run, providing immediate status feedback.
Figure 6.1: Deploying a configuration change to multiple devices at the same time. A useful function is the ability to specify different actions for various parts of the update process. With DeviceAuthority, you can specify a different update script for the start of the update process, before each device is updated, after each device is updated, and after the entire job finishes. As Figure 6.2 shows, a short script can be used to ping each device before trying to update it; a failure will cause that device to be skipped and the failure to be logged. After updating each device, the device configuration is backed up, ensuring that the new configuration is safely stored and that the organization's overall configuration management process is maintained.
Figure 6.2: Customizing deployment workflow. Figure 6.3 shows another important feature—the ability to provide detailed status reports for each step of the update process. Ideally, this status report is available both from the software's management interface, as Figure 6.3 shows, and through email notifications. The email notifications help provide real-time updates and status checks to administrators throughout your organization, helping ensure that everyone is kept up-to-date with the status of the change.
Figure 6.3: Viewing update results in DeviceAuthority. The following list highlights key features to look for in a software package:
| ||||||||||||||||||||||||||||||||||
