Speedy delivery! Distributing security patches and software updates

Software and patch distribution has become a vexing networking and business concern. Learn more about the issues and how Cisco Systems addressed the problem on its own internal network.

When Microsoft released its monthly security bulletins in September 2004, IT organizations everywhere sat up and took notice. The update files that constituted bulletin MS04-028 ranged between 1 and 10 MB in size, which didn't present much of a software distribution problem by itself. But the bulletin affected no fewer than 42 individual Microsoft applications. Which means it impacted nearly every user in an average enterprise.

While applying the patch, IT groups also needed to upgrade every user device to a minimum service-pack level. In the case of MS Office 2003, that amounts to 70 MB per device -- more than enough to choke a typical WAN if the upgrade were to be downloaded to every location simultaneously.

Making matters worse, a widely announced vulnerability such as this can be exploited quickly, so delaying distribution of the patch by even a few days might lead to serious trouble.

The September Microsoft bulletins are just one recent example of why software distribution has become such a vexing networking and business concern. In addition to desktop-level updates, enterprises typically have to distribute a number of different types of software content over the network on a regular basis. These content types include:


  • Security updates that are critical to protecting enterprise devices and assets. These updates include operating system and application updates, and antivirus dat and engine updates for products such as McAfee VirusScan and Symantec Norton Antivirus.
  • Application packages that may include hundreds of files -- approximately 5 to 500 MB per application -- and often contain combined binaries, making them difficult to compress. Distribution management software includes Windows Installer and Group Policy, IBM Tivoli, Altiris Notification Server, and Wise Package Studio.
  • Configuration policies in the form of .ini or XML files. Client devices are often configured to contact a server to check for updated policies, but frequent polling for updates can affect network connections. Distribution management applications include Altiris Software Delivery Solution and Network Associates ePolicy Orchestrator.
  • Disk images required to provision new user devices, update existing devices, and provide system rebuilds to repair corrupted disks. Distribution management products include Symantec Ghost, Altiris Deployment Solution, and PowerQuest DriveImage.

Other areas besides software content where the distribution infrastructure can impact network performance and user productivity are Web caching, Web application acceleration, and business video.

Current limitations
Today, if an enterprise has an automated software distribution system at all, it typically consists of Windows-based file servers that store content replicas at the various enterprise locations. These servers must be individually secured, managed, and maintained -- tasks that require significant staff time and increase operating expenses. Moreover, the servers are usually vulnerable to the same exploits that afflict other Windows platforms. So the distribution server infrastructure could be struck by the same sort of outage from which IT is seeking to shield the enterprise's desktop users.

The delivery hierarchy in many enterprises follows a multi-tier model, in which a file server at every layer of the network contains a software agent for each type of content that needs to be distributed. While this approach reduces redundant network traffic by localizing replication, it also adds complexity and administrative overhead. Several applications can replicate files on demand. In general, however, distribution and delivery policies are meager at best. Few of these applications can segment content for prioritization. And features such as usage throttling, multicast support, status monitoring, and error reporting are not included in most replication solutions.

Another aspect where current delivery systems are lacking is network presence. To remedy excessive and redundant WAN traffic, the distribution system must be able to identify the most appropriate -- that is to say, the closest -- server at each location, then direct client requests to that source. Scalable distribution solutions require some degree of "presence" intelligence, so that when end-user devices connect to the network they are automatically served from the most proximate location. The alternative is to rely on custom, manual, and static configurations to distribute content locally and to map users to the closest content servers. These solutions typically scale poorly. And the fixed configurations do not support users adequately when they move from place to place.

Optimally, each content-serving device should not only be accessible to local computers seeking software updates, it should also be well-protected against attacks. The whole system can be scaled in a secure, tiered manner by adding dedicated appliances rather than file servers at new sites, and by supporting load balancing at larger sites. One central content server -- the origin server -- can deliver content to every client through a centrally managed hierarchy of devices. By using dedicated devices that rely on a non-Windows operating system, IT can also avoid the many exploits that specifically target Windows.

A real-world scenario
At Cisco Systems, the IT department was able to quickly take action on the September Microsoft bulletin with no adverse impact on the network. "We upgraded 54,000 desktops worldwide, and never had data going over the WAN link more than once," says David Stafford, technical lead in the Cisco IT client platform security group. "If every one of those PCs had tried to pull the update over the network individually, our business could have come to a halt. Not because of a virus, but because we would have been launching a distributed denial-of-service attack against ourselves."

That sort of "friendly fire" problem, and many others associated with inefficient and redundant content delivery, have been alleviated at Cisco by an internal content-delivery network based on publicly available technology, specifically the Cisco Application and Content Networking System (ACNS) and Altiris6 from Altiris.

This system addresses the three crucial aspects of content delivery -- storage, replication, and presence -- with a three-tier architecture. High-use content is pre-positioned locally for immediate use and can be made accessible through a variety of protocols, including HTTP, FTP, and Common Internet File System (CIFS). Content can also be accessed by one machine over the WAN and cached transparently for local sourcing later. With pre-positioning, the administrator can proactively specify content to be distributed to remote sites in off-peak times.

Multipurpose content engines running Cisco ACNS software have been deployed at 280 remote sites (tier 3) and 17 major data centers (tier 2) for distributing software images. Because it is organized in tiers, the content delivery network helps ensure that the core server at tier 1 services only 17 clients at a time, rather than 280 -- or 54,000. The security-hardened, Linux-based content engine appliances are more secure than the Windows-based file servers they replaced. They are also easier to manage centrally through the Cisco Content Distribution Manager (CDM) at the data center.

Using the content engines, administrators can carry out tasks such as delivering a Windows XP image to laptops with all the relevant desktop tools preloaded without creating WAN congestion. In the future, Cisco IT users will be able to build or restore a disk image right at their desks. This is especially beneficial in a computer company where users sometimes run alpha-test software on their own machines.

Presence is critical to achieving effective content delivery. Each client needs to know how and where to get the desired content on the network. Ideally, all client requests should be intercepted at the edge of the LAN to prevent WAN overload.

In the Cisco corporate network, users are redirected transparently by one of two methods: 1) ACNS content routers employing simplified hybrid routing (HTTP 302 redirect), or 2) an enterprise router supporting Web Cache Communications Protocol (WCCP). This two-pronged approach to presence is much more scalable than subnet table mapping or site-specific scripts, and considerably more flexible on multiple operating systems than other solutions.

At headquarters, the CDM makes sure the content engines are configured specifically for the services they are providing and the content streams with which they need to be involved. The CDM also helps oversee the day-to-day administration of the content engines, which can be logically organized into device groups. These device groups are extremely helpful in determining which content engines should receive particular downloads or management settings. With the Cisco ACNS solution, the IT team can set policies and controls pertaining to distribution processes for unicast or multicast, bandwidth management and scheduling, content freshness and availability, and user authentication.

Altiris systems management software transforms all the user devices into managed end points to ease monitoring and offer visibility to each device. The Altiris software determines which devices have been updated, and which clients need applications. IT can install the missing software applications remotely over the network. Client accesses and downloaded files are logged and sent back for central reporting. On the back end, Altiris6 uses the Cisco ACNS infrastructure to optimize bandwidth for package distribution to the edges of the network, and also to provide file distribution points for LAN-based and mobile Altiris clients.

Bottom-line benefits
With this content delivery network in place, Cisco has been able to scale a single Microsoft IIS Web server to provide hundreds of thousands of bandwidth-throttled and automatic resumable software deliveries. This means that the IT department is able to protect the WAN from outages brought about by the distribution of software images. And if an outage were to occur, the system will resume distributing files from the point at which the outage occurred, rather than having to start over.

This virtualized system appears to each client as if it were a native Web server. Equally transparent is support for Microsoft Windows Installer through HTTP, and integrated security and pass-through authentication to Microsoft Active Directory. At the same time, distribution is localized without creating significant administrative overhead.

Individual clients are less at risk during the delivery of hot fixes because the system affords multiple levels of protection. Altiris6 prohibits content execution until the content is completely downloaded to the end device. A software package can be downloaded in its entirety instead of running off a remote share. Remote share could cause major difficulties if a software hot fix or service pack installation were to be interrupted.

At Cisco the content delivery system achieves 98% penetration for every software rollout, compared to only 60% to 70% penetration with the previous system. A typical security update can be disseminated within 72 hours, and emergency updates are distributed within one business day. Additionally, IT staff members have more time to devote to high-priority projects, with no need to worry about congested servers or saturated WAN links.

Best of all, any enterprise can set up a similar distribution system using products and technologies available today.

Getting ready to deploy
When preparing to deploy a software content delivery system, make sure your organization has laid the appropriate groundwork:


  • Create a cross-functional IT team representing desktop, server, and network interests. The staff devoted to managing the desktops and servers should focus on issues such as what technology to use for content storage, replication, and presence, whether that technology can support software updates across different applications and protocols, and how client devices can find the closest server. Other desktop and server issues include administration reduction, speed of delivery, user mobility, impact on the WAN, content prioritization, and error reporting. The network-oriented group should consider how network topology, bandwidth usage, and device upgrades will be affected by the new system, and whether WCCP can be used on the routers.
  • Involve all the enterprise stakeholders in the process, including business and network security managers. Business managers can concentrate on cost justification and how to measure the success of the installation once it has been implemented. A security manager will be most interested in how quickly and efficiently security upgrades and patches can be distributed.
  • Identify and characterize all the content to be distributed. This means categorizing the files and determining their size and how often they are updated. Content that requires the most bandwidth offers the greatest potential for cost savings and helps an implementation team create the strongest business case. Be sure to take into account video and Web application acceleration needs as well.
  • Define "channels" to pre-position the various categories of content. In the case of the Cisco CDM, channels and policies may be defined using either the GUI or XML-based API supported by the CDM. Files to be pre-positioned can be injected programmatically into the network using the API. Once a channel is defined, IT can subscribe content engines to that channel.
  • When piloting, begin slowly with one or two application packages. Test in a limited, controlled environment as much as possible to keep networking variables from skewing the results. Cache-hit counters can determine that the local content engine is receiving the hits, an aid in determining return on investment.
  • Educate internal support personnel about the system, including content engine names, channels where software updates are stored, how clients are being directed to the channels, the origin server involved, and how to download a file directly from a content engine to make sure it is current.

A well-designed, efficient content-delivery network gives the enterprise a potent tool for keeping mission-critical software current, optimizing WAN performance, safeguarding networked resources, leveraging IT expertise, reducing administrative and maintenance costs, and helping to ensure employee productivity across the organization.

About the author:
Baruch Deutsch is Product Marketing Director at Cisco Systems Inc.

This was last published in February 2005

Dig Deeper on Network management software and network analytics

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.