Recently, in several visits to the sites of prospective disaster recovery planning clients, certain trends have begun to coalesce. For one, the monies that were allocated for data protection a year or two ago, but frozen in response to the economic slowdown, have begun to be freed up for planning. With this fiscal "spring thaw," companies have signaled that are ready to begin planning, strategizing, and implementing a data protection capability. That's the good news.
The bad news is, now that they have money to spend, these companies realize that they confront a challenge of nightmarish proportions, particularly with respect to storage. In one instance, the issue is linked to SANs. In another, it has to do with multi-tier storage.
One of the companies facing this challenge is a medical institution of some repute. Two years ago, they hired a new Director of IT and a new CIO. Together, the managers let go most of the incumbent IT staff as part of an effort to consolidate resources and drive down IT costs. A leaner, meaner cadre of geeks, fresh out of college with the latest skills (and lower salary expectations), was hired and the new team has set about replacing aging infrastructure -- beginning with networks and storage -- with new technology.
The consolidation plan was crafted carefully and executed flawlessly in the first year and a half. The IT Director, who had previously worked in the dotcom world, plied his experience to build a new secure and internally redundant network for the hospital. His pride showed through as he explained how the new network design shored up the DMZ between internal hospital systems and data and "external users" -- doctors offices, clinics, and treatment centers in the professional buildings surrounding the main hospital campus, and insurance providers interfacing from even greater distances. HIPAA regulations required increased privacy and data protection, and from a networking perspective, this was built right into the architecture of the new network.
Being a network expert, the Director settled upon another kind of network for his storage consolidation effort: a storage area network (SAN). Knowing virtually nothing about storage technology, but a lot about networks, he went with a SAN from a name-brand vendor. It was a politically expedient approach, since vendor brand recognition streamlined the approval of the acquisition by the hospital's IT oversight council, a group comprised of the hospital's IT Director, CIO, representatives of finance and accounting, and representatives of the healthcare providers who delivered the actual services of the hospital.
And, as forklift upgrades go, the SAN implementation was as smooth as could be expected. Stumbling blocks included difficulties interfacing older servers – in particular, the core hospital system hosted on a Tandem Himalaya mainframe – to the fabric. A workaround was in the works, the Director said, for moving the storage on the Tandem onto the SAN "over time."
However, and unfortunately, less attention had been paid to the resiliency of the data storage infrastructure than to the resiliency and security of the LAN infrastructure. A self-described storage novice, the Director had conceived of the SAN merely as a scalable data repository. Beyond the internal point-in-time mirror splitting provided by the array vendor inside SAN-attached arrays, the Director hadn't given a lot of thought to any other aspects of data protection. Once the money allocated for disaster recovery was freed up, he just assumed that he would piggy back on a public SONET ring network and mirror his SAN at a friend's co-location facility about ten miles away on a duplicate of his local SAN.
This simple (albeit extremely expensive) strategy, like his overall data consolidation effort, was compromised, however, when another SAN appeared at the company. A large department of the hospital, helmed by a prestigious doctor of considerable political clout within the organization, had decided to field its own infrastructure to host its applications and data. No one on the IT advisory council questioned the move because the department contributed significantly to hospital revenues with the pricey services they provided (medical imaging, radiology, etc.).
The doctor, who fancied himself something of a repressed technology visionary, purchased a SAN offered by the same vendor who sold him his medical instrumentation. Compatibility with his medical imaging systems and data was assured, the sales rep had told him, because all the gear bore the same vendor name. What the rep didn't share with him was the fact that his company's FC fabric was not compatible with the FC fabric deployed in the IT department. With the deployment of the second SAN, an island of mission-critical data had been created.
Now, the IT Director's data protection strategy required the one-for-one replication not only of the components of his primary SAN, but also of the components of a second SAN. Even a cursory review revealed that the costs of such an infrastructure replacement strategy would quickly exhaust the dollars budgeted for overall disaster recovery, and that was before any capability whatsoever had been provided for keeping core hospital services up and running.
In this case, the organization had considered disaster prevention and recovery in its network re-design efforts -- building real resiliency and security capabilities directly into the architecture of the company LAN -- but had not applied the same criteria in vetting its new consolidated storage infrastructure design. Had data protection criteria been articulated together with a mandate that the criteria be considered in the vetting of any storage technology acquisition, the storage infrastructure might have had disaster recovery built in at the design level as well. Absent such a mandate, any cost-effective data protection capability will now need to be bolted-on to an already problematic storage infrastructure.
In part two of this tip, Jon looks looks at a financial services company that found out the price of SAN the hard way
For more information:
About the Author: Jon William Toigo has authored hundreds of articles on storage and technology along with his monthly SearchStorage.com "Toigo's Take on Storage" expert column and backup/recovery feature. He is also a frequent site contributor on the subjects of storage management, disaster recovery and enterprise storage. Toigo has authored a number of storage books, including Disaster recovery planning: Preparing for the unthinkable, 3/e. For detailed information on the nine parts of a full-fledged DR plan, see Jon's web site at www.drplanning.org/phases.html