the February 2002 issue of this magazine I spoke about disaster
recovery and how to plan an escape route. In the article I
discussed two steps: identifying elements for Failures &
Risks and drafting and implementing a Comprehensive Disaster
Policy. In this article I am going to highlight some precautions
one can take to avoid disasters and to cope with failure,
which pervades into infrastructure, client and application
systems. In these IT domains, all points of failures and disaster
situations need to be assessed separately. Business continuity
is ensured if you plan risk management strategies in advance,
plan resources to handle failures and disasters, and set procedures
for recovery. I think disaster and recovery planning should
be an integral part of design and deployment of any IT system.
sound disaster recovery/business continuity plan is essential
to protect the well being of an organization. by M. D.
SYSTEMS AND DISASTER
should not confuse disaster planning with design of fail-proof
systems, though the success of these banks on their interdependency.
There is need to architect the best and leave no scope for
failure while designing both applications and network systems.
Two key features of fail-proof design are:
a) Use of reliable hardware with in-built fault-tolerance
and redundant functionality; a network design with redundancy
both in carriers and at end-systems like switches and routers
the QOS approach.
b) Design and deployment of Software Systems with support
of a balanced architecture design, use of tested tools, and
adhering to known quality standards.
A good practice is to learn and find solutions from past failure.
There is no handbook that is the ultimate reference for failure
and disaster controls.
DISASTER RECOVERY PLANS
recovery plans vary greatly, depending on the size and scope
of a company and how critical IT is to its operation.
Business Managers and IT Managers should discuss what kind
of recovery plan is necessary, and which systems and business
units are most crucial to the company. They should decide
on people responsible for declaring a disruptive event and
mitigating its effects. Most importantly, the plan should
establish a process to locate and communicate with employees
after a disruptive event.
As per Gartner's recommendations, the best way to get started
is to conduct a BIA (Business Impact Analysis). This will
identify most crucial systems in the business and the effect
of an outage on business. The greater the potential impact,
the more money a company should spend to restore systems quickly.
For instance, a stock trading company may decide to pay for
completely redundant systems that will allow it to immediately
start processing trades at another location. On the other
hand, a manufacturing company may decide that it can wait
for 24 hours to resume shipping. A BIA will also help companies
set a restoration sequence to determine which parts of the
business should be restored first.
PLANNING AND TRAINING
You can form a comprehensive policy on risk management practices,
which lists the disaster recovery plan design, and selection
and deployment of hardware and software elements. There should
be a correct choice of systems for contingency. You can choose
between distributed systems and centralized systems. Some
times distributed systems (NASDAQ experience) offer a better
option for recovery.
You can then design and develop the fail-over applications.
Developing fail-proof systems for heterogeneous clients, servers,
and peer systems is a big task. The system administrator should
study the fixes, patches, and applications in detail to counter
the shortcomings and bugs, especially in desktop and server
Training programs for IT staff should also be incorporated
in the design to create awareness on the subject. I have seen
that in most cases, training and awareness on a critical subject
like ensuring data and network availability, is left solely
to the operation staff. And the basic infrastructure that
ensures data backup and fail-over is missing from the initial
plans. This is a wrong approach. It is necessary to include
redundancy and fail-over solutions in any IT system design.
The September 2001 attack on World Trade Center tested the
contingency plans of American businesses to an unanticipated
degree. Companies that had business continuity plans and contracts
in place with vendors of recovery services were able to continue
business at alternate sites with minimum downtime and loss
of data. And the alternate facilities provided by the vendors
performed rather efficiently.
NASDAQ recovered from the attack within six days. This was
possible because of the distributed systems, contingency planning,
and a redundant architecture. It is a matter of fresh planning
i.e. how to further reduce the recovery time and avoid business
We may witness disaster for any reason and in any form. It
may be a natural calamity or initiated by humans. Are we prepared
Organizations should not neglect to review and update its
business continuity plans and contracts. Only if organizations
learn from the experiences of these business houses will they
be able to add value to the lessons from the attack.
Role of outsourcing in disaster and recovery
Another option, which can turn out to be beneficial, is to
engage an outsourcing agency on encountering a disaster. Considering
the large amount of routine work involved, back to back arrangements
for recovery and backup with an outsource partner is viewed
as a better option. This will supplement backups kept at ISP
Option for contingency planning
Standby Computers: You can create standby computer systems
and locate them in air-conditioned computer rooms in trailers.
The equipment can be delivered to the nominated sites in the
event of a disaster. While it may not be practical to make
all systems redundant, do not compromise when it comes to
mission critical systems. An alternative is the outsource
route, especially if there is a very large requirement for
desktops and low-end servers.
Data Management: Efficient data management can provide facilities,
which automatically back up computer data to remote vaults
or mirror copies of databases held at a recovery center. This
may not provide fail-over for the system but will ensure data
Facilities and utilities: Supporting a data center like power
and network channels should be planned along with backup procedures.
The plan should encompass dual power source and supply, and
redundant network links from different utility feeds. The
team commissioned to design, install, and test the offsite
recovery of the data center, should give their feedback to
the project manager.
Legato published a report, which appeared in CIO Magazine,
on the pitfalls of IT systems during the September 11 attacks.
I will leave you with some lessons which we can learn from.
The most serious predicaments centered on:
Lack of documented recovery procedures.
Configuring replacement hardware without documentation detailing
the original systems configuration and setups.
Lack of tape documentation and tracking and inadequate tape
Lack of protection for departmental servers.
To deal with these problems, Legato has identified five key
Document system configurations: Many customers had not adequately
documented their system configurations. This fundamental precaution
takes form of a simple, but highly important document that
includes configuration addresses and settings required to
make the infrastructure operational. The document should be
Document and archive disaster Recovery procedures: It is important
to have a comprehensive, written disaster recovery procedure
in place. The documents should be archived off-site along
with business-critical data.
Safeguard, document and track tape media: From its interaction
with 18 customers that Legato worked with, following the September
11 disaster, it estimates that about 30 percent of business
data was lost because it had not been backed up or rotated
off-site quickly enough. Very few had documentation recording
the contents of the tapes. Tape contents must be documented
and, again, the documentation stored off-site.
Identify and protect all business critical servers: Restoring
business operations means that the servers that hosted business-critical
applications like email, ERP and database servers have to
be recovered. In addition to basic data protection, bare-metal
recovery solutions allow easier replacement and rebuilding
of critical application servers with minimal effort.
Online Data Protection Can Ease Recovery: The customers who
sailed through the disaster were the ones who had instant
access to online, off-site copies of their production data.
This second or third copy of critical data had been produced
in real-time by replication technologies.
These sorts of investments will be helpful for business continuity
like safety on the highway better late than never. There is
need for accountability. The choice is yours.
M. D. Agrawal is Chief Manager, IS Refinery
Petroleum Corporation Ltd.