Making a speedy recovery

Wipro Technologies' DR center at Chennai is fully equipped to keep the company's business up and running in case disaster strikes.

Abhinav Singh

Wipro Technologies' range of business and operations spans development centers, marketing and sales offices spread across the world in countries like USA, UK, and Japan.

The company is also one of the first in India to achieve a BS7799-2: 2002 certification as per the new standard released in September 2003. Critical aspects of Physical Security, Information Security & Business Continuity Planning (BCP) Security, Information Security and BCP have been addressed with the BS7799 framework. And keeping with the framework, the company has framed a comprehensive Disaster Recovery (DR) policy.

The need for DR

Wipro Technologies has 15 development centers in India out of which nine are in Bangalore, two in Chennai, two in Hyderabad, one in Pune, and one in Mumbai. The company also has offices in Mumbai and Bangalore in India, Santa Clara, New Jersey and Dallas in the US, Euston and Reading in the UK, and in Tokyo. Apart from this the company has near-shore development centers in the US, UK, Canada and Japan.

With such a vast scope of operations it became imperative for Wipro Technologies to frame a DR strategy which identified critical business processes and systems across the organization.

The business continuity plan

The Business Continuity (BC) plan at Wipro Technologies talks about managing availability risks for people, and the flow of processes and technology operations in case of a disaster.

Sunil P Rangreji, General Manager, Global IT infrastructure at Wipro Technologies said, "Business Continuity Planning (BCP) goes beyond disaster recovery planning and includes the recovery of critical business processes in the event of a disaster. Broadly, it is the process for developing, testing, and maintenance of a plan to recover critical IT systems when disaster strikes."

At Wipro Technologies, BCP includes aspects like the procedures for command and control of the organization, recovery procedures and resources, vital records, and infrastructure (facilities, telecom and IT).

The company has comprehensively defined a framework for its BC plan. As part of the framework, a business impact analysis and risk assessment is undertaken which in turn helps define the scope of recovery and priority of data restoration. Critical business processes are defined as part of the framework. Technology measures like fallback storage and server configuration needed for restoration of a given process and data, are also part of the BCP framework.

Ready to recover

Wipro Technologies' worldwide centers are well-equipped to recover and protect vital data in case of a disaster. There is a Chief Recovery Officer, damage assessment team, and an emergency response team at strategic centers of the company spread across the world.

The task of a damage assessment team is to classify the disaster and invoke the recovery process based on the level of disaster. Besides these there are critical people identified in all functions and business processes. All of them have visas to travel across countries in case of a disaster. The online project management and configuration management helps the company seamlessly move work between locations. The aim is to quickly respond to emergencies, minimize the impact of the disaster, analyze the impact, and prevent its reoccurrence through well-defined and well-documented policies and processes.

DR prioritized

Different types of data and applications have been prioritized on the basis of their importance to the company's business. Three classes have been created—the hot site, warm site, and cold site—for prioritizing the recovery timing for different applications and sites running across the company.

A hot site is kept operationally ready with specific hardware platforms and fallback connectivity for immediate availability in the event of disasters. WAN links, WAN equipment, and applications like SAP, CRM, SCM, and employee self-service servers are included in its category. The recovery time varies from immediate recovery to a maximum of eight hours.

The warm site is a partially equipped alternate site with backup tapes and basic connectivity infrastructure ready for use. Servers and desktops are pre-identified and can be installed upon notification of a disaster. Project servers, intranet servers, and quality function servers fall under this category. The recovery time is from 48 to 72 hours.

The cold site is an alternate site with basic facilities like space, power, and air-conditioning. Equipment and links are installed there to duplicate critical business functions.

Rangreji said, "Such a site can have all the basic infrastructure including a fire-proof storage facility to restart the process in case of a disaster. But the restoration time may be from 48 hours to a week."

Incremental backup is taken for mission critical applications every eight hours and may take longer for less mission critical applications. A full backup is taken at the end of every week.

Applications for DR

The company maintains a combination of Intel and Sun servers for DR. There are multiple 2 Mbps links through routes like fiber and satellite for DR. Inter-office connectivity uses 200 Mbps bandwidth.

Internet links are used from service providers like AT&T, Sprint, British Telecom, VSNL, STPI, BSNL, Bharti, Tata Teleservices, and Sify. The communication links are mostly automated.

Tape backups are carefully stored, and those containing important data are sent to Chennai. Tape backup is taken using Veritas' storage management software for servers and Altrinis for laptops.

The DR location

The company has identified Chennai as the location for its DR center and has a well-equipped facility there to run the operations of the entire company in a worst-case scenario.

Around eight personnel maintain the Chennai DR center round-the-clock. Chennai also acts as an offsite storage facility for tapes that are stored in a fireproof area. The center in Chennai replicates all the data being processed by any Wipro Technologies facility worldwide.

In case of a major disaster, the DR infrastructure has the capability to run the whole system within four hours of any other strategic center going down. Every six months Wipro Technologies undertakes a DR restoration exercise where for a whole week all the systems and processes of Wipro Technologies are shut down across the world and are run from its DR center at Chennai.

Abhinav Singh can be reached at abhinav@expresscomputeronline.com

A Phased WAN deployment

Going nationwide in phases

Federal Bank needed to share data between its Head Office, nationwide offices, branches, and ATMs. It designed a nationwide WAN called FedWide and deployed it in phases. Here's a look at the strategies used by the bank.

Soutiman Das Gupta

Federal Bank, a 70-year old private sector bank has a net worth of over Rs 560 crore. But in order to survive in the fiercely competitive banking environment of today, it had to introduce greater efficiency in its operations and workflow.

The bank realized that a good way to do that would be to share updated data across its nationwide locations. So, Federal Bank decided to link its nationwide locations on a WAN, and share data across systems.

Considering the organization's size, the entire exercise was no mean task. The bank is spread across 422 nationwide branches, 12 Regional Offices (ROs), and a Head Office (HO) in Alwaye, Kerala.

In order to link all locations in the quickest and most suitable way, the organization decided to deploy a WAN in six strategic phases. It decided to name the nationwide WAN infrastructure 'FedWide'.

Let's look into the FedWide strategy and architecture, and gain insights from KNC Nair, the CIO of Federal Bank.

FedWide design

A project plan was made with a lot of thought and care. Datacraft India Limited (DIL) was chosen as the implementation partner for the project. The plan documented the key roles and responsibilities for Federal Bank and DIL.

"Our basic goal in the initial phases was to set up a reliable leased line network with minimum expenditure. Keeping this in mind, we decided to use a hub-and-spoke WAN architecture," said Nair.

The design is such that every RO is connected to the HO through a 2 Mbps primary leased line link, and multiple 64 Kbps leased lines. This adds redundancy in the connectivity between the HO and the ROs, and acts as a fail-over path to the primary link.

The nationwide branches are connected to the nearest RO with 64 Kbps leased lines. This has reduced the per-branch cost of deploying a leased line link to the HO. The recurring charges for the backup ISDN links were optimized, because the nationwide ROs and branches were not very far from each other.

Six phases of deployment

The deployment of FedWide started in 2000 and was commenced in six planned phases.

First phase: The HO at Alwaye, Kerala was connected to the ROs at the five metro cities of Delhi, Mumbai, Kolkata, Chennai, and Bangalore, with 64 Kbps leased lines.

The Funds and Investment Branches at Mumbai, Delhi, and Chennai, the International Banking Department, and the Ernakulam location were connected to the respective ROs with leased lines. All other branches under each region were connected to the respective ROs with dial-up links.

Second phase: In the second phase, the ROs in Kerala were connected to the HO at Alwaye with a 64 Kbps leased line.

Third phase: Most of the large automated branches (70 branches) were connected to the network with 64 Kbps leased lines. ISDN lines were also provided at each location as backups to the leased lines.

In the fourth phase, 140 new branches were connected to the network with 64 Kbps leased lines. ISDN lines were used at the locations wherever feasible, as a backup measure.

Fifth phase: 70 more branches were connected on the network at the end of this phase.

In the sixth phase, they connected the remaining branches to the network by December 2003.

The entire project was tracked by DIL on a weekly basis. Reports were generated and submitted to the bank every week on the progress and any other issues.

Success factors

All key management staff of Federal Bank and DIL were involved in the project, right from the first phase. This helped to resolve all critical issues at the earliest.

To ensure quicker availability of leased lines with minimum accounting and administration hassles, DIL partnered with BSNL for the FedWide project. The top management from the BSNL HO in Delhi was informed about the project and it's criticality. BSNL, Delhi coordinated constantly with its offices in Kerala to address issues when they arose.

FedWide now

FedWide, the network, which was initially set up for collecting data only from the few ROs, has evolved into a mammoth architecture that connects 300 nationwide branches and offices.

FedWide is currently used for connectivity between the bank's ATMs, branch servers, and data transmission from the branches/offices to the HO. It allows access to the Bank's intranet server, e-mail server, and extranet server. The newest application to run on FedWide is the 'Anywhere Banking' service. VoIP has also been implemented on the network to ease communication.

FedWide has enabled the network to offer ATM services to a vast majority of its customers in Kerala, and has increased the customer base significantly. At the end of the final phase in December 2003, the bank will have connected all its 422 branches.

Network components

The nationwide branches are connected to the network with Cisco 1760 routers. The ROs are equipped with Cisco 3600 series routers. The procurement, installation, and integration of the equipment with the existing network was done by DIL. The integrator also signed a three-year support contract with the bank.

Bandwidth monitoring is performed with the help of a customized Multi Router Traffic Grapher (MRTG) solution, which runs on a Linux platform. The MRTG is a tool to monitor the traffic load on the network links. It generates HTML pages containing graphical images, which provide a live visual representation of the traffic. A network monitoring tool from Ipswitch is also used by the bank.

Management

The network functions with a distributed architecture in which, each branch has a dedicated server. There is no central data center. All offline bank ATMs have been connected with VSATs.

"We manage the network from a central location. And the system integrators manage the leased lines. The leased lines give an uptime of 96 percent," said Nair.

Security

The connectivity links between the HO and nationwide branches use IPsec for security. And all banking transactions take place through this setup. "We have also implemented other security devices and measures to take care of any unlawful entry into the network," explained Nair.

Future

"The bandwidth of the connectivity links between the ROs and the HO is being utilized to the maximum. We plan to double the bandwidth to accommodate new applications. However, the bandwidth of the links between the branches and ROs are yet to be fully utilized," explained Nair.

Soutiman Das Gupta can be reached at soutimand@networkmagazineindia.co