> Focus > Full Story
an Enterprise Storage Network
Storage Network (ESN) is an architecture used to extend
the reach and flexibility of a storage infrastructure in
order to leverage its value to enterprises. What this means
is that an ESN architecture consists of both enterprise
storage and storage networking technologies to deliver a
common way to manage, protect, and share information regardless
of distance or scale.
designing and implementing an ESN, you have to take into
consideration several aspects such as the possible expansion
of the enterprise in the future, the number of connections
required, security issues, the scalability factor, availability,
performance, and the services required. It is important
to understand at the outset that ESN represents an inclusive
strategy. It includes SAN, NAS, and direct attached connections,
only because the requirements of most practical situations
cannot be met by a single connection topology. ESN encompasses
both SANs and NAS in order to address the demands of realistic
Enterprise storage is at the heart of an Enterprise Storage
Network. In order to meet the demands placed upon an information
infrastructure by an organization where information is a
critical part of its success, the enterprise storage system
must perform certain functions.
connectivity-The application and computing environment
of an organization must be flexible to meet the broad
and evolving needs of that organization. In order to
permit this flexibility, the information infrastructure
must be capable of supporting any of the potential computing
environments the organization would need at present
and in the future, be it Unix, NT, Midrange, Linux,
or Mainframe operating systems and associated hardware
flexible-As information needs of an organization evolve
and grow, so will the applications of an enterprise
storage system. In order to extend its useful life and
reap the greatest RoI, enterprise storage must be scalable.
For instance, a system may start off by providing storage
to a legacy mainframe application that will eventually
become obsolete. The system can then be re-deployed
to provide network attached storage for a web environment.
protection-As information becomes more and more critical
to the success of an organization, the ability to ensure
continuous access to that information becomes equally
important. Enterprise storage must provide for fault
tolerance so as to avoid unplanned outages, information
availability in the event of unplanned outages, information
recovery following such outages, and enable the continuance
of information availability through planned interruptions
(e.g. backup or batch processes).
management-Information explosion and technological advances
have resulted in an increased demand for information
storage. As the amount of information organizations
are demanding continues to grow, the ability to manage
that information and the infrastructure that supports
it also becomes a very critical factor. Enterprise storage
systems need to not only facilitate the operation, maintenance,
and allocation of information storage; but in addition,
they need to facilitate the ability to access that information
from any system across the enterprise--all with fewer
and fewer resources per unit of data.
sharing-Speed and flexibility have become the dominant
drivers in the way organizations manage their information.
Hence, organizations will have to be able to leverage
the same information among many applications and hosts.
Enterprise storage needs to provide the functionality
to enable multiple systems to access the same information
for such applications as web hosting, messaging, data
warehousing and engineering systems.
Enterprise storage provides the foundation and functionality
for an Enterprise Storage Network. In order to leverage
the value of this foundation across the entire organization,
seamless access to information infrastructure must be
provided across the enterprise. Storage networking offers
two technologies to address this evolving connectivity
need: Storage Area Networks (SANs) and Network Attached
SANs today extend the point to point SCSI protocol used
to connect host servers to their storage
arrays into a network architecture by encapsulating it in
to the fibre channel protocol that supports a many to many
topology. In addition to enabling storage and the hosts
to be networked together, fibre channel also provides a
distance (10km) and speed (100MB) improvement over SCSI.
a result of these advantages offered by fibre channel, storage
infrastructures can now be designed to offer:
utilization of storage assets (disk and tape)
to access any storage system from any host
is important to note that while a fibre channel enables
these types of networked architectures, it is a technology
based on channel I/O. That is, each host virtually has its
own dedicated storage; it just happens that storage may
be in multiple systems or even in another building.
attached storage leverages the maturity and universality
of IP networks to provide access to information. While a
SAN essentially provides network topologies to do block-level
SCSI I/O, a NAS provides network access to files. In order
to accomplish this over IP, specialized file-serving protocols
such as NFS (for Unix) and CIFS (for NT) are used by the
hosts/servers to communicate with a file server. The file
server itself manages and accesses information and distributes
it over the network to the initiating host. When you access
a network drive through your workstation (e.g. a 'G' drive
on Windows), you are using network attached storage, where
the file server is some network server that you have been
given access to through your network login.
the file server provides a centralized point of management
for the file system that holds the files to be accessed
over the network, the same file could be accessed by any
of the hosts or clients logged in to the file server. This
makes network attached storage ideal for applications and
environments that require the sharing of files among multiple
hosts or clients even if they each have different operating
the other hand, IP and Ethernet networks do not provide
the same predictability of performance through a dedicated-like
service that channel based technologies such as SCSI or
fibre channel are designed to do. As a result, NAS is less
suited to those applications and environments that depend
on dedicated access to storage (e.g. databases or client-server
A number of factors have come together to initiate a fundamental
change in the way information infrastructures need to be
architected to meet the needs of organizations. These include
the sheer quantity of information, technology advancements,
limited human resources, and the need to be flexible. As
a result, traditional point to point storage topologies
are unable to meet the emerging needs of organizations because
they either don't take advantage of the available technology,
or are inflexible, and require more effort to manage and
storage networks, through both SAN and NAS technologies,
offer the ability to implement an information infrastructure
that addresses these factors.
The following challenges have converged to initiate the
need for storage networks:
of information growth.
for management efficiency.
of change in business requirements.
quantity and criticality of information that organizations
are storing, managing, and maintaining is growing at an
exponential rate. The contributing factors include the explosion
of Internet and web content, the emergence of digital media
assets (audio, video, and images), data warehousing and
CRM applications. As a result, IT shops are realizing that
to deploy storage in traditional (point to point or host
captive) architectures is becoming unwieldy and costly to
manage the multitude of storage pools that evolve (purchasing,
maintaining, providing access to applications, back up,
networks offer the ability to consolidate both the storage
and management for large environments, thereby reducing
the acquisition and management costs for the information
infrastructure. This is increasingly important in an economy
in which the ability to attract capable professionals to
manage and operate these information infrastructures is
becoming progressively more difficult.
conjunction with and perhaps fuelling the growth of information
are the advancements in storage technology. Both of these
enable the storage of large amounts of information and present
new challenges in terms of data protection, management,
and performance. For instance, disk drive densities tend
to double every 1.5 to 2 years--at the same cost! On the
other hand, enterprise storage arrays have a fixed number
of connections (SCSI, FC, or otherwise). Since the rise
in capacity of storage arrays has outpaced the number of
connectivity ports, we have arrived at a point where a storage
network is necessary to exploit the potential capacity (and
therefore value) of enterprise storage arrays.
the pace of change has accelerated to the point where any
infrastructure put in place must either be flexible enough
to address both current and anticipated needs, or it must
be rebuilt time and again as the needs of the organization
change. Applications, databases, and the computers that
host them are being re-engineered and re-deployed.
the content itself is the crux around which changing infrastructure
revolves. Information storage architectures are tied to
those applications, databases, and computers; and therefore,
must also be flexible enough to enable the re-engineering
and re-deployment of an organization's assets.
factors described above, result in two "situations"
that ultimately relate to issues of distance and connectivity.
These situations are:
storage' describes the situation where all connections in
a storage array are fully utilized, but the array itself
has **available capacity. This typically results from situations
where each host has small information storage requirements,
the disk drive density increases beyond the needs of the
number of hosts that can connect to an array, or when the
array has relatively few available connections.
instance, consider consolidating NT application and file
serving information on a storage array with 32 available
SCSI connections and an available capacity of 9 Tb of protected
storage. If the typical NT server requires 50 Gb of storage
and you could connect 32 servers, you would utilize 1.6
Tb or less than 20 per cent of the available capacity of
results in a higher than optimal cost per unit of storage
(because of the fixed costs associated with the array and
storage software). Further, that array is now unavailable
to the next server forcing it to go onto a second array
making it harder to pass information back and forth with
the other servers on the first array and increasing the
effort it takes to manage the environment.
'Captive host' describes the reverse situation as captive
storage. Essentially, the storage requirements of the hosts
connected to a storage array grow to exceed the capacity
of that array. This means that an additional array must
be deployed and the hosts and information re-deployed and
balanced across the two arrays. This involves substantial
effort and downtime associated with planning, executing,
and testing the migration of the hosts and storage from
the original array to the new configuration. In addition,
you end up underutilizing the storage assets if there is
available capacity elsewhere in the infrastructure that
is impractical to use due to distance or connectivity requirements
between host and storage.
information storage infrastructures are intimately coupled
with the application infrastructure (including processors,
software, etc.) they support. What this means is that when
you need to make a change in the application infrastructure,
the information infrastructure is affected as well (and
visa versa). For example, consolidating a number of applications
onto a single host platform would involve migration of applications
as well as configuring and implementing the new (consolidated)
host platform. In addition to this application infrastructure
change, the storage infrastructure must be revisited to
ensure that this new single host has access to the combined
information of all the original applications. Ultimately,
this coupling makes it more difficult for organizations
to be flexible and to rapidly deploy, change, or remove
applications to meet their evolving needs.
What is required?
storage networks present the ability for organizations to
de-couple their application and information infrastructures
and thereby generate both the flexibility and speed demanded
of information technology today. Practically speaking, this
means being able to move, add, and change application hosts
and/or information storage systems without having to rebuild
the infrastructure around the change. In order to achieve
this goal, three design concepts should be considered.
(enterprise) storage-the information infrastructure
must be functional enough to be capable of performing
information-related tasks (such as data replication)
independent of the application/host.
of storage-the information storage pool must appear
(to the applications/hosts) as if it were contained
in a single array making information accessible from
anywhere (in enterprise).
of applications (hosts)-applications software must be
host agnostic, that is, it must be capable of being
deployed on one processor today and on a different processor
implementation of these design concepts together would create
the most flexible infrastructure, it is often neither workable
nor necessary. More practically, organizations use the applicable
portions of these design concepts to strike a balance of
their current and anticipated needs.
Area Network or Network Attached Storage
the availability of the multitude storage networking technologies,
probably the first consideration should be whether to implement
a Storage Area Network or Network Attached Storage.
provide the characteristics of a dedicated storage connection
while, at the same time, allow robust network topologies
to be created to meet the complex connectivity needs. Despite
the 100 Mb bandwidth of a fibre channel connection, the
primary benefits of a fibre channel network are the flexibility
in distance and connectivity that it offers. Because of
these advantages, organizations can realize the benefits
of flexibility and speed in their information infrastructure.
For these reasons, SANs are well suited to those applications
where dedicated storage is required. These include-- client/server
and applications that require high performance storage.
enables sharing of one particular file with multiple hosts
in a heterogeneous environment at the same time. In addition,
NAS leverages the mature local area network technologies
to provide networking. These technologies are both cheaper
and more mature than their fibre channel counterparts. On
the other hand, local area network performance is not as
predictable as channel based architectures. For these reasons,
NAS suits file serving (network directory) consolidation
and file serving applications (e.g. CAD, SW development,
web hosting, or web mail).
both technologies offer advantages and disadvantages that
help decide which technology best meets the specific needs
of a given environment, it is often the case that enterprise
storage network implementations require both SAN and NAS.
For instance, a typical enterprise environment will have
network shared directories and web serving (well suited
for NAS); and at the same time, have client/server and database
applications (well suited for SAN). Further, in many cases,
the file server used for NAS can be connected on the SAN
to its storage.
Achieving Availability Service Levels
discussed above, various levels of availability can be achieved
through both switch and director architectures. Here, we'll
examine the spectrum of availability service levels and
how they can be attained using the available technologies.
extension can be achieved using any of the SAN technologies;
however, if the goal is merely to leverage an existing infrastructure
beyond the current distance or connectivity limitations,
FC-AL hub topologies represent the most inexpensive way
to accomplish this goal. The following diagram depicts an
existing infrastructure based on enterprise storage being
leveraged to a small number of departmental hosts that may
be in a separate server room some distance from the primary
can be layered on top of connectivity extension by deploying
a FC-SW switch or director in lieu of a hub. The topology
would be quite similar to the above connectivity extension
topology. However, it is important to note that in both
cases a single path to storage is used to achieve the objectives
of connectivity and performance. Such an architecture is
still subject to faults and is not classified as high availability.
availability is achieved through architectures with switches
or through component design with directors.
are, by design, highly available devices. This is achieved
through redundant internal components and the application
of highly available (not modular) fibre channel ports. A
high availability architecture using directors can be achieved
through single connections between devices and the director.
In this architecture, single points of failure in the connection
path still exist in the host bus adapter and the director
ports (even though they are of a high reliability design).
order to achieve high availability using switches, dual
paths through the fabric must be established in order to
achieve the same level of redundancy of switching components
as with the director architecture. While sometimes more
costly and complex to implement than the director architecture,
the switch architecture does eliminate all single points
of failure and thereby provides improved path availability
over the high availability director architecture.
tolerant (5-9s availability) architectures exceed high availability
by providing multiple highly available paths for data. Again,
this can be achieved through both a switch or director architecture.
The fault tolerant director architecture is achieved through
dual paths through the director (i.e., dual, high-availability
paths). Ideally, the dual paths would be implemented through
separate directors. In order to deliver this level of availability
through switch architecture, a minimum of three paths would
be required because each path does not provide high availability
by itself. A triple or quadruple path switch architecture
does provide fault tolerant levels of availability while
trading off additional network complexity.
An important design consideration for a SAN implementation
is how to accommodate growth and change in the environment.
With this in mind, there are three basic SAN architecture
philosophies that we will explore. These are Centralized,
Departmental, and Mixed (Core/Edge) architectures.
SAN's are characterized by consolidation of information
into a single point of control while allowing the application/host
resources to be distributed as necessary throughout the
enterprise. One goal that should be kept in mind is to provide
the ability to access any information to any application
both in the original implementation and maintaining that
ability as the environment grows and changes. If the environment
is small enough, this can be achieved, at least initially,
through a single-tier architecture as depicted below.
specific number of host and storage ports will determine
if a switch or director will be required (directors can
support larger numbers of host and storage connections due
to the sheer quantity of ports they carry). In order to
account for future growth (addition of hosts and storage
to the fabric), unused or open switch or director ports
need to be provided.
instance, consider an initial environment with 10 dually
connected hosts (20 fabric ports required) and a single
storage array with 5 connections (5 fabric ports). This
would require 25 fabric ports and could be implemented using
a single 25 or even 32 port director. However, if we want
to allow for 10 additional hosts and an additional storage
array to be connected, we would need 25 additional ports
in the future. And more importantly require these additional
hosts to access the original storage array and the original
hosts to access the new storage array.
order to accomplish this, we would need an implementation
using 2 directors, initially deployed with 12-16 ports each,
but with a capacity for up to a total of 64, while still
maintaining the "universal access" design goal.
single-tier architecture will not support larger environments.
This is because of the number of hosts and storage ports
required. In order to scale to larger environments and still
provide universal access and availability requirements,
it will be necessary to deploy a 2-tier architecture. The
first tier provides the connections to the application/host
infrastructure and the second tier provides the connections
to the information/storage infrastructure. Extensive use
of inter-switch links (ISLs) linking the 1st and 2nd tiers
enables the universal access to information for all hosts.
The number of ISLs must be carefully determined to allow
for both multi-pathing through the fabric (to allow for
high availability or fault tolerance) and to provide the
necessary bandwidth (to avoid network congestion). As with
the single tier architecture, unused ports must be designed
in to each switch or director; however, fewer free ports
are required per device because any given device need only
connect into the fabric in order to have access to all devices
SAN implementations apply when the organization has multiple
independent departments with relatively independent information
needs. In these situations, the application/host infrastructure
for each department for the most part leverages its own
information infrastructure. On the other hand, the need
for common infrastructure management, the occasional need
to share information across the organization, and the need
of the organization to re-organize and re-focus without
having to re-engineer its infrastructure all necessitate
that this be tied in to a single enterprise storage network.
These types of environments typically will evolve from separate
order to link these "SAN islands", it is most
practical to create a core fabric that each island can plug
management becomes simple and allows for future growth because
it is necessary to design the core for extension while each
department can be designed independently. In the above diagram,
switches were used to meet departmental needs, while a director
was used for the core to minimize complexity of design.
(Core/Edge) architectures are a hybrid of centralized and
departmental. They make the most sense when there is a core
central infrastructure that can be leveraged by distributed
relatively independent computing environments. In these
situations the core information infrastructure is architected
for high scalability and availability (typically using a
director) and services the primary data center information
and application resources.
distributed departments are typically implemented using
switch architectures that are connected back to the core
infrastructure in order to leverage the centralized resources
(e.g. centralized tape backup, data warehouse databases,
ERP systems, etc.)
Designing In Scalability
Network attached storage implementations in an organization
tend to begin as general purpose file servers providing
NAS to individual departments. As the organizational demand
and the mission criticality of NAS increases, more and more
of these departmental file servers are deployed. Eventually,
the NAS infrastructure reaches a point where it becomes
necessary to consolidate in order to achieve the availability
requirements and curb the growth of management effort required
to maintain the environment.
implementing the consolidated NAS infrastructure there are
several factors that should be considered in the design
to ensure it scales well:
Management of the infrastructure.
and load balancing.
is directly related to the level of consolidation (number
of systems) and the ability to provide enterprise management
of both the storage and the connectivity portions of the
infrastructure. In order to minimize the effort/resources
required to manage the infrastructure, it is important to
meet the needs of the infrastructure with the fewest number
of file servers.
the same time, each server must be capable of addressing
all enterprise storage arrays so that there can be a single
management framework for the entire infrastructure. This
is typically achieved by leveraging a cluster of high-powered
special purpose file servers in front of high-capacity enterprise
storage systems. Management efficiency can be achieved through
optimal clustering (where one standby file server services
multiple live file servers) and leveraging the management
capabilities of enterprise storage.
connectivity means being able to access any file in any
file system from anywhere in the enterprise. To best achieve
this, the IP network and file server connection must be
optimized to allow for initial infrastructure access needs
as well as future growth. As with SAN infrastructures, it
is important to design available unused network connections
on each file server to ensure it can be connected to network
expansion with a minimal of hops through the network.
file servers must expand their storage connections to allow
for growth of their file systems and connectivity to multiple
storage arrays. File servers that support high levels of
enterprise connectivity are characterized by multiple network
and storage (SCSI or fibre channel) connections. In addition,
enterprise connectivity means being able to provide access
to all flavors of hosts. Today, simultaneous access to Unix
and NT systems has become a standard.
continuity means being able to minimize both planned and
unplanned downtime (inaccessibility). This becomes more
and more critical as NAS is both centralized and consolidated
where a single outage could impact a large part of the organization.
downtime is typically addressed by eliminating single points
of failure in the network, storage, and file server. Enterprise
storage systems (unlike JBOD systems) have built in redundancy.
At the disk level, this is typically through mirroring or
RAID implementations. While any of these schemes eliminate
single points of failure, consideration should be given
to RAID group size as it directly impacts rebuild times
while recovering from a failure--thus extending the exposure
to a second failure leading to data loss.
servers typically utilize a combination of cluster capability
(for high availability) and remote mirroring (ideally synchronous,
for fault tolerance). Unplanned downtime is primarily due
to data backup and maintenance. In order to avoid backup
and maintenance from impacting operations, separate backup
systems (file servers and disk) must be incorporated to
perform that function. Typical implementations have a dedicated
backup file server that mounts copies of the "live"
file systems in order to perform zero-impact backups during
and load balancing capabilities of the NAS infrastructure
will directly affect its ability to provide flexibility
and growth. The more hosts/clients that a single file server
can service, the greater the level of consolidation, connectivity,
and management efficiency that can be attained.
as file systems are created, changed, and expanded, it becomes
necessary to balance the load across file servers in the
infrastructure so that a single file server is not a choke-point
while others remain idle. For these reasons, it is generally
less expensive to implement larger, faster file server systems
that can share access to all the NAS information.
Enterprise storage networks are a marriage of enterprise
storage, storage area networks, and network attached storage.
While there is much debate about the ultimate storage network
technology winner being either SAN or NAS, it is more practical
to consider where each technology can be best applied to
achieve the information needs of the organization. It is
often the case that the two represent complementary technologies.
More importantly, it is important to understand the organizational
forces that are driving the need to implement an enterprise
storage network. The information infrastructure should then
be designed leveraging the available storage, SAN, and NAS
technologies to best meet the current and future needs of
- T. Srinivasan, country manager, EMC India, can be reached