EMC VNX 5400 Storage Solution
EMC VNX 5400 Storage Solution
EMC VNX 5400 Storage Solution
EMC SOLUTIONS
Abstract
This white paper describes the technical validation of a 20,000-user Exchange
2013 storage solution deployed on EMC® VNX5400™ unified storage and
virtualized on Microsoft Hyper-V according to criteria specified by the Microsoft
Exchange Solution Reviewed Program (ESRP) – Storage program. The
performance results and best practices presented in this paper provide
validated guidelines for configuring the VNX5400 storage system for a large
enterprise Exchange Server 2013 environment.
March 2014
Copyright © 2014 EMC Corporation. All Rights Reserved.
EMC2, EMC, and the EMC logo are registered trademarks or trademarks of EMC
Corporation in the United States and other countries. All other trademarks used
herein are the property of their respective owners.
For the most up-to-date listing of EMC product names, see EMC Corporation
Trademarks on EMC.com.
Disclaimer ............................................................................................................................................ 5
Features ............................................................................................................................................... 6
EMC VNX series of unified storage platforms .................................................................................... 6
Architecture................................................................................................................................. 6
VNX software ............................................................................................................................... 7
VNX FAST VP ................................................................................................................................ 8
EMC Virtual Provisioning technology on VNX ............................................................................... 9
EMC VNX5400 specifications....................................................................................................... 9
Best practices..................................................................................................................................... 19
Hyper-V virtual Fibre Channel ......................................................................................................... 19
Mailbox server virtual machines ..................................................................................................... 19
Hypervisor host .............................................................................................................................. 20
Networking .................................................................................................................................... 20
Core storage................................................................................................................................... 20
Backup strategy ............................................................................................................................. 21
Conclusion ......................................................................................................................................... 25
Contact EMC .................................................................................................................................. 25
Disclaimer
This document has been produced independently of Microsoft Corporation. Microsoft
Corporation expressly disclaims responsibility for, and makes no warranty, express or
implied, with respect to, the accuracy of the contents of this document.
The information contained in this document represents the current view of EMC on
the issues discussed as of the date of publication. Due to changing market
conditions, it should not be interpreted as a commitment on the part of EMC. In
addition, EMC cannot guarantee the accuracy of any information presented after the
date of publication.
1
The ESRP—Storage program was developed by Microsoft Corporation to provide a common
storage testing framework for vendors to provide information on their storage solutions for
Microsoft Exchange Server software. For more details on the Microsoft ESRP—Storage
program, refer to http://technet.microsoft.com/en-us/exchange/ff182054.aspx.
EMC VNX series unified storage systems deliver uncompromising scalability and
flexibility for the mid-tier while providing market-leading simplicity and efficiency to
minimize total cost of ownership.
Architecture
Based on the powerful new family of Intel Xeon E5-2600 (Sandy Bridge) processors,
the EMC VNX implements a modular architecture that integrates hardware
components for block, file, and object with concurrent support for native network-
attached storage (NAS), Internet small computer system interface (iSCSI), Fibre
Channel (FC), and Fibre Channel over Ethernet (FCoE) protocols. The series delivers
file (NAS) functionality via two to eight X-Blade data movers and block (iSCSI, FCoE,
and FC) storage via dual storage processors (SPs) using full 6 Gb/s Serial Attached
SCSI (SAS) disk drive topology. The system includes the patented EMC MCx™
multicore storage software operating environment that delivers unparalleled
performance efficiency. You can start with block or file functionality and easily
upgrade to unified when needed. The unified configuration includes the following
rack-mounted enclosures:
• Block Services: Disk processor enclosure (includes disk drives) or SP
enclosure (no drives included) plus standby power system. Capacity for block
or file use cases is added via disk-array enclosures (DAEs).
• File and Unified Services: One or more data mover enclosures and a control
station to deliver file protocols.
VNX software
The VNX series includes six software suites and two software packs, making it easier
and simpler to attain the maximum overall benefits 3. All VNX software is managed
through Unisphere.
Software suites
The VNX software is also available in modular suites:
• FAST Suite—Automatically optimizes performance and cost to simultaneously
provide the highest system performance and the lowest storage cost
2
Features listed are based on the VNX Operating Environment version available at the time of
this solution validation. EMC constantly improves and updates its storage fleet with new
features and functionalities. For the latest features and updates visit www.emc.com.
3
Features of Suites and Packs might vary depending on the system. See the EMC VNX Series
Software Suites data sheet for details.
Software packs
The VNX series software is also available in two comprehensive packages to ensure
that customers have all of the necessary capabilities to protect and manage their
information.
• VNX Total Protection Pack—Includes local, remote, and application protection
suites
• VNX Total Efficiency Pack—Includes the suites in the VNX Total Protection
Pack, along with the FAST and Security & Compliance suites
For additional details about the EMC VNX series of unified storage systems, refer to
the following white papers available at www.emc.com:
VNX FAST VP
The FAST VP feature, which is included in the VNX FAST Suite, can lower the total cost
of ownership (TCO) and increase performance by intelligently managing data
placement according to activity level. When FAST VP is implemented, the storage
system measures, analyzes, and implements a dynamic storage-tiering policy much
faster and more efficiently than a human analyst could ever achieve.
Storage provisioning can be repetitive and time-consuming and, when estimates are
calculated incorrectly, it can produce uncertain results. It is not always obvious how
to match capacity to the performance requirements of a workload’s data. Even when a
match is achieved, requirements change and a storage system’s provisioning
requires constant adjustments.
Storage tiering allows a storage pool to use drives of varying levels of performance.
Logical unit numbers (LUNs) use the storage capacity needed from the pool on the
The VNX Operating Environment enables Microsoft Windows and Linux/UNIX clients
to share files in multiprotocol (NFS and CIFS) environments. At the same time, it
supports iSCSI, FC, and FCoE access for high-bandwidth and latency-sensitive block
applications.
Table 1 lists VNX5400 features for block storage. For additional VNX specifications for
both block and file storage, see EMC VNX Series Unified Storage Systems at
www.emc.com.
This solution employs virtualization to effectively use hardware resources that benefit
the Exchange application. In this solution, the physical hypervisor server hosts
multiple Exchange mailbox server virtual machines.
In this solution, 20,000 users are distributed across eight mailbox servers in a DAG.
The DAG has two RAID-protected copies of every Exchange database—a primary
(active) copy and a secondary (passive) copy—that are evenly split between
hypervisor hosts and storage pools on two VNX5400 arrays. Each Exchange database
replicates to an alternate mailbox server that resides on a different Hyper-V host
through the use of the Exchange native DAG host-based log shipping mechanism.
The solution is designed to eliminate a single point of failure and can handle the loss
of an array, hypervisor server, mailbox server virtual machine, database volume, host
bus adapter (HBA), or switch. For example, during a mailbox server failure or
maintenance operation, each mailbox server can handle the compute and storage
requirements of all active and passive databases. Similarly, during a hypervisor host
failure or maintenance activities, another host can support all mailbox servers
(20,000 users with four virtual mailbox servers per hypervisor host).
Note: You can compare this solution architecture to one in which Exchange Server 2013 is
deployed in a stand-alone configuration (without DAG) and a single VNX5400 storage array
with eight mailbox servers provides service to all 20,000 users.
Figure 2. EMC VNX5400 storage solution architecture for 20,000 Exchange 2013 users
Storage design for Storage design is an important element for ensuring the successful deployment of
Exchange Server Microsoft Exchange Server 2013. Sizing and configuring storage for use with
2013 Exchange Server 2013 can be a complicated process driven by many variables and
requirements that vary from organization to organization. Properly configured
Exchange storage, combined with optimally sized server and network infrastructures,
can guarantee a smooth Exchange operation. Defining a unit of measure—a mailbox
server building block—is one method that simplifies the sizing and configuration of
large amounts of storage on the EMC VNX series storage arrays for use with Exchange
Server 2013.
A mailbox server building block represents the amount of disk, network, and server
resources required to support a specific number of Exchange Server 2013 users. The
amount of required resources is derived from a specific user profile type, mailbox
size, and availability requirements. Using the building block approach simplifies the
design and implementation of Exchange Server 2013.
Once the initial building block is designed, it can be easily reproduced to support the
required number of users in your enterprise. By using this approach, EMC customers
can now create their own building blocks that are based on their company’s specific
Exchange environment requirements. This approach is helpful when future growth is
expected because it makes Exchange environment expansion simple and
straightforward. EMC best practices involving the building block approach for
Exchange Server design have proven to be highly successful in many customer
implementations.
This building block is made up of 14 x 3 TB NL-SAS drives (12 for databases and 2 for
logs) and supports I/O and capacity for a single mailbox server, which in turn
supports 5,000 users (2,500 active and 2,500 passive).
The solution uses four of these building blocks (56 disks—48 disks for databases and
8 disks for logs) on the primary storage array to scale the configuration up to 20,000
users. The four building blocks are then duplicated on the secondary storage array.
Multiple building blocks are grouped together in storage pools. In this solution we
grouped two building blocks in a RAID 1/0 storage pool with 24 x 3 TB NL-SAS disks,
thus supporting database I/O and mailbox capacity for two mailbox server virtual
machines. We also added 2 x 100 GB flash disks to the pool for FAST VP to allow for
absorption of metadata. For transactional logs, we configured separate RAID 1/0
pools with four 3 TB NL-SAS disks, each supporting two building blocks. See Figure 3
on page 14 for more details.
Item Value/Description
Number of users supported by a 5,000 (2,500 active and 2,500 passive)
single mailbox server
Mailbox size 2 GB
Drive type and capacity 3 TB NL-SAS drives; 7,200 rpm; 100 GB flash
* For mailbox server CPU and memory requirements, use the Exchange 2013 Server
Role Requirements Calculator on Microsoft TechNet.
By using storage pools, you can take advantage of FAST VP, which can lower the total
cost of ownership (TCO) and increase performance by intelligently managing data
placement according to activity level. Also, using storage pools provides the
capability to use VNX Snapshots for data replication and protection. EMC AppSync®
uses VNX Snapshots replication technology to provide fully automated application-
consistent replicas of Exchange data. To learn more about protecting Exchange data
with EMC AppSync and VNX Snapshots, review the white paper titled Advanced
Protection for Microsoft Exchange 2010 on EMC VNX Storage available at
www.emc.com.
Figure 3 illustrates the VNX5400 storage solution using building block methodology,
designed to provide the best performance and sufficient capacity for virtualized
Exchange 2013 with 20,000 users at 2 GB mailbox capacity.
Any combination of these factors can affect the total number of mailboxes supported
by a single mailbox server. Because all of these factors fall outside the scope of
ESRP—Storage program solution validation, a single mailbox server might not
necessarily support 5,000 users (as validated for the solution described in this
document) in every customer deployment.
Targeted customer This solution is designed for any medium-to-large enterprise that is planning to
profile consolidate and virtualize its Exchange Server 2013 environment, and requires a
highly reliable, scalable storage solution. The design of this solution, which supports
20,000 Exchange users, has the following characteristics:
• 20,000 active mailboxes
• 0.10 IOPS per user (150 messages sent and received per user/day, with an
additional 20 percent overhead—0.12 IOPS validated)
• 2 GB mailbox size
• Eight Exchange servers per DAG, each supporting 5,000 users (2,500 active
and 2,500 passive)
• Native DAG replication mechanism for mailbox resiliency and high availability
with two database copies maintained
Note: If you require more than two database copies, EMC recommends placing two RAID-
protected DAG copies on one storage array and placing a third RAID-protected copy on
another array.
Attribute Value
Number of Exchange mailboxes 20,000
Number of DAGs 1
Number of copies/database 2
Number of servers/DAG 8
4
Storage performance characteristics change based on the percentage utilization of the
individual disks. Tests that use a small percentage of the storage (~25 percent) might exhibit
reduced throughput if the storage capacity utilization is significantly increased beyond what is
reported in this paper.
Component Description
Storage connectivity FCoE
Storage model and OS/firmware EMC VNX5400, VNX Block Operating Environment
revision version 05.33.000.5.038
Windows Hardware Compatibility List
Number of HBAs per host 4 (2 for boot and 2 for virtual Fibre Channel)
Number of disks used in this 56 (48 for databases and 8 for logs)
solution as validated
Component Description
HBA driver 2.3.0.20
Attribute Description
Attribute Description
Disk type and speed 3 TB NL-SAS 7,200 rpm
Link type IP
5
Capacity of drives, and therefore the raw data capacity of the disk array, based on binary
representation of 1KB = 1024 bytes. This means that 1 GB = 1,073,741,824 bytes.
Hyper-V virtual In this solution we used virtual FC within the guest OS on the Hyper-V server. The
Fibre Channel virtual FC feature in Hyper-V on Windows Server 2012 and Windows Server 2012 R2
provides virtual FC ports within the guest operating system, which allows you to
connect to virtual FC directly from within virtual machines. Therefore, a virtual
machine can connect directly to a SAN through a virtualized FC adapter.
For more information, requirements, and best practices about Hyper-V virtual FC, refer
to Hyper-V Virtual Fibre Channel Overview on Microsoft TechNet.
Mailbox server Follow these recommendations to ensure the best possible mailbox server
virtual machines performance:
• Format Windows Server NTFS volumes to be used for Exchange databases and
logs with an allocation unit size of 64 KB.
Networking For high availability deployments that use DAGs, we recommend multiple physical
NICs connected to different networks to isolate user MAPI traffic from database
replication traffic.
For iSCSI deployments, we prefer multiple network switches for fault tolerance and
performance. Where this is not possible, use VLANs to isolate iSCSI traffic.
Core storage EMC VNX5400 is already optimized for Exchange Server 2013 workloads and requires
only a few minor adjustments. Follow these guidelines to ensure optimal storage
performance:
• Isolate the Microsoft Exchange server database workload from other
I/O-intensive applications or workloads (assign the Exchange workload to its
own set of disks). This ensures the highest level of performance for Exchange
and simplifies troubleshooting in the event of a disk-related performance
issue.
• For VNX models that preceded next-generation systems (VNX5100, VNX5300,
VNX5500, VNX 5700, VNX7500), set the VNX storage array page size
parameter to 16 KB, and allocate 1 GB of available VNX array SP memory to
read cache and the remaining available memory to write cache.
Note: On next-generation VNX systems such as the VNX5400 model, you do not
have to set the page size and the memory for read and write cache. These settings
are now managed automatically.
• Ensure that write cache is enabled for all database and log LUNs.
• Plan for performance even in a failover situation. Ensure SP utilization does
not exceed 70–75 percent.
• Ensure that I/O load and bandwidth are evenly balanced and distributed
across the VNX5400 front-end ports and back-end buses for failover and load
balancing.
• When sizing storage, always prioritize IOPS over capacity. After calculating
the IOPS requirements, assess the capacity that is required.
For additional information about configuring VNX storage for best performance,
review the white paper titled EMC VNX Unified Best Practices for Performance
available on www.emc.com.
For a complete list of EMC storage best practices for Microsoft Exchange, as well as
using EMC FAST VP, EMC FAST Cache, and EMC XtremCache™ technologies with
Exchange workloads, review the white paper titled Microsoft Exchange Server Best
Practices and Design Guidelines on EMC Storage available on www.emc.com.
Backup strategy When you are using intelligent VNX storage arrays for Exchange storage, we
recommend using EMC AppSync to take application-consistent Volume Shadow Copy
Service (VSS) snapshots to protect your Exchange data.
Information resources
For more information on EMC solutions for Microsoft Exchange Server, visit
www.emc.com/exchange.
Note: Detailed test result reports are attached to this PDF in HTML format.
We validated this solution using Microsoft Jetstress 2013 to ensure that the storage
design satisfies the disk I/O and capacity requirements for the target profile. Jetstress
simulates Exchange Server 2013 I/O types at the database level by interacting
directly with the database Extensible Storage Engine (ESE) technology (also known as
Jet) on which Exchange is built.
Jetstress can be configured to test the maximum I/O throughput available to the disk
subsystem within the performance constraints of Exchange. Alternatively, Jetstress
can be configured to accept a specific user profile (user count, IOPS per user) and
validate that the disk subsystem is capable of maintaining an acceptable level of
performance with such a profile. Both configuration options produce similar results.
Note: The Jetstress tool is designed to test performance of an Exchange storage subsystem
before the subsystem is placed in a production environment. Jetstress is not designed to
test server CPU, memory configuration, or the impact of user client activity. To test Exchange
server functionality and end-to-end deployment, it is a best practice to use Microsoft Load
Generator (LoadGen).
The documentation for Jetstress describes how to configure and execute I/O validation
or evaluation on your server hardware. The Microsoft Jetstress tool is available at
www.microsoft.com/en-us/download/details.aspx?id=36849.
Storage reliability The reliability (stress) test runs for 24 hours. The goal of this test is to validate that
(stress test) the storage can handle high I/O load for a long period of time. Both log and database
files are analyzed for integrity after the stress tests to ensure that no database or log
corruption occurs. The 24-hour stress test results reveal that:
• No errors were reported in the saved event log file.
• No errors were reported for the database and log checksum processes.
Storage The purpose of the primary storage performance test is to exercise the storage with
performance the maximum sustainable Exchange I/O for 2 hours with 100 percent user
concurrency. The test reveals the amount of time it takes the storage to respond to
I/O requests under load.
This section presents the sum of all database transactional I/O and the average I/O
latencies for all databases and logs recorded during the 2-hour test duration. Data for
each server is presented separately. Aggregates across servers are also included.
I/O database average write latency (ms) 1.72 2.07 1.81 1.76
I/O log average write latency (ms) 0.59 0.61 0.55 0.58
Database backup Two tests are reported in this category. The first test measures the sequential read
and recovery rate of database files, and the second test measures recovery/replay (the playing of
performance transaction logs against databases) performance.
Metric Value
MBs read/sec per database 35.18
Metric Value
Average time to play one log file (in seconds) 2.78
How to view Click the paper clip icon in the left-hand pane of Adobe Reader to reveal the list of
Jetstress reports Jetstress reports (HTML files). Double-click each file to open the corresponding report
in your browser.
EMC recommends that you use a building-block approach when designing storage
solutions for Exchange Server 2013. This solution uses a building block of 5,000
users with 0.12 IOPS per user (including 20 percent overhead) and a 2 GB mailbox
size for validation.
The configuration is scaled to 20,000 users through the utilization of four building
blocks on an EMC VNX5400 storage array. This configuration meets all of the
recommended Microsoft Exchange Server 2013 metrics for performance and capacity.
In addition, the performance (I/O and latency) test results presented in this
Microsoft-approved ESRP document demonstrate that this solution provides
significant headroom to accommodate future user growth.
EMC periodically publishes Proven Solutions white papers that demonstrate that the
EMC VNX series of storage arrays can handle extremely heavy Exchange Server 2013
workloads. For more information, visit www.emc.com/exchange.
Contact EMC EMC recommends that you consult with EMC Professional Services to assist with the
design and deployment of a similar solution. For information regarding this or any
other EMC solution, use the following numbers:
• United States: (800) 782-4362 (SVC-4EMC)
• Canada: (800) 543-4782 (543-4SVC)
• Worldwide: (508) 497-7901
For additional information on EMC products and services available to customers and
partners, visit www.EMC.com or http://Powerlink.EMC.com.