VSAN 2 Node Guide

vSAN 2 Node Guide
First Published On: 03-07-2018

Last Updated On: 06-18-2019
Copyright © 2019 VMware, Inc. All rights reserved.

vSAN 2 Node Guide
Table Of Contents
1. Overview
1.1.Introduction
2. Support Statements
2.1.vSphere Versions
2.2.vSphere & vSAN
2.3.Hybrid and All-Flash Support
2.4.vSAN Witness Host
2.5.Supported on vSAN but not vSAN 2 Node Clusters
3. New Concepts in vSAN - 2 Node Clusters
3.1.The Witness Host
3.2.Read Locality in vSAN 2 Node Clusters
3.3.Witness Traffic Separation (WTS)
4. Requirements
4.1.VMware vCenter Server
4.2.A Witness Host
4.3.Networking and Latency Requirements
5. Configuration Minimums and Maximums
5.1.Virtual Machines Per Host
5.2.Witness Host
5.3.vSAN Storage Policies
5.4.Fault Domains
6. Design Considerations
6.1.Cluster Compute Resource Utilization
6.2.Network Design Considerations
6.3.The Role of vSAN Heartbeats
6.4.Bandwidth Calculation
6.5.Using the vSAN Witness Appliance as a vSAN Witness Host
6.6.Using a Physical Host as a vSAN Witness Host
6.7.vSAN Witness Host Networking Examples
6.8.vSAN Witness Appliance Sizing
6.9.vSAN Witness Host Versioning & Updating
7. Cluster Settings – vSphere HA
7.1.Cluster Settings – vSphere HA
7.2.Turn on vSphere HA
7.3.Host Monitoring
7.4.Admission Control
7.5.Host Hardware Monitoring – VM Component Protection
7.6.Datastore for Heartbeating
7.7.Virtual Machine Response for Host Isolation

vSAN 2 Node Guide
7.8.Advanced Options
8. Cluster Settings - DRS
8.1.Cluster Settings - DRS
8.2.Partially Automated or Fully Automated DRS
9. Using a vSAN Witness Appliance
9.1.Setup Step 1: Deploy the vSAN Witness Appliance
9.2.Setup Step 2: vSAN Witness Appliance Management
9.3.Setup Step 3: Add Witness to vCenter Server
9.4.Setup Step 4: Config vSAN Witness Host Networking
10. Configuring 2 Node vSAN
10.1.Pre-Requisites
10.2.VMkernel Interfaces
10.3.VMkernel Interfaces - Witness Traffic
10.4.VMkernel Interfaces - vSAN Traffic - VSS
10.5.VMkernel Interfaces - vSAN Traffic - VDS
10.6.Creating a New 2 Node vSAN Cluster
10.7.Upgrading a older 2 Node vSAN Cluster
10.8.Converting a 2 Node Cluster with WTS to a 3 Node Cluster
11. Management and Maintenance
11.1.Management and Maintenance
11.2.Maintenance Mode Consideration
12. Failure Scenarios
12.1.Failure Scenarios
12.2.Individual Drive Failure
12.3.Host Failure / Network Partitions
12.4.VM Provisioning When a Host is Offline
12.5.Multiple Simultaneous Failures
12.6.Replacing a Failed vSAN Witness Host
12.7.Failure Scenario Matrices
13. Appendix A
13.1.Appendix A: Additional Resources
13.2.Location of the vSAN Witness Appliance OVA
14. Appendix B
14.1.Appendix B: Commands for vSAN 2 Node Clusters

vSAN 2 Node Guide
1. Overview
vSAN 2 Node is a specific configuration typically implemented in environments
where a minimal configuration is required.

vSAN 2 Node Guide
1. 1 Introduction
The vSAN 2 Node configuration was introduced in vSAN 6.1. A vSAN 2 Node
Cluster is a specific configuration implemented in environments where a
minimal configuration is a key requirement. This guide was developed to
provide additional insight and information for installation, configuration and
operation of a vSAN 2 Node Cluster infrastructure in conjunction with VMware
vSphere. This guide will explain how vSphere handles specific failure scenarios
and discuss various design considerations and operational procedures for 2
Node vSAN Releases including 6.1 through 6.7.
VMware vSAN 2 Node with a Witness Host refers to a deployment where a

user sets up a vSAN cluster with 2 nodes in a single site. The 2 vSAN network
is connected to a switch, or in some editions via a Direct Connection.
A vSAN Witness Host that provides quorum for the 2 Nodes can be hosted on
a third site via low bandwidth/high latency links, or on alternate infrastructure
in the same location.
Each node is configured as a vSAN Fault Domain. The supported configuration

is 1+1+1 (2 nodes + vSAN Witness Host).
There is only one witness host in any configuration.

vSAN 2 Node Guide
With two Fault Domains available, virtual machines deployed on vSAN 2 Node
Clusters typically have mirrored data protection, with one copy of data on
Node 1, a second copy of data on Node 2, and the Witness component placed
on the vSAN Witness Host.
In the event of a node or device failure, a full copy of the virtual machine data
is still available on the alternate node. Because the alternate replica and
Witness component are still available, the virtual machine remains accessible
on the vSAN datastore.
In cases where the virtual machine was running on the failed node, vSphere HA
can handle this task if properly configured.

vSAN 2 Node Guide
2. Support Statements
vSAN Stretched Cluster configurations require vSphere 6.0 Update 1 (U1) or
greater.

vSAN 2 Node Guide
2. 1 vSphere Versions
VMware vSAN 2 Node Cluster configurations require vSphere 6.0 Update 1

(U1) or greater. This implies both vCenter Server 6.0 U 1 and ESXi 6.0 U1. This
version of vSphere includes vSAN version 6.1. This is the minimum version
required for vSAN 2 Node support.
Advanced feature support in 2 Node vSAN requires a combination of vSAN

version, On-Disk format version, architecture, and host count.
vSAN 6.1 vSAN 6.2 vSAN 6.5 vSAN 6.6 vSAN 6.7
Feature Requireme Requireme Requireme Requireme Requireme
nts nts nts nts nts
v2 On- v2 On- v2 On- v2 On- v2 On-

vSAN 2
Disk Disk Disk Disk Disk
Node
format format format format Format
v3 On- v3 On- v3 On-

Disk Disk Disk
format format format v3 On-
Deduplicat All-Flash All-Flash All-Flash Disk
ion & architectur architectur architectur format
Compressi e e e All-Flash
on vSAN vSAN vSAN architectur
Advanced Advanced Advanced e
or Higher or Higher or Higher
Licensing Licensing Licenising
v3 On- v3 On- v3 On- v3 On-

Checksum Disk Disk Disk Disk
format format format format
v3 On- v3 On- v3 On- v3 On-

IOPS
Disk Disk Disk Disk
Limits
format format format format

vSAN 2 Node Guide
vSAN 6.1 vSAN 6.2 vSAN 6.5 vSAN 6.6 vSAN 6.7
Feature Requireme Requireme Requireme Requireme Requireme
nts nts nts nts nts
Witness vSphere vSphere vSphere

vSphere
Traffic 6.0 U3 6.5d 6.7
6.5
Separation or higher or higher or higher
v3 On- v3 On- v3 On-

iSCSI
Disk Disk Disk
Service
format format format
v5 On- v5 On-
Encryption Disk Disk
format format

vSAN 2 Node Guide
The latest On-Disk format is generally recommended per version of vSAN.
2. 2 vSphere & vSAN
VMware vSAN 6.1 introduced several features including All-Flash and 2 Node
Cluster functionality. There are no limitations on the edition of vSphere used
for vSAN. However, for vSAN 2 Node functionality, vSphere Distributed
Resource Scheduler (DRS) is very desirable . DRS will provide initial placement
assistance, load balance the environment when there's an imbalance, and will
also automatically migrate virtual machines to their correct site in accordance
to VM/Host affinity rules. It can also help with migrating virtual machines to
their correct site when a node recovers after a failure. Otherwise the
administrator will have to manually carry out these tasks.
Note: DRS is only available in Enterprise edition and higher of vSphere.
2. 3 Hybrid and All-Flash Support
VMware vSAN 2 Node Clusters are supported on both Hybrid configurations

(hosts with local storage comprised of both magnetic disks for capacity and
flash devices for cache) and All-Flash configurations (hosts with local storage
made up of flash devices for capacity and flash devices for cache).
2. 4 vSAN Witness Host
Both physical ESXi hosts and vSAN Witness Appliances are supported as a
vSAN Witness Host.
VMware provides a vSAN Witness Appliance for those customers who do not
wish to use a physical host for this role. The vSAN Witness Appliance must run
on an ESXi 5.5 or higher host. *If the vSAN Witness Appliance is used for a
vSAN 6.7 2 Node cluster, the ESXi 5.5 or higher host must also meet the CPU
requirements of vSphere 6.7.
This can include an ESXi Free licensed host, a vSphere licensed (ESXi) host, or a
host residing in OVH (formerly vCloud Air), a vCloud Air Network (VCAN)
partner, or any hosted ESXi installation.
vSAN Witness Hosts (physical or the vSAN Witness Appliance) cannot be
10

vSAN 2 Node Guide
shared between multiple vSAN 2 Node Clusters.
2. 5 Supported on vSAN but not vSAN 2 Node Clusters
The following are limitations on a vSAN 2 Node Cluster implementation:
In a vSAN 2 Node Clusters, there are only 3 Fault Domains. Two fault
domains are configured (typically referred to as the Preferred and
Secondary) and the Witness fault domain is implied. Standard vSAN
configurations can be comprised of up to 32 Fault Domains.
Support statements specific to using vSAN 2 Node Cluster implementations:
SMP-FT, the new Fault Tolerant VM mechanism introduced in vSphere 6.0:

Is supported when using 2 Node configurations in the same physical
location. SMP-FT requires appropriate vSphere licensing. *The vSAN
Witness Appliance managing a 2 Node cluster may not reside on the
cluster it is providing quorum for. SMP-FT is not a feature that
removes this restriction.
11

vSAN 2 Node Guide
3. New Concepts in vSAN - 2 Node

Clusters
A common question is how 2 Node vSAN differ from Fault Domains, which is a
vSAN feature that was introduced with vSAN version 6.0.
12

vSAN 2 Node Guide
3. 1 The Witness Host
vSAN Witness Host Purpose
The vSAN Witness Host is a dedicated ESXi host, which may be a physical ESXi
host or vSAN Witness Appliance, whose purpose is to host the Witness
component of vSAN objects.
The vSAN Witness must have connectivity to both vSAN nodes to join the
cluster. In steady state operations, the master node is the host that resides in
the “Preferred site”; the backup node resides in the “Secondary site”. Unless
the vSAN Witness Host connects to both the master and the backup nodes, it
will not join the vSAN cluster.
vSAN Witness Host Connectivity
The vSAN Witness Host must be managed by the same vCenter Server
managing the vSAN Cluster. There must be connectivity between vCenter
Server and the vSAN Witness Host in the same fashion as vCenter controlling
other vSphere hosts.
The vSAN Witness Host must also have connectivity between the to each vSAN
node.
In vSAN 6.5 VMware publicly introduced the feature known as Witness Traffic
Separation, where a separately tagged VMkernel interface may be used for
connectivity to the vSAN Witness Host, rather than requiring direct
connectivity to the vSAN Data Network. This capability can only be enabled
from the command line, and is supported when used with 2 Node Direct
Connect configurations.
These will be covered more thoroughly in a later section.
Updating the vSAN Witness Appliance
The vSAN Witness Appliance can easily be maintained/patched using vSphere

Update Manager in the same fashion as traditional vSphere hosts. It is not
required to deploy a new vSAN Witness Appliance when updating or patching
vSAN hosts. Normal upgrade mechanisms are supported on the vSAN Witness
Appliance.
13

vSAN 2 Node Guide
3. 2 Read Locality in vSAN 2 Node Clusters
In traditional vSAN clusters, a virtual machine’s read operations are distributed

across all replica copies of the data in the cluster. In the case of a policy
setting of NumberOfFailuresToTolerate =1, which results in two copies of the
data, 50% of the reads will come from replica1 and 50% will come from
replica2. In the case of a policy setting of NumberOfFailuresToTolerate =2 in
non-stretched vSAN clusters, results in three copies of the data, 33% of the
reads will come from replica1, 33% of the reads will come from replica2 and
33% will come from replica3.
In a vSAN 2 Node Clusters, 100% of reads, occur in the site the VM resides on.
This aligns with the behavior of vSAN Stretched Clusters. Read locality
overrides the NumberOfFailuresToTolerate=1 policy’s behavior to distribute
reads across the components.
This is not significant to consider in All-Flash configurations, but should be

considered in Hybrid vSAN configurations. To understand why, it is important
to know how write and read operations behave in 2 Node Virtual SAN
configurations.
Writes Are Synchronous

In vSAN write operations are always synchronous. The image of a Hybrid vSAN
cluster, shows that writes are being written to Node 1 and Node 2, with
Metadata updates being written to the vSAN Witness Host. This is due to a
Number of Failures to Tolerate policy of 1 .
Notice the blue triangle in the cache devices? That’s 30% of the cache device
being allocated as the write buffer. The other 70% of the cache device is green
, demonstrating the read cache. It is important to note that All-Flash vSAN
14

vSAN 2 Node Guide
clusters do not use the cache devices for read caching.
Writes go to the write buffer on both Nodes 1 and 2. This is always the case
because writes occur on both nodes simultaneously.
Default Stretched Cluster / 2 Node Read Behavior
By default, reads are only serviced by the host that the VM is running on.
The image shows a typical read operation. The virtual machine is running on
Node 1 and all reads are serviced by the cache device of Node 1’s disk group.
By default, reads do not traverse to the other node. This behavior is default in 2
Node configurations, as they are mechanically similar to Stretched Cluster
configurations. This behavior is preferred when the latency between sites is at
the upper end of the supported boundary of 5ms round-trip-time (RTT).
This is advantageous in situations where the two sides of a Stretched Cluster

are connected by an inter-site link, because it removes additional overhead of
reads traversing the inter-site link.
2 Node Reads after vMotion
Read operations after a vMotion, are going to behave differently.
15

vSAN 2 Node Guide
Because the cache has not been warmed on the host the virtual machine is
now running on, reads will have to occur on the capacity drives as the cache is
warmed.
The image shows only part of the cache device as green, indicating that as
reads occur, they are cached in the read cache of the disk group.
The process of invoking a vMotion could be from various DRS events, such as
putting a host in maintenance mode or balancing workloads. The default
Stretched Cluster recommendation, is to keep virtual machines on one site or
the other, unless there is a failure event.
2 Node Reads after a Disk Failure
Read operations after a disk failure, are going to behave similarly to those of a
vMotion. A single disk in the disk group has failed in the image on the right.
Reads are going to come from Node 2, and the cache device on Node 2 is
going to start caching content from the virtual machine’s disk.
Since only a capacity device failed, and there are others still contributing to the
capacity, reads will also traverse the network, as data is rewritten to one of the
surviving capacity devices on Node 1, if there is sufficient capacity.
Once data has been reprotected on Node 1, the cache will have to rewarm on
Node 1 again.
16

vSAN 2 Node Guide
2 Node Reads after a Cache Device/Disk Group Failure
Read operations after a disk group failure, are also going to behave like that of
a disk failure.
In configurations where the host with a failed disk group has an additional disk
group, rebuilds will occur on the surviving disk group provided there is
capacity.
In hosts with a single disk group, rebuilds will not occur, as the disk group is
unavailable.
What can be done to prevent the need for rewarming the read cache?
There is an advanced setting which will force reads to always be serviced by

both hosts.
The vSAN “DOMOwnerForceWarmCache” setting can be configured to force

reads on both Node 1 and Node 2 in a 2 Node configuration.
Forcing the cache to be read across Stretched Cluster sites is bad, because
additional read latency is introduced.
vSAN 2 Node configurations are typically in a single location, directly
17

vSAN 2 Node Guide
connected or connected to the same switch, just as a traditional vSAN

deployment.
When DOMOwnerForceWarmCache setting is True (1), it will force reads

across all mirrors to most effectively use cache space. This means reads would
occur across both nodes in a 2 Node config.
When it is False (0), site locality is in effect and reads are only occurring on the
site the VM resides on.
In short, DOM Owner Force Warm Cache:
Doesn’t apply to traditional vSAN clusters

Stretched Cluster configs with acceptable latency & site locality enabled –
Default 0 (False)
2 Node (typically low very low latency) – Modify 1 (True)
Not only does this help in the event of a virtual machine moving across hosts,
which would require cache to be rewarmed, but it also allows reads to occur
across both mirrors, distributing the load more evenly across both hosts.
To check the status, run the following command on each ESXi host:
esxcfg-advcfg -g /VSAN/DOMOwnerForceWarmCache
To set it for 2 Node clusters:

esxcfg-advcfg -s 1 /VSAN/DOMOwnerForceWarmCache
A PowerCLI script is available to connect to each host in a vSphere cluster, and

set the value to 1, which forces warm cache across both nodes.
That script can be found here: https://code.vmware.com/samples/1662
18

vSAN 2 Node Guide
Forcing reads to be read across both nodes in cases where the Number of
Failures to Tolerate policy is 1 can prevent having to rewarm the disk group
cache in cases of vMotions, host maintenance, or device failures.
Client Cache
VMware vSAN 6.2 introduced Client Cache, a mechanism that allocates 0.4%
of host memory, up to 1GB, as an additional read cache tier. Virtual machines
leverage the Client Cache of the host they are running on. Client Cache is not
associated with Stretched Cluster read locality, and runs independently.
3. 3 Witness Traffic Separation (WTS)
By default, when using vSAN 2 Node configurations, the Witness VMkernel

interface tagged for vSAN traffic must have connectivity with each vSAN data
node's VMkernel interface tagged with vSAN traffic.
In vSAN 6.5, an alternate VMkernel interface can be designated to carry traffic

destined for the Witness rather than the vSAN tagged VMkernel interface. This
feature allows for more flexible network configurations by allowing for
separate networks for node-to-node and node-to-witness traffic.
2 Node Direct Connect

This Witness Traffic Separation provides the ability to directly connect vSAN
data nodes in a 2 Node configuration. Traffic destined for the Witness host can
be tagged on an alternative interface from the directly connected vSAN tagged
interface.
19

vSAN 2 Node Guide
In the illustration above, the configuration is as follows:
Host 1
vmk0 - Tagged for Management Traffic
vmk1 - Tagged for Witness Traffic - This must* be done using esxcli
vsan network ip add -i vmk1 -T=witness
vmk2 - Tagged for vSAN Traffic
vmk3 - Tagged for vMotion Traffic
Host 2
vmk0 - Tagged for Management Traffic
vmk1 - Tagged for Witness Traffic - This must* be done using esxcli
vsan network ip add -i vmk1 -T=witness
20

vSAN 2 Node Guide
vmk2 - Tagged for vSAN Traffic

vmk3 - Tagged for vMotion Traffic
vSAN Witness Appliance
vmk0 - Tagged for Management Traffic***
vmk1 - Tagged for vSAN Traffic****
*Enabling Witness Traffic is not available from the vSphere Web Client.
**Any VMkernel port, not used for vSAN Traffic, can be used for Witness
Traffic. In a more simplistic configuration, the Management VMkernel interface
(vmk0) could be tagged for Witness Traffic. The VMkernel port used, will be
required to have connectivity to the vSAN Traffic tagged interface on the vSAN
Witness Appliance.
***The vmk0 VMkernel Interface, which is used for Management traffic may
also be used for vSAN Traffic. In this situation, vSAN Traffic must be
unchecked from vmk1.
****The vmk1 VMkernel interface must not have an address that is on the same
subnet as vmk0. Because vSAN uses the default tcp/ip stack, in cases where
vmk0 and vmk1 are on the same subnet, traffic will use vmk0 rather than
vmk1. This is detailed in KB 2010877 . Vmk1 should be configured with an
address on a different subnet than vmk0.
The ability to connect 2 Nodes directly removes the requirement for a high
speed switch. This design can be significantly more cost effective when
deploying tens or hundreds of 2 Node clusters.
vSAN 2 Node Direct Connect was announced with vSAN 6.5, and is available
with vSAN 6.6, 6.5, and 6.2*. *6.2 using vSphere 6.0 Patch 3 or higher without
an RPQ.
21

vSAN 2 Node Guide
4. Requirements
List of requirements for implementing vSAN 2 Node Clusters
22

vSAN 2 Node Guide
4. 1 VMware vCenter Server
A vSAN 2 NodeCluster configuration can be created and managed by a single

instance of VMware vCenter Server.
4. 2 A Witness Host
In a vSAN 2 NodeCluster, the Witness components are only ever placed on the
vSAN Witness Host. Either a physical ESXi host, or the vSAN Witness Appliance
provided by VMware, can be used as the vSAN Witness Host.
If a vSAN Witness Appliance is used for the Witness host, it will not consume
any of the customer’s vSphere licenses. A physical ESXi host that is used as a
vSAN Witness Host will need to be licensed accordingly, as this can still be
used to provision virtual machines should a customer choose to do so.
It is important that the vSAN Witness Host is not added to the vSAN cluster.
The vSAN Witness Host is selected during the creation of a vSAN 2 Node
Cluster.
The vSAN Witness appliance will have a unique identifier in the vSphere UI to
assist with identifying that a host is in fact a vSAN Witness Appliance. It is
shown as a “blue” host, as highlighted below:
Note: This is only visible when the vSAN Witness Appliance is deployed. If a
physical host is used as a vSAN Witness Host, then it does not change its
appearance in the web client. A dedicated vSAN Witness Host is required for
23

vSAN 2 Node Guide
each 2 Node Cluster.
4. 3 Networking and Latency Requirements
When vSAN is deployed in a 2 Node Cluste there are certain networking

requirements that must be adhered to.
Layer 2 and Layer 3 Support
Both Layer 2 (same subnet) and Layer 3 (routed) configurations are used in a
recommended vSAN 2 Node Cluster deployment.
VMware recommends that vSAN communication between the vSAN nodes

be over L2.
VMware recommends that vSAN communication between the vSAN nodes
and the Witness Host is
Layer 2 for configurations with the Witness Host in the same site
Layer 3 for configurations with the Witness Host in an alternate site.
vSAN traffic between nodes is multicast for versions 6.5 and previous, while
unicast is used for version 6.6 or later . Witness traffic between each node in a
2 Node cluster and the Witness Host is unicast .
vSAN Node to Witness Network Latency
This refers to the communication between vSAN hosts and the Witness
Host/Site.
In typical 2 Node configurations, such as Remote Office/Branch Office

deployments, this latency or RTT is supported up to 500msec (250msec one-
way).
The latency to the Witness Host is dependent on the number of objects in the
cluster.
vSAN Node to Witness Network Bandwidth
Bandwidth between vSAN Nodes hosting VM objects and the Witness Host are
dependent on the number of objects residing on vSAN. It is important to size
data site to witness bandwidth appropriately for both availability and growth. A
standard rule of thumb is 2Mbps for every 1000 components on vSAN.
Because vSAN nodes have a maximum number of 9000 components per host,
24

vSAN 2 Node Guide
the maximum bandwidth requirement from a 2 Node cluster to the Witness

Host supporting it, is 18Mbps.
Please refer to the Design Considerations section of this guide for further
details on how to determine bandwidth requirements.
Inter-Site MTU Consistency
Knowledge Base Article 2141733 details a situation where data nodes have
an MTU of 9000 (Jumbo Frames) and the vSAN Witness Host has an MTU of
1500. The vSAN Health Check looks for a uniform MTU size across all
VMkernel interfaces that are tagged for traffic related to vSAN, and reports
any inconsistencies. It is important to maintain a consistent MTU size across all
vSAN VMkernel interfaces on data nodes and the vSAN Witness Host in a
vSAN 2 Node cluster to prevent traffic fragmentation.
As KB 2141733 indicates, the corrective actions are either to reduce the MTU
from 9000 on the data node VMkernel interfaces, or increase the MTU value
on the vSAN Witness Host's VMkernel interface that is tagged for vSAN Traffic.
Either of these are acceptable corrective actions.
The placement of the vSAN Witness Host will likely be the deciding factor in
which configuration will be used. Network capability, control, and cost to/from
the vSAN Witness Host as well as overall performance characteristics on data
nodes are items to consider when making this design decision.
In situations where the vSAN Witness Host VMkernel interface tagged for
vSAN traffic is not configured to use Jumbo Frames (or cannot due to
underlying infrastructure), VMware recommends that all vSAN VMkernel
interfaces use the default MTU of 1500.
As a reminder, there is no requirement to use Jumbo Frames with VMkernel

interfaces used for vSAN.
Multiple vSAN Witness Hosts sharing the same VLAN
For customers who have implemented multiple 2 Node vSAN deployments, a

common question is whether the Witness traffic from each of the remote sites
requires its own VLAN.
The answer is no.
Multiple 2 Node vSAN deployments can send their witness traffic on the same
25

vSAN 2 Node Guide
shared VLAN.
26

vSAN 2 Node Guide
5. Configuration Minimums and

Maximums
The maximum number of virtual machines per ESXi host is unaffected by the
vSAN 2 Node configuration.
27

vSAN 2 Node Guide
5. 1 Virtual Machines Per Host
The maximum number of virtual machines per ESXi host is unaffected by the
vSAN 2 Node Cluster configuration. The maximum is the same as normal vSAN
deployments.
VMware recommends that customers should run their hosts at 50% of

maximum number of virtual machines supported in a standard vSAN cluster to
accommodate a full site failure.
In the event of node failure the virtual machines on the failed node can be
restarted on the surviving node.
5. 2 Witness Host
There is a maximum of 1 vSAN Witness Host per vSAN 2 Node Cluster.
The vSAN Witness Host requirements are discussed in the Design

Considerations section of this guide.
VMware provides a fully supported vSAN Witness Appliance, in the Open

Virtual Appliance (OVA) format. This is for customers who do not wish to
dedicate a physical ESXi host as the witness. This OVA is essentially a pre-
licensed ESXi host running in a virtual machine, and can be deployed on a
physical ESXi host at the 3rd site or host.
5. 3 vSAN Storage Policies
Number of Failures To Tolerate (FTT) - Pre-vSAN 6.6

Primary Number of Failures To Tolerate (PFTT) - vSAN 6.6 and forward
The FTT/PFTT policy setting, has a maximum of 1 for objects.
In Pre-vSAN 6.6 2 Node Clusters FTT may not be greater than 1. In vSAN 6.6 or
higher 2 Node Clusters, PFTT may not be greater than 1. This is because 2
Node Clusters are comprised of 3 Fault Domains.
Failure Tolerance Method (FTM)
28

vSAN 2 Node Guide
Failure Tolerance Method rules provide object protection with RAID-1

(Mirroring). Mirroring is the only FTM rule that can be satisfied, due to the 3
fault domains present. This means that data is Mirrored across both sites.
Affinity
Affinity rules are used when the PFTT rule value is 0. This rule has 2 values,
Preferred or Secondary. This determines which site an Affinity based vmdk
would reside on.
Other Policy Rules
Other policy settings are not impacted by deploying vSAN in a 2 Node Cluster
configuration and can be used as per a non-stretched vSAN cluster.
5. 4 Fault Domains
Fault Domains play an important role in vSAN 2 Node Clusters.
Similar to the Number Of Failures To Tolerate (FTT) policy setting discussed

previously, the maximum number of Fault Domains in a vSAN 2 Node Cluster is
2.
The "Preferred" Fault Domain is selected in the vSphere UI, and assigns the
"vSAN Master" node role to the host in that Fault Domain. The alternate Fault
Domain assigns the "vSAN Backup" node role to the host in that Fault Domain.
These Fault Domains are typically named "Preferred" and "Secondary" but are
not required to be. One of these Fault Domains must be designed with the
"Preferred" setting. The vSAN Witness Host provides a 3rd implied fault
domain.
29

vSAN 2 Node Guide
6. Design Considerations
The witness host must be capable of running the same version of ESXi as vSAN
data nodes.
30

vSAN 2 Node Guide
6. 1 Cluster Compute Resource Utilization
For full availability, VMware recommends that customers should be running at

50% of resource consumption across the vSAN 2 Node Cluster. In the event of
a node failure, all of the virtual machines could be run on the surviving node.
VMware understands that some customers will want to run levels of resource
utilization higher than 50%. While it is possible to run at higher utilization in
each site, customers should understand that in the event of failure, not all
virtual machines will be restarted on the surviving node.
vSA FT S
Capacity Capacity Capacity
N Protec T/P FT F
Required in Required in Require
Versi tion FT M T
Preferred Site Secondary Site ment
on T T
Across Mir
6.1- N
Sites 1 ror 100% 100% 200%
6.6 A
Only ing
31

vSAN 2 Node Guide
6. 2 Network Design Considerations
2 Node vSAN Network Design

Considerations
Sites
A vSAN 2 Node Cluster typically runs in a single site, with the vSAN Witness Host residing in an alternate location
or on an alternate host in the same site.
vSAN Data nodes
Preferred Node - Specified to be the primary owner of vSAN objects.

This is an important designation specifically in cases of connectivity
disruptions.
Secondary Node - Backup owner of vSAN objects.
Witness Site - Contains vSAN Witness host - Could be in a different site, or the
small site as the 2 Node cluster.
Maintains Witness Component data from Preferred/Secondary sites

when applicable
Connectivity and Network Types
Preferred Host Secondary Host Witness Site
Managemen
Layer 2 or 3
t Network to Layer 2 or 3 Layer 2 or 3
vCenter
Managemen
t Network to
Typically Layer 2 Typically Layer 2
Alternate
Host
32

vSAN 2 Node Guide
No requirement
for VM Network.
Running VMs on
the vSAN Witness
Appliance is not
VM Network Typically Layer 2 Typically Layer 2
supported.
Running VMs on
a Physical
Witness Host is
supported.
Typically Layer 2 Typically Layer 2 There is no

vMotion is not vMotion is not requirement for
vMotion
required between required between vMotion
Network
this Data site & the this Data site & the networking to the
Witness Site Witness Site Witness site.
Layer 2
Layer 2
Multicast: vSAN
vSAN Multicast: vSAN
6.1-6.5
Network 6.1-6.5
Unicast: vSAN
Unicast: vSAN 6.6+
6.6+
Witness
Traffic Using To the vSAN To the vSAN
To the Data Site:
WTS & vSAN Witness Site: Witness Site: Layer
Layer 3
Witness Layer 3 3
Offsite
Witness
Traffic Using To the vSAN To the vSAN To the vSAN
WTS & vSAN Witness: Layer 2 Witness: Layer 2 or Nodes: Layer 2 or
Witness or 3 3 3
Locally
33

vSAN 2 Node Guide
Port Requirements
VMware vSAN requires these ports to be open, both inbound and outbound:
Po Prot Connectivity
rt ocol To/From
12
vSAN 34
Clustering 5, UDP vSAN Hosts
Service 23
45
vSAN 22
TCP vSAN Hosts
Transport 33
vSAN VASA
80 vSAN Hosts
Vendor TCP
80 and vCenter
Provider
vSAN Hosts
vSAN Unicast 12
and vSAN
Agent (to 32 UDP
Witness
Witness Host) 1
Appliance
34

vSAN 2 Node Guide
TCP/IP Stacks, Gateways, and Routing
TCP/IP Stacks
At this time, the vSAN traffic does not have its own dedicated TCP/IP stack.
Custom TCP/IP stacks are also not applicable for vSAN traffic.
Default Gateway on ESXi Hosts
ESXi hosts come with a default TCP/IP stack. As a result, hosts have a single default gateway. This default
gateway is associated with the Management VMkernel interface (typically vmk0).
It is a best practice to implement separate storage networking, in this case

vSAN networking, on an alternate VMkernel interface, with alternate
addressing.
vSAN networking uses the same TCP/IP stack as the Management VMkernel interface. If the vSAN Data Network
were to attempt to use a Layer 3 network, static routing would be need to be added for the VMkernel interfaces
tagged for vSAN Traffic.
The addition of Witness Traffic separation allows vSAN interfaces to be directly

connected across hosts with communication to the vSAN Witness handled by
an alternate interface that has traffic tagged as "Witness" traffic.
Communication with the vSAN Witness via an alternate VMkernel interface can
be performed by using a separate VMkernel interface, or the Management
interface, as long as the interface used has traffic tagged as "Witness Traffic".
It is important to remember that vSAN uses the same TCP/IP stack as the
Management interface, and therefore if an alternate interface is used, static
routing must be in place for the vSAN node to properly communicate with the
vSAN Witness Host
The Witness Host on the Witness Site will require a static route added so that
requests to reach the 2 Node Cluster are routed out the WitnessPg VMkernel
interface
Static routes are added via the esxcli network ip route or esxcfg-route commands. Refer to the appropriate
vSphere Command Line Guide for more information.
Caution when implementing Static Routes: Using static routes requires

administrator intervention. Any new ESXi hosts that are added to the cluster at
35

vSAN 2 Node Guide
either site 1 or site 2 needed to have static routes manually added before they
can successfully communicate to the witness, and the other data site. Any
replacement of the witness host will also require the static routes to be
updated to facilitate communication to the data sites.
Sample configuration using Witness Traffic Separation using a dedicated VMkernel

Interface on each ESXi Host
In the illustration below, each vSAN Host's vmk1 VMkernel interface is tagged
with "witness" traffic. Each vSAN Host must have a static route configured for
vmk1 able to properly access vmk1 on the vSAN Witness Host, which is tagged
with "vsan" traffic. The vmk1 interface on the vSAN Witness Host must have a
static route configured to be able to properly communicate with vmk1 on each
vSAN Host. This is a supported configuration.
Sample configuration using Witness Traffic Separation using only the Management
VMkernel Interface on each ESXi Host
In the illustration below, each vSAN Host's vmk0 VMkernel interface is tagged
with both "Management" and "witness" traffic. The vSAN Witness Host has the
vmk0 VMkernel interface tagged with both "Management" and "vsan" traffic.
This is also a supported configuration.
36

vSAN 2 Node Guide
Sample misconfiguration with vSAN Witness vmk0/vmk1 on the same subnet
In this illustration, the vmk0 and vmk1 VMkernel interfaces are on the same
network. Vmk1 is tagged for "vsan" traffic and vmk0 is tagged for
"Managment" traffic. Because vSAN uses the default TCP/IP stack, vSAN
traffic does not properly flow form vmk1, which is tagged for "vsan" traffic, but
rather from vmk0, which is NOT tagged for "vsan" traffic. This causes an error
with the vSAN Health Check indicating that the vSAN Witness Host does not
have proper tagging.
37

vSAN 2 Node Guide
The issue is not unique to vSAN, but rather occurs when using any VMkernel
interfaces using the default TCP/IP stack. KB Article 2010877 addresses this
condition.
Though not represented here, this is also true for the vSAN Data network.
6. 3 The Role of vSAN Heartbeats
As mentioned previously, when vSAN is deployed in a 2 Node Cluster

configuration, the vSAN Master node is found in the Fault Domain designated
as Preferred and the vSAN backup node is found in the alternate Fault Domain.
The vSAN Master node and the vSAN Backup node send heartbeats every
second. If communication is lost for 5 consecutive heartbeats (5 seconds)
between the master and the backup due to an issue with the backup node, the
master chooses a different ESXi host as a backup on the remote site. This is
repeated until all hosts on the remote site are checked. If there is a complete
site failure, the master selects a backup node from the “Preferred” site.
A similar scenario arises when the master has a failure.
When a node rejoins an empty site after a complete site failure, either the
master (in the case of the node joining the primary site) or the backup (in the
case where the node is joining the secondary site) will migrate to that site.
38

vSAN 2 Node Guide
If communication is lost for 5 consecutive heartbeats (5 seconds) between the

master and the Witness, the Witness is deemed to have failed. If the Witness
has suffered a permanent failure, a new Witness host can be configured and
added to the cluster.
6. 4 Bandwidth Calculation
As stated in the requirements section, the bandwidth requirement between the

two main sites is dependent on workload and in particular the number of write
operations per ESXi host. Other factors such as read locality not in operation
(where the virtual machine resides on one site but reads data from the other
site) and rebuild traffic, may also need to be factored in.
Requirements Between 2 Node vSAN and the Witness Site
Hosts designated as a vSAN Witness Host do not maintain any VM data, but
rather only component metadata, the requirements are much smaller than that
of the backend vSAN data network.
Virtual Machines on vSAN are comprised of multiple objects, which can potentially be split into multiple
components, depending on factors like policy and size. The number of components on vSAN have a direct
impact on the bandwidth requirement between the data sites and the witness.
The required bandwidth between the vSAN Witness Host and each node is
equal to ~1138 B x Number of Components /5s
1138 B x NumComp / 5 seconds
The 1138 B value comes from operations that occur when the Preferred Site
goes offline, and the Secondary Site takes ownership of all of the components.
When the Preferred Node goes offline, the Secondary Node becomes the
Master Node. The vSAN Witness Host sends updates to the new Master,
followed by the new Master replying to the vSAN Witness Host as ownership is
updated.
The 1138 B requirement for each component comes from a combination of a

payload from the vSAN Witness Host to the backup agent, followed by
metadata indicating that the Preferred Site has failed.
In the event of a Preferred Node failure, the link must be large enough to allow
39

vSAN 2 Node Guide
for the cluster ownership to change, as well ownership of all of the

components within 5 seconds.
Witness to Site Examples
Workload 1
With a VM being comprised of
3 objects {VM namespace, vmdk (under 255GB), and vmSwap}

Failure to Tolerate of 1 (FTT=1)
Stripe Width of 1
Approximately 25 VMs with the above configuration would require the vSAN
Witness Host to contain 75 components.
To successfully satisfy the Witness bandwidth requirements for a total of 75

components on vSAN, the following calculation can be used:
Converting Bytes (B) to Bits (b), multiply by 8

B = 1138 B * 8 * 75 / 5s = 136,560 Bits per second = 136.56 Kb
ps
VMware recommends adding a 10% safety margin and round up.
B + 10% = 136.56 Kbps + 13.656 Kbps = 0.150 Kbps
With the 10% buffer included, a rule of thumb can be stated that for every
1,000 components, 2 Mbps is appropriate.
Workload 2
With a VM being comprised of
3 objects {VM namespace, vmdk (under 255GB), and vmSwap}

Failure to Tolerate of 1 (FTT=1)
Stripe Width of 2
2 Snapshots per VM
Approximately 25 VMs with the above configuration would require up to 250

components to be stored on the vSAN Witness Host.
To successfully satisfy the Witness bandwidth requirements for 250

components on vSAN, the resulting calculation is:
40

vSAN 2 Node Guide
B = 1138 B * 8 * 250 / 5s = 455,200 Bits per second = 455.2 Kb

ps
B + 10% = 455.2 Kbps + 45.52 Kbps = 500.7 Kbps
Using the general equation of 2Mbps for every 1,000 components,

(NumComp/1000) X 2Mbps, it can be seen that 250 components does in fact
require 0.5 Mbps.
6. 5 Using the vSAN Witness Appliance as a vSAN Witness Host
VMware vSAN 2 Node Clusters require a vSAN Witness Host.
This section will address using the vSAN Witness Appliance as a vSAN Witness
Host. The vSAN Witness Appliance is available in an OVA (Open Virtual
Appliance) format from VMware. The vSAN Witness Appliance does need to
reside on a physical ESXi host.
Minimal Requirements to Host the vSAN Witness Appliance
The vSAN Witness Appliance must on an ESXi 5.5 or greater VMware host.
The ESXi 5.5 or greater host must meet the minimum requirements for the
vSAN Witness Host for the version of vSAN 2 Node that the vSAN Witness
Appliance will support.
Networking must be in place that allows for the vSAN Witness Appliance
to properly communicate with the vSAN 2 Node Cluster.
Where can the vSAN Witness Appliance run?
In addition to the minimal requirements for hosting the vSAN Witness

Appliance, several supported infrastructure choices are available:
On a vSphere environment backed with any supported storage (vmfs datastore, NFS datastore, vSAN
Cluster)
On vCloud Air/OVH backed by supported storage
Any vCloud Air Network partner hosted solution
On a vSphere Hypervisor (free) installation using any supported storage (vmfs datastore or NFS datastore)
Support Statements specific to placement of the vSAN Witness Appliance on a

vSAN cluster:
41

vSAN 2 Node Guide
The vSAN Witness Appliance is supported running on top of another non-Stretched vSAN cluster.
The vSAN Witness Appliance is supported on a Stretched Cluster vSAN for another vSAN 2 Node cluster.
vSAN 2-node cluster hosting witness for another vSAN 2-node cluster witness, and vice versa, is not
recommended and requires an RPQ.
The next question is how to implement such a configuration, especially if the

witness host is on a public cloud? How can the interfaces on the hosts in the
data sites, which communicate to each other over the vSAN network,
communicate to the witness host?
CPU Requirements
The vSAN Witness Appliance is a virtual machine that has special vSphere
installation that is used to provide quorum/tiebreaker services for a 2 Node
vSAN Cluster. The underlying CPU architecture must be supported by the
vSphere installation inside the vSAN Witness Appliance.
As an example, a vSphere/vSAN 6.7 2 Node vSAN Cluster will require a vSAN

Witness Appliance that is running vSphere 6.7. The ESXi host that the vSAN
Witness Appliance runs on top of, could run any version of vSphere 5.5 or
higher. With vSphere/vSAN 6.7 having different CPU requirements, the ESXi
host that the vSAN Witness Appliance runs on, must support the CPU
requirements of vSphere 6.7, regardless of the version of vSphere the ESXi
host is running.
In cases where a vSAN Witness Appliance is deployed to an ESXi host that

does not meet the CPU requirements, it may be deployed, but not powered on.
The vSAN Witness Appliance, patched like a normal vSphere Host, cannot be
upgraded to vSphere 6.7 if the underlying CPU does not support vSphere 6.7.
This consideration is important to take into account when upgrading 2 Node

vSAN Clusters. The vSAN Witness is a critical part of the patching and upgrade
process. It is strenuously recommended by VMware to keep the vSAN Witness
version consistent with the vSphere version of the 2 Node vSAN Cluster.
Networking
The vSAN Witness Appliance contains two network adapters that are
connected to separate vSphere Standard Switches (VSS).
The vSAN Witness Appliance Management VMkernel is attached to one VSS,

and the WitnessPG is attached to the other VSS. The Management VMkernel
(vmk0) is used to communicate with the vCenter Server for normal managment
42

vSAN 2 Node Guide
of the vSAN Witness Appliance. The WitnessPG VMkernel interface (vmk1) is

used to communicate with the vSAN Data Nodes. This is the recommended
configuration.
Alternatively, the Management VMkernel (vmk0) interface could be tagged to

include vSAN traffic as well as Management traffic. In this case, vmk0 would
require connectivity to both vCenter Server and the vSAN Witness Network.
A Note About Promiscuous Mode
In many ESXi in a VM environments, there is a recommendation to enable

promiscuous mode to allow all Ethernet frames to pass to all VMs that are
attached to the port group, even if it is not intended for that particular VM. The
reason promiscuous mode is enabled in many ESXi in a VM environments is to
prevent a virtual switch from dropping packets for (nested) vmnics that it does
not know about on the ESXi in a VM hosts. ESXi in a VM deployments are not
supported by VMware other than the vSAN Witness Appliance.
The MAC addresses of the vSAN Witness Appliance's VMkernel interfaces

vmk0 & vmk1 are configured to match the MAC addresses of the ESXi host's
physical NICs, vmnic0 and vmnic1. Because of this, packets destined for either
the Management VMkernel interface (vmk0) or the WitnessPG VMkernel
interface, are not dropped.
43

vSAN 2 Node Guide
Because of this, promiscuous mode is not required when using a vSAN Witness
Appliance.
Since the vSAN Witness Appliance will be deployed on a physical ESXi host the
underlying physical ESXi host will need to have a minimum of one VM network
preconfigured. This VM network will need to reach both the management
network and the vSAN network shared by the ESXi hosts on the data sites. An
alternative option that might be simpler to implement is to have two
preconfigured VM networks on the underlying physical ESXi host, one for the
management network and one for the vSAN network. When the virtual ESXi
witness is deployed on this physical ESXi host, the network will need to be
attached/configured accordingly.
44

vSAN 2 Node Guide
6. 6 Using a Physical Host as a vSAN Witness Host
Physical ESXi host used as a vSAN Witness Host:
When using a physical ESXi host as a vSAN Witness Host, the VMkernel
interface that will be tagged for "vsan" traffic must have connectivity to the
vSAN Data Node VMkernel interace that is tagged for "witness" traffic. This
could be over Layer 3, or even over Layer 2 if desired.
If using a physical host as the vSAN Witness Host there are some requirements
to consider.
Requirement
s
If using a physical host as a vSAN Witness Host, it must be

licensed with a valid vSphere license. This does not require
Licensing
the same licensed edition as the vSAN Cluster it is
supporting.
If using a physical host as a vSAN Witness Host, it must be

vSphere Build running the same build of vSphere as the 2 Node Cluster
that it is participating with.
45

vSAN 2 Node Guide
The minimum specifications required for ESXi meet the

minimum requirements for use as a vSAN Witness Host.
Compute & Minimum requirements for vSphere are dependent on the
Memory build of vSphere, and can be found in the documentation
section for each edition in VMware Documentation:
https://www.vmware.com/support/pubs/
Storage requirements do not change for a physical host

being used as a vSAN Witness Host in comparison to the
vSAN Witness Appliance. An ESXi boot device, a cache
device, and one or more capacity devices are still required.
Required
1st device - vSphere Boot Device - Normal vSphere

Storage
Requirements
2nd device - vSAN Cache Device - No requirement for
Flash/SSD, but it must be tagged as Flash/SSD in ESXi
to be used as though it were a Flash/SSD device. This
must be at least 10GB in size.
3rd device - Can be up to 350GB and will support
metadata for up to 21,000 components on a vSAN
Cluster
If using a physical host as a vSAN Witness Host, it may run

other workloads. Because the physical vSAN Witness Host is
external to the vSAN Cluster it is contributing to, those
workloads will not be part of the vSAN Cluster. The vSAN
Disk Group, and the disks it includes, may not be used for
those workloads.
Other
Workloads
*Important consideration: Multiple vSAN Witness
Appliances can run on a single physical host. Using vSAN
Witness Appliances is typically more cost effective than
dedicating physical hosts for the purpose of meeting the
vSAN Witness Host requirement.
46

vSAN 2 Node Guide
6. 7 vSAN Witness Host Networking Examples
In both options, either a physical ESXi host or vSAN Witness

Appliance may be used as vSAN Witness Host.
Option 1: 2 Node Configuration for Remote Office/Branch Office

Deployment using Witness Traffic Separation with the vSAN Witness
in a central datacenter
In the use case of Remote Office/Branch Office (ROBO) deployments, it is

common to have 2 Node configurations at one or more remote offices. This
deployment model can be very cost competitive when a running a limited
number of virtual machines no longer require 3 nodes for vSAN.
Management traffic for the data nodes is typically automatically routed to the
vCenter server at the central data center. Because they reside in the same
physical location, vSAN traffic networking between data nodes is consistent
with that of a traditional vSAN cluster.
These vSAN Nodes are still required to communicate with the vSAN Witness
Host residing in the central datacenter. Witness Traffic Separation, allows for a
separate VMkernel interface for "witness" traffic. This VMkernel must be able
to communicate with the vSAN Witness Host.
47

vSAN 2 Node Guide
In cases were the vSAN Witness Appliance uses the WitnessPG (vmk1) to
communicate with vSAN Data Nodes over Layer 3, static routing will be
required to ensure proper connectivity.
Adding static routes is achieved using the esxcfg-route –a command on the

ESXi hosts and witness VM.
The below illustration shows a vSAN Witness Appliance with the Managment
(vmk0) and WitnessPg (vmk1) VMkernel interfaces on different networks. This
is a typical configuration.
In the illustration above, the vSAN Witness Appliance in the central datacenter
has an Management (vmk0) IP address of 192.168.109.23. This VMkernel
interface will use the default gateway to communicate with vCenter Server.
The WitnessPG VMkernel interface (vmk1) has an IP address of
192.168.110.23.
48

vSAN 2 Node Guide
The vSAN 2 Node configuration has the Managment (vmk0) IP addresses of

192.168.1.21 and 192.168.1.22 for Host 1 and Host 2 respectively. As these
are the Management interfaces for these hosts, they will also use the default
gateway to communicate with the vCenter Server. Traffic is tagged as
"witness" for vmk1 in both Host 1 and Host 2 and are configured with IP
addresses of 192.168.15.21 and 192.168.15.22 respectively.
Because vSAN based traffic (tagged as "vsan" or "witness") use the default
gateway, static routes must be used in this configuration. A static route on
Host 1 and Host 2 for their vmk1 VMkernel interfaces to properly connect to
the WitnessPg VMkernel interface on the vSAN Witness Appliance. The
routing command on Host 1 and Host 2 would look like this:
esxcfg-route -a 192.168.110.0/24 192.168.15.1
Additionally, a static route would need to be configured on the vSAN Witness

Appliance as well:
esxcfg-route -a 192.168.15.0/24 192.168.110.1
*Note the address of x.x.x.1 is the gateway in each of these networks for this
example. This may differ from your environment.
The below illustration shows a vSAN Witness Appliance with the Managment
(vmk0) and WitnessPg (vmk1) VMkernel interfaces on the same network. This
is a not a typical configuration, but is supported if configured properly.
49

vSAN 2 Node Guide
Notice that the WitnessPg (vmk1) VMkernel interface is NOT tagged with
"vsan" traffic, but rather the Management (vmk0) VMkernel interface has both
"Management" and "vsan" traffic tagged, In cases were vmk0 and vmk1 would
happen to reside on the same network, a multi-homing issue will occur if the
WitnessPg (vmk1) continues to have "vsan" traffic tagged.
The correct configuration in this situation, would be untag "vsan" traffic from
the WitnessPg (vmk1) VMkernel interface, and tag the Management (vmk0)
interface instead. This is also a supported configuration.
*Some concerns have been raised in the past about exposing the vSAN Data
Network to the WAN in 2 Node vSAN Clusters. Witness Traffic Separation
mitigates these concerns because only vSAN Obect metadata traverses the
WAN to the vSAN Witness Host.
Option 2: 2 Node Configuration for Remote Office/Branch Office
50

vSAN 2 Node Guide
Deployment using Witness Traffic Separation with the vSAN Witness

in the same location
Some VMware customers that have deployed 2 Node vSAN in remote

office/branch office locations have decided to include a 3rd node for use as a
vSAN Witness Host, or as a location to run the vSAN Witness Appliance.
Other customers have asked "why?" A 3rd host running locally could
potentially be a lesser capable host that has enough resources to run a minimal
workload that includes the vSAN Witness Host role, possibly local backups, as
well as other resources such as networking based virtual machines.
In the below illustration, the 2 Node vSAN Cluster Hosts, Host 1 and Host 2, are
both connected to the same Management network as the physical host that is
running the vSAN Witness Appliance.
In this configuration, each vSAN Data Node's has a dedicated VMkernel
51

vSAN 2 Node Guide
interface tagged for "witness" traffic on the 192.168.15.x network. The vSAN
Witness Appliance has the WitnessPg (vmk1) VMkernel interface tagged for
"vsan" traffic, also on the 192.168.15.x network. In this configuration, static
routing is not required because Layer 2 networking is use.
This is a supported configuration.
In the below illustration, the 2 Node vSAN Cluster Hosts, Host 1 and Host 2, are
both connected to the same Management network as the physical host that is
running the vSAN Witness Appliance.
In this configuration, each vSAN Data Node's Management (vmk0) VMkernel

interface is tagged for "witness" traffic on the 192.168.1.x network. The vSAN
Witness Appliance has the Management (vmk0) VMkernel interface tagged for
"vsan" traffic, also on the 192.168.1.x network. In this configuration, static
routing is not required because Layer 2 networking is in use.
This is a supported configuration.
52

vSAN 2 Node Guide
6. 8 vSAN Witness Appliance Sizing
vSAN Witness Appliance Size
When using a vSAN Witness Appliance, the size is dependent on the

configurations and this is decided during the deployment process. vSAN
Witness Appliance deployment options are hard coded upon deployment and
there is typically no need to modify these.
Compute Requirements
The vSAN Witness Appliance, regardless of configuration, uses at least two

vCPUs.
Memory Requirements
Memory requirements are dependent on the number of components.
Storage Requirements
Cache Device Size: Each vSAN Witness Appliance deployment option has a
cache device size of 10GB. This is sufficient for each for the maximum of
45,000 components. In a typical vSAN deployment, the cache device must be a
Flash/SSD device. Because the vSAN Witness Appliance has virtual disks, the
10GB cache device is configured as a virtual SSD. There is no requirement for
this device to reside on a physical flash/SSD device. Traditional spinning drives
are sufficient.
Capacity Device Sizing: First consider that a capacity device can support up to
21,000 components. Also consider that a vSAN 2 Node Cluster can support a
maximum of 27,000 components. Each Witness Component is 16MB, as a
result, the largest capacity device that can be used for storing of Witness
Components is approaching 350GB.
vSAN Witness Appliance Deployment Sizes & Requirements Summary
Tiny - Supports up to 10 VMs/750 Witness Components -Typical for 2

Node vSAN deployments
Compute - 2 vCPUs
Memory - 8GB vRAM
ESXi Boot Disk - 12GB Virtual HDD
Cache Device - 10GB Virtual SSD
53

vSAN 2 Node Guide
Capacity Device - 15GB Virtual HDD
Normal - Supports up to 500 VMs/21,000 Witness Components -

Alternative for 2 Node vSAN deployments with a significant number of
components
Compute - 2 vCPUs
Memory - 16GB vRAM
Capacity Device - 350GB Virtual HDD
Large - Supports over 500 VMs/45,000 Witness Components -

Unnecessary for 2 Node vSAN deployments.
Compute: 2 vCPUs
Memory - 32 GB vRAM
Capacity Devices - 3x350GB Virtual HDD
8GB ESXi Boot Disk*, one 10GB SSD, three 350GB HDDs
Supports a maximum of 45,000 witness components
6. 9 vSAN Witness Host Versioning & Updating
vSAN Witness Appliance Version
A vSAN Witness Appliance is provided with each release of vSAN. Upon initial
deployment of the vSAN Witness Appliance, it is required to be the same as the
version of vSAN. The vSphere host that the vSAN Witness Appliance runs on, is
not required to be the same version.
Example 1: A new vSAN 6.6.1 deployment
Requires vSphere 6.5 Update 1 hosts

Requires vSAN 6.6.1 based vSAN Witness Appliance
Underlying host can be vSphere 5.5 or higher
Example 2: A new vSAN 6.7 deployment
Requires vSphere 6.7 hosts
54

vSAN 2 Node Guide
Requires vSAN 6.7 based vSAN Witness Appliance

Underlying host can be vSphere 5.5 or higher, but the CPU must be
supported by vSphere 6.7
This is because the vSAN Cluster is vSphere 6.7 based and the vSAN
Witness Appliance is running vSphere 6.7.
When upgrading the vSAN Cluster, upgrade the vSAN Witness Appliance in the
same fashion as upgrading vSphere . This keeps the versions aligned. Be
certain to ensure that the underlying hosts and the vCenter Server version
supports the version of vSAN being upgraded to.
Successful Example: Upgrade to vSAN 6.6 from vSAN 6.2
Upgrade vCenter to 6.5 Update 1 using the VCSA embedded update

mechanism
Upgrade vSAN hosts to vSphere 6.5 Update 1 using VMware Update
Manager
Upgrade vSAN Witness Host using VMware Update Manager.
Unsuccessful Example 1: Upgrade to vSAN 6.7 from vSAN 6.6
Do not upgrade vCenter to 6.7

Upgrade vSAN hosts to vSphere 6.7 using VMware Update Manager
Do not upgrade vSAN Witness Host
Upgrade vCenter to 6.7 using the VCSA embedded update mechanism

Do not upgrade vSAN Witness Host
Upgrade vCenter to 6.7 using the VCSA embedded update mechanism

Attempt to upgrade a vSAN Witness Host that is running a vSphere 5.5-6.5
host with CPUs that are not supported by vSphere 6.7
55

vSAN 2 Node Guide
7. Cluster Settings – vSphere HA

Certain vSphere HA behaviors have been modified especially for vSAN.
56

vSAN 2 Node Guide
7. 1 Cluster Settings – vSphere HA
Certain vSphere HA behaviors have been modified especially for vSAN. It

checks the state of the virtual machines on a per virtual machine basis.
vSphere HA can make a decision on whether a virtual machine should be failed
over based on the number of components belonging to a virtual machine that
can be accessed from a particular partition.
When vSphere HA is configured on a vSAN 2 Node Cluster, VMware

recommends the following:
vSphere HA Turn on
Host Monitoring Enabled
Host Hardware Monitoring – VM

Component Protection: “Protect
Disabled (default)
against Storage Connectivity
Loss”
Customer Preference – Disabled by

Virtual Machine Monitoring
default
Admission Control Set to 50%
Host Isolation Response Power off and restart VMs
“Use datastores only from the specified

list”, but do not select any datastores
Datastore Heartbeats
from the list. This disables Datastore
Heartbeats
Advanced Settings:
das.usedefaultisolationaddress False
das.isolationaddress0 IP address on vSAN network
57

vSAN 2 Node Guide
Note that a 2-node Direct Connect configuration is a special case. In this

situation, it is impossible to configure a valid external isolation address within
the vSAN network. VMware recommends disabling the isolation response for a
2-node Direct Connect configuration. If however, a host becomes isolated,
vSAN has the ability to kill VMs which are "ghosted" (no access to any of the
components). This will allow for vSphere HA to safely restart the impacted
VMs, without the need to use the Isolation Response. Do note that in the case
of a direct connect configuration the setting das.usedefaultisolationaddress
does not need to be set, if it has been set it should be configured to "true".
7. 2 Turn on vSphere HA
To turn on vSphere HA, select the cluster object in the vCenter inventory,
Manage, then vSphere HA. From here, vSphere HA can be turned on and off
via a check box.
7. 3 Host Monitoring
Host monitoring should be enabled on vSAN 2 Node Cluster configurations.
58

vSAN 2 Node Guide
Host monitoring should be enabled on vSAN 2 Node Cluster configurations.

This feature uses network heartbeat to determine the status of hosts
participating in the cluster, and if corrective action is required, such as
restarting virtual machines on other nodes in the cluster.
7. 4 Admission Control
Admission control ensures that HA has sufficient resources available to restart

virtual machines after a failure. As a full site failure is one scenario that needs
to be taken into account in a resilient architecture, VMware recommends
enabling vSphere HA Admission Control. Availability of workloads is the
primary driver for most stretched cluster environments. Sufficient capacity
must therefore be available for a host failure. Since virtual machines could be
equally divided across both hosts in a vSAN 2 Node Cluster, and to ensure that
all workloads can be restarted by vSphere HA, VMware recommends
configuring the admission control policy to 50 percent for both memory and
CPU.
VMware recommends using the percentage-based policy as it offers the most

flexibility and reduces operational overhead. For more details about admission
control policies and the associated algorithms we would like to refer to the
vSphere 6.5 Availability Guide.
The following screenshot shows a vSphere HA cluster configured with

admission control enabled using the percentage based admission control
policy set to 50%.
59

vSAN 2 Node Guide
It should be noted that vSAN is not admission-control aware. There is no way

to inform vSAN to set aside additional storage resources to accommodate fully
compliant virtual machines running on a single site. This is an additional
operational step for administrators if they wish to achieve such a configuration
in the event of a failure.
7. 5 Host Hardware Monitoring – VM Component Protection
vSphere 6.0 introduced a new enhancement to vSphere HA called VM

Component Protection (VMCP) to allow for an automated fail-over of virtual
machines residing on a datastore that has either an “All Paths Down” (APD) or
a “Permanent Device Loss” (PDL) condition.
A PDL, permanent device loss condition, is a condition that is communicated

by the storage controller to ESXi host via a SCSI sense code. This condition
indicates that a disk device has become unavailable and is likely permanently
unavailable. When it is not possible for the storage controller to communicate
back the status to the ESXi host, then the condition is treated as an “All Paths
Down” (APD) condition.
60

vSAN 2 Node Guide
In traditional datastores, APD/PDL on a datastore affects all the virtual

machines using that datastore. However, for vSAN this may not be the case. An
APD/PDL may only affect one or few VMs, but not all VMs on the vSAN
datastore. Also in the event of an APD/PDL occurring on a subset of hosts,
there is no guarantee that the remaining hosts will have access to all the virtual
machine objects, and be able to restart the virtual machine. Therefore, a
partition may result in such a way that the virtual machine is not accessible on
any partition.
Note that the VM Component Protection (VMCP) way of handling a failover is

to terminate the running virtual machine and restart it elsewhere in the cluster.
VMCP/HA cannot determine the cluster-wide accessibility of a virtual machine
on vSAN, and thus cannot guarantee that the virtual machine will be able to
restart elsewhere after termination. For example, there may be resources
available to restart the virtual machine, but accessibility to the virtual machine
by the remaining hosts in the cluster is not known to HA. For traditional
datastores, this is not a problem, since we know host-datastore accessibility for
the entire cluster, and by using that, we can determine if a virtual machine can
be restarted on a host or not.
At the moment, it is not possible for vSphere HA to understand the complete

inaccessibility vs. partial inaccessibility on a per virtual machine basis on vSAN;
hence the lack of VMCP support by HA for vSAN.
VMware recommends leaving VM Component Protection (VMCP) disabled.
7. 6 Datastore for Heartbeating
vSphere HA provides an additional heartbeating mechanism for determining

the state of hosts in the cluster. This is in addition to network heartbeating, and
is called datastore heartbeating. In many vSAN environment no additional
datastores, outside of vSAN, are available, and as such in general VMware
recommends disabling Heartbeat Datastores as the vSAN Datastore cannot be
used for heartbeating. However, if additional datastores are available, then
using heartbeat datastores is fully supported.
What do Heartbeat Datastores do, and when does it come in to play? The
heartbeat datastore is used by a host which is isolated to inform the rest of the
cluster what its state is and what the state of the VMs is. When a host is
61

vSAN 2 Node Guide
isolated, and the isolation response is configured to "power off" or "shutdown",

then the heartbeat datastore will be used to inform the rest of the cluster when
VMs are powered off (or shutdown) as a result of the isolation. This allows the
vSphere HA master to immediately restart the impacted VMs.
To disable datastore heartbeating, under vSphere HA settings, open the

Datastore for Heartbeating section. Select the option “ Use datastore from only
the specified list ”, and ensure that there are no datastore selected in the list, if
any exist. Datastore heartbeats are now disabled on the cluster. Note that this
may give rise to a notification in the summary tab of the host, stating that the
number of vSphere HA heartbeat datastore for this host is 0, which is less than
required:2. This message may be removed by following KB Article 2004739
which details how to add the advanced setting
das.ignoreInsufficientHbDatastore = true.
7. 7 Virtual Machine Response for Host Isolation
This setting determines what happens to the virtual machines on an isolated

host, i.e. a host that can no longer communicate to other nodes in the cluster,
nor is able to reach the isolation response IP address. VMware recommends
that the Response for Host Isolation is to Power off and restart VMs. The
reason for this is that a clean shutdown will not be possible as on an isolated
host the access to the vSAN Datastore, and as such the ability to write to disk,
is lost.
62

vSAN 2 Node Guide
Note that a 2-node direct connect configuration is a special case. In this

situation it is impossible to configure a valid external isolation address within
the vSAN network. VMware recommends disabling the isolation response for a
2-node direct connect configuration. If however a host becomes isolated, vSAN
has the ability to kill VMs which are "ghosted" (no access to any of the
components). This will allow for vSphere HA to safely restart the impacted
VMs, without the need to use the Isolation Response.
7. 8 Advanced Options
When vSphere HA is enabled on a vSAN Cluster, uses a heartbeat mechanisms

to validate the state of an ESXi host. Network heart beating is the primary
63

vSAN 2 Node Guide
mechanism for HA to validate availability of the hosts.
If a host is not receiving any heartbeats, it uses a fail safe mechanism to detect
if it is merely isolated from its HA master node or completely isolated from the
network. It does this by pinging the default gateway.
In vSAN environments, vSphere HA uses the vSAN traffic network for

communication. This is different to traditional vSphere environments where
the management network is used for vSphere HA communication. However,
even in vSAN environments, vSphere HA continues to use the default gateway
on the management network for isolation detection responses. This should be
changed so that the isolation response IP address is on the vSAN network, as
this allows HA to react to a vSAN network failure.
In addition to selecting an isolation response address on the vSAN network,

additional isolation addresses can be specified manually to enhance reliability
of isolation validation.
Network Isolation Response and Multiple Isolation Response

Addresses
When using vSAN 2 Node clusters in the same location, there is no need to
have a separate das.isolationaddress for each of the hosts.
64

vSAN 2 Node Guide
8. Cluster Settings - DRS

vSphere DRS is used in many environments to distribute load within a cluster.
65

vSAN 2 Node Guide
8. 1 Cluster Settings - DRS
vSphere DRS is used in many environments to distribute load within a cluster.

vSphere DRS offers many other features which can be very helpful in stretched
environments.
If administrators wish to enable DRS on vSAN 2 Node Cluster, there is a

requirement to have a vSphere Enterprise license edition or higher.
With vSphere DRS enabled on the cluster, the virtual machines can simply be
deployed to the cluster, and then the virtual machine is powered on, DRS will
move the virtual machines to either host based on utilization.
8. 2 Partially Automated or Fully Automated DRS
Customers can decide whether to place DRS in partially automated mode or

fully automated mode. With partially automated mode, DRS will handle the
initial placement of virtual machines. However, any further migration
recommendations will be surfaced up to the administrator to decide whether
or not to move the virtual machine. The administrator can check the
recommendation, and may decide not to migrate the virtual machine.
Recommendations should be for hosts on the same site.
With fully automated mode, DRS will take care of the initial placement and on-
going load balancing of virtual machines. DRS should balance virtual machines
across the hosts.
In Hybrid 2 Node vSAN Clusters, It is important to consider read locality, which

implies that they will cache on the host the virtual machine is running on. If the
virtual machine is migrated by DRS to the alternate host, the cache will need to
be warmed on the alternate host before the virtual machine reaches its
previous levels of performance. Changing the
/VSAN/DOMOwnerForceWarmCache value to 1 can mitigate this issue.
66

vSAN 2 Node Guide
9. Using a vSAN Witness Appliance

vSAN StretchedCluster supports the use of a vSAN Witness Appliance
67

vSAN 2 Node Guide
9. 1 Setup Step 1: Deploy the vSAN Witness Appliance
The vSAN Witness Appliance must be deployed on different infrastructure

than the 2 Node vSAN Cluster itself. This step will cover deploying the vSAN
Witness Appliance to a different cluster.
The first step is to download and deploy the vSAN Witness Appliance, or
deploy it directly via a URL, as shown below. In this example it has been
downloaded:
Select a Datacenter for the vSAN Witness Appliance to be deployed to and

provide a name (Witness-01 or something similar).
68

vSAN 2 Node Guide
Select a cluster for the vSAN Witness Appliance to reside on.
69

vSAN 2 Node Guide
Review the details of the deployment and press next to proceed.
70

vSAN 2 Node Guide
The license must be accepted to proceed.
71

vSAN 2 Node Guide
At this point a decision needs to be made regarding the expected size of the 2
Node vSAN Cluster configuration. There are three options offered. If you
expect the number of VMs deployed on the vSAN 2 Node Cluster to be 10 or
fewer, select the Tiny configuration. If you expect to deploy more than 10 VMs,
but less than 500 VMs, then the Normal (default option) should be chosen. On
selecting a particular configuration, the resources consumed by the appliance
and displayed in the wizard (CPU, Memory and Disk):
72

vSAN 2 Node Guide
Select a datastore for the vSAN Witness Appliance. This will be one of the
datastore available to the underlying physical host. You should consider when
the vSAN Witness Appliance is deployed as thick or thin, as thin VMs may grow
over time, so ensure there is enough capacity on the selected datastore.
73

vSAN 2 Node Guide
Select a network for the Management Network and for the Witness Network.
74

vSAN 2 Node Guide
Give a root password for the vSAN Witness Appliance:
75

vSAN 2 Node Guide
At this point, the vSAN Witness Appliance is ready to be deployed. It will need
to be powered on manually via the vSphere web client UI later:
76

vSAN 2 Node Guide
Once the vSAN Witness Appliance is deployed and powered on, select it in the
vSphere web client UI and begin the next steps in the configuration process.
9. 2 Setup Step 2: vSAN Witness Appliance Management
Once the vSAN Witness Appliance has been deployed, select it in the vSphere
web client UI, open the console.
The console of the vSAN Witness Appliance should be access to add the
correct networking information, such as IP address and DNS, for the
management network.
On launching the console, unless you have a DHCP server on the management
network, it is very likely that the landing page of the DCUI will look something
similar to the following:
77

vSAN 2 Node Guide
Use the <F2> key to customize the system. The root login and password will
need to be provided at this point. This is the root password that was added
during the OVA deployment earlier.
Select the Network Adapters view. There will be two network adapters, each
corresponding to the network adapters on the virtual machine. You should
note that the MAC address of the network adapters from the DCUI view match
the MAC address of the network adapters from the virtual machine view.
Because these match, there is no need to use promiscuous mode on the
network, as discussed earlier.
Select vmnic0, and if you wish to view further information, select the key <D>
to see more details.
78

vSAN 2 Node Guide
Navigate to the IPv4 Configuration section. This will be using DHCP by default.
Select the static option as shown below and add the appropriate IP address,
subnet mask and default gateway for this vSAN Witness Appliance
Management Network.
79

vSAN 2 Node Guide
The next step is to configure DNS. A primary DNS server should be added and
an optional alternate DNS server can also be added. The FQDN, fully qualified
domain name, of the host should also be added at this point.
One final recommendation is to do a test of the management network. One

can also try adding the IP address of the vCenter server at this point just to
make sure that it is also reachable.
80

vSAN 2 Node Guide
When all the tests have passed, and the FQDN is resolvable, administrators can
move onto the next step of the configuration, which is adding the vSAN
Witness Appliance ESXi instance to the vCenter server.
9. 3 Setup Step 3: Add Witness to vCenter Server
There is no difference to adding the vSAN Witness Appliance ESXi instance to

vCenter server when compared to adding physical ESXi hosts. However, there
are some interesting items to highlight during the process. First step is to
provide the name of the Witness. In this example, vCenter server is managing
multiple data centers, so we are adding the host to the witness data center.
81

vSAN 2 Node Guide
Provide the appropriate credentials. In this example, the root user and
password.
82

vSAN 2 Node Guide
Acknowledge the certificate warning:
83

vSAN 2 Node Guide
There should be no virtual machines on the vSAN Witness Appliance. Note: It

can never run VMs in a vSAN 2 Node Cluster configuration. Note also the
mode: VMware Virtual Platform. Note also that builds number may differ to the
one shown here.
84

vSAN 2 Node Guide
The vSAN Witness Appliance also comes with its own license. You do not need
to consume vSphere licenses for the witness appliance:
85

vSAN 2 Node Guide
Lockdown mode is disabled by default. Depending on the policies in use at a

customer’s site, the administrator may choose a different mode to the default:
86

vSAN 2 Node Guide
Click Finish when ready to complete the addition of the Witness to the vCenter
server:
87

vSAN 2 Node Guide
One final item of note is the appearance of the vSAN Witness Appliance ESXi
instance in the vCenter inventory. It has a light blue shading, to differentiate it
from standard ESXi hosts. It might be a little difficult to see in the screen shot
below, but should be clearly visible in your infrastructure. (Note: In vSAN 6.1
and 6. 2 deployments, the “No datastores have been configured” message is
because the nested ESXi host has no VMFS datastore. This can be ignored.)
88

vSAN 2 Node Guide
One final recommendation is to verify that the settings of the vSAN Witness
Appliance matches the Tiny, Normal or Large configuration selected during
deployment. For example, the Normal deployment should have an 12GB HDD
for boot in vSAN 6.5 (8GB for vSAN 6.1/6.2), a 10GB Flash that will be
configured later on as a cache device and another 350 HDD that will also be
configured later on as a capacity device.
89

vSAN 2 Node Guide
Once confirmed, you can proceed to the next step of configuring the vSAN
network for the vSAN Witness Appliance.
9. 4 Setup Step 4: Config vSAN Witness Host Networking
The next step is to configure the vSAN network correctly on the vSAN Witness
Appliance. When the Witness is selected, navigate to Configure > Networking >
Virtual switches as shown below.
90

vSAN 2 Node Guide
The Witness has a portgroup pre-defined called witnessPg. Here the VMkernel
port to be used for vSAN traffic is visible. If there is no DHCP server on the
vSAN network (which is likely), then the VMkernel adapter will not have a valid
IP address. Select VMkernel adapters > vmk1 to view the properties of the
witnessPg. Validate that "vSAN" is an enabled service as depicted below.
91

vSAN 2 Node Guide
* Engineering note: A few things to consider when configuring vSAN traffic on

the vSAN Witness Appliance.
The default configuration has vmk0 configured for Management Traffic

and vmk1 configured for vSAN Traffic.
The vmk1 interface cannot be configured with an IP address on the same
range as that of vmk0. This is because Management traffic and vSAN
traffic use the default TCP/IP stack. If both vmk0 and vmk1 are configured
on the same range, a multihoming condition will occur and vSAN traffic
will flow from vmk0, rather than vmk1. Health Check reporting will fail
because vmk0 does not have vSAN enabled. The multihoming issue is
detailed in KB 2010877 ( https://kb.vmware.com/kb/2010877 ).
In the case of 2 Node vSAN, If it is desired to have vSAN traffic on the
same subnet as vmk0, (or simply use a single interface for simplicity), it is
recommended to disable vSAN services on vmk1 (WitnessPg) and enable
vSAN services on vmk0 (Management). This is a perfectly valid and
supported configuration.
92

vSAN 2 Node Guide
Configure the network address
Select the witnessPg and edit the properties by selecting the pencil icon.
If vSAN is not an enabled service, select the witnessPg portgroup, and then
select the option to edit it. Tag the VMkernel port for vSAN traffic, as shown
below:
Next, ensure the MTU is set to the same value as the vSAN Data Node hosts’
vSAN VMkernel interface.
93

vSAN 2 Node Guide
In the IPV4 settings, a default IP address has been allocated. Modify it for the
vSAN traffic network.
Static routes are still required by the witnessPg VMkernel interface (vmk1) as in
vSAN 6.1 or 6.2. The "Override default gateway for this adapter" setting is not
supported for the witness VMkernel interface (vmk1).
Once the witnessPg VMkernel interface address has been configured, click OK.
94

vSAN 2 Node Guide
10. Configuring 2 Node vSAN

Configuring 2 Node vSAN
95

vSAN 2 Node Guide
10. 1 Pre-Requisites
Pre-requisites:
To properly configure 2 Node vSAN, it is important to have a few pre-requisites

in place.
Determine whether Witness Traffic Separation will be used or not.
Determine whether the vSAN Witness Host will be a vSAN Witness

Appliance or a Physical vSphere Host
Determine where the vSAN Witness Appliance will be hosted (if used)
Ensure connectivity between the 2 Node cluster and the vSAN Witness
Host
If using Witness Traffic Separation - A VMkernel interface other than
the vSAN Network will be required
If not using Witness Traffic Separation - The vSAN Network will be
required to have connectivity to the vSAN Witness Host
Configure appropriate networking
For vCenter
vCenter must be able to communicate with the Management
interface on each vSAN Host
vCenter must be able to communicate with the Management
interface for the vSAN Witness Host
For vSAN Hosts
The vSAN Host Management interface must be able to
communicate with vCenter
vSAN Hosts must be able to communicate with each other
vSAN Hosts must be able to communicate with the vSAN Witness
Host vSAN Tagged interface
For vSAN Witness Host
The vSAN Witness Host Management interface must be able to
communicate with vCenter
The vSAN Witness Host vSAN Tagged interface must be able to
communicate with the vSAN Nodes
96

vSAN 2 Node Guide
10. 2 VMkernel Interfaces
VMkernel Interfaces
VMkernel interfaces provide connectivity for different inter-node
communication between vSpehre hosts. IP-based storage protocols typically
have dedicated VMkernel interfaces on isolated networks. It is considered a
best practice and VMware recommendation to isolate storage traffic. vSAN
may only use VMkernel interfaces that are appropriate tagged for vSAN traffic
types.
In most vSAN configurations, each vSAN tagged VMkernel interface must be

able to commuicate with each and every other vSAN tagged VMkernel
interface. This is true for normal vSAN clusters as well as 2 Node vSAN Clusters
up to version 6.2 using vSphere 6.0 Update 2.
Witness Traffic Separation was publicly introduced with vSAN 6.5, but is also
included in vSAN 6.2 as of vSphere 6.0 Update 3 (vCenter and vSphere
versions). Witness Traffic Separation removes the requirement for traffic to
and from the vSAN Witness Host to be available from the vSAN data network.
A "front-end facing" VMkernel interface may be used, allowing the "back-end"
of 2 Node vSAN to use VMkernel ports that are directly connected across
hosts. The back-end vSAN network can still be connected to a switch if desired.
Using Witness Traffic Separation is the preferred method for configuring 2

Node vSAN as of vSphere 6.0 Update 3, wether using direct connect or using a
switch for the back-end vSAN data network. Because of this, the previously
required method will not be addressed.
Configuring the back-end VMkernel ports may be accomplished using

Configuration Assist, but the Witness Traffic Separation ports must still be
configured manually.
It is also recommended to configure 2 Node vSAN networking before

configuring the 2 Node vSAN cluster.
VMkernel Interfaces
97

vSAN 2 Node Guide
VMkernel interfaces provide connectivity for different inter-node

communication between vSpehre hosts. IP-based storage protocols typically
have dedicated VMkernel interfaces on isolated networks. It is considered a
best practice and VMware recommendation to isolate storage traffic. vSAN
may only use VMkernel interfaces that are appropriate tagged for vSAN traffic
types.
In most vSAN configurations, each vSAN tagged VMkernel interface must be

able to commuicate with each and every other vSAN tagged VMkernel
interface. This is true for normal vSAN clusters as well as 2 Node vSAN Clusters
up to version 6.2 using vSphere 6.0 Update 2.
Witness Traffic Separation was publicly introduced with vSAN 6.5, but is also
included in vSAN 6.2 as of vSphere 6.0 Update 3 (vCenter and vSphere
versions). Witness Traffic Separation removes the requirement for traffic to
and from the vSAN Witness Host to be available from the vSAN data network.
A "front-end facing" VMkernel interface may be used, allowing the "back-end"
of 2 Node vSAN to use VMkernel ports that are directly connected across
hosts. The back-end vSAN network can still be connected to a switch if desired.
Using Witness Traffic Separation is the preferred method for configuring 2

Node vSAN as of vSphere 6.0 Update 3, wether using direct connect or using a
switch for the back-end vSAN data network. Because of this, the previously
required method will not be addressed.
Configuring the back-end VMkernel ports may be accomplished using

Configuration Assist, but the Witness Traffic Separation ports must still be
configured manually.
It is also recommended to configure 2 Node vSAN networking before

configuring the 2 Node vSAN cluster.
10. 3 VMkernel Interfaces - Witness Traffic
Using Witness Traffic Separation for vSAN Witness

Traffic
Since the public introduction of Witness Traffic Separation in vSphere 6.5, this
has become the preferred method of allowing communication with the vSAN
Witness Host.
98

vSAN 2 Node Guide
As previously discussed, Witness Traffic Separation separates the networking

requirements for vSAN data and vSAN metadata communication to the vSAN
Witness Host. Communication to the vSAN Witness Host is performed through
a different VMkernel interface than the interface used to communicate
between vSAN Data nodes.
Data nodes can use a direct connection between nodes or may be connected
to a traditional switch, which could be part of an isolated data network.
Communication with the vSAN Witness Host is configured separately from the
back-end vSAN Data network.
Creating the vSAN Witness Traffic VMkernel interfaces can be accomplished in

the following fashion:
1. Create a new VMkernel port for use as a Witness Traffic interface
2. This will typically be on the same virtual switch as the Management

interface (vSwitch0)
99

vSAN 2 Node Guide
3. Give the VMkernel interface a descriptive label and be certain to select the
appropriate VLAN if necessary.
100

vSAN 2 Node Guide
Do not enable any services on the VMkernel interface.
4. Enter the appropriate network settings
101

vSAN 2 Node Guide
*Note: the Default gateway may not be set. vSAN uses the same TCP/IP
stack as the Management interface and may not use an alternate gateway.
It is important to also ensure that this VMkernel interface is NOT on the
same TCP/IP network as the Management interface.
5. Select Finish to complete creating the VMkernel interface.
102

vSAN 2 Node Guide
6. Reviewing the VMkernel adapters, the newly created VMkernel interface

does not have the proper tagging in place.
1. In the HTML5 client (vSphere 6.7):
2. From the ESXi console:
7. To enable this vSAN Witness traffic on the VMkernel interface in the

illustration (vmk1) there are a couple methods that can be used to enable
vSAN Witness Traffic.
1. Enable SSH on the vSAN Host and execute the following command:
103

vSAN 2 Node Guide
esxcli vsan network ip add -i <VMkernel for Witness Tra

ffic> -T=witness
example: esxcli vsan network ip add -i vmk1 -T=witness
2. Use the vSphere CLI to connect to the vSAN Host and execute the
following command:
esxcli vsan network ip add -i <VMkernel for Witness Tra

ffic> -T=witness
example: esxcli vsan network ip add -i vmk1 -T=witness
3. Execute the following single line PowerCLI script against the vSAN
Host:
Connect-VIServer <vSAN Host>; $EsxCli = Get-EsxCli -V2

; $VMkArgs = $EsxCli.vsan.network.ip.add.CreateArgs()
; $VMkArgs.interfacename = <VMkernel for Witness Traffi
c> ; $VMkArgs.traffictype = "witness" ; $EsxCli.vsan.ne
twork.ip.add.Invoke($VMkArgs) ; Disconnect-VIServer -Co
nfirm:$false
example: Connect-VIServer "host1.demo.local"; $EsxCli =

Get-EsxCli -V2; $VMkArgs = $EsxCli.vsan.network.ip.add
.CreateArgs(); $VMkArgs.interfacename = "vmk1"; $VMkArg
s.traffictype = "witness" ; $EsxCli.vsan.network.ip.add
.Invoke($VMkArgs); Disconnect-VIServer -Confirm:$false
8. Reviewing the VMkernel adapters again, the newly tagged VMkernel

interface does have the proper tagging in place.
104

vSAN 2 Node Guide
9. The "Witness Tagged" VMkernel interface will need to have connectivity to

the "vSAN Traffic" tagged interface on the vSAN Witness Host. In cases
where this connectivity is over Layer 3, static routes must be added. This is
because vSAN uses the default TCP/IP stack and will attempt to use the
Management VMkernel's default gateway.
To add a static route on the vSAN data nodes to the vSAN Witness Host's
"vSAN Tagged" VMkernel interface, perform one of the following methods:
1. Enable SSH on the vSAN Host, login, and execute the following
command:
esxcfg-route -a <target address/prefixlength> <gateway

to use>
example: esxcfg-route -a "192.168.110.0/24 192.168.15.1

"
2. Use the vSphere CLI to connect to the vSAN Host and execute the
following command:
esxcfg-route -a <target address/prefixlength> <gateway

to use>
example: esxcfg-route -a "192.168.110.0/24 192.168.15.1

"
Host:
Connect-VIServer <vSAN Host>; New-VMHostRoute -Destinat

ion <target address> -PrefixLength <prefixlength> -Gate
105

vSAN 2 Node Guide
way <gateway to use> -Confirm:$false; Disconnect-VIServ

er -Confirm:$false
example: Connect-VIServer "host1.demo.local"; New-VMHos

tRoute -Destination "192.168.110.0" -PrefixLength "24"
-Gateway "192.168.15.1" -Confirm:$false; Disconnect-VIS
erver -Confirm:$false
10. Confirm the Witness Tagged VMkernel interface can ping the vSAN
Witness Host vSAN Traffic tagged VMkernel interface:
1. Enable SSH on the vSAN Host, login, and execute the following
command:
vmkping -I <host Witness Tagged VMkernel interface name

> <vSAN Witness Host vSAN Tagged interface IP address>
example: vmkping -I vmk1 192.168.110.23
Host:
Connect-VIServer <vSAN Host> -user root; EsxCli = Get-E

sxCli -V2; $VMkArgs = $EsxCli.network.diag.ping.CreateA
rgs(); $VMkArgs.interface = <Witness Tagged interface n
ame>; $VMkArgs.host = <vSAN Witness Host vSAN Tagged in
terface IP address>; $EsxCli.network.diag.ping.Invoke($
VMkArgs).Summary; Disconnect-VIServer -Confirm:$false
example: Connect-VIServer "host1.demo.local" -user root

; EsxCli = Get-EsxCli -V2; $VMkArgs = $EsxCli.network.d
iag.ping.CreateArgs(); $VMkArgs.interface = "vmk1"; $VM
kArgs.host = "192.168.110.23"; $EsxCli.network.diag.pin
g.Invoke($VMkArgs).Summary; Disconnect-VIServer -Confir
m:$false
Once Witness Traffic Separation networking has been configured on the first
host, repeat the process for the second host.
*Important things to consider/remember:
The VMkernel interface for Witness Traffic may not be on the same
network as the Management interface (vmk0). If Witness Traffic must be
run on the same network as vmk0, simply skip steps 1-6 and tag vmk0 in
step 7 instead.
Witness Traffic from data nodes to the vSAN Witness Host contain no
106

vSAN 2 Node Guide
virtual machine data, but rather vSAN metadata, such as vSAN component
placement, vSAN node ownership, etc. Tagging vmk0 for Witness Traffic is
fully supported.
10. 4 VMkernel Interfaces - vSAN Traffic - VSS
vSAN Data Networking

The back-end vSAN Data network is configured in the same fashion for 2 Node
vSAN as any other vSAN network.
vSAN back-end networking may use either a vSphere Standard Switch or

vSphere Distributed Switch. This document will only focus on the setup
process and not the feature set differences between each of these.
Using a vSphere Standard Switch
Creating the vSAN Traffic VMkernel interfaces can be accomplished in the

following fashion:
1. Create a new VMkernel port for use as a vSAN Traffic interface on the first
host.
107

vSAN 2 Node Guide
2. This will typically be on a different virtual switch from the Management

interface (vSwitch0)
108

vSAN 2 Node Guide
3. When creating a new vSphere Standard Switch, adapters will have to be

assigned. Click the plus to assign adapters to this vSphere Standard
Switch.
109

vSAN 2 Node Guide
4. Select one or more adapters that will be used for vSAN Traffic and select
OK.
110

vSAN 2 Node Guide
5. When all adapters have been added, select Next.
111

vSAN 2 Node Guide
6. Give the VMkernel interface a descriptive label and be certain to select the
appropriate VLAN if necessary.
Enable vSAN services on the VMkernel interface.
112

vSAN 2 Node Guide
113

vSAN 2 Node Guide

114

vSAN 2 Node Guide
Notice vmk2 has vSAN Traffic Tagged and vmk1 has Witness Traffic
Tagged.
10. Repeat this process for the second host.
10. 5 VMkernel Interfaces - vSAN Traffic - VDS
vSAN Data Networking

The back-end vSAN Data network is configured in the same fashion for 2 Node
vSAN as any other vSAN network. vSAN licensing includes the vSphere
Distributed Switch feature, regardless of vSphere licensed edition.
vSAN back-end networking may use either a vSphere Standard Switch or

vSphere Distributed Switch. This document will only focus on the setup
process and not the feature set differences between each of these.
Using a vSphere Distributed Switch
Creating the vSAN Witness Traffic VMkernel interfaces can be accomplished in
115

vSAN 2 Node Guide
the following fashion:
1. Create a new VMkernel port for use as a vSAN Traffic interface on the first
host.
2. When using a vSphere Distributed Switch, select an existing network that

has been previously created.
116

vSAN 2 Node Guide
3. Select browse and choose the vSAN Port Group on the VDS and select OK.
117

vSAN 2 Node Guide
4. Select Next.
118

vSAN 2 Node Guide
5. Select the appropriate VLAN if necessary and check vSAN in the Available
services. This is "tagging" this VMkernel interface for vSAN Traffic.
119

vSAN 2 Node Guide
120

vSAN 2 Node Guide

121

vSAN 2 Node Guide
Notice vmk2 has vSAN Traffic Tagged and vmk1 has Witness Traffic
Tagged.
9. Repeat this process for the vMotion Network.
10. Repeat this process for the vSAN and vMotion Network for the second
host.
10. 6 Creating a New 2 Node vSAN Cluster
Creating the vSAN Cluster
The following steps should be followed to install a new 2 Node vSAN Cluster.
In this example, there are 2 nodes available: host1.demo.local and

host2.demo.local. Both hosts reside in a vSphere cluster called 2 Node. vSAN
Witness Host, witness.demo.central. is in its own data center and is not added
to the cluster.
122

vSAN 2 Node Guide
To setup 2 Node vSAN, he Manage > vSAN > Services . Click Configure to
begin the vSAN wizard.
Create Step 1 Configure vSAN as a 2 Node vSAN Cluster
The initial wizard allows for choosing various options like enabling
Deduplication and Compression (All-Flash architectures only with Advanced or
greater licensing) or Encryption (Enterprise licensing required) for vSAN. Select
Two host vSAN Cluster and Next.
123

vSAN 2 Node Guide
Create Step 2 Configure Services
Select the services desired that are compatible with the hardware and licensing
options for the cluster.
124

vSAN 2 Node Guide
Deduplication and Compression will require vSAN Advanced licensing or

higher and All-Flash hardware
Encryption will require vSAN Enterprise Licensing and an available KMIP
1.1 compliant Key Management Server (KMS).
In 2 Node vSAN it is important to get into the habit of using Allow Reduced
Redundancy. This is because any time there is an requirement to
create/remove/recreate a vSAN Disk Group on a 2 Node vSAN Cluster, the
Allow Reduced Redundancy option is required. Without this setting checked,
changes will likely not occur due to the cluster not being able to maintain
storage policy compliance.
Create Step 3 Claim Disks
Disks should be selected for their appropriate role in the vSAN cluster.
125

vSAN 2 Node Guide
Click Next .
Create Step 4 Create Fault Domains
The Create fault domain wizard will allow hosts to be selected for either the
Preferred or Secondary fault domain. The default naming of these two fault
domains are Preferred and Secondary.
126

vSAN 2 Node Guide
Select one host and choose >> to move it to the Secondary fault domain.
Click Next .
Create Step 5 Select Witness Host
The vSAN Witness host detailed earlier must be selected to act as the vSAN
Witness for the 2 Node vSAN Cluster.
127

vSAN 2 Node Guide
Click Next .
Create Step 6 Claim Disks for Witness Host
Just like physical vSAN hosts, the vSAN Witness needs a cache tier and a
capacity tier. * Note: The vSAN Witness does not actually require SSD backing
and may reside on a traditional mechanical drive.
128

vSAN 2 Node Guide
Be certain to select the 10GB device as the cache device and the 15GB device
as a capacity device.
Select Next .
Create Step 7 Complete
Review the vSAN 2 Node Cluster configuration for accuracy and click Finish.
129

vSAN 2 Node Guide
10. 7 Upgrading a older 2 Node vSAN Cluster
Upgrading a vSAN 2 Node Cluster is very easy. It is important though to follow

a sequence of steps to ensure the upgrade goes smoothly.
Upgrading Step 1: Upgrade vCenter Server
As with any vSphere upgrades, it is typically recommended to upgrade vCenter

Server first. While vCenter Server for Windows installations are supported, the
steps below describe the process when using the vCenter Server Appliance
(VCSA).
Log in to the VAM I interface and select Update from the Navigator pane to
begin the upgrade process.
130

vSAN 2 Node Guide
*Refer to the documentation for vCenter Server for Windows to properly

upgrade to a newer release of vCenter Server.
After the upgrade has completed, the VCSA will have to be rebooted for the
updates to be applied. It is important to remember that vSAN Health Check will
not be available until after hosts have been upgraded.
Upgrading Step 2: Upgrade Each Host
Upgrading each host is the next task to be completed. There are a few
considerations to remember when performing these steps.
As with any upgrade, hosts will be required to be put in maintenance mode,

remediated, upgraded, and rebooted. Maintenance mode will require the
“ensure accessibility” method is selected, and read operations will be
performed by the alternate host. In Hybrid configurations it may be
advantageous to enable the /VSAN/DOMOwnerForceWarmCache setting to
131

vSAN 2 Node Guide
ensure reads are always distributed across hosts, which will mitigate the
warming process required when a virtual machine moves to the alternate host.
With vSphere DRS in place, will ensure that virtual machines are moved to the
alternate host. If DRS is set to “fully automated” virtual machines will vMotion
to the other host automatically, while “partially automated” or “manual” will
require the virtualization admin to vMotion the virtual machines to the other
host manually.
Upgrading Step 3: Upgrade the Witness Host
After both vSAN Data Nodes have been upgraded, the vSAN Witness Host will
also need to be upgraded. The Health Check will show that the vSAN Witness
Host has not been upgraded.
Upgrading the vSAN Witness Host is done in the same way that any other ESXi
hosts are updated. It is important to remember that as the vSAN Witness Host
is upgraded, the witness components will no longer be available and objects
will be noncompliant. Objects will report that the Witness component is not
found.
132

vSAN 2 Node Guide
After the upgrade is complete, the Witness component will return, and will
reside on the vSAN Witness Host.
Upgrading Step 4: Upgrade the on-disk Format if necessary
The final step in the upgrade process, will be to upgrade the on-disk format. To use the new features, from time
to time the on-disk format must be upgraded to the required version. The Health Check will assume that the
vSphere version will prefer a native on-disk format for that version, and as a result, it will throw an error until the
format is upgraded.
133

vSAN 2 Node Guide
To upgrade the on-disk format from an older version to a newer version, select
M anage > General under vSAN. Then click Upgrade under the On-disk Format
Version section.
The depending on the on-disk format version an upgrade will either perform a
rolling upgrade across the hosts in the cluster, or make a small metadata
update. In the event a rolling upgrade is required, each host’s disk groups will
be removed and recreated with the new on-disk format. The amount of time
required to complete the on-disk upgrade will vary based on the cluster
hardware configuration, how much data is on the cluster, and whatever over
134

vSAN 2 Node Guide
disk operations are occurring on the cluster. The on-disk rolling upgrade
process does not interrupt virtual machine disk access and is performed
automatically across the hosts in the cluster.
The witness components residing on the vSAN Witness Host will be deleted
and recreated. This process is relatively quick given the size of witness objects.
10. 8 Converting a 2 Node Cluster with WTS to a 3 Node Cluster
How can a 2 Node Cluster be

converted to a 3 Node Cluster when
using Witness Traffic Separation
(WTS)?
This is a very common question asked by customers who start small with 2
Node vSAN and wish to grow the cluster to 3 or more hosts at a later time.
2 Node vSAN is essentially a 1+1+1 Stretched Cluster configuration, which

requires a vSAN Witness Host. Alternatively, "traditional" vSAN Clusters do not
require a vSAN Witness Host.
To convert a 2 Node Cluster to a larger cluster with 3 or more hosts, a few

steps must be accomplished in a particular order of operations.
Basic Workflow
The process of converting a 2 Node Cluster to a Cluster with 3 or more hosts is
135

vSAN 2 Node Guide
as follows:
1. Ensure Node to Node connectivity for the vSAN Data Nodes

2. Remove the vSAN Witness Host
3. Add the 3rd, 4th, 5th, and subsequent Nodes to the Cluster
Some additional workflow considerations

Some additional items to consider when planning to move from a 2 Node vSAN
Cluster to a larger "traditional" cluster:
Are the 2 Nodes connected directly to each other, as in a Direct Connect

configuration?
This is important, because all hosts in a cluster will have to be
connected to a switch
Is Witness Traffic Separation (WTS) being used?
WTS was publicly introduced in vSAN 6.5, but works with vSAN 6.0 U3
or higher.
Traditional vSAN clusters do not use WTS
What version of vSphere/vSAN is being used?
vSphere 6.0/6.5/6.7, and vSAN 6.1/6.2/6.5/6.6/6.7 respectively, hosts
can simply be added to a vSphere cluster
When using Cluster Quickstart, vSphere 6.7 Update 1 requires hosts to
be added to a vSphere cluster using the Cluster Quickstart
Networking - Hosts are directly connected when

using WTS
Because vSAN clusters that have more than 2 Nodes must be connected to a
switch for vSAN traffic, it is important to migrate the direct connection to a
switch.
These steps will ensure availability of vSAN data and virtual machines when
migrating to a switch for vSAN traffic.
1. Migrate any virtual machines to the host that is in the Fault Domain that is
designated as "Preferred".
1. This Fault Domain is often named "Preferred" but could have been
given a different name during configuration.
2. The asterisk in the Fault Domains UI will indicate which host is
136

vSAN 2 Node Guide
"Preferred"
2. When all workloads have been migrated to the preferred host, place the
other host in Maintenance Mode, choosing Ensure Accessibility
3. Disconnect the direct connected uplink(s) from the alternate host (non-
preferred) and connect it to an appropriate switch
1. This will connect the preferred host to the switch
4. Connect the non-preferred node to the switch, just as the preferred node
is connected
1. Confirm connectivity between the preferred and non-preferred nodes
2. On each node, from the ESXi shell, use vmkping to confirm
connectivity
1. vmkping -I vmX (vSAN tagged VMkernel interface) <IP Address of
alternate host vSAN tagged VMkernel>
2. vmkping -I vmX (vMotion tagged VMkernel interface) <IP Address
of alternate host vMotion tagged VMkernel>
5. When connectivity is confirmed, exit maintenance mode on the non-
preferred node
Proceed to the next section.
Converting from 2 Node to 3 Node when all data

nodes are connected to a switch
When all vSAN Nodes are connected to a switch, and vSAN VMkernel
interfaces are properly communicating between nodes, the process to move
from 2 Node vSAN to 3 or more nodes is very straightforward.
1. Disable Stretched Clustering

1. If In the Fault Domains UI, select Disable
137

vSAN 2 Node Guide
2. Remove the vSAN Witness Host
3. Remove the Fault Domains because they are not necessary

1. Select each Fault Domain from the Fault Domains UI, and Remove
138

vSAN 2 Node Guide
4. If the hosts are using Witness Traffic Separation (WTS) it is important

to untag "witness" traffic
1. This can be accomplished using the following command from the
ESXi shell/console: esxcli vsan network remove -i vmkX (vmkX is
the VMkernel interface that has "witness" traffic tagged)
2. Alternatively, using PowerCLI, the following one-liner may be used
(must be connected to vCenter first): Get-Cluster -Name
CLUSTERNAME | Get-VMHost | Get-EsxCli -v2 | % {
$_.vsan.network.remove.Invoke(@{interfacename='vmkX'})}
2. Configure networking on 3rd host

1. Configure the vSAN Network on the 3rd host
139

vSAN 2 Node Guide
2. Configure the vMotion Network on the 3rd host
3. Add 3rd host to the vSAN cluster & claim disks

1. If using the Cluster Quickstart:
140

vSAN 2 Node Guide
1. Use the Cluster Quickstart to add the host to the cluster
2. Use the Cluster Quickstart to create the disk group(s)
2. If not using the Cluster Quickstart

1. Hosts are added to the Cluster by
1. Right clicking the host and selecting Move to
2. Dragging a host into the Cluster
3. Right clicking the Cluster and adding a host not previously
added to vCenter
2. Disk Groups can be added using the Add Disk Group option in
Disk Management
141

vSAN 2 Node Guide
4. Rebuild the vSAN Witness Components on the 3rd Node

1. Because the Witness Components were previously on the vSAN
Witness Host, they must be recreated for vSAN Objects to be
compliant.
2. These will be recreated automatically when the CLOM Repair Timer
expires, or an administrator may immediately recreate them through
the vSAN Health Check
1. Select Monitor from the vSAN Cluster UI
2. Select vSAN - Health
3. Expand the Data alarm and select "Repair Objects Immediately"
Below is a narrated video that goes through the process for a 2 Node vSAN 6.7
Update 1 cluster:
142

vSAN 2 Node Guide
Click to see topic

media
Summary
The process to convert a 2 Node vSAN Cluster to a 3 (or more) Node vSAN
Cluster is relatively easy, but does require a particular steps to be taken.
Any vSAN 2 Node cluster can be converted to 3 or more nodes.
There is no architectural or licensing limitation when converting from 2 Node

to 3 or more node vSAN provided appropriate licensing is assigned to the
cluster.
143

vSAN 2 Node Guide
11. Management and Maintenance

The following section of the guide covers considerations related to
management and maintenance of a vSAN Stretched Cluster configuration.
144

vSAN 2 Node Guide
11. 1 Management and Maintenance
The following section of the guide covers considerations related to

management and maintenance of 2 Node vSAN.
11. 2 Maintenance Mode Consideration
In any situation where a 2 Node vSAN cluster has an inaccessible host or disk group, vSAN objects are at risk of
becoming inaccessible should another failure occur.
Maintenance Mode on the Witness Host
When a host fails, vSAN cannot rebuild data on another host to protect against another failure.
If a host must enter maintenance mode, vSAN cannot evacuate data from the host to maintain policy
compliance. While the host is in maintenance mode, data is exposed to a potential failure or inaccessibility
should an additional failure occur.
Maintenance Mode on a Data Node
When placing a vSAN host in maintenance mode, the vSAN data migration options are:
o Full data migration – Not available for two-node vSAN clusters using the default storage policy, as policy
compliance requires two for data and one for the witness object
o Ensure accessibility – The preferred option for two-host or three-host vSAN clusters using the default
storage policy. Ensure accessibility guarantees the enough components of the vSAN object are available for
the object to remain available. Though still accessible, vSAN objects on two-host or three-host clusters are no
longer policy
compliant. When the host is no longer in maintenance mode, objects will be rebuilt to ensure policy
compliance. During this time however, vSAN objects are at risk because they will become inaccessible if
another failure occurs.
Any objects that have a non-standard single copy storage policy (FTT=0) will be moved to an available host in
the cluster. If there is not sufficient capacity on any alternate hosts in the cluster, the host will not enter
maintenance mode.
145

vSAN 2 Node Guide
o No data migration – This is not a recommended option for vSAN clusters. vSAN objects that use the default
vSAN Storage Policy may continue to be accessible, but vSAN does not ensure their accessibility.
Any objects that have a non-standard single copy storage policy (FTT=0) will become inaccessible until the
host exits maintenance mode.
Maintenance Mode on the vSAN Witness Host
Maintenance mode on the vSAN Witness Host is typically an infrequent event.

Different considerations should be taken into account, depending on the type
of vSAN Witness Host used:
o vSAN Witness Appliance (recommended) – No virtual machine workloads may run here. The only purpose
of this appliance is to provide quorum for the 2 Node vSAN Cluster. Maintenance mode should be brief,
typically associated with updates and patching.
o Physical ESXi Host – While not typical, a physical host may be used as a vSAN Witness Host. This
configuration will support virtual machine workloads. Keep in mind that a vSAN Witness Host may not be a
member of any vSphere cluster, and as a result virtual machines will have to be manually moved to an
alternate host for them to continue to be available.
When maintenance mode is performed on the witness host, the witness

components cannot be moved to either site. When the witness host is put in
maintenance mode, it behaves as the No data migration option would on site
hosts. It is recommended to check that all virtual machines are in compliance
and there is no ongoing failure, before doing maintenance on the witness host.
146

vSAN 2 Node Guide
12. Failure Scenarios

In this section, we will discuss the behavior of the vSAN Stretched Cluster
when various failures occur.
147

vSAN 2 Node Guide
12. 1 Failure Scenarios
Failure Scenarios and Component

Placement
Understanding component placement is paramount in understanding failure
scenarios. The illustration shows the the placement of a vSAN Object's
components in a 2 Node vSAN Cluster Scenario.
The virtual machine's virtual disk (vmdk) has one component placed in the
Preferred Host, one component placed in the Secondary Host, and a Witness
component in the Witness Site that houses the vSAN Witness Host.
148

vSAN 2 Node Guide
The illustration shows a storage policy that protect vSAN Objects across sites.
This is the default protection policy used with vSAN 2 Node Clusters for
versions 6.1 through 6.7.
vSAN 2 Node Clusters can support up to a single host failure.
It is important to remember that the vSAN Witness Host, residing in the

Witness site, is a single host also.
149

vSAN 2 Node Guide
The scenarios in this section will cover different failure behaviors.
12. 2 Individual Drive Failure
What happens when a drive
Fails?
vSAN will mark the component degraded and the VM will continue to
run.
Reads are immediately serviced from the alternate node
The component will be rebuilt within the same host if there is an
additional drive or disk group available.
If there are no additional drives or disk groups available in the host, the
component will only be rebuilt if the failed/isolated device comes back
online.
In cases where the component cannot be rebuilt, reads will continue
to be serviced from the alternate host.
150

vSAN 2 Node Guide
Goes Offline?
vSAN will mark the component absent and the VM will continue to run.
151

vSAN 2 Node Guide
Reads continue to be serviced from the alternate node

After 60 minutes:
The component will be rebuilt within the same host if there is an
additional drive or disk group available.
If there are no additional drives or disk groups available in the
host, the component will only be rebuilt if the failed/isolated
device comes back online.
In cases where the component cannot be rebuilt, reads will continue
to be serviced from the alternate host.
12. 3 Host Failure / Network Partitions
What happens when a host goes

offline, or loses connectivity?
A typical 2 Node vSAN Cluster configuration can be seen here:
152

vSAN 2 Node Guide
Preferred Host Failure or Completely Partitioned
In the event the Preferred Host fails or is partitioned, vSAN powers the virtual
machines running in that Host off. The reason for this, is because the virtual
machine's components are not accessible due to the loss of quorum. The vSAN
2 Node has now experienced a single Host failure. The loss of either Host in
addition to the vSAN Witness is two failures, will take the entire cluster offline.
153

vSAN 2 Node Guide
An the Secondary Node will be elected as the HA master, which will validate
which virtual machines are to be powered on. Because quorum has been
formed between the vSAN Witness Host and the Secondary Host, virtual
machines on the Secondary Host will have access to their data, and can be
powered on.
154

vSAN 2 Node Guide
Secondary Host Failure or Partitioned
In the event the Secondary Host fails or is partitioned, vSAN powers the virtual
machines running in that host off. The reason for this, is because the virtual
machine's components are not accessible due to the loss of quorum. The vSAN
2 Node Cluster has now experienced a single site failure. The loss of either
node in addition to the witness is two failures, will take the entire cluster
offline.
155

vSAN 2 Node Guide
The Preferred Host, will validate which virtual machines are to be powered on.
Virtual machines which have been moved to the Preferred Host will now have
access to their data, and can be powered on.
156

vSAN 2 Node Guide
vSAN Witness Host Failure or Partitioned
Virtual machines running in both nods of a 2 Node Cluster are not impacted by
the vSAN Witness Host being partitioned. Virtual machines continue to run.
The vSAN 2 Node Cluster has now experienced a single failure. The loss of
either host in addition to the witness is two failures, will take the entire cluster
offline.
157

vSAN 2 Node Guide
In the event the vSAN Witness Host has failed, the behavior is the same as if
the vSAN Witness Host has been partitioned from the cluster. Virtual machines
continue to run at both locations. Because the 2 Node vSAN Cluster has now
experienced a single site failure, it is important to either get the vSAN Witness
Host back online, or deploy a new one for the cluster.
158

vSAN 2 Node Guide
When the existing vSAN Witness Host comes back online, metadata changes
are resynchronized between the 2 Node vSAN Cluster sites and the vSAN
Witness Host.
If a new vSAN Witness Host is deployed after 60 minutes the metadata

changes will automatically be created on the vSAN Witness Host and
synchronized across the 2 Node Cluster. If the new vSAN Witness Host is
deployed before 60 minutes, selecting Repair Objects in the vSAN Health
Check UI will be required to create the vSAN Witness Component metadata.
159

vSAN 2 Node Guide
The amount of data that needs to be transmitted depends on a few items such
as the number of objects and the number of changes that occurred while the
vSAN Witness Host was offline. However, this amount of data is relatively small
considering it is metadata, not large objects such as virtual disks.
12. 4 VM Provisioning When a Host is Offline
If there is a failure in the cluster, i.e. one of the hosts is down; new virtual
160

vSAN 2 Node Guide
machines can still be provisioned. The provisioning wizard will however warn
the administrator that the virtual machine does not match its policy as follows:
In this case, when one node is offline and there is a need to provision virtual
machines, the ForceProvision capability is used to provision the VM. This
means that the virtual machine is provisioned with a
NumberOfFailuresToTolerate = 0, meaning that there is no redundancy.
Administrators will need to rectify the issues on the failing site and bring it
back online. When this is done, vSAN will automatically update the virtual
machine configuration to NumberOfFailuresToTolerate = 1, creating a second
copy of the data and any required witness components.
12. 5 Multiple Simultaneous Failures
What happens if there are failures at

multiple levels?
The scenarios thus far have only covered situations where a single failure has
occurred.
Votes and their contribution to object accessibility
The vSAN Design and Sizing Guide goes into further detail about how
component availability determines access to objects. In short, each component
has a vote, and a quorum of votes must be present for an object to be
accessible. Each site will have an equal number of votes and there will be an
even distribution of votes within a site. If the total number of votes is an even
number, a random vote will be added.
In the illustration below, a 2 Node vSAN Cluster (4+4+1) has an object mirrored
across hosts.
161

vSAN 2 Node Guide
If the vSAN Witness Host fails, the vmdk is still accessible.
162

vSAN 2 Node Guide
If the Secondary Node, or the link between the Nodes, were to also fail, the
vSAN Objects would not be accessible
163

vSAN 2 Node Guide
The proper mechanism to recover would be to bring the Secondary Node back
online and ensure it can communicate with the Preferred Node. With the
Preferred and Secondary Nodes online, vSAN Objects will be accessible, but
will not have policy compliance.
164

vSAN 2 Node Guide
The vSAN Witness Host may be brought back online, or a new vSAN Witness
Host may be deployed.
When either the vSAN Witness Host or a replacement vSAN Witness Host is
brought online, objects may be repaired through the vSAN Health Check UI to
achieve policy compliance.
165

vSAN 2 Node Guide
12. 6 Replacing a Failed vSAN Witness Host
Should a vSAN Witness Host fail in the vSAN 2 Node Cluster, a new vSAN
Witness Host can easily be introduced to the configuration.
Replacing the vSAN Witness Host in vSAN 6.7 or higher using the vSphere Client
Navigate to Cluster > Configure > vSAN > Fault Domains & Stretched Cluster
166

vSAN 2 Node Guide
Select "Change "
Use the Change Witness Host Wizard in the same fashion as adding an initial
vSAN Witness Host.
Select the new vSAN Witness Host to be used
167

vSAN 2 Node Guide
Select the 10GB disk for the cache device and the 15GB for capacity
Finish the wizard
168

vSAN 2 Node Guide
Replacing the vSAN Witness Host in vSAN 6.6 or higher using the vSphere Web Client
Navigate to Cluster > Configure > vSAN > Fault Domains & Stretched Cluster
169

vSAN 2 Node Guide
Select "Change Witness Host"
Use the Change Witness Host Wizard in the same fashion as adding an initial
vSAN Witness Host.
Select the new vSAN Witness Host to be used
Select the 10GB disk for the cache device and the 15GB for capacity
170

vSAN 2 Node Guide
Finish the Wizard.
171

vSAN 2 Node Guide
Replacing the vSAN Witness Host Pre-6.6
Navigate to Cluster > Manage > vSAN > Fault Domains.
The failing witness host can be removed from the vSAN Stretched Cluster via
the UI (red X in fault domains view).
The next step is to rebuild the vSAN stretched and selecting the new witness
host. In the same view, click on the “configure stretched cluster” icon. Align
hosts to the preferred and secondary sites as before. This is quite simple to do
since the hosts are still in the original fault domain, so simply select the
secondary fault domain and move all the hosts over in a single click:
172

vSAN 2 Node Guide
Select the new witness host:
Create the disk group and complete the vSAN Stretched Cluster creation.
On completion, verify that the health check failures have resolved. Note that
the vSAN Object health test will continue to fail as the witness component of
VM still remains “Absent”. When CLOMD (Cluster Level Object Manager
Daemon) timer expires after a default of 60 minutes, witness components will
173

vSAN 2 Node Guide
be rebuilt on new witness host. Rerun the health check tests and they should
all pass at this point, and all witness components should show as active.
12. 7 Failure Scenario Matrices
Failure Scenario Matrices

Hardware/Physical Failures
Impact/Observed
VMware HA
Scenario vSAN Behaviour
Behaviour
Cache disk failure Disk Group is marked as VM will continue

failed and all running.
components present on
it will rebuild on another
Disk Group after 60
minutes.
Capacity disk failure Disk Group is marked as VM will continue

(Dedupe and failed and all running.
Compression ON) components present on
it will rebuild on another
Disk Group after 60
minutes.
Capacity disk failure Disk marked as failed and VM will continue

(Dedupe and all components present running.
Compression OFF) on it will rebuild on
another disk after 60
minutes.
174

vSAN 2 Node Guide
Impact/Observed
VMware HA
Behaviour
Disk Group failure/offline All components present VM will continue

on the Disk Group will running.
rebuild on another Disk
Group after 60 minutes.
RAID/HBA card failure All Disk Groups backed VM will continue

by the HBA/RAID card running.
will be marked absent
and all components
present will rebuild on
other Disk Groups after
60 minutes.
Host failure Component on the host VM will continue

will be marked as absent running if on the
by vSAN alternate host.
If the VM was
running on the same
host as the failure a
HA restart of the VM
will take place if HA
is enabled.
vSAN Witness Host loss / vSAN Witness Host loss VM will continue
failed / isolated from one counts as a site (PFTT) running.
or both nodes failure, as such the
cluster will be placed in a
degraded state until the
witness comes back
online or is redeployed.
175

vSAN 2 Node Guide
Impact/Observed
VMware HA
Behaviour
Node failure or partition, Host is declared lost, VMs running on the

ISL failure / connectivity quorum is established partitioned/failed
loss between the vSAN node are powered
Witness Host and the off.
remaining node.
HA will restart the
VMs from the
secondary fault
domain/host on the
preferred fault
domain/host.
If the preferred fault

domain/host has
failed, VMs will
restart on the
secondary fault
domain/host.
Loss of both hosts Cluster offline. VMs stop running.

HA cannot restart
VMs until quorum is
re-established.
13. Appendix A
176

vSAN 2 Node Guide
A list of links to additional vSAN resources is included below.

13. 1 Appendix A: Additional Resources
A list of links to additional vSAN resources is included below.
vSAN 6.0 Proof Of Concept Guide

vSAN 6.1 Health Check Plugin Guide
vSAN Stretched Cluster Bandwidth Sizing Guidance
Tech note: New VSAN 6.0 snapshot format vsanSparse
vSAN 6.2 Design and Sizing Guide
vSAN Troubleshooting Reference Manual
RVC Command Reference Guide for vSAN
vSAN Administrators Guide
vSAN 6.0 Performance and Scalability Guide
13. 2 Location of the vSAN Witness Appliance OVA
The vSAN Witness Appliance OVA is located on the Drivers & Tools tab of each
vSAN edition's download page.
There you will find a section called VMware vSAN Tools, Plug-ins, and
Appliances.
The vSAN Witness Appliance may be found there.
177

vSAN 2 Node Guide
14. Appendix B
New ESXCLI commands for vSAN Stretched Cluster.
178

vSAN 2 Node Guide
14. 1 Appendix B: Commands for vSAN 2 Node Clusters
Commands for 2 Node vSAN Clusters

ESXCLI commands
esxcli vsan cluster preferredfaultdomain
Display the preferred fault domain for a host:
[root@host1:~] esxcli vsan cluster preferredfaultdomain

Usage: esxcli vsan cluster preferredfaultdomain {cmd} [cmd op
tions]
Available Commands:
get Get the preferred fault domain for a stretched cluster
.
set Set the preferred fault domain for a stretched cluster
.
[root@host1:~] esxcli vsan cluster preferredfaultdomain get

Preferred Fault Domain Id: a054ccb4-ff68-4c73-cb c2-d272d4
5e32df
Preferred Fault Domain Name: Preferred
[root@host1:~]
esxcli vsan network
Display, add, modify, remove vSAN VMkernel interfaces for use by vSAN
[root@host1:~] esxcli vsan network

Usage: esxcli vsan network {cmd} [cmd options]
Available Namespaces:
ip Commands for configuring IP network fo
r vSAN.
ipv4 Compatibility alias for "ip"
Available Commands:
clear Clear the vSAN network configuration.
list List the network configuration current
ly in use by vSAN.
remove Remove an interface from the vSAN netw
179

vSAN 2 Node Guide
ork configuration.
restore Restore the persisted vSAN network con
figuration.
[root@host1:~] esxcli vsan network list

Interface
VmkNic Name: vmk2
IP Protocol: IP
Interface UUID: 7c80645b-73d6-60f7-faa1-001b2193c268
Agent Group Multicast Address: 224.2.3.4
Agent Group IPv6 Multicast Address: ff19::2:3:4
Agent Group Multicast Port: 23451
Master Group Multicast Address: 224.1.2.3
Master Group IPv6 Multicast Address: ff19::1:2:3
Master Group Multicast Port: 12345
Host Unicast Channel Bound Port: 12321
Multicast TTL: 5
Traffic Type: vsan
[root@host1:~] esxcli vsan network ip add -i vmk1 -T=witness
[root@host1:~] esxcli vsan network list

Interface
VmkNic Name: vmk2
IP Protocol: IP
Interface UUID: 7c80645b-73d6-60f7-faa1-001b2193c268
Multicast TTL: 5
Traffic Type: vsan
Interface
VmkNic Name: vmk1
IP Protocol: IP
Interface UUID: 1f825f5b-5018-f75c-a99d-001b2193c268
180

vSAN 2 Node Guide

Multicast TTL: 5
Traffic Type: witness
[root@host1:~]
esxcfg-advcfg
Advanced Settings for ESXi
[root@host2:~] esxcfg-advcfg
This usage
Usage: esxcfg-advcfg <options> [<adv cfg Path>]
-g|--get Get the value of the VMkernel advan
ced
configuration option
-s|--set <value> Set the value of the VMkernel advan
ced
configuration option
Get DOMOwnerForceWarmCache to determine if Read Locality is enabled

(Only relevant in Hybrid 2 Node vSAN)
[root@host1:~] esxcfg-advcfg -g /VSAN/DOMOwnerForceWarmCache

Value of DOMOwnerForceWarmCache is 0
Set DOMOwnerForceWarmCache to ensure cache based reads occur on both

nodes (Only relevant in Hybrid 2 Node vSAN)
[root@host1:~] esxcfg-advcfg -s 1 /VSAN/DOMOwnerForceWarmCache

Set DOMOwnerForceWarmCache to restore cache based reads behavior (Only

relevant in Hybrid 2 Node vSAN)
[root@host1:~] esxcfg-advcfg -s 0 /VSAN/DOMOwnerForceWarmCache

Get the Sparse Swap setting (Introduced in vSphere 6.0 Update 2, enabled by
default in vSphere 6.5 Update 2 or higher)
[root@host1:~] esxcfg-advcfg -g /VSAN/SwapThickProvisionDisabl
181

vSAN 2 Node Guide
ed
Set the Sparse Swap setting (Introduced in vSphere 6.0 Update 2, enabled by
default in vSphere 6.5 Update 2 or higher)
[root@host1:~] esxcfg-advcfg -s 1 /VSAN/SwapThickProvisionDisa

bled
RVC–Ruby vSphere Console
The following are the new stretched cluster RVC commands:

vsan.stretchedcluster.config_witness
Configure a witness host. The name of the cluster, the witness
host and the preferred fault domain must all be provided as a
rguments.
/localhost/Site-A/compu ters> vsan.stretchedcluster.config_wi

tness -h
usage: config_witness cluster witness_host preferred_fault
_domain
Configure witness host to form a vSAN Stretched Cluster
cluster: A cluster with vSAN enabled
witness_host: Witness host for the stretched cluster
preferred_fault_domain: preferred fault domain for witness
host
--help, -h: Show this message
/localhost/Site-A/computers>
vsan.stretchedcluster.remove_witness
Remove a witness host. The name of the cluster must be provided as an
argument to the command.
/localhost/Site-A/compu ters> vsan.stretchedcluster.remove_wi

tness -h
usage: remove_witness cluster
Remove witness host from a vSAN Stretched Cluster
cluster: A cluster with vSAN stretched cluster enabled
--help,-h: Show this message
vsan.stretchedcluster.witness_info
Display information about a witness host. Takes a cluster as an argument.
/localhost/Site-A/compu ter s> ls
182

vSAN 2 Node Guide
0 Site-A (cluster): cpu 100 GHz, memory 241 GB

1 cs-ie-dell04.ie.local (standalone): cpu 33 GHz, memory 81
GB
/localhost/Site-A/compu ters> vsan.stretchedcluster.witness_in
fo 0
Found witness host for vSAN stretched cluster.
+------------------------+
--------------------------------------+
| Stretched Cluster | Site-A |
+------------------------+-----------------------------------
---+
| Witness Host Name | cs-ie-dell04.ie.local
|
| Witness Host UUID | 55684ccd-4ea7-002d-c3a 9-ecf4bbd59
370 |
| Preferred Fault Domain | Preferred
|
| Unicast Agent Address | 172.3.0.16
|
+------------------------+-----------------------------------
---+
183

VSAN 2 Node Guide

Uploaded by

Document Informationclick to expand document informationvmware vsan document for vsan 2 nodes cluster

Document Informationclick to expand document information

Copyright:

Available Formats

VSAN 2 Node Guide

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

VSAN 2 Node Guide

Uploaded by

Copyright:

Available Formats

vSAN 2 Node Guide

First Published On: 03-07-2018

Copyright © 2019 VMware, Inc. All rights reserved.

Copyright © 2019 VMware, Inc. All rights reserved.

Copyright © 2019 VMware, Inc. All rights reserved.

Copyright © 2019 VMware, Inc. All rights reserved.

VMware vSAN 2 Node with a Witness Host refers to a deployment where a

Each node is configured as a vSAN Fault Domain. The supported configuration

Copyright © 2019 VMware, Inc. All rights reserved.

Copyright © 2019 VMware, Inc. All rights reserved.

Copyright © 2019 VMware, Inc. All rights reserved.

VMware vSAN 2 Node Cluster configurations require vSphere 6.0 Update 1

Advanced feature support in 2 Node vSAN requires a combination of vSAN

v2 On- v2 On- v2 On- v2 On- v2 On-

v3 On- v3 On- v3 On-

v3 On- v3 On- v3 On- v3 On-

v3 On- v3 On- v3 On- v3 On-

Copyright © 2019 VMware, Inc. All rights reserved.

Witness vSphere vSphere vSphere

v3 On- v3 On- v3 On-

Copyright © 2019 VMware, Inc. All rights reserved.

The latest On-Disk format is generally recommended per version of vSAN.

2. 2 vSphere & vSAN

Note: DRS is only available in Enterprise edition and higher of vSphere.

2. 3 Hybrid and All-Flash Support

VMware vSAN 2 Node Clusters are supported on both Hybrid configurations

2. 4 vSAN Witness Host

vSAN Witness Hosts (physical or the vSAN Witness Appliance) cannot be

Copyright © 2019 VMware, Inc. All rights reserved.

shared between multiple vSAN 2 Node Clusters.

2. 5 Supported on vSAN but not vSAN 2 Node Clusters

The following are limitations on a vSAN 2 Node Cluster implementation:

Support statements specific to using vSAN 2 Node Cluster implementations:

SMP-FT, the new Fault Tolerant VM mechanism introduced in vSphere 6.0:

Copyright © 2019 VMware, Inc. All rights reserved.

3. New Concepts in vSAN - 2 Node

Copyright © 2019 VMware, Inc. All rights reserved.

3. 1 The Witness Host

vSAN Witness Host Purpose

vSAN Witness Host Connectivity

These will be covered more thoroughly in a later section.

Updating the vSAN Witness Appliance

The vSAN Witness Appliance can easily be maintained/patched using vSphere

Copyright © 2019 VMware, Inc. All rights reserved.

3. 2 Read Locality in vSAN 2 Node Clusters

In traditional vSAN clusters, a virtual machine’s read operations are distributed

This is not significant to consider in All-Flash configurations, but should be

Writes Are Synchronous

Copyright © 2019 VMware, Inc. All rights reserved.

clusters do not use the cache devices for read caching.

Default Stretched Cluster / 2 Node Read Behavior

This is advantageous in situations where the two sides of a Stretched Cluster

2 Node Reads after vMotion

Read operations after a vMotion, are going to behave differently.

Copyright © 2019 VMware, Inc. All rights reserved.

2 Node Reads after a Disk Failure

Copyright © 2019 VMware, Inc. All rights reserved.

2 Node Reads after a Cache Device/Disk Group Failure