Nutanix TechNote-VMware VSphere Networking With Nutanix

VMware vSphere
Networking
Nutanix Best Practices
Version 1.1May 2017BP-2074

VMware vSphere Networking
Copyright
Copyright 2017 Nutanix, Inc.
Nutanix, Inc.
1740 Technology Drive, Suite 150
San Jose, CA 95110
All rights reserved. This product is protected by U.S. and international copyright and intellectual
property laws.
Nutanix is a trademark of Nutanix, Inc. in the United States and/or other jurisdictions. All other
marks and names mentioned herein may be trademarks of their respective companies.
Copyright | 2
Contents
1. Executive Summary................................................................................ 5
2. Nutanix Enterprise Cloud Platform Overview.......................................6

2.1. Nutanix Acropolis Overview...........................................................................................6
2.2. Distributed Storage Fabric.............................................................................................7
2.3. App Mobility Fabric........................................................................................................7
2.4. AHV................................................................................................................................8
2.5. Third-Party Hypervisors................................................................................................. 8
2.6. Nutanix Acropolis Architecture...................................................................................... 8
3. Introduction to VMware vSphere Networking.....................................11

3.1. vSphere Standard Switch............................................................................................ 11
3.2. vSphere Distributed Switch......................................................................................... 11
3.3. VMware NSX............................................................................................................... 13
4. Discovery Protocols..............................................................................14
5. Network I/O Control (NIOC).................................................................. 16

5.1. NIOC Version 2........................................................................................................... 16
5.2. vSphere 6.x: NIOC Version 3......................................................................................19
5.3. NIC Teaming and Failover.......................................................................................... 20
5.4. Security........................................................................................................................ 23
5.5. Virtual Networking Configuration Options....................................................................23
5.6. Jumbo Frames.............................................................................................................26
6. Multicast Filtering in vSphere 6.0........................................................30
7. Conclusion............................................................................................. 31
Appendix......................................................................................................................... 32
References.......................................................................................................................... 32
3
About the Authors............................................................................................................... 32

About Nutanix......................................................................................................................33
List of Figures................................................................................................................34
List of Tables................................................................................................................. 35
4
1. Executive Summary
With the VMware vSphere hypervisor, you can dynamically configure, balance, or share logical
networking components across various traffic types. Therefore, its imperative that we take virtual
networking into consideration when architecting a network solution for Nutanix hyperconverged
appliances.
This best practices guide addresses the fundamental features of VMwares vSphere switching
technology and details the configuration elements you need to run vSphere on the Nutanix
Enterprise Cloud Platform. The guidance in this document can help you quickly and easily
develop an appropriate design strategy for deploying the Nutanix hyperconverged platform with
VMware vSphere 6.x as the hypervisor.
Table 1: Document Version History
Version Number Published Notes

1.0 March 2017 Original publication.
May 2017 Updated platform overview, virtual networking
1.1
option diagrams, and Nutanix VLAN IDs.
1. Executive Summary | 5
2. Nutanix Enterprise Cloud Platform Overview
2.1. Nutanix Acropolis Overview

Nutanix delivers a hyperconverged infrastructure solution purpose-built for virtualization and
cloud environments. This solution brings the performance and economic benefits of web-scale
architecture to the enterprise through the Nutanix Enterprise Cloud Platform, which is composed
of two product familiesNutanix Acropolis and Nutanix Prism.
Attributes of this solution include:
Storage and compute resources hyperconverged on x86 servers.
System intelligence located in software.
Data, metadata, and operations fully distributed across entire cluster of x86 servers.
Self-healing to tolerate and adjust to component failures.
API-based automation and rich analytics.
Simplified one-click upgrade.
Native file services for hosting user profiles.
Native backup and disaster recovery solutions.
Nutanix Acropolis provides data services and can be broken down into three foundational
components: the Distributed Storage Fabric (DSF), the App Mobility Fabric (AMF), and AHV.
Prism furnishes one-click infrastructure management for virtual environments running on
Acropolis. Acropolis is hypervisor agnostic, supporting three third-party hypervisorsESXi,
Hyper-V, and XenServerin addition to the native Nutanix hypervisor, AHV.
2. Nutanix Enterprise Cloud Platform Overview | 6

Figure 1: Nutanix Enterprise Cloud Platform
2.2. Distributed Storage Fabric

The Distributed Storage Fabric (DSF) delivers enterprise data storage as an on-demand
service by employing a highly distributed software architecture. Nutanix eliminates the need for
traditional SAN and NAS solutions while delivering a rich set of VM-centric software-defined
services. Specifically, the DSF handles the data path of such features as snapshots, clones, high
availability, disaster recovery, deduplication, compression, and erasure coding.
The DSF operates via an interconnected network of Controller VMs (CVMs) that form a Nutanix
cluster, and every node in the cluster has access to data from shared SSD, HDD, and cloud
resources. The hypervisors and the DSF communicate using the industry-standard NFS, iSCSI,
or SMB3 protocols, depending on the hypervisor in use.
2.3. App Mobility Fabric

The Acropolis App Mobility Fabric (AMF) collects powerful technologies that give IT professionals
the freedom to choose the best environment for their enterprise applications. The AMF
encompasses a broad range of capabilities for allowing applications and data to move freely
between runtime environments, including between Nutanix systems supporting different
hypervisors, and from Nutanix to public clouds. When VMs can migrate between hypervisors (for
example, between VMware ESXi and AHV), administrators can host production and development
or test environments concurrently on different hypervisors and shift workloads between them as
needed. AMF is implemented via a distributed, scale-out service that runs inside the CVM on
every node within a Nutanix cluster.

2.4. AHV
Nutanix ships with AHV, a built-in enterprise-ready hypervisor based on a hardened version of
proven open source technology. AHV is managed with the Prism interface, a robust REST API,
and an interactive command-line interface called aCLI (Acropolis CLI). These tools combine to
eliminate the management complexity typically associated with open source environments and
allow out-of-the-box virtualization on Nutanixall without the licensing fees associated with other
hypervisors.
2.5. Third-Party Hypervisors

In addition to AHV, Nutanix Acropolis fully supports Citrix XenServer, Microsoft Hyper-V, and
VMware vSphere. These options give administrators the flexibility to choose a hypervisor that
aligns with the existing skillset and hypervisor-specific toolset within their organization. Unlike
AHV, however, these hypervisors may require additional licensing and, by extension, incur
additional costs.
2.6. Nutanix Acropolis Architecture

Acropolis does not rely on traditional SAN or NAS storage or expensive storage network
interconnects. It combines highly dense storage and server compute (CPU and RAM) into a
single platform building block. Each building block is based on industry-standard Intel processor
technology and delivers a unified, scale-out, shared-nothing architecture with no single points of
failure.
The Nutanix solution has no LUNs to manage, no RAID groups to configure, and no complicated
storage multipathing to set up. All storage management is VM-centric, and the DSF optimizes I/
O at the VM virtual disk level. There is one shared pool of storage composed of either all-flash
or a combination of flash-based SSDs for high performance and HDDs for affordable capacity.
The file system automatically tiers data across different types of storage devices using intelligent
data placement algorithms. These algorithms make sure that the most frequently used data is
available in memory or in flash for optimal performance. Organizations can also choose flash-
only storage for the fastest possible storage performance. The following figure illustrates the data
I/O path for a write in a hybrid model (mix of SSD and HDD disks).

Figure 2: Information Life Cycle Management
The figure below shows an overview of the Nutanix architecture, including the hypervisor of
your choice (AHV, ESXi, Hyper-V, or XenServer), user VMs, the Nutanix storage CVM, and its
local disk devices. Each CVM connects directly to the local storage controller and its associated
disks. Using local storage controllers on each host localizes access to data through the DSF,
thereby reducing storage I/O latency. Moreover, having a local storage controller on each
node ensures that storage performance as well as storage capacity increase linearly with node
addition. The DSF replicates writes synchronously to at least one other Nutanix node in the
system, distributing data throughout the cluster for resiliency and availability. Replication factor
2 creates two identical data copies in the cluster, and replication factor 3 creates three identical
data copies.
Figure 3: Overview of the Nutanix Architecture

Local storage for each Nutanix node in the architecture appears to the hypervisor as one large
pool of shared storage. This allows the DSF to support all key virtualization features. Data
localization maintains performance and quality of service (QoS) on each host, minimizing the
effect noisy VMs have on their neighbors performance. This functionality allows for large, mixed-
workload clusters that are more efficient and more resilient to failure than traditional architectures
with standalone, shared, and dual-controller storage arrays.
When VMs move from one hypervisor to another, such as during live migration or a high
availability (HA) event, the now local CVM serves a newly migrated VMs data. While all write I/
O occurs locally, when the local CVM reads old data stored on the now remote CVM, the local
CVM forwards the I/O request to the remote CVM. The DSF detects that I/O is occurring from a
different node and migrates the data to the local node in the background, ensuring that all read I/
O is served locally as well.
The next figure shows how data follows the VM as it moves between hypervisor nodes.
Figure 4: Data Locality and Live Migration

3. Introduction to VMware vSphere Networking

VMware vSphere supports two main types of virtual switches: the vSphere Standard Switch
(vSwitch) and the vSphere Distributed Switch (vDS). Both switches provide basic functionality,
which includes the ability to forward layer 2 frames, provide 802.1q VLAN encapsulation, manage
traffic shape, and utilize more than one uplink (NIC teaming). The main differences between
these two virtual switch types pertain to switch creation, management, and the more advanced
functionality they provide.
3.1. vSphere Standard Switch

The vSwitch is available in all versions of VMware vSphere and is the default method for
connecting VMs, as well as management and storage traffic, to the external (or physical)
network. The vSwitch is part of the standard vSphere licensing model and comes with the
following advantages and disadvantages.
Advantages:
Simple and easy to configure.
No reliance on VirtualCenter (vCenter) availability.
Disadvantages:
No centralized management.
No support for Network I/O Control (NIOC).
No automated network load balancing.
No backup, restore, health check, or network visibility capabilities.
3.2. vSphere Distributed Switch

To address many of the vSwitchs disadvantages, the vDS separates the networking data
plane from the control plane, enabling advanced networking features such as load balancing,
traffic prioritization, backup and restore capabilities, and so on. These features can certainly be
compelling to an organization, but its important to note the following disadvantages of the vDS:
Requires Enterprise Plus licensing from VMware.
Because VMware vCenter stores the control plane in this configuration, you must have
vCenter server availability to configure and manage the vDS.
Complexity of management configuration.
3. Introduction to VMware vSphere Networking | 11

The vDS has matured over the several version iterations of the vSphere product, providing new
functionality as well as changes to existing components and behaviors.
vSphere 5.x provided the following capabilities and enhancements:
Inter-VM traffic visibility via the Netflow protocol.
LLDP (link layer discovery protocol) support, to provide upstream switch visibility.
Enhanced link aggregation support, including a variety of hashing algorithms.
Single-root I/O virtualization (SR-IOV) support.
Support for 40 Gb network interface cards (NICs).
Network I/O Control (NIOC) version 2.
In vSphere 6.x, VMware has introduced improvements as well as new features and functionality,
including:
NIOC version 3.
Support for IGMP snooping for IPv4 packets and MLD snooping for IPv6 packets.
Multiple TCP/IP stacks for vMotion.
The following table concisely compares the vSwitch and the vDS.
Table 2: Summary Comparison between vSwitch and vDS
Switch Management Private

License NIOC Teaming LLDP LACP
Type Method VLANs
Originating port ID
IP hash
Host or
vSwitch Standard No No Source MAC hash No No
vCenter
Explicit failover
order

Switch Management Private

License NIOC Teaming LLDP LACP
Type Method VLANs
Originating port ID
IP hash
Source MAC hash
Enterprise
vDS vCenter Yes Yes Yes Yes
Plus Explicit failover
order
Route based on
physical NIC load
3.3. VMware NSX

Running VMware NSX for vSphere on Nutanix improves control and increases the agility of the
compute, storage, virtualization, and network layers. Nutanix has evaluated and tested VMware
NSX for vSphere and developed a technical guide for customers who want to move toward SDN
architecture with Nutanix.

4. Discovery Protocols
From vSphere 5.0 onward, the vDS has supported two discovery protocols: LLDP and CDP
(Cisco discovery protocol). Discovery protocols give vSphere administrators visibility into
connectivity from the virtual network to the physical network, which simplifies troubleshooting in
the event of cabling issues, MTU (maximum transmission unit) or packet fragmentation, or other
problems. The only disadvantage to using discovery protocols is a potential security issue, as
they make both the topology of the network and switch-specific information visible.
VMware vSphere offers three configuration options or attributes for LLDP, as presented in the
table below. Each option varies the information that the selected discovery protocol sends and
receives. CDP is either enabled or notit does not have additional options.
Table 3: LLDP Discovery Options
Attribute Description
In this mode, ESXi detects and displays information from the upstream switch,
Listen
but it does not advertise the vDS configuration to the upstream network device.
In this mode, ESXi advertises information about the vDS to the switch
Advertise administrator, but it does not detect or display information about the physical
switch.
This attribute listens and advertises information between the vDS and upstream
Both
switch provider.
Note: Nutanix recommends the "Both" option, which ensures that ESXi collects and
displays information from the physical switch and sends information about the vDS to
the physical switch.
The following information is visible in the vSphere client when enabling discovery protocol data:
The physical switch interface connected to the dvUplink.
MTU size (in other words, whether or not jumbo frames are enabled).
The switch management IP address.
The switch name, description, software version, and capabilities.
4. Discovery Protocols | 14
The following table presents some general guidelines, but Nutanix recommends that all
customers carefully consider the advantages and disadvantages of discovery protocols for their
specific security and discovery requirements.
Table 4: Recommendations for Discovery Protocol Configuration
Dependent on your switching infrastructure. Use CDP for Cisco-based

Type
environments and LLDP for non-Cisco environments.
Both CDP and LLDP are generally acceptable in most environments and
provide maximum operational benefits. Configuring an attribute (listen,
Operation advertise, or both) in LLDP allows vSphere and the physical network to openly
exchange information. You can also choose not to configure an attribute in
LLDP, if you dont want this exchange to occur.
4. Discovery Protocols | 15
5. Network I/O Control (NIOC)

NIOC is a vDS feature that has been available since the introduction of vSphere 4.1. The feature
initially used network resource pools to determine the bandwidth that different network traffic
types could receive in the event of contention. With the introduction of vSphere 6.0, we must
now also take shares, reservations, and limits into account. The following sections detail our
recommendations.
5.1. NIOC Version 2

NIOC Version 2 was introduced in vSphere 5.1 and became generally available in vSphere 5.5.
Enabling NIOC divides vDS traffic into the following predefined network resource pools:
Fault tolerance
iSCSI
vMotion
Management
vSphere Replication (VR)
NFS
VM
The physical adapter shares assigned to a network resource pool determine what portion of the
total available bandwidth is guaranteed to the traffic associated with that network resource pool
in the event of contention. It is important to understand that NIOC only impacts network traffic
if there is contention. Thus, when the network is less than 100 percent utilized, NIOC has no
advantage or disadvantage.
Comparing a given network resource pool's shares to the allotments for the other network
resource pools determines its available bandwidth. You can apply limits to selected traffic types,
but Nutanix recommends against it, as limits may inadvertently constrain performance for given
workloads when there is available bandwidth. Using NIOC shares ensures that burst workloads
such as vMotion (migrating a VM to a different ESXi host) can complete as quickly as possible
when bandwidth is available, while also protecting other workloads from significant impact if
bandwidth becomes limited.
The following table shows the Nutanix-recommended network resource pool share values.
Note: None of the network resource pools configure artificial limits or reservations,
so leave the host limit set to unlimited.
5. Network I/O Control (NIOC) | 16

Table 5: Recommended Network Resource Pool Share Values
Network Resource Pool Share Value Physical Adapter Shares

Management traffic 25 Low
vMotion traffic 50 Normal
Fault tolerance traffic 50 Normal
VM traffic 100 High
NFS traffic 100 High
Nutanix CVM traffic (1) 100 High
iSCSI traffic 100 High
vSphere Replication traffic 50 Normal
vSphere Storage Area Network traffic (2) 50 Normal
Virtual SAN traffic (3) 50 Normal
(1) This is a custom network resource pool created manually.

(2) This is exposed in vSphere 5.1, but as it has no function and no impact on NIOC, we can
ignore it.
(3) This is exposed in vSphere 5.5, but as Nutanix environments do not use it, it has no impact
on NIOC here.
The sections below describe our rationale for setting the share values in this table.
Note: The calculations of minimum bandwidth per traffic type presented here
assume that the system is not using iSCSI, vSphere Replication, vSphere Storage
Area Network traffic, or Virtual SAN traffic. Also, we assume that NFS traffic remains
with the host for the default "NFS traffic" resource pool.
Management Traffic
Management traffic requires minimal bandwidth. A share value of 25 (low) and two 10 Gb
interfaces ensure a minimum of approximately 1.5 Gbps for management traffic, which exceeds
the minimum bandwidth requirement of 1 Gb.
vMotion Traffic
vMotion is a burst-type workload that uses no bandwidth until the distributed resource scheduler
(DRS) (a vCenter service that actively rebalances VMs across hosts) invokes it or a vSphere

administrator starts a vMotion or puts a host into maintenance mode. As such, it is unlikely to
have any significant ongoing impact on the network traffic. A share value of 50 (normal) and two
10 Gb interfaces provide a minimum of approximately 3 Gbps, which is sufficient to complete
vMotion in a timely manner without degrading VM or storage performance and well above the
minimum bandwidth requirement of 1 Gb.
Fault Tolerance Traffic

Fault tolerance (FT) traffic depends on the number of fault-tolerant VMs you have per host
(current maximum is four per host). FT is generally a consistent workload (as opposed to a burst-
type workload like vMotion), as it needs to keep the primary and secondary VMs CPU resources
synchronized. We typically use FT for critical VMs, so to protect FT traffic (which is also sensitive
to latency) from impact during periods of contention, we use a share value of 50 (normal) and
two 10 Gb interfaces. This setting provides a minimum of 3 Gbps, which is well above VMware's
recommended minimum of 1 Gb.
VM Traffic
VM traffic is the reason we have datacenters in the first place, so this traffic is always important,
if not critical. As slow VM network connectivity can quickly impact end users and reduce
productivity, we must ensure that this traffic has a significant share of the available bandwidth
during periods of contention. With a share value of 100 (high) and two 10 Gb interfaces, VM
traffic receives a minimum of approximately 6 Gbps. This bandwidth is more than what is
required in most environments and ensures a good amount of headroom in case of unexpected
burst activity.
NFS Traffic
NFS traffic is always critical, as its essential to the Acropolis DSF and to CVM and VM
performance, so it must receive a significant share of the available bandwidth during periods
of contention. However, as NFS traffic is serviced locally, it does not impact the physical
network card unless the Nutanix CVM fails or is offline for maintenance. Thus, under normal
circumstances, no NFS traffic crosses the physical NICs, so the NIOC share value has no impact
on other traffic. As such, we have excluded it from our calculations.
In case of network contention, CVM maintenance, or failure, we can assign NFS traffic a share
value of 100 (high) as a safety measure, ensuring a minimum of 6 Gbps bandwidth.
Nutanix CVM Traffic

For the Acropolis DSF to function, it must have connectivity to the other CVMs in the cluster for
tasks such as synchronously writing I/O across the cluster and Nutanix cluster management. It
is important to note that, under normal circumstances, minimal or no read I/O traffic crosses the
physical NICs; however, write I/O always utilizes the physical network.

To ensure optimal DSF performance in the unlikely circumstance wherein both 10 Gb adapters
are saturated, we assign a share value of 100 (high), guaranteeing each CVM a minimum of
6 Gbps bandwidth. This bandwidth is more than what is required in most environments and
ensures a good amount of headroom in case of unexpected burst activity.
iSCSI Traffic
Like NFS traffic, iSCSI traffic is not used by default; however, if you are using it, it may be critical
to your environment. As such, we have given this traffic type a share value of 100.
Note: NIOC does not cover in-guest iSCSI. If you are using in-guest iSCSI, we
recommend creating a dvPortGroup for in-guest iSCSI traffic and assigning it to a
custom network resource pool called "In-Guest iSCSI." Assign this pool a share value
of 100 (high).
vSphere Replication Traffic

In a non-Nutanix environment, vSphere Replication (VR) traffic may be critical to your
environment if you choose to use VR (with or without VMware Site Recovery Manager (SRM)).
However, if you are using SRM, we strongly recommend using the Nutanix Storage Replication
Adaptor (SRA) instead of VR, as it is more efficient. If youre using VR without SRM, the default
share value of 50 (normal) should be suitable for most environments, ensuring approximately 3
Gbps of network bandwidth.
vSphere Storage Area Network and Virtual SAN Traffic

vSphere Storage Area Network traffic is visible as a parameter in vSphere 5.1, but it is not used.
Therefore, simply leave the default share value, as vSphere Storage Area Network traffic has no
impact on NIOC.
Virtual SAN traffic is visible as a parameter from vSphere 5.5 onward, but Nutanix environments
do not use it. Therefore, simply leave the default share value as it is, as Virtual SAN traffic also
has no impact on the environment.
5.2. vSphere 6.x: NIOC Version 3

VMware vSphere versions 6.0 and later employ a newer iteration of NIOC: version 3. NIOC v3
allows an administrator to not only set shares, but also artificial limits and reservations on system
traffic such as vMotion, management, NFS, VMs, and so on. Administrators can apply limits and
reservations to a VMs network adapter via the same constructs they used when allocating CPU
and memory resources to the VM. Admission control is now part of the vDS, which integrates
with VMware HA and DRS to actively balance network resources across hosts.

Although setting artificial limits and reservations on various traffic types guarantees quality of
service (QoS) and SLAs, it prevents other workloads from using the total available bandwidth.
Reserving a certain amount of bandwidth strictly for one workload or purpose constrains the
networks ability to sustain an unreserved workloads short bursts of traffic.
In the course of testing and evaluating the functionality of NIOC v3, Nutanix has encountered
various abnormalities in its usability and behavior as well as generally higher CPU overhead.
Because of these results, we recommend that customers implement NIOC v2 instead of NIOC
v3.
To implement NIOC v2, provision a vDS with 5.5.0 selected as the Distributed Switch version,
following the proportional shares we described above for NIOC v2. The section below offers
detailed instructions for provisioning your vDS.
Creating a vSphere Distributed Switch with NIOC Version 2 Compatibility

For guidance on creating a new vDS on your cluster and configuring its settings, please refer to
the VMware vSphere 5.5 Documentation Center.
5.3. NIC Teaming and Failover

vSphere provides several NIC teaming and failover options. Each of these settings has different
goals and requirements, and we need to understand and carefully consider them before using
them in our vSphere deployments.
The available options are:
1. Route based on originating virtual port.
2. Route based on physical NIC load (also called load-based teaming or LBT).
a. Only available with the vDS.
3. Route based on IP hash.
4. Route based on source MAC hash.
5. Use explicit failover order.
Recommendation for vSwitch: Route Based on Originating Virtual Port

The Route based on originating virtual port option is the default load balancing policy and
does not require an advanced switching configuration such as LACP, Cisco EtherChannel, or HP
teaming, making it simple to implement, maintain, and troubleshoot. It only requires 802.1q VLAN
tagging for secure separation of traffic types. This options main disadvantage is that it does not
support load balancing based on network load, so the system always sends traffic from a single
VM to the same physical NIC unless a NIC or upstream link failure causes a failover event.

Recommendation for vDS: Route Based on Physical NIC Load

For environments using vDS, the Route based on physical NIC load option (LBT) also does not
require an advanced switching configuration such as LACP, Cisco Ether channel, or HP teaming.
It offers fully automated load balancing, which takes effect when one or more NICs reach and
sustain 75 percent utilization for a period of at least 30 seconds, based on egress and ingress
traffic. LBTs only requirement is 802.1q VLAN tagging for secure separation of traffic types, so it
is also simple to implement, maintain, and troubleshoot.
Note: The vDS requires VMware vSphere Enterprise Plus licensing.
LBT is a very simple and effective solution that works very well in Nutanix deployments.
Network Failover Detection

ESXi uses one of two network failover detection methods: beacon probing or link status.
Beacon probing sends out and listens for beacon probes, which are made up of broadcast
frames. Because beacon probing must have three network connections to function, we do not
recommend it for Nutanix solutions, which currently have two NICs of each speed (1 Gb and 10
Gb).
Link status depends on the state that the physical NIC reports for the link. Link status can detect
failures such as a cable disconnection or a physical switch power failures, but it cannot detect
configuration errors or upstream failures which that may also result in connectivity issues for
VMs.
To avoid these link status limitations related to upstream failures, enable Link state tracking
(if the physical switches support it) or an equivalent, which then enables the switch to pass
upstream link state information back to ESXi which allows link status to trigger a link down on
ESXi where appropriate.
Notify Switches
The purpose of the notify switches policy setting is to enable or disable communication by ESXi
with the physical switch in the event of a failover. Choosing Yes for this policy setting means that
when a failover event routes a virtual Ethernet adapters traffic over a different physical Ethernet
adapter in the team, ESXi sends a notification to the physical switch to update the lookup tables.
This configuration ensures that failover occurs in a timely manner and with minimal interruption to
network connectivity.
Failback
For customers not using Enterprise Plus or the vDS with load-based teaming, failback can help
rebalance network traffic across the original NIC, which may improve network performance.
The only significant disadvantage of setting failback to Yes is that, in the unlikely event of

network instability (or "flapping"), having network traffic fail back to the original NIC may result in
intermittent or degraded network connectivity.
Nutanix recommends setting failback to Yes when using vSwitch and No when using vDS.
Failover Order
Using failover order allows the vSphere administrator to specify the order in which NICs fail
over by assigning a pNIC to one of three groups: active adapters, standby adapters, or unused
adapters. For example, if all active adapters lose connectivity, the system uses the highest
priority standby adapter, and so on. This feature is only necessary in a Nutanix environment
when performing multi-NIC vMotion.
When configuring multi-NIC vMotion, set the first dvPortGroup used for vMotion to have one
dvUplink active and the other standby. Configure the reverse for the second dvPortGroup used
for vMotion.
For more information, see Multiple-NIC vMotion in vSphere 5 (KB2007467).
Summary of Recommendations for NIC Teaming and Failover
Table 6: Recommendations for NIC Teaming and Failover
vSphere Distributed Switch (vDS)

Load Balancing Route based on physical NIC load (LBT)
Network Failover Detection Link status only (1)
Notify Switches Yes
Failback No
vSphere Standard Switch (vSwitch)
Load Balancing Route based on originating virtual port
Network Failover Detection Link status only (1)
Notify Switches Yes
Failback Yes
(1) Enable link state tracking or the equivalent on physical switches.

5.4. Security
When configuring a vSwitch or vDS, there are three configurable options under security that can
be set to Accept or Reject: promiscuous mode, MAC address changes, and forged transmits.
In general, the most secure and appropriate setting for each of these options is Reject unless
you have a specific requirement for Accept. Two examples of situations in which you might
consider configuring Accept on forged transmits and MAC address changes are:
1. Microsoft load balancing in Unicast mode.
2. iSCSI deployments on select storage types.
The following table offers general guidelines, but Nutanix recommends that all customers
carefully consider their requirements for specific applications.
Table 7: Recommendations for VMware Virtual Networking Security
Promiscuous Mode Reject

MAC Address Changes Reject
Forged Transmits Reject
5.5. Virtual Networking Configuration Options

The following two virtual networking options cover the Nutanix recommended configurations for
both vSwitch and vDS solutions. For each option, we discuss the advantages, disadvantages,
and example use cases. In both of the options below, Nutanix recommends setting all vNICs as
active on the portgroup and dvPortGroup unless otherwise specified.
The following table defines our naming convention, portgroups, and corresponding VLANs as
used in the examples below for various traffic types.
Table 8: Portgroups and Corresponding VLANs
Portgroup VLAN Description

MGMT_10 10 VM kernel interface for host management traffic
VMOT_20 20 VM kernel interface for vMotion traffic
FT_30 30 Fault tolerance traffic

Portgroup VLAN Description

VM_40 40 VM traffic
VM_50 50 VM traffic
Nutanix CVM to CVM cluster communication traffic (public
NTNX_10 10
interface)
Svm-iscsi-pg N/A Nutanix CVM to internal host traffic
VM kernel port for CVM to hypervisor communication
VMK-svm-iscsi-pg N/A
(internal)
All Nutanix deployments use an internal-only vSwitch for the NFS communication between the
ESXi host and the Nutanix CVM. This vSwitch remains unmodified regardless of the virtual
networking configuration for ESXi management, VM traffic, vMotion, and so on.
Tip: Do not modify the internal-only vSwitch (vSwitch-Nutanix). vSwitch-Nutanix

facilitates communication between the CVM and the internal hypervisor.
The configurations for vSwitch and vDS solutions look very similarthe main difference is the
location of the control plane.
With the default vSwitch, there is a control plane on each host, and each operates
independently. This independence requires an administrator to configure, maintain, and
operate each individual vSwitch separately.
With the vDS, the control plane is located within vCenter. This centrality requires the system to
keep vCenter online to ensure that configuration and maintenance activities propagate to the
ESXi hosts that are part of the vDS instance.
Both options offer the advantage of reduced cabling and switching requirements (because they
do not require 1 GbE ports). Moreover, they represent a simple solution that only requires 802.1q
configured on the physical network. These options are suitable for environments where logical
separation of management and VM traffic is acceptable.
Option 1: vSphere Standard Switch with Nutanix vSwitch Deployment

Option 1 is the default configuration for a Nutanix deployment and suits most use cases. This
option is a good fit for customers who do not have VMware vSphere Enterprise Plus licensing or
those who prefer not to use the vDS.
Customers may wish to incorporate the 1 GbE ports into the vSwitch as a means of providing
additional redundancy; however, this configuration requires additional cabling and switch ports. If
you connect the 1 GbE ports, set them to Standby and configure Failback as noted above.

Option 1 offers several benefits; the most important benefit is that the control plane is located
locally on every ESXi host, so there is no dependency on vCenter.
A major disadvantage of the vSwitch lies in its load balancing capabilities. Although both physical
uplinks are active, traffic across these pNICs does not actively rebalance unless the VM has
gone through a power cycle or a failure has occurred in the networking stack. Furthermore, a
standard vSwitch cannot manage contention or prioritize network traffic across various traffic
classes (for example, storage, vMotion, VM, and so on).
Figure 5: Virtual Networking Option 1: Single vSwitch with Nutanix Internal-Only vSwitch
Recommended use cases:

1. When vSphere licensing is not Enterprise Plus.
2. When physical and virtual server deployments are not large enough to warrant a vDS.
Option 2: vSphere Distributed Switch (vDS) with Nutanix vSwitch Deployment

Option 2 is a good fit for customers using VMware vSphere Enterprise Plus licensing and offers
several benefits, including:
Advanced networking features, such as:
Network I/O Control (NIOC).
Load-based teaming (Route based on physical NIC load).
Actively rebalancing flows across physical NICs.
Ability for all traffic types to "burst" where required, up to 10 Gbps.
Effective with both egress and ingress traffic.
Lower overhead than IP hash load balancing.

Reduced need for NIOC to intervene during traffic congestion.

Because the control plane is located within vCenter, vDS administration requires vCenter to
be highly available and protected, as its corresponding database stores all configuration data.
Particularly during disaster recovery operations, the requirement for vCenter availability can pose
additional challenges; many administrators see this constraint as the primary disadvantage to
using the vDS.
Recommended use cases:
1. When vSphere licensing is Enterprise Plus.
2. Where there is a requirement for corresponding VMware products or solutions.
3. Large physical and virtual server deployments.
Figure 6: Virtual Networking Option 2: vDS with Nutanix Internal-Only vSwitch
5.6. Jumbo Frames

Jumbo frames are Ethernet frames with a payload greater than 1,500 bytes. Typically,
administrators set jumbo frame payloads to 9,000, which is the maximum payload size for an
Ethernet frame.
Jumbo frames are useful because, as network speeds increase, a 1,500-byte frame is
unnecessarily small. With solutions such as IP-based storage (iSCSI or NFS) using converged

networks, larger frames assist with both greater throughput and reduced network overhead.
By default, the network moves large numbers of 1,500-byte frames; however, with larger,
9,000-byte frames, the network can process six times fewer packets, thus reducing the CPU
overhead. Solutions such as VXLAN require a payload greater than 1,500 bytes; thus, VMware
recommends using jumbo frames for VXLAN to avoid packet fragmentation.
When considering jumbo frames, be sure to confirm that all network devices support them end-
to-end and that you can enable this support on all switches globally. If you can confirm both of
these capabilities, consider using jumbo frames.
Note: You can choose to enable jumbo frames for only specific traffic types;
however, ensure it is configured end-to-end
Nutanix AOS 4.0.3 or Later

Beginning with AOS 4.0.3, Nutanix does not require jumbo frames, though you can still enable
them if you wish.
Regardless of AOS version, jumbo frames are enabled on the internal adapters for both the CVM
and the host. As this is a network internal to the host, it does not require any modifications to the
physical networking components.
Example End-to-End Configuration with Jumbo Frames

The following diagram shows an MTU size of 9,216 configured on the physical network, with two
ESXi hosts configured with jumbo frames where appropriate.
Figure 7: End-to-End Configuration with Jumbo Frames
Tip: Nutanix supports using a standard (not jumbo) MTU of 1,500. Performance
using this MTU is still excellent.
The figure above illustrates how jumbo frames and standard frames can coexist and
communicate in a Nutanix deployment.

Table 9: Traffic Types
Suited to jumbo frames (1) (MTU 9,000) Nutanix CVM internal traffic
vMotion / multi-NIC vMotion
Fault tolerance
iSCSI
NFS
VXLAN (2)
Not suited to jumbo frames (MTU 1,500) ESXi management

VM traffic (3)
(1) Assumes that traffic types are NOT routed.

(2) Minimum MTU supported is 1,524; Nutanix recommends >=1,600.
(3) Can be beneficial in selected use cases, but generally not advantageous.
The following table shows the various traffic types in a Nutanix environment with VMware
vSphere, as well as the Nutanix-recommended MTU.
In summary, jumbo frames offer the following benefits:
Reduced number of interrupts for the switches to process.
Lower CPU utilization on the switches and on Nutanix CVMs.
Increased performance for some workloads, including the Nutanix CVM, vMotion, and fault
tolerance.
Performance with jumbo frames is either the same or better than without them.
Implications of Jumbo Frames

The converged network must be configured for jumbo frames, and administrators must validate
the configuration to ensure that jumbo frames are enabled properly end-to-end, so no packet
fragmentation occurs. Fragmentation typically results in degraded performance, due to the higher
overhead it brings to the network.
The following are general guidelines; Nutanix advises all customers to carefully consider the
requirements for their specific applications. When jumbo frames are supported on all devices in
your network, Nutanix recommends these settings:
Physical switches and interfaces: configure an MTU of 9,216.

VMK for NFS or iSCSI: configure an MTU of 9,000 (if applicable).

VMK for vMotion or fault tolerance: configure an MTU of 9,000 (if applicable).

6. Multicast Filtering in vSphere 6.0

When using vSphere 6.0 or later, administrators have the option to configure advanced multicast
filtering within the vDS. Prior to vSphere 6.0, VMs connected to a vSwitch encountered issues in
forwarding and receiving their multicast trafficVMs that may not have originally subscribed to
the multicast group sending traffic received that traffic anyway.
To overcome these traffic concerns, vSphere 6.0 introduced two configurable filtering modes:
Basic Mode: Applies to the vSwitch and the vDS. This mode forwards multicast traffic for VMs
according to the destination MAC address of the multicast group.
Multicast Snooping: Applies only to the vDS and uses IGMP snooping to route multicast traffic
only to VMs that have subscribed to its source. This mode listens for IGMP network traffic
between VMs and hosts and maintains a map of the groups destination IP address and the
VMs preferred source IP address (IGMPv3).
When a VM does not renew its membership to a group within a certain period of time, the
vDS removes the groups entry from the lookup records.
Note: Enabling multicast snooping on a vDS with a Nutanix CVM attached affects
its ability to discover and add new nodes in the cluster (the Cluster Expand option in
Prism and the Nutanix CLI).
If you need to implement multicast snooping, your standard operational procedures should
include manual steps for adding additional nodes to a cluster. For more information, refer to
Nutanix KB 3493 and VMwares vSphere 6.0 documentation on Multicast Filtering Modes.
6. Multicast Filtering in vSphere 6.0 | 30

7. Conclusion
Virtual networking within a vSphere 6.x environment plays an important role in infrastructure
availability, scalability, performance, management, and security. With hyperconverged
technology, customers need to evaluate their networking requirements carefully against the
various configuration options, implications, and features within the VMware vSphere 6.x
hypervisor.
In most deployments, Nutanix recommends the vSphere Distributed Switch (vDS) coupled with
Network I/O Control (Version 2) and load-based teaming, as this combination provides simplicity,
ensures traffic prioritization in the event of contention, and reduces operational management
overhead.
With a strategically networked vSphere deployment on Nutanix, customers can be confident that
their networking configuration conforms to field-tested best practices.
For feedback or questions, please contact us using the Nutanix NEXT Community forums.
7. Conclusion | 31
Appendix
References
1. VMware vSphere 6.0 Documentation
2. Nutanix Jumbo Frames
3. Performance Evaluation of NIOC in VMware vSphere 6.0
About the Authors

Richard Arsenian is a Senior Solution Architect within the Solutions and Performance R&D
engineering team at Nutanix, Inc. As a Nutanix Platform Expert (NPX #09) and VMware Certified
Expert (VCDX #126), Richard brings a wealth of experience within the datacenter across the
various elements of the systems development life cycle (presales, consulting, and support), due
to his experience working at key technology vendors including VMware and Cisco.
While maintaining his passion within the datacenter, Richard also integrates his R&D efforts
with autonomous aerial and motor vehicles; he most recently released the concept of the
Nutanix Acropolis 1 (patent pending), a platform for delivering applications autonomously, via his
Community Edition with Intel NUC project.
Follow Richard on Twitter @richardarsenian.
Josh Odgers has a proven track record in virtualization and cloud computing based on over
13 years of IT industry experience, including seven years focused on virtualization. His
experience includes leading SMB and enterprise customers through their cloud computing and
hyperconvergence journeys. Josh is a virtualization evangelist and industry thought leader
who enjoys giving back to the community by speaking at industry events and via blogging and
tweeting. Josh has been awarded the titles of vExpert in 2013 and vChampion (20112014).
In his role as a Senior Solutions and Performance Engineer at Nutanix, Josh is responsible for
developing reference architectures for solutions, mainly focused on virtualizing business-critical
applications. Josh works with customers at all levels of business, including C-level executives
and senior management, to ensure that the chosen solution delivers technical benefits and
excellent return on investment (ROI).
Follow Josh on Twitter @josh_odgers.
Appendix | 32
About Nutanix
Nutanix makes infrastructure invisible, elevating IT to focus on the applications and services that
power their business. The Nutanix Enterprise Cloud Platform leverages web-scale engineering
and consumer-grade design to natively converge compute, virtualization, and storage into
a resilient, software-defined solution with rich machine intelligence. The result is predictable
performance, cloud-like infrastructure consumption, robust security, and seamless application
mobility for a broad range of enterprise applications. Learn more at www.nutanix.com or follow up
on Twitter @nutanix.
Appendix | 33
List of Figures
Figure 1: Nutanix Enterprise Cloud Platform.....................................................................7
Figure 2: Information Life Cycle Management.................................................................. 9
Figure 3: Overview of the Nutanix Architecture................................................................ 9
Figure 4: Data Locality and Live Migration......................................................................10
Figure 5: Virtual Networking Option 1: Single vSwitch with Nutanix Internal-Only vSwitch 25
Figure 6: Virtual Networking Option 2: vDS with Nutanix Internal-Only vSwitch..............26
Figure 7: End-to-End Configuration with Jumbo Frames................................................ 27
34
List of Tables
Table 1: Document Version History.................................................................................. 5
Table 2: Summary Comparison between vSwitch and vDS............................................12
Table 3: LLDP Discovery Options................................................................................... 14
Table 4: Recommendations for Discovery Protocol Configuration.................................. 15
Table 5: Recommended Network Resource Pool Share Values.....................................17
Table 6: Recommendations for NIC Teaming and Failover............................................ 22
Table 7: Recommendations for VMware Virtual Networking Security............................. 23
Table 8: Portgroups and Corresponding VLANs............................................................. 23
Table 9: Traffic Types......................................................................................................28
35

Nutanix TechNote-VMware VSphere Networking With Nutanix

Uploaded by

Copyright:

Available Formats

Nutanix TechNote-VMware VSphere Networking With Nutanix

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Nutanix TechNote-VMware VSphere Networking With Nutanix

Uploaded by

Copyright:

Available Formats

VMware vSphere

Version 1.1May 2017BP-2074

2. Nutanix Enterprise Cloud Platform Overview.......................................6

3. Introduction to VMware vSphere Networking.....................................11

5. Network I/O Control (NIOC).................................................................. 16

6. Multicast Filtering in vSphere 6.0........................................................30

About the Authors............................................................................................................... 32

Table 1: Document Version History

Version Number Published Notes

2. Nutanix Enterprise Cloud Platform Overview

2.1. Nutanix Acropolis Overview

2. Nutanix Enterprise Cloud Platform Overview | 6

Figure 1: Nutanix Enterprise Cloud Platform

2.2. Distributed Storage Fabric

2.3. App Mobility Fabric

2. Nutanix Enterprise Cloud Platform Overview | 7

2.5. Third-Party Hypervisors

2.6. Nutanix Acropolis Architecture

2. Nutanix Enterprise Cloud Platform Overview | 8

Figure 2: Information Life Cycle Management

Figure 3: Overview of the Nutanix Architecture

2. Nutanix Enterprise Cloud Platform Overview | 9

Figure 4: Data Locality and Live Migration

2. Nutanix Enterprise Cloud Platform Overview | 10

3. Introduction to VMware vSphere Networking

3.1. vSphere Standard Switch

3.2. vSphere Distributed Switch

3. Introduction to VMware vSphere Networking | 11

Table 2: Summary Comparison between vSwitch and vDS

Switch Management Private

3. Introduction to VMware vSphere Networking | 12

Switch Management Private

3.3. VMware NSX

3. Introduction to VMware vSphere Networking | 13

Table 3: LLDP Discovery Options

Table 4: Recommendations for Discovery Protocol Configuration

Dependent on your switching infrastructure. Use CDP for Cisco-based

5. Network I/O Control (NIOC)

5.1. NIOC Version 2

5. Network I/O Control (NIOC) | 16

Table 5: Recommended Network Resource Pool Share Values

Network Resource Pool Share Value Physical Adapter Shares

(1) This is a custom network resource pool created manually.

5. Network I/O Control (NIOC) | 17

Fault Tolerance Traffic

Nutanix CVM Traffic

5. Network I/O Control (NIOC) | 18

vSphere Replication Traffic

vSphere Storage Area Network and Virtual SAN Traffic

5.2. vSphere 6.x: NIOC Version 3

5. Network I/O Control (NIOC) | 19

Creating a vSphere Distributed Switch with NIOC Version 2 Compatibility

5.3. NIC Teaming and Failover

Recommendation for vSwitch: Route Based on Originating Virtual Port

5. Network I/O Control (NIOC) | 20

Recommendation for vDS: Route Based on Physical NIC Load

Note: The vDS requires VMware vSphere Enterprise Plus licensing.

Network Failover Detection

5. Network I/O Control (NIOC) | 21

Summary of Recommendations for NIC Teaming and Failover