Mellanox OFED Linux Release Notes 3 4-1 0 0 0

Mellanox OFED for Linux Release Notes
Rev 3.4-1.0.0.0
www.mellanox.com
Rev 3.4-
NOTE:
THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED
DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY
KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE
THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST ENVIRONMENT
HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCT(S)
AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES CANNOT AND DOES NOT
GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY. ANY
EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED.
IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT,
INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES OF ANY KIND (INCLUDING, BUT NOT
LIMITED TO, PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA,
OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
ARISING IN ANY WAY FROM THE USE OF THE PRODUCT(S) AND RELATED DOCUMENTATION EVEN IF
ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Mellanox Technologies
350 Oakmead Parkway Suite 100
Sunnyvale, CA 94085
U.S.A.
www.mellanox.com
Tel: (408) 970-3400
Fax: (408) 970-3403
© Copyright 2016. Mellanox Technologies Ltd. All Rights Reserved.
Mellanox®, Mellanox logo, Accelio®, BridgeX®, CloudX logo, CompustorX®, Connect-IB®, ConnectX®, CoolBox®,
CORE-Direct®, EZchip®, EZchip logo, EZappliance®, EZdesign®, EZdriver®, EZsystem®, GPUDirect®, InfiniHost®,
InfiniScale®, Kotura®, Kotura logo, Mellanox Federal Systems®, Mellanox Open Ethernet®, Mellanox ScalableHPC®,
Mellanox TuneX®, Mellanox Connect Accelerate Outperform logo, Mellanox Virtual Modular Switch®, MetroDX®,
MetroX®, MLNX-OS®, NP-1c®, NP-2®, NP-3®, Open Ethernet logo, PhyX®, PSIPHY®, SwitchX®, Tilera®, Tilera logo,
TestX®, TuneX®, The Generation of Open Ethernet logo, UFM®, Virtual Protocol Interconnect®, Voltaire® and Voltaire
logo are registered trademarks of Mellanox Technologies, Ltd.
All other trademarks are property of their respective owners.
For the most updated list of Mellanox trademarks, visit http://www.mellanox.com/page/trademarks
2 Mellanox Technologies
Rev 3.4-
Table of Contents
Table of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Release Update History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Chapter 1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.1 Content of Mellanox OFED for Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2 Supported Platforms and Operating Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.2.1 Supported Hypervisors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.2.2 Supported Non-Linux Virtual Machines. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.2.3 Tested Hypervisors in Paravirtualized and SR-IOV Environments . . . . . . . . . . . . 7
1.3 Hardware and Software Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.4 Supported HCAs Firmware Versions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.5 Compatibility Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.6 RoCE Modes Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Chapter 2 Changes and New Features in Rev 3.4-1.0.0.0. . . . . . . . . . . . . . . . . . . . . . . . 11
2.1 Future API Changes in MLNX_OFED . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Chapter 3 Known Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.1 Driver Installation/Loading/Unloading/Start Known Issues . . . . . . . . . . . . . . . . 13
3.1.1 Installation Known Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.1.2 Driver Unload Known Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.1.3 Driver Start Known Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.1.4 System Time Known Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.1.5 UEFI Secure Boot Known Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.2 Performance Known Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.3 HCAs Known Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.3.1 mlx5 Driver Known Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.4 Ethernet Network. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.4.1 Ethernet Known Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.4.2 Port Type Management Known Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3.4.3 Flow Steering Known Issues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3.4.4 Quality of Service Known Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3.4.5 Ethernet Performance Counters Known Issues. . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3.5 InfiniBand Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3.5.1 IPoIB Known Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3.5.2 eIPoIB Known Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.5.3 XRC Known Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.5.4 Verbs Known Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.5.5 RoCE Known Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.5.6 iSCSI over IPoIB Known Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.5.7 SDP Known Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
3.6 Storage Protocols Known Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
3.6.1 Storage Known Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
3.6.2 SRP Known Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
Mellanox Technologies 1
Rev 3.4-
3.6.3 SRP Interop Known Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

3.6.4 DDN Storage Fusion 10000 Target Known Issues . . . . . . . . . . . . . . . . . . . . . . . . 30
3.6.5 Oracle Sun ZFS Storage 7420 Known Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
3.6.6 iSER Initiator Known Issues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
3.6.7 iSER Target Known Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
3.6.8 ZFS Appliance Known Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.6.9 Erasure Coding Verbs Known Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.7 Virtualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.7.1 SR-IOV Known Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.8 Resiliency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.8.1 Reset Flow Known Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.9 Miscellaneous Known Issues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.9.1 General Known Issues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.9.2 ABI Compatibility Known Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
3.9.3 Connection Manager (CM) Known Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
3.9.4 Fork Support Known Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
3.9.5 Uplinks Known Issues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
3.9.6 Resources Limitation Known Issues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
3.9.7 Accelerated Verbs Known Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
3.10 InfiniBand Fabric Utilities Known Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
3.10.1 Performance Tools Known Issues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
3.10.2 Diagnostic Utilities Known Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
3.10.3 Tools Known Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
Chapter 4 Bug Fixes History. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Chapter 5 Change Log History. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
Chapter 6 API Change Log History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
Rev 3.4-1.0.0.0
Release Update History
Release Date Description

Rev 3.4-1.0.0.0 September 29, 2016 Initial release of this version
Rev 3.4-1.0.0.0 Overview
1 Overview
These are the release notes of Mellanox OFED for Linux Driver, Rev 3.4-1.0.0.0 which operates
across all Mellanox network adapter solutions supporting the following uplinks to servers:
Uplink/HCAs Uplink Speed
ConnectX®-4 • InfiniBand: SDR, QDR, FDR, FDR10, EDR
• Ethernet: 1GigE, 10GigE, 25GigE, 40GigE, 50GigE, 56GigEa,
and 100GigE
ConnectX®-4 Lx Ethernet: 1GigE, 10GigE, 25GigE, 40GigE, and 50GigE
Connect-IB® InfiniBand: SDR, QDR, FDR10, FDR
ConnectX®-3/ConnectX®-3 Pro • InfiniBand: SDR, QDR, FDR10, FDR
• Ethernet: 10GigE, 40GigE and 56GigEa
ConnectX®-2 • InfiniBand: SDR, DDR
• Ethernet: 10GigE, 20GigE
PCI Express 2.0 2.5 or 5.0 GT/s
PCI Express 3.0 8 GT/s
a. 56 GbE is a Mellanox propriety link speed and can be achieved while connecting a Mellanox adapter
cards to Mellanox SX10XX switch series or connecting a Mellanox adapter card to another Mellanox
adapter card.
1.1 Content of Mellanox OFED for Linux

Mellanox OFED for Linux software contains the following components:
Components Description
OpenFabrics core and ULPs • InfiniBand and Ethernet HCA drivers (mlx4, mlx5)
• core
• Upper Layer Protocols: IPoIB, SRP, iSER and iSER Initiator and Target
OpenFabrics utilities • OpenSM: IB Subnet Manager with Mellanox proprietary Adaptive
Routing
• Diagnostic tools
• Performance tests
• SSA (SLES12): libopensmssa plugin for OpenSM, ibssa, ibacm
MPI • Open MPI stack 1.6.5 and later supporting the InfiniBand interface
• MPI benchmark tests (OSU benchmarks, Intel MPI benchmarks, Presta)
PGAS • HPC-X OpenSHMEM v2.2 supporting InfiniBand, MXM and FCA
• HPC-X UPC v2.2 supporting InfiniBand, MXM and FCA
HPC Acceleration packages • Mellanox MXM v3.0 (p2p transport library acceleration over Infini-
band)
• Mellanox FCA v2.5 (MPI/PGAS collective operations acceleration
library over InfiniBand)
• KNEM, Linux kernel module enabling high-performance intra-node
MPI/PGAS communication for large messages
Extra packages • ibutils2
• ibdump
• MFT
Rev 3.4-1.0.0.0
Components Description
Sources of all software modules
(under conditions mentioned in
the modules' LICENSE files)
except for MFT, OpenSM
plugins, ibutils2, and ibdump
HCAs • ConnectX-4 EN driver Rev 3.4-1.0.0.0
• ConnectX-3 EN driver Rev 3.4-1.0.0.0
Documentation
1.2 Supported Platforms and Operating Systems

The following are the supported OSs in MLNX_OFED Rev 3.4-1.0.0.0:
Operating System Platform
RHEL6.2/ CentOS6.2 x86_64
RHEL6.6/ CentOS6.6 x86_64/PPC64
RHEL7.1/ CentOS7.1 x86_64/PPC64/PPC64LE (Power8)
RHEL7.2/ CentOS7.2 x86_64/PPC64/PPC64LE (Power8)/ARMv8 (AMD)
Debian 7.6 x86_64
Debian 8.0 x86_64
Debian 8.1 x86_64
Debian 8.2 x86_64
Debian 8.3 x86_64
Fedora 19 x86_64
Fedora 20 x86_64
Fedora 21 x86_64
Fedora 22 x86_64
Fedora 23 x86_64/PPC64LE (Power8)
Fedora 24 x86_64
OL 6.5 x86_64
OL 6.6 x86_64
OL 6.7 x86_64 (UEK)
OL 6.8 x86_64
OL 7.1 x86_64 (UEK 3)
OL 7.2 x86_64 (UEK 4)
SLES10 SP3 x86_64
SLES11 SP1 x86_64
SLES11 SP2 x86_64
SLES11 SP3 x86_64/PPC64 (Power7)
SLES11 SP4 x86_64/PPC64 (Power8)
SLES11 SP4 SAP PowerVM with little Endian
Operating System Platform

SLES12 x86_64/PPC64LE (Power8)
SLES12 SP1 x86_64/PPC64LE
Ubuntu 12.04.5 x86_64
Ubuntu 14.04 x86_64/PPC64LE (Power 8)
Ubuntu 14.10 x86_64/PPC64LE (Power8)
Ubuntu 16.04 x86_64/PPC64LE (Power8)/ARMv8 (Qualcomm, Cavium)
WindRiver 6.0 x86_64
PowerKVM 3.1.0 PPC64LE
Kernels 4.5-4.8
For RPM based distributions, if you wish to install OFED on a different kernel, you need to
create a new ISO image, using mlnx_add_kernel_support.sh script.
See the MLNX_OFED User Manual for instructions.
Upgrading MLNX_OFED on your cluster requires upgrading all of its nodes to the newest ver-
sion as well.
1.2.1 Supported Hypervisors

The following are the supported and tested Hypervisors in MLNX_OFED Rev 3.4-1.0.0.0:
Architecture Operating Systems
x86_64 RedHat/CentOS 6.6
RedHat/CentOS 6.7
RedHat/CentOS 6.8
RedHat/CentOS 7.1
RedHat/CentOS 7.2
SLES 11 SP3
SLES 11 SP4
SLES 12
SLES 12.1
Ubuntu 14.04
Ubuntu 15.10
Ubuntu 16.04
WindRiver 6.0
PPC64LE Ubuntu 16.04
PowerKVM 3.1.0 (PV only)
Rev 3.4-1.0.0.0
1.2.2 Supported Non-Linux Virtual Machines

The following are the supported Non-Linux (InfiniBand only) Virtual Machines in
MLNX_OFED Rev 3.4-1.0.0.0:
• Windows Server 2012 R2
1.2.3 Tested Hypervisors in Paravirtualized and SR-IOV Environments

Tested Hypervisors HCAs Operating System
SR-IOV ConnectX-4 Lx RHEL 6.7 KVM
RHEL 6.8 KVM
RHEL 7.1 KVM
RHEL 7.2 KVM
SLES11 SP3 KVM
SLES11 SP4 KVM
Ubuntu 15.10 KVM
Ubuntu 16.04 KVM
WindRiver 6.0 KVM
ConnectX-4 RHEL 6.6 KVM
RHEL 6.7 KVM
RHEL 7.2 KVM
SLES11 SP3 KVM
SLES11 SP4 KVM
SLES12 KVM
Ubuntu 14.04 KVM
Ubuntu 15.10 KVM
Ubuntu 16.04 KVM
ConnectX-3 Pro RHEL 6.7 KVM
RHEL 7.2 KVM
SLES11 SP4 KVM
SLES 12 KVM
Ubuntu 14.04 KVM
Ubuntu 15.04 KVM
Ubuntu 15.10 KVM
WindRiver 6.0 KVM
ConnectX-3 SLES 11 SP4 KVM
Paravirtualized ConnectX-4 Lx RHEL 6.7 KVM
SLES11 SP3 KVM
SLES11 SP4 KVM
Ubuntu 15.10 KVM
Ubuntu 16.04 KVM
WindRiver6.0
Tested Hypervisors HCAs Operating System

ConnectX-4 RHEL 6.7 KVM
SLES11 SP3 KVM
SLES11 SP4 KVM
Ubuntu 15.10 KVM
Ubuntu 16.04 KVM
WindRiver6.0
ConnectX-3 & RHEL 6.7 KVM
ConnectX-3 Pro SLES11.4 KVM
Ubuntu 15.10 KVM
Rev 3.4-1.0.0.0
1.3 Hardware and Software Requirements

The following are the hardware and software requirements of MLNX_OFED Rev 3.4-1.0.0.0.
• Linux operating system
• Administrator privileges on your machine(s)
• Disk Space: 1GB
For the OFED Distribution to compile on your machine, some software packages of your operat-
ing system (OS) distribution are required.
To install the additional packages, run the following commands per OS:
Operating System Required Packages Installation Command
RHEL/OL/Fedora yum install perl pciutils python gcc-gfortran libxml2-python
tcsh libnl.i686 libnl expat glib2 tcl libstdc++ bc tk gtk2 atk
cairo numactl pkgconfig ethtool
XenServer yum install perl pciutils python libxml2-python libnl expat
glib2 tcl bc libstdc++ tk pkgconfig ethtool
SLES 10 SP3 zypper install pkgconfig pciutils python libxml2-python libnl
lsof expat glib2 tcl libstdc++ bc tk ethtool
SLES 11 SP2 zypper install perl pciutils python libnl-32bit libxml2-python
tcsh libnl libstdc++46 expat glib2 tcl bc tklibcurl4 gtk2 atk
cairo pkg-config ethtool
SLES 11 SP3 zypper install perl pciutils python libnl-32bit libxml2-python
tcsh libstdc++43 libnl expat glib2 tcl bc tk libcurl4 gtk2 atk
cairo pkg-config ethtool
SLES 12 zypper install pkg-config expat libstdc++6 libglib-2_0-0 lib-
gtk-2_0-0 tcl libcairo2 tcsh python bc pciutils libatk-1_0-0
tk python-libxml2 lsof libnl3-200 ethtool
Ubuntu/Debian apt-get install perl dpkg autotools-dev autoconf libtool auto-
make1.10 automake m4 dkms debhelper tcl tcl8.4 chrpath swig
graphviz tcl-dev tcl8.4-dev tk-dev tk8.4-dev bison flex dpatch
zlib1g-dev curl libcurl4-gnutls-dev python-libxml2 libvirt-bin
libvirt0 libnl-dev libglib2.0-dev libgfortran3 automake m4
pkg-config libnuma logrotate ethtool
Debian 8 apt-get install libnl-3-200 automake debhelper curl dkms
logrotate libglib2.0-0 python-libxml2 graphviz tk tcl libvirt-
bin coreutils pkg-config autotools-dev flex autoconf pciutils
quilt module-init-tools libvirt0 libstdc++6 dpkg libgfortran3
procps lsof libltdl-dev gcc dpatch chrpath grep m4 gfortran
bison libnl-route-3-200 swig perl make ethtool
1.4 Supported HCAs Firmware Versions

MLNX_OFED Rev 3.4-1.0.0.0 supports the following Mellanox network adapter cards firmware
versions:
HCA Recommended Firmware Rev. Additional Firmware Rev. Supported
Connect-IB® 10.16.1020 10.16.1006
ConnectX®-4 Lx 14.17.1010 14.16.1006
ConnectX®-4 12.17.1010 12.16.3044
HCA Recommended Firmware Rev. Additional Firmware Rev. Supported

ConnectX®-3 Pro 2.36.5150 2.36.5150
ConnectX®-3 2.36.5150 2.36.5150
ConnectX®-2 2.9.1000 2.9.1000
For the official firmware versions, please see:
http://www.mellanox.com/content/pages.php?pg=firmware_download
1.5 Compatibility Matrix

MLNX_OFED Rev 3.4-1.0.0.0 is compatible with the following:
Mellanox Product Description/Version
MLNX-OS® MSX6036 w/w MLNX-OS® version 3.4.3202a
Grid Director™ 4036 w/w Grid Director™ version 3.9.1-985
Unified Fabric Manager (UFM®) v5.5
MXM v3.5
HPC-X UPC v2.22
HPC-X OpenSHMEM v1.8.3
FCA v2.5 and v3.5
OpenMPI v1.10
a. MLNX_OFED Rev 3.4-1.0.0.0 was tested with this switch however, additional switches might be supported as
well.
1.6 RoCE Modes Matrix

The following is RoCE modes matrix:
RoCEv1 & RoCEv2
RoCEv1 (IP Based GIDs) RoCEv2
(RoCE per GID) Supported as
Software Stack / Supported as of Version Supported as of Version
of Version
Inbox Distribution
ConnectX-3 & ConnectX-4 & ConnectX-3 ConnectX-4 & ConnectX-3 Pro, ConnectX-4
ConnectX-3 Pro ConnectX-4 Lx Pro ConnectX-4 Lx & ConnectX-4 Lx
MLNX_OFED 2.1-x.x.x 3.0-x.x.x 2.3-x.x.x 3.0-x.x.x 3.0-x.x.x
Kernel.org 3.14 4.4 4.4 4.4 4.4
RHEL 6.6, 7.0 - - - -
SLES 12 - - - -
Ubuntu 14.04.4, 16.04, - - - -
15.10
Rev 3.4-1.0.0.0
2 Changes and New Features in Rev 3.4-1.0.0.0

HCAs Feature/Change Description
ConnectX®-3/ VST Q-in-Q Added support for Q-in-Q encapsulation per VF in Linux
ConnectX®-3 Pro (VST) for ConnectX-3 Pro adapters.
Package Content SR-IOV enabled firmware binaries for ConnectX-3 has
been removed from MLNX_OFED package (the installa-
tion flag “--enable-sriov” has been deprecated).
To configure SR-IOV, please use the “mlxconfig” or “mst-
config” utility.
MLNX_OFED repository metadata files has been moved
to the folder holding the binary packages (named “RPMS”
in rpm based OS, and “DEBS” in Debian based OS).
Please update your repository configuration file accord-
ingly (refer to the MLNX_OFED User Manual for more
details about setting up MLNX_OFED as a repository).
ConnectX®-4/ Raw Ethernet Added new APIs for enhanced raw Ethernet programming:
ConnectX®-4 Lx Programming • Packet Pacing
• TCP Segmentation Offload (TSO)
• ToS based steering
• Flow ID based steering (beta)
• VxLAN based steering (beta)
For further information, refer to the “Programming” sec-
tion in OFED User Manual.
Enhanced PCIe Error Enhanced PCIe error recovery by adding the following
Recovery behaviors to the flow:
• In case SR-IOV is enabled during the recovery process,
it will not get automatically disabled and will require
the administrator that enabled it to disable it.
• When the driver goes down, VF PCI function will not
be removed.
• Ethernet interface attributes (MTU, state, ring size,
etc...) will be recovered after the error recovery stage is
completed.
• The net device kernel layer will not be aware of any
ongoing PCI error recovery process.
SR-IOV Max Rate Added the ability to rate-limit traffic per Virtual Function
Limit Ethernet/RoCE in SR-IOV mode.
(beta level)
Dynamically tuned Added support for dynamically controlling the interrupts
Interrupt Moderation per channel to ensure maximum packet rate with minimum
(DIM) interrupt rate. This feature is enabled by default.
Dump Configuration Added support for dump configuration which helps dump-
ing driver and firmware configuration using ethtool. It cre-
ates a backup of the configuration files into a specified
dump file.
Ethernet Counters Updated the list of counters the can be retrieved via ethtool
for mlx5 driver, changed counters names and added new
counters.
Rev 3.4-1.0.0.0 Changes and New Features in Rev 3.4-1.0.0.0
HCAs Feature/Change Description

ConnectX®-3/ Mellanox PeerDirect The experimental PeerDirect Async APIs have been
ConnectX®-3 Pro/ Async (beta level) changed to make the implementation of peer clients sim-
ConnectX®-4/ pler. Note the following:
ConnectX®-4 Lx • These changes are not backward compatible.
• The code adds CQ polling support for peer devices that
do not support the NOR operation, replacing it with a
GEQ operation.
To see the API changes, refer to the man page.
ABI Incompatibility Added the ability to fix the issue of when a new kernel
may not be compatible with the MLNX_OFED modules,
preventing to load them.
Connect-IB/Con- Mellanox Scalable This technology improves the performance of MPI opera-
nectX®-3/Con- Hierarchical Aggre- tion by offloading collective operations from the CPU and
nectX®-3 Pro/ gation Protocol dispatching to the switch network, and eliminating the
ConnectX®-4/ (SHArP™) need to send data multiple times between endpoints. This
ConnectX®-4 Lx
approach decreases the amount of data traversing the net-
work as aggregation nodes are reached, and dramatically
reduces the MPI operation time.
For further information on SHArP and its configuration,
see SHArP Deployment Guide.
Bug Fixes Refer to Section 4, “Bug Fixes History”, on page 41.
For additional information on the new features, please refer to the MLNX_OFED User Manual.
2.1 Future API Changes in MLNX_OFED
Note that the following APIs will be deprecated and replaced with the new APIs as of MLNX-
_OFED version 4.0, as listed in the table below.
Feature Type Current API New API

Rereg MR Verb ibv_exp_rereg_mr ibv_rereg_mr
Memory Win- Verb ibv_exp_bind_mw ibv_bind_mw
dow Structure ibv_exp_send_wr -> bind_mw ibv_send_wr -> bind_mw
Opcodes IBV_EXP_WR_SEND_WITH_INV IBV_WR_SEND_WITH_INV
IBV_EXP_WR_LOCAL_INV IBV_WR_LOCAL_INV
IBV_EXP_WR_BIND_MW IBV_WR_BIND_MW
Capability IBV_EXP_DEVICE_MEM_WIN- IBV_DEVICE_MEM_WIN-
DOW DOW
Completion IBV_EXP_WC_WITH_INV IBV_WC_WITH_INV
Rev 3.4-1.0.0.0
3 Known Issues
The following is a list of general limitations and known issues of the various components of this
Mellanox OFED for Linux release.
3.1 Driver Installation/Loading/Unloading/Start Known Issues

3.1.1 Installation Known Issues
Index Internal Reference Number: Description Workaround

1. When upgrading from an earlier Mellanox OFED Stop the old OFED stack (/etc/init.d/
version, the installation script does not stop the ear- openibd stop) before upgrading to this new
lier version prior to uninstalling it. version.
2. When using bonding on Ubuntu OS, the "ifenslave" -
package must be installed.
3. On PPC systems, the ib_srp module is not installed If your system does not require the ibmvscsi
by default since it breaks the ibmvscsi module. module, run the mlnxofedinstall script with the
"--with-srp" flag.
4. #679801: Updating MLNX_OFED via Yum (e.g. Remove the mpitests packages manually:
running "yum update mlnx-ofed-all") can fail # rpm -e --allmatches $(rpm -qa |
with the following error: grep mpitests_)
--> Finished Dependency Resolution
Error: Package: mpitests_open-
mpi__1_8_8-3.2.16-fe5387c.x86_64
(installed)
Requires: liboshmem.so.3()(64bit)
Removing: openmpi-1.8.8-1.x86_64
(installed)
liboshmem.so.3()(64bit)
Updated By: openmpi-1.10.2rc4-
1.32008.x86_64 (mlnx_ofed)
~liboshmem.so.9()(64bit)
5. #690799: OpenSM package removal fails with the 1. Create the missing link by running this com-
following error on Ubuntu12.04: mand:
Removing opensm ... # ln -s /usr/lib/insserv/insserv
/sbin/insserv: No such file or direc- /sbin/insserv
tory 2. Remove the package.
6. #764204: Weak Updates (KMP) support is broken As of MLNX_OFED v3.3, use the mlnx_ad-
on RHEL PPC64LE with errata kernels. MLNX- d_kernel_support.sh script, or simply provide
_OFED installation will pass, but no links will be the --add-kernel-support flag to mlnxofedin-
created under the weak-updates directory for the stall script.
new kernel. Therefore, the driver load will fail.
Rev 3.4-1.0.0.0 Known Issues

7. #785119: When upgrading ConnectX-4/ConnectX-4 -
Lx firmware version from v12/14.14.2036 to a
newer one (for example:12/14.16.1xxx), power
cycle is necessary to enable working in Pass-
Through mode. Using mlxfwreset instead of power
cycle will print messages similar to the following
when Passing-Through the device to Virtual
Machine:
"-device vfio-pci,host=04:00.0,id=host-
dev0,bus=pci.0,addr=0x7: vfio: Error:
Failed to setup INTx fd: No such device
2016-05-22T06:46:39.164786Z qemu-kvm: -
device vfio-pci,host=04:00.0,id=host-
dev0,bus=pci.0,addr=0x7: Device ini-
tialization failed."
3.1.2 Driver Unload Known Issues

1. "openibd stop" can sometime fail with the error: Re-run "openibd stop"
Unloading ib_cm [FAILED]
ERROR: Module ib_cm is in use by ib_i-
poib
3.1.3 Driver Start Known Issues

1. Failure to load a 4K page on ARM architecture. Enlarge the CMA area by adding cma=256M
or more to grub.conf.
2. “Out-of-memory” issues may rise during drivers -
load depending on the values of the driver module
parameters set (e.g. log_num_cq).
3. When reloading/starting the driver using the /etc/ Remove the third party RPM/non MLNX-
init.d/openibd the following messages are dis- _OFED drivers directory, run:
played if there is a third party RPM or driver "depmod" and then rerun "/etc/init.d/
installed: openibd restart"
"Module mlx4_core does not belong to MLNX-
_OFED"
or
"Module mlx4_core belong to <rpm name> which is
not a part of MLNX_OFED"
4. Occasionally, when trying to repetitively reload the -
NES hardware driver on SLES11 SP2, a soft lockups
occurs that required reboot.
5. When downgrading from MLNX_OFED 3.0-x.x.x, The issues will be resolved automatically after
driver reload might fail with the following error in system reboot or by invoking the following
dmeg: commands:
[166271.886407] compat: exports dupli- rmmod mlx_compat
cate symbol __ethtool_get_settings depmod -a
(owned by mlx_compat) /etc/init.d/openibd restart
Rev 3.4-1.0.0.0

6. #773774: When downgrading from MLNX_OFED The issues will be resolved automatically after
3.3-x.x.x, driver reload might fail with the following system reboot or by invoking the following
error in dmeg: commands:
rmmod: ERROR: Module mlx_compat is in rmmod ib_netlink
use by: ib_netlink depmod -a
/etc/init.d/openibd restart
7. In ConnectX-2, (when the debug_level module -
parameter for module mlx4_core is non-zero), if the
driver load succeeds, the message below is pre-
sented:
“mlx4_core 0000:0d:00.0: command
SET_PORT (0xc) failed:
in_param=0x120064000, in_mod=0x2, op_-
mod=0x0, fw status = 0x40”
This message is simply part of the learning process
for setting the maximum port VLs compatible with a
4K port mtu, and should be ignored.
8. “openibd start” unloads kernel modules that were -
loaded from initrd/initramfs upon boot.
This affects only kernel modules which come with
MLNX_OFED and are included in initrd/initramfs.
9. If a Lustre storage is used, it must be fully unloaded 1. Unmount any mounted Lustre storages:
before restarting the driver or rebooting the # umount<lustre_mount_point>
machine, otherwise machine might get stuck/panic. 2. Unload all Lustre modules:
# lustre_rmmod
10. Driver unload fails with the following error mes- Make sure that there are no mount points over
sage: NFS/RDMA prior to unloading the driver and
Unloading rdma_cm [FAILED] run:
rmmod: ERROR: Module rdma_cm is in use # modprobe -r xprtrdma
by: xprtrdma In case that the xprtrdma module keeps getting
loaded automatically even though it is not
used, add a pre-stop hook for the openibd ser-
vice script to always unload it.
Create an executable file
"/etc/infiniband/pre-stop-hook.sh"
with the following content:
#!/bin/bash
modprobe -r xprtrdma
11. When loading or unloading the driver on HP Pro- If you are *not* running SR-IOV on your sys-
liant systems, you may see log messages like: tem, you may eliminate these messages by
dmar: DMAR:[DMA Write] Request device removing the term "intel_iommu=on" from
[07:00.0] fault addr 3df7f000 the boot line in file /boot/grub/menu.lst.
DMAR:[fault reason 05] PTE Write access For SR-IOV systems, this term must remain,
is not set you can ignore the log messages.
This is a known issue with ProLiant systems (see
their support notice for Emulex adapters: http://
h20564.www2.hpe.com/hpsc/doc/public/dis-
play?docId=emr_na-c04446026&lang=en-
us&cc=us
)The messages are harmless, and may be ignored.
12. #677998: False alarms errors may be printed to -
dmesg

13. #610395: On RHEL 7.1, after updating to kernel Use the mlnx_add_kernel_support.sh
version 3.10.0-229.14.1.el7 or later, driver load fails script to compile MLNX_OFED drivers
with unknown symbols errors in dmesg. against the new kernel.
3.1.4 System Time Known Issues

1. Loading the driver using the openibd script when no -
InfiniBand vendor module is selected (for example
mlx4_ib), may cause the execution of the
/sbin/start_udev’ script.
In RedHat 6.x and OL6.x this may change the local
system time.
3.1.5 UEFI Secure Boot Known Issues

1. On RHEL7 and SLES12, the following error is dis- For further information, please refer to the
played in dmesg if the Mellanox's x.509 Public Key User Manual section "Enrolling Mellanox's
is not added to the system: x.509 Public Key On your Systems".
[4671958.383506] Request for unknown

module key 'Mellanox Technologies sign-
ing key:
61feb074fc7292f958419386ffdd9d5-
ca999e403' err -11
This error can be safely ignored as long as Secure

Boot is disabled on the system.
2 Ubuntu12 requires update of user space -
open-iscsi to v2.0.873
3 The initiator does not respect interface parameter Configure each interface on a different subnet.
while logging in.
3.2 Performance Known Issues

1. #765777: Low VxLAN throughput due to broken Use kernel version 4.6 or above.
GRO offload in most kernels older than kernel v4.6.
2. On machines with irqbalancer daemon turned off, Execute the following script as root:
the default InfiniBand interrupts will be routed to a set_irq_affinity.sh <interface or
single core which may cause overload and software/ IB device> [2nd interface or IB
hardware lockups. device]
3. #414827: Out-of-the-box throughput performance in For additional performance tuning, please refer
Ubuntu14.04 is not optimal and may achieve results to Performance Tuning Guide.
below the line rate in 40GE link speed.
Rev 3.4-1.0.0.0

4. UDP receiver throughput may be lower then Disable adaptive interrupt moderation and set
expected, when running over mlx4_en Ethernet lower values for the interrupt coalescing manu-
driver. ally.
This is caused by the adaptive interrupt moderation ethtool -C <eth>X adaptive-rx off
routine, which sets high values of interrupt coalesc- rx-usecs 64 rx-frames 24
ing, causing the driver to process large number of
packets in the same interrupt, leading UDP to drop Values above may need tuning, depending the
packets due to overflow in its buffers. system, configuration and link speed.
5. #417751: Performance degradation might occur -
when bonding Ethernet interfaces.
6. #656415: In RHEL7.0, when the irqbalance service -
is started or restarted, it incorrectly re-balances the
IRQs, including the banned ones.
7. #651322: In RH7.0/RH7.1, performance issue with Use RH7.2 to avoid such performance issues.
ConnectX-4 cards over 100GbE link might occur
when the process of forwarding the packets between
the ports, which is done by the kernel, fib_table_-
lookup() function is called.
For further information, please refer to:
http://comments.gmane.org/gmane.linux.network/
344243
8. #754646: The default RX coalescing values yield to Increase the RX microseconds and frames
high CPU utilization when using VXLAN on VMs coalescing parameters for a better utilization
over PV. using the ethtool -C command.
9. #783496: When using a VF over RH7.X KVM, low Install the following packages:
throughput is expected. • qemu-img-1.5.3-
105.el7_2.1.bz1299846.0.x86_64.rpm
• qemu-kvm-1.5.3-
105.el7_2.1.bz1299846.0.x86_64.rpm
• qemu-kvm-common-1.5.3-
105.el7_2.1.bz1299846.0.x86_64.rpm
3.3 HCAs Known Issues

3.3.1 mlx5 Driver Known Issues
1. #860311: An allocation of high-order page in mlx- No action is required on user’s end. A frag-
5e_alloc_striding_rx_wqe fails with a call-trace. mented fallback flow will handle this failure.
2. Atomic Operations in Connect-IB® are fully sup- -
ported on big-endian machines (e.g. PPC). Their
support is limited on little-endian machines (e.g.
x86)
3. #435583: EEH events that arrive while the mlx5 -
driver is loading may cause the driver to hang.
4. #434570: The mlx5 driver can handle up to 5 EEH If more events are received, cold reboot the
events per hour. machine.
5. #554120: When working with Connect-IB® firm- Upgrade Connect-IB firmware to the latest
ware v10.10.5054, the following message would available version.
appear in driver start.
command failed, status bad system
state(0x4), syndrome 0x408b33
The message can be safely ignored.
6. Changing the link speed is not supported in Ethernet -
driver when connected to a ConnectX-4 card.
7. #538843: Bonding "active-backup" mode does not -
function properly.
8. Rate, speed and width using IB sysfs/tools are avail- -
able in RoCE mode in
ConnectX-4 only after port physical speed configu-
ration is done.
9. #598092: Since MLNX_OFED's openibd does not Restart OpenSM
unload modules while OpenSM is running, remov-
ing mlx5_core manually while OpenSM is run-
ning, may cause it to be out of sync when probed
again.
10. #563022: ConnectX-4 port GIDs table shows a -
duplicated RoCE v2 default GID.
Rev 3.4-1.0.0.0
3.4 Ethernet Network

3.4.1 Ethernet Known Issues

1. #843306: [ConnectX-4/ConnectX-4 Lx] When -
configuring ETS, bandwidth values are limited
between 1-100, and 0 is an invalid value.
2. #704750: [ConnectX-4/ConnectX-4 Lx] First -
ICMP6 packet may be lost as a result of first IP frag-
ment loss when packets size is significantly bigger
than MTU.
3. When creating more than 125 VLANs and SR-IOV -
mode is enabled, a kernel warning message will be
printed indicating that the native VLAN is created
but will not work with RoCE traffic.
kernel warning: mlx4_core 0000:07:00.0:

vhcr command ALLOC_RES (0xf00) slave:0
in_param 0x7e in_mod=0x107, op_mod=0x1
failed with error:0, status -28
4. Kernel panic might occur during FIO splice in ker- Use kernel v2.6.34-rc4
nels before 2.6.34-rc4. which provides the following solution:
baff42a net: Fix
oops from
tcp_collapse() when
using splice()
5. In PPC systems when QoS is enabled a harmless -
Kernel DMA mapping error messages might appear
in kernel log (IOMMU related issue).
6. Transmit timeout might occur on RH6.3 as a result -
of lost interrupt (OS issue). In this case, the follow-
ing message will be shown in dmesg:
do_IRQ: 0.203 No irq handler for vector
(irq -1)
7. Mixing ETS and strict QoS policies for TCs in -
40GbE ports may cause inaccurate results in band-
width division among TCs.
8. Creating a VLAN with user priority >= 4 on -
ConnectX®-2 HCA is not supported.
9. Affinity hints are not supported in Xen Hypervisor To overcome this issues, run:
(an irqblancer issue). This causes a non-optimal IRQ set_irq_affinity.sh eth<x>
affinity.
10. #433366: Reboot might hang in SR-IOV when using -
the “probe_vf” parameter with many Virtual
Functions. The following message is logged in the
kernel log:
"waiting for eth to become free. Usage
count =1"
11. In ConnectX®-2, RoCE UD QP does not include
VLAN tags in the Ethernet header

12. VXLAN may not be functional when configured -
over Linux bridge in RH7.0 faceor Ubuntu14.04.
The issue is within the bridge modules in those ker-
nels. In Vanilla kernels above 3.16 issues is fixed.
13. In RH6.4, ping may not work over VLANs that are -
configured over Linux bridge when the bridge has a
mlx4_en interface attached to it.
14. The interfaces LRO needs to be set to "OFF" manu- Run: ethtool -K ethX lro off
ally when there is a bond configured on Mellanox
interfaces with a Bridge over that bond.
15. #539117: On SLES12, the bonding interface over 1. Set "STARTMODE=hotplug" in the bonding
Mellanox Ethernet slave interfaces does not get IP slave's ifcfg files.
address after reboot. More details can be found in the SUSE doc-
umentations page: https://www.suse.com/
documentation/sles-12/book_sle_admin/
?page=/documentation/sles-12/
book_sle_admin/data/sec_bond.html
2. Enable the “nanny” service to support hot-
plugging:
Open the "/etc/wicked/common.xml"
file.
Change: "<use-nanny>false</use-
nanny>" to "<use-nanny>true</use-
nanny>"
3. Run:
# systemctl restart wickedd.ser-
vice wicked
16. ethtool -x command does not function in SLES OS. -
17. #516136: Ethertype proto 0x806 not supported by -
ethtool
18. ETS configuration is not supported in the following
kernels:
• 3.7
• 3.8
• 3.9
• 3.10
• 3.2.37-94_fbk17_01925_g8e3b329
• 3.14
• 3.2.55-106_fbk22_00877_g6902630
• 3.2.28-76_fbk14_00230_g3c40d9e
19. ETS is not supported in kernels that do not have -
MQPRIO as QDISC_KIND option in the tc tool.
20. #592229: When NC-SI is ON, the port’s MTU can- -
not be set to lower than 1500.
21. #600242: GRO is not functional when using -
VXLAN in ConnectX-3 adapter cards.
22. #596075: ethtool -X: The driver supports only the -
'equal' mode and cannot be set by using weight
flags.
Rev 3.4-1.0.0.0

23. #600752: Q-in-Q infrastructure in the kernel is sup- -
ported only in kernel version 3.10 and up.
24. #596537: When SLES11 SP4 is used as a DHCP cli- -
ent over ConnectX-3 or ConnectX-3 adapters, it
might fail to get an IP from the DHCP server.
25. #560575: When using a hardware that has Time -
Stamping enabled, the system time might be higher
than the expected variance.
26. #597758: In Q-in-Q, ping failed when sending traf- -
fic with package size > 1468
27. #665131: Call trace may occur when configuring -
VXLAN or under high traffic stress.
28. HW LRO does not function in ConnectX-4 adapter -
cards.
29. #685069/689607: ethtool header does not currently -
support the link speeds of 25/50/100. Therefore,
these speeds cannot be seen as advertised/supported.
30. #835239: While running Q-in-Q packets with stag Check the wire or a switch between the hosts,
offloading, tcpsump/wireshark on host may show the wireshark will show 0x88A8 ethertype as
svlan ethertype as 0x8100 instead of 0x88A8. expected.
3.4.2 Port Type Management Known Issues

1. OpenSM must be stopped prior to changing the port -
protocol from InfiniBand to Ethernet.
2. After changing port type using connectx_port_- Use udev rules for persistent naming configu-
config interface ports’ names can be changed. For ration.
example. ib1 -> ib0 if port1 changed to be Ethernet For further information, please refer to the
port and port2 left IB. User Manual
3. A working IP connectivity between the RoCE -
devices is required when creating an address handle
or modifying a QP with an address vector.
4. IPv4 multicast over RoCE requires the MGID for- -
mat to be as follow ::ffff:<Multicast IPv4
Address>
5. IP routable RoCE does not support Multicast Lis- -
tener Discovery (MLD) therefore, multicast traffic
over IPv6 may not work as expected.
6 DIF: When running IO over FS over DM during -
unstable ports, block layer BIOS merges may cause
false DIF error.
7 connectx_port_config configurations is not saved Re-run "connectx_port_config"
after unbind/bind.
3.4.3 Flow Steering Known Issues

1. Flow Steering is disabled by default in firmware ver- To enable it, set the parameter below as follow:
sion < 2.32.5100. log_num_mgm_entry_size should set
to -1
2. IPv4 rule with source IP cannot be created in -
SLES 11.x OSes.
3. RFS does not support UDP. -
4. When working in DMFS:A0 mode and VM/hyper- -
visor is MLNX_OFED 2.3-x.x.x, the second side
(hypervisor/VM respectively) should be MLNX-
_OFED 2.3-x.x.x as well.
5. #516136: Setting ARP flow rules through ethtool is -
not allowed.
3.4.4 Quality of Service Known Issues

1. QoS is not supported in XenServer, Debian 6.0 and -
6.2 with uek kernel
2. When QoS features are not supported by the kernel, -
mlnx_qos tool may crash.
3. #448981: QoS default settings are not returned after -
configuring QoS.
3.4.5 Ethernet Performance Counters Known Issues

1. In ConnectX®-3, in a system with more than 61 -
VFs, the 62nd VF and onwards is assigned with the
SINKQP counter, and as a result will have no statis-
tics, and loopback prevention functionality for SINK
counter.
2. In ConnectX®-3, since each VF tries to allocate 2 -
more QP counter for its RoCE traffic statistics, in a
system with less than 61 VFs, if there is free
resources it receives new counter otherwise receives
the default counter which is shared with Ethernet. In
this case RoCE statistics is not available.
3. In ConnectX®-3, when we enable function-based -
loopback prevention for Ethernet port by default
(i.e., based on the QP counter index), the dropped
self-loopback packets increase the IfRxErrorFrames/
Octets counters.
Rev 3.4-1.0.0.0
3.5 InfiniBand Network

3.5.1 IPoIB Known Issues

1. #854235: IPoIB bonding interface has to be Toggle bonding interface to state “down” and
restarted in order to work on some operating sys- then to state “up”.
tems.
2. When user increases receive/send a buffer, it might -
consume all the memory when few child's interfaces
are created.
3. The size of send queue in Connect-IB® cards cannot -
exceed 1K.
4. In 32 bit devices, the maximum number of child -
interfaces that can be created is 16. Creating more
that might cause out-of-memory issues.
5. In RHEL7.0, the Network-Manager can detect when Set “ignore-carrier” for the corresponding
the carrier of one of the IPoIB interfaces is OFF and device in NetworkManager.conf.
can decide to disable its IP address. For further information, please refer to "man
NetworkManager.conf"
6. IPoIB interface does not function properly if a third We recommend modifying PKey tables via
party application changes the PKey table. OpenSM.
7. Fallback to the primary slave of an IPoIB bond does -
not work with ARP monitoring. (https://bugs.open-
fabrics.org/show_bug.cgi?id=1990)
8. Out-of memory issue might occur due to overload of To calculate the allowed memory per each
interfaces created. IPoIB interface check the following:
• Num-rings = min(num-cores-on-that-
device, 16)
• Ring-size = 512 (by default, it is module
parameter)
• UD memory: 2 * num-rings * ring-size *
8K
• CM memory: ring-size * 64k
• Total memory = UD mem + CM mem
9. Connect-IB does not reach the bidirectional line rate Optimize the IPoIB performance in
Connect-IB:
cat /sys/class/net/<interface>/
device/local_cpus > /sys/class/net/
<interface>/queues/rx-0/rps_cpus
10. If the CONNECTED_MODE parameter is set to “no” or Set the CONNECTED_MODE=yes parameter
missing from the ifcfg file for Connect-IB® IPoIB in the ifcfg file for Connect-IB® interface.
interface then the "service network restart"
will hang.

11. Whenever the IOMMU parameter is enabled in the To avoid such issue:
kernel it can decrease the number of child interfaces • Decrease the amount of the RX receive buf-
on the device according to resource limitation. fers (module parameter, the default is 512)
The driver will stuck after unknown amount of child • Decrease the number of RX rings (sys/fs or
interfaces creation. ethtool in new kernels)
• Avoid using IOMMU if not required
For further information, please see:
https://access.redhat.com/site/articles/66747 For KVM users:
http://support.citrix.com/article/CTX136517 Run:
http://www.novell.com/support/kb/ echo 1 > /sys/module/kvm/parame-
doc.php?id=7012337 ters/allow_unsafe_assigned_inter-
https://bugzilla.redhat.com/show_- rupts
bug.cgi?id=1044595
To make this change persist across reboots, add
the following to the /etc/modprobe.d/
kvm.conf file (or create this file, if it does not
exist):
options kvm allow_unsafe_as-
signed_interrupts=1 kernel parame-
ters
12. System might crash in skb_checksum_help() Use UD mode in IPoIB
while performing TCP retransmit involving packets
with 64k packet size.
A similar out to the below will be printed:
kernel BUG at net/core/dev.c:1707!
invalid opcode: 0000 [#1] SMP
RIP: 0010:[<ffffffff81448988>] skb_-
checksum_help+0x148/0x160
Call Trace:
<IRQ>
[<ffffffff81448d83>] dev_hard_start_x-
mit+0x3e3/0x530
[<ffffffff8144c805>] dev_queue_x-
mit+0x205/0x550
[<ffffffff8145247d>] neigh_connect-
ed_output+0xbd/0x1
…
…
13. When InfiniBand ports are removed from the host To avoid it and have persistent IPoIB network
(e.g when changing port type from IB to Eth or devices names for ConnectX ports, add to the
removing a card from the PCI bus) the remaining /etc/udev/rules.d/70-persistent-
IPoIB interface might be renamed. net.rules file:
SUBSYSTEM=="net", ACTION=="add",
DRIVERS=="?*",
ATTR{address}=="*<Port GID>",
NAME="ibN"
Where N is the IPoIB required interface index

14. After releasing a bond interface that contains IPoIB -
slaves, a call trace might be printed to the dmesg.
Rev 3.4-1.0.0.0

15. IPoIB interfaces are loaded without an IP address on 1. Open the "/etc/wicked/common.xml" file.
SLES 12. 2. Change:
"<use-nanny>false</use-nanny>"
to
"<use-nanny>true</use-nanny>"
3. Run:
# systemctl restart wickedd.ser-
vice wicked
# ifup all
16. In RHEL7.0, running ifdown then ifup on an inter- Reload the driver "/etc/init.d/openibd
face after changing CONNECTED_MODE in its restart" or reboot the system.
ifcfg file, will cause the interface bring up to fail.
17. Clone interfaces receive a duplicated IPv6 address -
when a child interface with the same PKey (a.k.a
clone interface) is used for all the clones.
18. eth_ipoib module is not loaded after reloading the To restart the IPoIB driver, run "/etc/
ib_ipoib module. init.d/openibd restart". Do not restart
it by manually restarting each module.
19. In Ubuntu and Debian, the default of the recv_- -
queue_size and send_queue_size is 128
according to the io_mmu issue.
20. In RHEL6.7, when the Network Manager service is Either disable the Network Manager, or add
enabled and an IPoIB interface is configured using "DEVICE=<interface name>" to the inter-
the "nm-connection-editor" tool, the generated face's ifcfg file.
ifcfg file is missing the "DEVICE=<interface
name>" parameter. Hence, changing the CONNECT-
ED_MODE in the ifcfg file, will cause a failure in the
interface bring up.
21. #552840: ifdown command does not function in -
RH7.x
22. #665143: Kernel Oops may occur after reboot. -
23. #555632: Kernel panic may occur while re-assign- -
ing LIDs.
24. #556352: ICMP traffic might be lost after Vnic -
restart
25. #560575: Spikes may occur while running PTP pro- -
tocol over ConnectX-3/ConnectX-3 Pro.
26. #684720: ifdown fails on SLES12SP0/SP1 with the To see the list of all dependent interfaces, run:
following errors # wicked --debug all ifdown ib0
# ifdown ib0 ..
wicked: ifdown: no matching interfaces ..
The error indicates that there are active interfaces wicked: skipping ib0 interface:
using the interface you are trying to bring down, and unable to ifdown due to lowerdev
you must ifdown all dependent interfaces. dependency to: ib0.8001
wicked: ifdown: no matching inter-
faces
wicked: Exit with status: 0
27. #766451: Occasionally, in kernel 3.10, under heavy -
load, the kernel fails to get free page.
For more details, please refer to:
https://bugs.centos.org/view.php?id=10245
3.5.2 eIPoIB Known Issues

1. #383034: On rare occasions, upon driver restart the -
following message is shown in the dmesg:
'cannot create duplicate filename '/
class/net/eth_ipoib_interfaces'
2. No indication is received when eIPoIB is non-func- Run ‘ps -ef | grep ipoibd’ to verify its
tional. functionality.
3. eIPoIB requires libvirtd, python -
4. eIPoIB supports only active-backup mode for bond- -
ing.
5. eIPoIB supports only VLAN Switch Tagging (VST) -
mode on guests.
6. IPv6 is currently not supported in eIPoIB -
7. #384279: eIPoIB cannot run when Flow Steering is -
enabled
8. eIPoIB daemon requires the following libs in order -
to run: python-libxml2, libvirt-bin, libvirt0
9. The eIPoIB driver in ConnectX®-3 and Connect-IB -
is currently at beta level.
3.5.3 XRC Known Issues

1. Legacy API is deprecated, thus when recompiling -
applications over MLNX_OFED v2.0-3.x.x, warn-
ings such as the below are displayed.
rdma.c:1699: warning: 'ibv_open_xrc_do-
main' is deprecated (declared at /usr/
include/infiniband/ofa_verbs.h:72)
rdma.c:1706: warning: 'ibv_create_x-
rc_srq' is deprecated (declared at /
usr/include/infiniband/ofa_verbs.h:89)
These warnings can be safely ignored.
2. XRC is not functional in heterogeneous clusters -
containing non Mellanox HCAs.
3. XRC options do not work when using qperf tool. Use perftest instead
4. Out-of memory issue might occur due to overload of -
XRC receive QP with non zero receive queue size
created.
XRC QPs do not have receive queues.
Rev 3.4-1.0.0.0
3.5.4 Verbs Known Issues

1. #855362: A compilation error will occur in kernel Assign the enum field explicitly. For example:
space application when setting the wr_id field upon wr.wr_id = MY_WR_ID;
initializing any of the following structures: ib_wc,
ib_send_wr, or ib_recv_wr. This is caused due to
wr_id insertion into an anonymous union.
2. Using libnl1_1_3~26 or earlier, requires ibv_cre- -
ate_ah protection by a lock for multi-threaded appli-
cations.
3. In MLNX_OFED v2.4-1.0.0, if several CQEs are When getting an event, poll the CQ until it is
received on a CQ, they will be coalesced and a user- empty.
space event will be triggered only once.
4. #420847: ibv_task_pingpong over ConnectX-2 -
adapter cards in not supported.
5. #835061: According to the verbs header (/usr/ Run QP command query to verify the value.
include/infiniband/verbs.h), the static rate field in
the Address handler can take value from 0 to 18.
The values 11 to 18 (inclusive) are not supported for
Connect-X 4 and Connect-X 3.
3.5.5 RoCE Known Issues

1. #869158: Occasionally, UC|UD traffic over default -
GIDs, with high iterations may get stuck.
2. #854517: Driver restart while having RDMA-CM -
running applications may hang.
3. #866072: [mlx5] RoCE v2 multicast traffic using -
RDMA-CM with IPv4 address will not be received.
4. Not configuring the Ethernet devices or independent Restart the driver
VMs with a unique IP address in the physical port,
may result in RoCE GID table corruption.
5. If RDMA_CM is not used for connection manage- -
ment, then the source and destination GIDs used to
modify a QP or create AH should be of the same
type - IPv4 or IPv6.
6. #392592: On rare occasions, the driver reports a -
wrong GID table (read from /sys/class/infiniband/
mlx4_*/ports/*/gids/*). This may cause communi-
cation problems.
7. MLNX_OFED v2.1-1.0.0 and onwards is not -
interoperable with older versions of MLNX_OFED.

8. Since the number of GIDs per port is limited to 128, -
there cannot be more than the allowed IP addresses
configured to Ethernet devices that are associated
with the port. Allowed number is:
• “127” for a single function machine
• “15” for a hypervisor in a multifunction
machine
• “(127-15)/n” for a guest in a multifunction
machine (where n is the number of virtual func-
tions)
Note also that each IP address occupies 2 entries
when RoCE mode is set to 4 (RoCEv1 + RoCE v2).
This further reduces the number of allowed IP
addresses.
9. A working IP connectivity between the RoCE -
devices is required when creating an address handle
or modifying a QP with an address vector.
10. IPv4 multicast over RoCE requires the MGID -
format to be as follow ::ffff:<Multicast IPv4
Address>
11. IP RoCEv2 does not support Multicast Listener Dis- -
covery (MLD) therefore, multicast traffic over IPv6
may not work as expected.
12. Using GID index 0 (the default GID) is possible -
only if the matching IPv6 link local address is con-
figured on the net device of the port.
This behavior is possible even though the default
GID is configured regardless of the presence of the
IPv6 address.
13. Using IPv6 link local address (GID0) when VLANs -
are configured is not supported.
14. Using GID index 0 (the default GID) on port 2 is -
currently not supported on kernel 3.14 and below.
15. #559276/591244: Dynamically Connected (DC) in -
RoCE in ConnectX®-4 is currently not supported.
16. Enslaving a Mellanox device to a bond with already 1. Enslave the Mellanox device.
configured IPs (or configured upper devices), pre- 2. Configure IP devices.
vents these IPs from being configured as GIDs.
17. #517825: ibv_create_ah_from_wc is not sup- -
ported for multicast messages.
18. #609950/649407: Occasionally, when the Bonding Add slave devices to the master before giving it
Mode is set to other than active/backup mode (mode an IP address.
1), the GID table is not populated correctly.
19. #667399: In ConnectX-4 adapter cards, when the -
port speed is lower than 10Gbps, the IB tools will
present a higher rate.
Rev 3.4-1.0.0.0

20. #778492: RoCE requires that when a bonding mod- Work in fail_over_mac mode (bonding).
ule enslaves 2 Ethernet interfaces, the GID for any
IP address on bond0 will appear only on the port of
the active interface
Due to kernel limitations, the information about
active slave is unknown, therefore, any IP address
on bond0 will appear on both ports.
21. #781383: Creating Address Handler (AH) may run -
slow or may hang under a heavy load on all nodes’
cores (for example: MPI All2All cases).
3.5.6 iSCSI over IPoIB Known Issues

1. When working with iSCSI over IPoIB, LRO must be -
disabled (even if IPoIB is set to connected mode)
due to a a bug in older kernels which causes a kernel
panic.
3.5.7 SDP Known Issues

1. #664110: SDP is currently not supported in mlx5 -
driver (Connect-IB and Connect-X 4 adapter cards)
3.6 Storage Protocols Known Issues

3.6.1 Storage Known Issues
1. Older versions of rescan_scsi_bus.sh may not If encountering such issues, it is recommended
recognize some newly created LUNs. to use the '-c' flag.
3.6.2 SRP Known Issues

1. MLNX_OFED SRP installation breaks the -
ibmvstgt and ibmvscsi symbol resolution in
RHEL7.0
2. SRP daemon does not start at boot. Add “service srpd start” to rc.local or
start it manually.
3. srp_daemon fails to connect on ConnectX-4 VF. -
3.6.3 SRP Interop Known Issues

1. The driver is tested with Storage target vendors rec- -
ommendations for multipath.conf extensions (ZFS,
DDN, TMS, Nimbus, NetApp).
3.6.4 DDN Storage Fusion 10000 Target Known Issues

1. DDN does not accept non-default P_Key connection -
establishment.
3.6.5 Oracle Sun ZFS Storage 7420 Known Issues

1. Ungraceful power cycle of an initiator connected -
with Targets DDN, Nimbus, NetApp may result in
temporary "stale connection" messages when initia-
tor reconnects.
Rev 3.4-1.0.0.0
3.6.6 iSER Initiator Known Issues

1. On SLES OSs, the ib_iser module does not load Add a dummy interface using iscsiadm:
on boot. • # iscsiadm -m iface -I ib_iser -
o new
• # iscsiadm -m iface -I ib_iser -
o update -n iface.trans-
port_name -v ib_iser
2 Ubuntu12 requires update of user space -
open-iscsi to v2.0.873
3 The initiator does not respect interface parameter Configure each interface on a different subnet.
while logging in.
4 iSCSID v2.0.873 can enter an endless loop on bind -
error.
5 iSCSID may hang if target crashes during logout -
sequence (reproducible with TCP).
6 #440756: SLES12: Logging in with PI disabled fol- -
lowed by a log out and re-log in with PI enabled,
without flushing multipath might cause the block
layer to panic.
7 #453232: Ubuntu14.04: Stress login/logout might -
cause block layer to invoke a WARN trace.
8 #683370: iSER small read IO (< 8k) performance Set module param always_register=N
degrades compared to previous versions. $ modprobe ib_iser always_regis-
iSER performs memory registration for each IO and ter=N
avoids sending a global memory key to the target.
Sending the global memory key to the wire should
only be done in a trusted environment and is not rec-
ommended to use over the Internet protocol.
3.6.7 iSER Target Known Issues

1. iSER Target currently supports only the following -
OSs (distribution kernel):
• RHEL 7.0/7.1/7.2
• SLES12/12.1
• Ubuntu14.04, Ubuntu15.04
2 RHEL/CentOS 7.0: Discovery over RDMA is not -
supported.
3 ib_isert is unavailable on custom kernels after run- 1. Add "isert=y" to the mlnx_add_ker-
ning the mlnx_add_kernel_support.sh script. nel_support.sh script after "cat <<
EOF > ofed.conf".
2. Use the updated script to build MLNX-
_OFED for the custom kernel.
3.6.8 ZFS Appliance Known Issues

1. Connection establishment occurs twice which may -
cause iSER to log a stack trace.
3.6.9 Erasure Coding Verbs Known Issues

1. The Erasure-coding logical block size must be -
aligned to 64 bytes
2 Only w=1,2,3,4 are supported (w corresponds to the -
Galois symbol size - GF(2^w))
3 ibv_exp_ec_mem must pass with the following -
restrictions:
• num_data_sge must be equal to K (property of
the EC calc)
• num_code_sge must be equal to M (property of
the EC calc)
3.7 Virtualization
3.7.1 SR-IOV Known Issues
1. #858628: PCI error handling is not supported during -
driver reload. This might cause a kernel panic or
calltrace.
2. #860385: Creating 127 VFs may cause kernel panic -
in SLES11 SP4 KVM with Kernel 3.0.101-63
because of a IOMMU kernel bug.
3. #866875: During VM shutdown, kernel panic may -
occur as a result of using the ndo_get_-
phys_port_id callback during shutdown.
4. #822781: SR-IOV is not supported in systems with a -
page size greater than 16KB since this is the maxi-
mal VF uar size supported.
5. #795697: [mlx4] While spoof-check filters the The driver must be restarted for the disable-
incoming traffic to a VM, when this feature is dis- ment of the feature to take effect and all traffic
abled, traffic still does not reach the VM. to be reached to the VM.
6. #791101: [mlx4] Spoof-check may be turned on for -
MAC address 00:00:00:00:00:00
7. #835065: [mlx5] When working with InfiniBand -
QoS, the bandwidth for VFs that are attached to
VMs might not be spread according to the QoS con-
figuration if not enough cores are assigned to the
VM.
Rev 3.4-1.0.0.0

8. #784940: Currently, the firmware cannot process -
many page requests in parallel as the driver pro-
cesses page requests serially. Therefore, enabling/
disabling a large number of VFs will often cause an
driver slowdown.
9. #784954: When SR-IOV is disabled, the VF driver -
receives pci_err_detected event and a teardown
flow will be started. During the teardown flow, all
firmware commands will fail because the function is
already deleted.
10. #819595: [ConnectX-3 Pro] In case a VF is set to -
VST mode on the same port following QinQ config-
uration, that VF will insert C-VLAN not only to
untagged packets, but also to tagged packets. The
packets that are tagged twice will be dropped by the
switch or by the destination host since they have two
C-VLANs.
11. #775944: Bonding VFs on the same physical port -
using bonding mode 0 requires configuration of
fail_over_mac=1.
12. When using legacy VMs with MLNX_OFED 2.x -
hypervisor, you may need to set the
'enable_64b_cqe_eqe' parameter to zero on the
hypervisor.
It should be set in the same way that other module
parameters are set for mlx4_core at module load
time.
For example, add “options mlx4_core
enable_64b_cqe_eqe=0” as a line in the file /
etc/modprobe.d/mlx4_core.conf.
13. #381754: mlx4_port1_mtu sysfs entry shows a -
wrong MTU number in the VM.
14. #426988: When at least one port is configured as Use both the num_vfs and the probe_vf in
InfiniBand, and the num_vfs is provided but the the modprobe line.
probe_vf is not, HCA initialization fails.
15. #385750/378528: When working with a bonding Unload the module mlx4_ib and reload it in the
device to enslave the Ethernet devices in active- VM.
backup mode and failover MAC policy in a Virtual
Machine (VM), establishment of RoCE connections
may fail.
16. Attaching or detaching a Virtual Function on Unload the mlx4_core module in the hypervi-
SLES11 SP3 to a guest Virtual Machine while the sor before attaching or detaching a function to
mlx4_core driver is loaded in the Virtual Machine or from the guest.
may cause a kernel panic in the hypervisor.
17. #392172: When detaching a VF without shutting Shut down the driver in the VM before detach-
down the driver from a VM and reattaching it to ing the VF.
another VM with the same IP address for the Mella-
nox NIC, RoCE connections will fail

18. Enabling SR-IOV requires appending the -
“intel_iommu=on” option to the relevant OS in
file /boot/grub/grub.conf or /boot/grub2/
grub.cfg, depending on the OS installed.
Without that SR-IOV cannot be loaded.
19. On various combinations of Hypervisor/OSes and Attach/detach VFs to/from a VM only while
Guest/OSes, an issue might occur when attaching/ that VM is down.
detaching VFs to a guest while that guest is up and
running.
20. The known PCI BDFs for all VFs in kernel com- -
mand line should be specified by adding xen-pci-
back.hide
For further information, please refer to
http://wiki.xen.org/wiki/Xen_PCI_Passthrough
21. The inbox qemu version (2.0) provided with Ubuntu -
14.04 does not work properly when more than 2
VMs are run over an Ubuntu 14.04 Hypervisor.
22. SR-IOV UD QPs are forced by the Hypervisor to -
use the base GID (i.e., the GID that the VF sees in its
GID entry at its paravirtualized index 0). This is
needed for security, since UD QPs use Address Vec-
tors, and any GID index may be placed in such a
vector, including indices not belonging to that VF.
23. Attempting to attach a PF to a VM when SR-IOV is -
already enabled on that PF may result in a kernel
panic.
24. osmtest on the Hypervisor fails when SR-IOV is -
enabled. However, only the test fails, OpenSM will
operate correctly with the host. The failure reason is
that if an mcg is already joined by the host, a subse-
quent join request for that group succeeds automati-
cally (even if the join parameters in the request are
not correct). This success does no harm.
25. If a VM does not support PCI hot plug, detaching an -
mlx4 VF and probing it to the hypervisor may cause
the hypervisor to crash.
26. QPerf test is not supported on SR-IOV guests in -
Connect-IB cards.
27. On ConnectX®-3 HCAs with firmware version -
2.32.5000 and later, SR-IOV VPI mode works only
with Port 1 = ETH and Port 2 = IB.
28. Occasionally, the lspci | grep Mellanox com- 1. Locate the file:
mand shows incorrect or partial information due to $locate pci.ids
the current pci.ids file on the machine. 2. Manually update the file according to the lat-
est version available online at:
https://pci-ids.ucw.cz/v2.2/pci.ids
This file can also be downloaded (using the fol-
lowing command: update-pciids).
29. SR-IOV is not supported in AMD architecture. -
Rev 3.4-1.0.0.0

30. #506512: Setting 1 Mbit/s rate limit on Virtual -
Functions (Qos Per VF feature) may cause TX
queue transmit timeout.
31. DC transport type is not supported on SR-IOV VMs. -
32. #567908: Attaching a VF to a VM before unbinding -
it from the hypervisor and then attempting to destroy
the VM, may cause the system to hang for a few
minutes.
33. When using SR-IOV make sure to set interface to -
down and unbind BEFORE unloading driver/remov-
ing VF/restarting VM or kernel will lock. (reboot
needed)
Basically, clean-up might not work perfectly so user
should do it manually.
34. #601749: Since the guest MAC addresses are con- -
figured to be all zeroes by default, in ConnectX-4
the administrator must explicitly set the VFs’ MAC
addresses. otherwise the Guest VM will see MAC
zero and traffic is not passed.
35. #649366: Restarting the PF (Hypervisor) driver -
while Virtual Functions are assigned is not allowed
in RH7 and above due to a vfio-pci bug.
36. #639046: Due to an issue with SR-IOV loopback, -
prevention "Duplicate IPv6 detected" are
seen in the VF driver.
37. #655410: [ConnectX-4/Connect-IB] Failed to 1. Add pci=realloc=on to the grub com-
enable SR-IOV due to errors in PCI or BIOS. mand line.
2. Add more memory to the server.
3. Upgrade BIOS version.
38. #651119: Kernel panic may occur while running -
IPv6 UDP on SR-IOV ConnectX-4 environment
39. #669910: Bind/Unbind over ConnectX-4 Hypervi- -
sor may cause system lockup.
40. #650458: Occasionally, IPv6 might not function -
properly and cause lockup on SR-IOV ConnectX-4
environment.
41. #688551: In ConnectX-3 adapter cards, the extended -
counter port_rcv_data_64 on the VF may not be
updated in some flows.
42. #690656/690674: When the physical link is down, -
any traffic from the PF to any VF on the same port
will be dropped.
43. #691661: When in LAG mode and the Virtual Func- Probe one of the VFs in the hypervisor and use
tions are present (VF LAG), the IP address given to for RoCE.
the bonding interface (in the hypervisor) cannot be
used for RoCE as well.
44. #691661: Ethernet SR-IOV in ConnectX-4 requires -
firmware version 12.14.1100 and higher

45. #737434: VF vport statistics are not cleared upon -
ifconfig up/down.
46. #738464: In SLES11 SP4, user cannot open all VFs -
announced in sriov_totalvfs. However he can
set the num_vfs up to maximum sriov_totalvfs-1
vfs.
47. #784127: While disabling SR-IOV, all firmware -
teardown flow commands are expected to fail and
error messages will be reported in the dmesg.
48. #784146: Creating/destroying as many as 64 VFs -
may sometimes take longer time than usual on some
setups.
49. #766105: Due to a bug in some QEMU versions, Upgrade to the latest version of QEMU in the
interrupts do not function properly for Virtual Func- hypervisor.
tions. This causes the driver initialization to fail, and
such error message will be printed: "mlx4_core
0000:0b:00.0: command 0x31 timed out
(go bit not cleared)
mlx4_core 0000:0b:00.0: NOP command
failed to generate interrupt (IRQ 57),
aborting".
50. When working with InfiniBand QoS, the bandwidth -
for Virtual Functions (VF) that are attached to Vir-
tual Machines (VM) might not be spread according
to the QoS configuration if not enough cores are
assigned to the Virtual Machine.
3.8 Resiliency
3.8.1 Reset Flow Known Issues

1. SR-IOV non persistent configuration (such as VGT, Reset Admin configuration post Reset Flow
VST, Host assigned GUIDs, and QP0-enabled VFs)
may be lost upon Reset Flow.
2. Upon Reset Flow or after running restart driver, Reset the VLANs using the ifup command.
Ethernet VLANs are lost.
3. Restarting the driver or running connectx_port_- -
config when Reset Flow is running might result in
a kernel panic
4. Networking configuration (e.g. VLANs, IPv6) -
should be statically defined in order to have them set
after Reset Flow as of after restart driver.
5. After recovering from an EEH event, mlx5_core/ -
mlx4_core unload may fail due to a bug in some ker-
nel versions. The bug is fixed in Kernel 3.15
Rev 3.4-1.0.0.0
3.9 Miscellaneous Known Issues

3.9.1 General Known Issues
1. #856033: The following PCIe bus error on Qual- 1.Edit the kernel parameters (in grub) and add
comm ARM processor might appear when mapping qiommu.identity_map_qiom-
a large number of DMA addresses: mus=PCIE0_MMU,PCIE4_MMU
“AER: Corrected error received: id=0000 (The bus numbers depend on the ConnectX-4
slot.)
PCIe Bus Error: severity=Corrected, type=Transac-
2.Reboot the server.
tion Layer, id=0000(Receiver ID)
device [17cb:0400] error status/mask=00002000/
00004000
[13] Advisory Non-Fatal
mlx5_warn:mlx5_0:dump_cqe:257:(pid 0): dump
error cqe
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 12007806 25000063 8728c8d3”
3. On ConnectX-2/ConnectX-3 Ethernet adapter cards, N/A.
there is a mismatch between the GUID value Please use the GUID value returned by the fab-
returned by firmware management tools and that ric/driver utilities (not 0xfffff).
returned by fabric/driver utilities that read the GUID
via device firmware (e.g., using ibstat). Mlxburn/
flint return 0xffff as GUID while the utilities return a
value derived from the MAC address. For all driver/
firmware/software purposes, the latter value should
be used.
4. #552870/548518: On rare occasions, under -
extremely heavy MAD traffic, MAD (Management
Datagram) storms might cause soft-lockups in the
UMAD layer.
5. Packets are dropped on the SM server on big clus- Increase the recv_queue_size of ib_mad
ters. module parameter for SM server to 8K.
The recv_queue_size default size (4K)
6. #663434: On ConnectX-4/ConnectX-4 Lx, when Run update-pciids
running "lspci" in RH7.0/7.1, the device informa-
tion is displayed incorrect or the device is unnamed.
7. #767016: Resetting hardware counters after netdev -
goes up can break statistics scripts.
3.9.2 ABI Compatibility Known Issues

1. MLNX_OFED v2.3-1.0.1 is not ABI compatible Recompile the application over the new
with previous MLNX_OFED/OFED versions. MLNX_OFED version
3.9.3 Connection Manager (CM) Known Issues

1. When 2 different ports have identical GIDs, All ports must have different GIDs.
the CM might send its packets on the wrong
port.
2. #781382: The number of local ports that Modify the range of available ports for binding, run:
rdma_cm ID can bind to is limited. This lim- sysctl net.ipv4.ip_lo-
itation depends on the OS dynamics. cal_port_range="MIN MAX"
The MIN and MAX values can range from 0 to 65535.
Note: Modifying the range also affects the range of
available ports for socket applications (TCP/IP) even
though the pool is not mutual between the RDMA
stack and the TCP/IP stack.
3.9.4 Fork Support Known Issues

1. Fork support from kernel 2.6.12 and above is avail- -
able provided that applications do not use threads.
fork() is supported as long as the parent process
does not run before the child exits or calls exec().
The former can be achieved by calling
wait(childpid), and the latter can be achieved by
application specific means. The Posix system() call
is supported.
3.9.5 Uplinks Known Issues

1. On rare occasions, ConnectX®-3 Pro adapter card Restart the driver
may fail to link up when performing parallel detect
to 40GbE.
3.9.6 Resources Limitation Known Issues

1. The device capabilities reported may not be reached -
as it depends on the system on which the device is
installed and whether the resource is allocated in the
kernel or the userspace.
2. #387061: mlx4_core can allocate up to 64 MSI-X -
vectors, an MSI-X vector per CPU.
Rev 3.4-1.0.0.0

3. Setting more IP addresses than the available GID -
entries in the table results in failure and the
"update_gid_table error message is displayed:
GID table of port 1 is full. Can't add
<address>" message.
4. #553657: Registering a large amount of Memory -
Regions (MR) may fail because of DMA mapping
issues on RHEL 7.0.
5. Occasionally, a user process might experience some To free memory to allow it to be allocated in a
memory shortage and not function properly due to user process, run the drop_caches procedure
Linux kernel occupation of the system’s free mem- below.
ory for its internal cache. Performing the following steps will cause the
kernel to flush and free pages, dentries and
inodes caches from memory, causing that
memory to become free.
Note: As this is a non-destructive operation

and dirty objects are not freeable, run `sync'
first.
• To free the pagecache:

echo 1 > /proc/sys/vm/drop_caches
• To free dentries and inodes:
• To free pagecache, dentries and inodes:
3.9.7 Accelerated Verbs Known Issues

1. On ConnectX®-4 Lx, the following may not be sup- -
ported when using Multi-Packet WR flag (IBV_EX-
P_QP_BURST_CREATE_ENABLE_MULTI_PAC
KET_SEND_WR) on QP-burst family creation:
• ACLs
• SR-IOV (eSwitch offloads)
• priority and dscp forcing
• Loopback decision.
• VLAN insertion
• encapsulation (encap/decap)
• sniffer
• Signature
3.10 InfiniBand Fabric Utilities Known Issues

3.10.1 Performance Tools Known Issues
1. perftest package in MLNX_OFED v2.2-1.0.1 and -
onwards does not work with older versions of the
driver.
3.10.2 Diagnostic Utilities Known Issues

1. When running the ibdiagnet check nodes_info on Run ibdiagnet --skip nodes_info
the fabric, a warning specifying that the card does
not support general info capabilities for all the
HCAs in the fabric will be displayed.
2. ibdump does not work when IPoIB device managed
Enable IPoIB Flow Steering and restart the
Flow Steering is OFF and at least one of the ports is
driver.
configured as InfiniBand. For further information, please refer to MLNX-
_OFED User Manual section Enable/Disable
Flow Steering.
3. #736136: The maximum number of HCAs shown by -
ibstat is 32 HCAs.
3.10.3 Tools Known Issues

1. Running ibdump in InfiniBand mode with firmware Run ibdump with firmware v2.33.5000 and
older than v2.33.5000, may cause the server to hang higher
due to a firmware issue.
Rev 3.4-1.0.0.0
4 Bug Fixes History

This table lists the bugs fixed in this release.
Discovered Fixed in
# Issue Internal Reference Number: Description
in Release Release
1. irqbalancer #854344: Fixed the issue where mlnx_affinity 3.3-1.0.0.0 3.4-1.0.0.0
script on RHEL/CentOS7.x host did not disable or
enable irqbalancer.
2. QoS #824736: Fixed wrong skprio2UP mapping by 3.3-1.0.0.0 3.4-1.0.0.0
removing it and its scripts, such as tc_wrap, from the
driver. This mapping should now be done using the
kernel's set_egress_map commands.
Note: Only for RDMACM over old kernels, the
original skprio2UP mapping in tc_wrap remains
valid as these kernels do not support set_egress_-
map.
3. ibacm #824775: Fixed the issue where starting ibacm dae- 3.3-1.0.0.0 3.4-1.0.0.0
mon failed on Debian based distributions with the
following message:
“/etc/init.d/ibacm: line 37: /sbin/start_daemon: No
such file or directory”.
4. IPoIB #799004: Fixed the issues of when establishing 3.0-2.0.0.0 3.4-1.0.0.0
IPoIB CM connection, a race could occur if there
were many CM connections taking place while the
driver was going up and down. This race in the
IPoIB driver could have caused memory corruption.
5. ARM #777733/778099: Fixed the issue where in ARM 3.3-1.0.0.0 3.4-1.0.0.0
architecture, multiple kernel panics of mlx4 and
mlx5 drivers were observed as a result of undefined
behavior of vmap(virt_to_page(dma_alloc_coher-
ent)) call sequence on driver load, by allocating con-
tiguous memory instead of vmapping it.
6. mlx4_en #826686: Fixed the issue where server reboot could 3.3-1.0.0.0 3.4-1.0.0.0
get stuck because of kernel panic in mlx-
4_en_get_drvinfo() that is called from asynchronous
event handler.
7. #824130: Fixed the issue where ethtool self test used 3.3-1.0.0.0 3.4-1.0.0.0
to fail on interrupt test after timeout if mlx4_ib mod-
ule was not loaded.
8. mlx4 driver #855311: Fixed the issue of when using RDMA 3.0-1.0.1 3.4-1.0.0.0
READ with a higher value than 30 SGEs in the WR,
this might have lead to “local length error”.
9. mlx5 driver #786720: Fixed a crash that used to occur when try- 3.3-1.0.0.0 3.4-1.0.0.0
ing to bring the interface up in a kernel that did not
support accelerated RFS (aRFS).
Rev 3.4-1.0.0.0 Bug Fixes History
Discovered Fixed in
in Release Release
10. SR-IOV #781747: Fixed the issue of when attempting to dis- 3.3-1.0.0.0 3.4-1.0.0.0
able SR-IOV while there are any VF netdevs open,
the operation would fail and the driver would hang.
11. #568602: Fixed the issue of when repeating change 3.0-2.0.0 3.4-1.0.0.0
of the mlx5_num_vfs value from 0 to non-zero
might have caused kernel panic in the PF driver.
12. IPoIB #814941: Fixed the issue where as a result of the 3.3-1.0.0 3.3-1.0.4.0
ndo_set_mac_address support in IPoIB, a memory
corruption issue was exposed in the bonding driver
over IPoIB interface. This memory corruption issue
could have caused unpredictable behavior, such as
ports becoming dysfunctional in RedHat, kernel pan-
icking in Ubuntu devices, and other behaviors.
13. IB Core #781183: Fixed a potential security breach in the 3.2-2.0.0 3.3-1.0.0
InfiniBand stack that was caused due to wrong reli-
ance on the write system call. For more information,
please see CVE-2016-4565 tracker.
14. RoCE #592652: Fixed the issue where the InfiniBand error 3.1-0.0.7 3.3-1.0.0
counters found under
/sys/class/infiniband/<mlx5_dev>/ports/
<port>/ did not function properly in ConnectX-4
adapter cards.
15. Installation #765162: Fixed the issue where dapl package was 2.0.2.0.5 3.3-1.0.0
missing in MLNX_OFED for Ubuntu PPC64LE.
16. TX Queue Counter #748308: Changed TX queue counter format to: 3.2-2.0.0 3.3-1.0.0
xq_[tc]*[ring/channel].
17. RDMA Sniffer #751097: Fixed RDMA sniffer functionality issues. 3.2-2.0.0 3.3-1.0.0
18. IPoIB #751096: Fixed IPoIB Connected Mode in Con- 3.2-2.0.0 3.3-1.0.0
nectX-3 functionality issues.
19. #769688: Fixed the issue where in order to change 3.2-2.0.0 3.3-1.0.0
the IPoIB mode (connected/datagram), the interface
had to be taken down (via ifconfig ibX down or
ifdown ibX). Now, the mode can be changed
regardless of the interface’s state (“up” or “down”).
20. mlx4_en #704756: Added DCB PFC support through CEE 3.2-2.0.0 3.3-1.0.0
netlink commands to prevent Priority Flow Control
mode functionality issues on the host side.
21. SR-IOV #648680/655070: Fixed an issue which added error 3.1-1.0.5 3.3-1.0.0
messages to the dmesg when a VF used ethtool facil-
ities.
22. #690772/690656: Fixed an issue which cased any 3.2-1.0.1.1 3.3-1.0.0
traffic from PF to any VF on the same port to drop
when the physical link was down.
Rev 3.4-1.0.0.0
Discovered Fixed in
in Release Release
23. mlx5 driver #708299: Fixed kernel’s back-ports of XPS and 3.2-1.0.1.1 3.2-2.0.0
affinity that did not have CONFIG_CPUMASK_OFF-
STACK
24. #685082: Added support for Rate Limit 0 to enable 3.2-1.0.1.1 3.2-2.0.0
unlimited rate limiter and to prevent max rate zero
traffic lose.
25. SR-IOV #667559: Fixed an issue which enabled SR-IOV on 3.2-1.0.1.1 3.2-2.0.0
RHEL 6.7 although SR-IOV was already enabled. A
check was added to make sure SR-IOV is not
enabled before enabling it.
26. eIPoIB #682750: Fixed race between the udev that changes 3.0-1.0.1 3.2-2.0.0
the interface name of eth_ipoib driver and the
eIPoIB daemon that configured the same interface.
27. Ethernet traffic #692520: Fixed an issue which prevented ConnectX- 3.2-1.0.1.1 3.2-2.0.0
4/ConnectX-4 Lx adapter cards from running Ether-
net traffic on Big Endian arch machines.
28. Performance #668346: Set close NUMA node as default for RSS. 3.2-1.0.1.1 3.2-2.0.0
29. mlx4_en #696150: Fixed an issue where the ARP request 3.2-1.0.1.1 3.2-2.0.0
packets destined for a proxy VXLAN interface were
not handled correctly when GRO was enabled.
30. Counters #698795: Fixed an issue which prevented the calcu- 3.0-1.0.1 3.2-2.0.0
lated software counters (the correct ones) from being
shown and provided the error counters that were pre-
viously inactive.
31. Virtualization #597110: Fixed an issue which prevented the driver 3.1-1.0.3 3.2-1.0.1.1
from reaching VLAN when the VLAN was created
over a Linux bridge.
32. mlx5 driver # 656298: Fixed an issue in the driver (in ConnectX- 3.1-1.0.3 3.2-1.0.1.1
4) that discarded s-tag VLAN packets when in Pro-
miscuous Mode.
33. # 647865: Fixed an issue which prevented 3.0-1.0.1 3.2-1.0.1.1
PORT_ERR event to be propagated to the user-space
application when the port state was changed from
Active to Initializing.
34. HPC Acceleration # 663975: Fixed a rare issue which allowed the 3.1-1.0.3 3.2-1.0.1.1
packages knem package to run depmod on the wrong kernel
version.
35. IB/Core # 666992: Fixed a race condition in the IB/umad 3.0-2.0.1 3.2-1.0.1.1
layer that caused NULL pointer dereference.
36. IPoIB # 657718: Fixed an IPoIB issue that caused connec- 3.1-1.0.3 3.2-1.0.1.1
tivity lost after server’s restart in a cluster.
37. Driver un-installation # 619272: Fixed an issue causing MLNX_OFED to 3.1-1.0.3 3.2-1.0.1.1
remove the “mutt” package upon driver uninstall.
Discovered Fixed in
in Release Release
38. PFC # 613514: Added a warning message in dmesg, noti- 3.1-1.0.3 3.2-1.0.1.1
fying the user that the PFC RX/TX cannot be
enabled simultaneously with Global Pauses. In this
case Global Pauses will be disabled.
39. IB MAD #606916: Fixed an issue causing MADs to drop in 3.1-1.0.0 3.1-1.0.3
large scale clusters.
40. SR-IOV #367410: Fixed InfiniBand counters which were 2.1-1.0.0 3.1-1.0.0
unavailable in the VM.
41. RoCE #549687: Fixed InfiniBand traffic counters that are 3.0-1.0.1 3.1-1.0.0
found under /sys/class/infiniband/<mlx-
5_dev>/ports/<port>/ which dis not function
properly in ConnectX-4 adapter cards.
42. Virtualization #589247/591877: Fixed VXLAN functionality 3.0-2.0.1 3.1-1.0.0
issues.
43. Performance TCP/UDP latency on ConnectX®-4 was higher than 3.0-2.0.1 3.1-1.0.0
expected.
44. TCP throughput on ConnectX®-4 achieved full line 3.0-2.0.1 3.1-1.0.0
rate.
45. #568718: Fixed an issue causing inconsistent perfor- 3.0-2.0.1 3.1-1.0.0
mance with ConnectX-3 and PowerKVM 2.1.1.
46. #552658: Fixed ConnectX-4 traffic counters. 3.0-2.0.1 3.1-1.0.0
47. num_entries #572068: Updated the desired num_entries in 3.0-1.0.1 3.1-1.0.0
each iteration, and accordingly updated the offset of
the WC in the given WC array.
48. mlx5 driver #536981/554293: Fixed incorrect port rate and port 3.0-2.0.1 3.1-1.0.0
speed values in RoCE mode in ConnectX-4.
49. IPoIB #551898: In RedHat7.1 kernel 3.10.0-299, when 3.0-2.0.1 3.1-1.0.0
sending ICMP/TCP/UDP traffic over Connect-IB/
ConnectX-4 in UD mode, the packets were dropped
with the following error:
UDP: bad checksum...
50. openibd #596458: Fixed an issue which prevented openibd 3.0-2.0.1 3.1-1.0.0
from starting correctly during boot.
51. Ethernet #589207: Added a new module parameter to control 3.0-2.0.1 3.1-1.0.0
the number of IRQs allocated to the device.
52. mlx5 driver #576326: Fixed an issue on PPC servers which pre- 3.0-2.0.1 3.1-1.0.0
vented PCI from reloading after EEH error recovery.
53. OpenSM #569369: Fixed an issue which prevented the 3.0-2.0.1 3.1-1.0.0
OpenSM package from being fully removed when
uninstalling MLNX_OFED v3.0-2.0.1
Rev 3.4-1.0.0.0
Discovered Fixed in
in Release Release
54. mlx5_en #568169: Added the option to toggle LRO ON/OFF 3.0-2.0.1 3.1-1.0.0
using the “-K” flags. The priv flag hw_lro will
determine the type of LRO to be used, if the flag is
ON, the hardware LRO will be used, otherwise the
software LRO will be used.
55. #568168: Added the option to toggle LRO ON/OFF 3.0-2.0.1 3.1-1.0.0
using the “-K” flags.
56. #551075: Fixed race when updating counters. 3.0-2.0.1 3.1-1.0.0
57. #550275: Fixed scheduling while sending atomic 3.0-2.0.1 3.1-1.0.0
dmesg warning during bonding configuration.
58. #550824: Added set_rx_csum callback implemen- 3.0-2.0.1 3.1-1.0.0
tation.
59. mlx4_ib #535884: Fixed mismatch between SL and VL in 3.0-1.0.1 3.1-1.0.0
outgoing QP1 packets, which caused buffer overruns
in attached switches at high MAD rates.
60. SR-IOV/RoCE #542722: Fixed a problem on VFs where the RoCE 2.3-1.0.1 3.1-1.0.0
driver registered a zero MAC into the port's MAC
table (during QP1 creation) because the ETH driver
had not yet generated a non-zero random MAC for
the ETH port.t
61. #561866: Removed BUG_ON assert when checking 3.0-1.0.1 3.1-1.0.0
if the ring is full.
62. libvma #541149: Added libvma support for Debian 8.0 3.0-2.0.1 3.1-1.0.0
x86_64 and Ubuntu 15.04
63. IPoIB Fixed an issue which prevented the failure to destroy 3.0-1.0.1 3.0-2.0.0
QP upon IPoIB unload on debug kernel.
64. Configuration Fixed an issue which prevented the driver version to 3.0-1.0.1 3.0-2.0.0
be reported to the Remote Access Controller tools
(such as iDRAC)
65. SR-IOV Passed the correct port number in port-change event 2.4-1.0.0 3.0-2.0.0
to single-port VFs, where the actual physical port
used is port 2.
66. Enabled OpenSM, running over a ConnectX-3 HCA, 3.0-1.0.1 3.0-2.0.0
to manage a mixed ConnectX-3/ConnectX-4 net-
work (by recognizing the "Well-known GID" in mad
demux processing).
67. Fixed double-free memory corruption in case where 3.0-1.0.1 3.0-2.0.0
SR-IOV enabling failed (error flow).
68. Start-up sequence Fixed a crash in EQ's initialization error flow. 3.0-1.0.1 3.0-2.0.0
69. Installation #554253: MLNX_OFED v3.0-1.0.1 installation 3.0-1.0.1 3.0-2.0.0
using yum fails on RH7.1
Discovered Fixed in
in Release Release
70. mlx5 driver #542686: In PPC systems, when working with Con- 3.0-1.0.1 3.0-2.0.0
nectX®-4 adapter card configured as Ethernet,
driver load fails with BAD INPUT LENGTH.
dmesg:
command failed, status bad input
length(0x50), syndrome 0x9074aa
71. Error counters such as: CRC error counters, RX out 3.0-1.0.1 3.0-2.0.0
range length error counter, are missing in the
ConnectX-4 Ethernet driver.
72. Changing the RX queues number is not supported in 3.0-1.0.1 3.0-2.0.0
Ethernet driver when connected to a ConnectX-4
card.
73. Ethernet Hardware checksum call trace may appear when 3.0-1.0.1 3.0-2.0.0
receiving IPV6 traffic on PPC systems that uses
CHECKSUM COMPLETE method.
74. mlx4_en Fixed ping/traffic issue occurred when RXVLAN 2.4-1.0.4 3.0-1.0.1
offload was disabled and CHECKSUM COM-
PLETE was used on ingress packets.
75. Security CVE-2014-8159 Fix: Prevented integer overflow in 2.0-2.0.5 2.4-1.0.4
IB-core module during memory registration.
76. mlx5_ib Fixed the return value of max inline received size in 2.3-2.0.1 2.4-1.0.0
the created QP.
77. Resolved soft lock on massive amount of user mem- 2.3-2.0.1 2.4-1.0.0
ory registrations
78. InfiniBand Counters Occasionally, port_rcv_data and port_xmit_- 2.3-1.0.1 2.4-1.0.0
data counters may not function properly.
79. mlx4_en LRO fixes and improvements for jumbo MTU. 2.3-2.0.1 2.4-1.0.0
80. Fixed a crash occurred when changing the number of 2.2-1.0.1 2.4-1.0.0
rings (ethtool set-channels) when interface con-
nected to netconsole.
81. Fixed ping issues with IP fragmented datagrams in 2.2-1.0.1 2.4-1.0.0
MTUs 1600-1700.
82. The default priority to TC mapping assigns all prior- 2.3-1.0.1 2.4-1.0.0
ities to TC0. This configuration achieves fairness in
transmission between priorities but may cause unde-
sirable PFC behavior where pause request for prior-
ity “n” affects all other priorities.
83. mlx5_ib Fixed an issue related to large memory regions regis- 2.3-2.0.1 2.3-2.0.5
tration. The problem mainly occurred on PPC sys-
tems due to the large page size, and on non PPC
systems with large pages (contiguous pages).
84. Fixed an issue in verbs API: fallback to glibc on con- 2.3-2.0.1 2.3-2.0.5
tiguous memory allocation failure
85. IPoIB Fixed a memory corruption issue in multi-core sys- 2.3-2.0.1 2.3-2.0.5
tem due to intensive IPoIB transmit operation.
Rev 3.4-1.0.0.0
Discovered Fixed in
in Release Release
86. IB MAD Fixed an issue to prevent process starvation due to 2.3-2.0.1 2.3-2.0.5
MAD packet storm.
87. IPoIB #433348: Fixed an issue which prevented the spread 2.3-1.0.1 2.3-2.0.0
of events among the closet NUMA CPU when only a
single RX queue existed in the system.
88. Returned the CQ to its original state (armed) to pre- 2.3-1.0.1 2.3-2.0.0
vent traffic from stopping
89. Fixed a TX timeout issue in CM mode, which 2.1-1.0.0 2.3-2.0.0
occurred under heavy stress combined with ifup/
ifdown operation on the IPoIB interface.
90. mlx4_core Fixed "sleeping while atomic" error occurred when 2.3-1.0.1 2.3-2.0.0
the driver ran many firmware commands simultane-
ously.
91. mlx4_ib Fixed an issue related to spreading of completion 2.1-1.0.0 2.3-2.0.0
queues among multiple MSI-X vectors to allow bet-
ter utilization of multiple cores.
92. Fixed an issue that caused an application to fail 2.3-1.0.1 2.3-2.0.0
when attaching Shared Memory.
93. mlx4_en Fixed dmesg warnings: "NOHZ: local_soft- 2.3-1.0.1 2.3-2.0.0
irq_pending 08".
94. Fixed erratic report of hardware clock which caused 2.1-1.0.0 2.3-2.0.0
bad report of PTP hardware Time Stamping.
95. mlx5_core Fixed race when async events arrived during driver 2.3-1.0.1 2.3-2.0.0
load.
96. Fixed race in mlx5_eq_int when events arrived 2.3-1.0.1 2.3-2.0.0
before eq->dev was set.
97. Enabled all pending interrupt handlers completion 2.3-1.0.1 2.3-2.0.0
before freeing EQ memory.
98. mlnx.conf Defined mlnx.conf as a configuration file in mlnx- 2.1-1.0.0 2.3-2.0.0
ofa_kernel RPM
99. SR-IOV Fixed counter index allocation for VFs which 2.3-1.0.1 2.3-2.0.0
enables Ethernet port statistics.
100. iSER Fixed iSER DIX sporadic false DIF errors caused in 2.3-1.0.1 2.3-2.0.0
large transfers when block merges were enabled.
101. RoCE v2 RoCE v2 was non-functional on big Endian 2.3-1.0.1 2.3-2.0.0
machines.
102. Verbs Fixed registration memory failure when fork was 2.3-1.0.1 2.3-2.0.0
enabled and contiguous pages or ODP were used.
103. Installation Using both '-c|--config' and '--add-kernel- 2.2-1.0.1 2.3-2.0.0
support' flags simultaneously when running the
mlnxofedinstall.sh script caused installation
failure with the following on screen message
"--config does not exist".
Discovered Fixed in
in Release Release
104. IPoIB Changing the GUID of a specific SR-IOV guest after 2.1-1.0.0 2.3-1.0.1
the driver has been started, causes the ping to fail.
Hence, no traffic can go over that InfiniBand inter-
face.
105. XRC XRC over ROCE in SR-IOV mode is not functional 2.0-3.1.0 2.2-1.0.1
106. mlx4_en Fixed wrong calculation of packet true-size reporting 2.1-1.0.0 2.2-1.0.1
in LRO flow.
107. Fixed kernel panic on Debian-6.0.7 which occurred 2.1-1.0.0 2.2-1.0.1
when the number of TX channels was set above the
default value.
108. Fixed a crash incidence which occurred when 2.0-2.0.5 2.2-1.0.1
enabling Ethernet Time-stamping and running
VLAN traffic.
109. IB Core Fixed the QP attribute mask upon smac resolving 2.1-1.0.0 2.1-1.0.6
110. mlx5_ib Fixed a send WQE overhead issue 2.1-1.0.0 2.1-1.0.6
111. Fixed a NULL pointer de-reference on the debug 2.1-1.0.0 2.1-1.0.6
print
112. Fixed arguments to kzalloc 2.1-1.0.0 2.1-1.0.6
113. mlx4_core Fixed the locks around completion handler 2.1-1.0.0 2.1-1.0.6
114. mlx4_core Restored port types as they were when recovering 2.0-2.0.5 2.1-1.0.0
from an internal error.
115. Added an N/A port type to support port_type_array 2.0-2.0.5 2.1-1.0.0
module param in an HCA with a single port
116. SR-IOV Fixed memory leak in SR-IOV flow. 2.0-2.0.5 2.0-3.0.0
117. Fixed communication channel being stuck 2.0-2.0.5 2.0-3.0.0
118. mlx4_en Fixed ALB bonding mode failure when enslaving 2.0-3.0.0 2.1-1.0.0
Mellanox interfaces
119. Fixed leak of mapped memory 2.0-3.0.0 2.1-1.0.0
120. Fixed TX timeout in Ethernet driver. 2.0-2.0.5 2.0-3.0.0
121. Fixed ethtool stats report for Virtual Functions. 2.0-2.0.5 2.0-3.0.0
122. Fixed an issue of VLAN traffic over Virtual Machine 2.0-2.0.5 2.0-3.0.0
in paravirtualized mode.
123. Fixed ethtool operation crash while interface down. 2.0-2.0.5 2.0-3.0.0
124. IPoIB Fixed memory leak in Connected mode. 2.0-2.0.5 2.0-3.0.0
125. Fixed an issue causing IPoIB to avoid pkey value 0 2.0-2.0.5 2.0-3.0.0
for child interfaces.
Rev 3.4-1.0.0.0
5 Change Log History

Release Category Description
3.3-1.0.0.0 VF MAC Address [ConnectX-4/ConnectX-4 Lx] Also known as MAC spoof-check, the
Anti-Spoofing VF MAC Address Anti-Spoofing prevents malicious VFs from
faking their MAC addresses.
VF All-multi Mode [ConnectX-4/ConnectX-4 Lx] Added support for the VF to enter all-
multi RX mode, meaning that in addition to the traffic originally
targeted to the VF, it will receive all the multicast traffic sent from/
to the other functions on the same physical port.
Note: Only privileged/trusted VFs can enter the all-multi RX
mode.
VF Promiscuous [ConnectX-4/ConnectX-4 Lx] Added support for the VF to enter pro-
Mode miscuous RX mode, meaning that in addition to the traffic origi-
nally targeted to the VF, it will receive the unmatched traffic and
all the multicast traffic that reaches the physical port.
The unmatched traffic is any traffic’s DMAC that does not match
any of the VFs’ or PFs’ MAC addresses.
Note: Only privileged/trusted VFs can enter the promiscuous RX
mode.
Privileged VF [ConnectX-4/ConnectX-4 Lx] Added support for determining privi-
leged/trusted VFs so security sensitive features can be enabled for
these VFs, such as entering promiscuous and all-multi RX modes.
DCBX [ConnectX-4/ConnectX-4 Lx] Added support for standard DCBX
CEE API.
Per Priority Count- [ConnectX-4/ConnectX-4 Lx] Exposed performance counters per pri-
ers ority.
IB Error Counters [ConnectX-4/ConnectX-4 Lx] Exposed IB sysfs error counters for
mlx5 driver.
Accelerated Receive [ConnectX-4/ConnectX-4 Lx] Boosts the speed of RFS by adding
Flow Steering hardware assistance. RFS is an in-kernel-logic responsible for load
(aRFS) balancing between CPUs by attaching flows to CPUs that are used
by flow’s owner applications.
Packet Pacing for [ConnectX-4/ConnectX-4 Lx] Performs rate limit per UDP/TCP con-
UDP/TCP nection.
OFED Scripts Renamed the UP name that appears in mlnx_perf report to “TC”,
as the mlnx_perf script counts the packets and calculates the band-
width on rings that belong to the same Traffic Class (TC).
Physical Memory Added support for Physical Address Memory Region (PA-MR)
Allocation which allows managing physical memory used for posting send
and receive requests.
MAD Congestion Added an SA MAD congestion control mechanism that is configu-
Control rable using sysfs entries.
IB Router Added the ability to send traffic between two or more subnets.
Rev 3.4-1.0.0.0 Change Log History

PeerDirect Async Mellanox PeerDirect Async™ sub-system gives peer hardware
devices, such as GPU cards, and dedicated AS accelerators the
ability to take control over HCA in critical path offloading CPU.
Physical MR Allows the user to use physical addresses instead of virtual
addresses in critical path. Thus enhances performance since there is
no need in addresses translation.
RoCE v1 (Layer 2) Added the option to connect between nodes running MLNX-
Compatibility _OFED and nodes running RoCE with Layer 2 GID format.
3.2-2.0.0.0 API Changes • Support FCS scattering for Raw Packet QPs and WQs.
• Indication of L4 packet type on the receive side completions
• Support CVLAN insertion for WQs
IPoIB • Added support for the following IPoIB UD QP offloads:
• RX check summing (AKA RX csu)
• Large Send Offloads (AKA LSO)
To see the new IPoIB UD mode, run: "ethtool -k <inter-
face>"
Rev 3.4-1.0.0.0

3.2-1.0.1.1 VXLAN Hard- [ConnectX-4 / ConnectX-4 Lx] Provides scalability and security
ware Stateless Off- challenges solutions.
loads
Priority Flow Con- [ConnectX-4 / ConnectX-4 Lx] Applies pause functionality to spe-
trol (PFC) cific classes of traffic on the Ethernet link.
Offloaded Traffic [ConnectX-4 / ConnectX-4 Lx] Allows bypass kernel traffic (such
Sniffer/TCP Dump as, RoCE, VMA, DPDK) to be captured by existing packet ana-
lyzer such as tcpdump.
Ethernet Time [ConnectX-4 / ConnectX-4 Lx] Keeps track of the creation of a
Stamping packet. A time-stamping service supports assertions of proof that a
datum existed before a particular time.
Custom RoCE [ConnectX-4 / ConnectX-4 Lx] Provide a clear indication on
Counters RDMA send/receive statistics and errors.
LED Beaconing [ConnectX-4 / ConnectX-4 Lx] Enables visual identification of the
port by LED blinking.
Enhanced Trans- [ConnectX-4 / ConnectX-4 Lx] Exploits the time periods in which
mission Selection the offered load of a particular Traffic Class (TC) is less than its
standard (ETS) minimum allocated bandwidth.
Strided WQE User [ConnectX-4 / ConnectX-4 Lx] Striding RQ is a receive queue
Space comprised by work queue elements (i.e. WQEs), where multiple
packets of LRO segments (i.e. message) are written to the same
WQE.
VLAN Stripping in [ConnectX-4 / ConnectX-4 Lx] Adds access to the device's ability
Linux Verbs to offload the Customer VLAN (cVLAN) header stripping from an
incoming packet.
iSER: Remote [ConnectX-4 / ConnectX-4 Lx] Improves performance by enabling
invalidation sup- the hardware to perform implicit memory region invalidation.
port (target and ini-
tiator)
iSER: Zero-Copy [ConnectX-4 / ConnectX-4 Lx] Reduces the latency of small writes
ImmediateData by avoiding an extra memory copy in the iSER target stack.
iSER: Indirect [ConnectX-4 / ConnectX-4 Lx] Uses ConnectX®-4 adapter card’s
Memory Registra- Indirect Memory Registration capabilities to avoid bounce buffer
tion strategy implementation and to reduce the latency of highly
unaligned vectored IO operations, and also in cases of BIO merg-
ing.
Vector Calculation/ [ConnectX-4 / ConnectX-4 Lx] Uses the HCA for offloading era-
Erasure coding off- sure coding calculations.
load
Virtual Guest Tag- [ConnectX-3 / ConnectX-3 Pro] VGT+ is an advanced mode of
ging (VGT+) Virtual Guest Tagging (VGT), in which a VF is allowed to tag its
own packets as in VGT, but is still subject to an administrative
VLAN trunk policy.
Link Aggregation [ConnectX-3 / ConnectX-3 Pro] Protects a VM with an attached
for Virtual Func- ConnectX-3 VF from VF port failure, when VFs are present and
tions RoCE Link Aggregation is configured in the Hypervisor.

3.1-1.0.3 User Access Allows the ConnectX-3 driver to operate on PPC machines without
Region (UAR) requiring a change to the MMIO area size.
CQE Compression Saves PCIe bandwidth by compressing a few CQEs into a smaller
amount of bytes on PCIe
Bug fixes See Section 4, “Bug Fixes History”, on page 41
3.1-1.0.0 Wake-on-LAN Wake-on-LAN (WOL) is a technology that allows a network pro-
(WOL) fessional to remotely power on a computer or to wake it up from
sleep mode.
Hardware Acceler- Q-in-Q tunneling allows the user to create a Layer 2 Ethernet con-
ated 802.1ad nection between two servers. The user can segregate a different
VLAN (Q-in-Q VLAN traffic on a link or bundle different VLANs into a single
Tunneling) VLAN.
ConnectX-4 ECN ECN in ConnectX-4 enables end-to-end congestions notifications
between two end-points when a congestion occurs, and works over
Layer 3.
RSS Verbs Sup- Receive Side Scaling (RSS) technology allows spreading incoming
port for ConnectX- traffic between different receive descriptor queues. Assigning each
4 HCAs queue to different CPU cores allows better load balancing of the
incoming traffic and improve performance.
Minimal Band- The amount of bandwidth (BW) left on the wire may be split
width Guarantee among other TCs according to a minimal guarantee policy.
(ETS)
SR-IOV Ethernet SR-IOV Ethernet at Beta level
3.0-2.0.1 Virtualization Added support for SR-IOV for ConnectX-4/Connect-IB adapter
cards.
3.0-1.0.1 HCAs Added support for ConnectX®-4 Single/Dual-Port Adapter sup-
porting up to 100Gb/s.
RoCE per GID RoCE per GID provides the ability to use different RoCE versions/
modes simultaneously.
RoCE Link Aggre- RoCE Link Aggregation (available in kernel 4.0 only)
gation (RoCE provides failover and link aggregation capabilities for mlx4 device
LAG): physical ports. In this mode, only one IB port that represents the
ConnectX-3/Con- two physical ports, is exposed to the application layer.
nectX-3 Pro only
Rev 3.4-1.0.0.0

Resource Domain Resource domain is a verb object which may be associated with QP
Experimental and/or CQ objects on creation to enhance data-path performance.
Verbs
Alias GUID Sup- Enables the query_gid verb to return the admin desired value
port in InfiniBand instead of the value that was approved by the SM, to prevent a case
where the SM is unreachable or a response is delayed, or if the VF
is probed into a VM before their GUID is registered with the SM.
Denial Of Service Denial Of Service MAD prevention is achieved by assigning a
(DOS) MAD Pre- threshold for each agent's RX. Agent's RX threshold provides a
vention protection mechanism to the host memory by limiting the agents'
RX with a threshold.
QoS per VF (Rate Virtualized QoS per VF, (supported in ConnectX-3/ConnectX-3
Limit per VF) Pro adapter cards only with firmware v2.33.5100 and above), lim-
its the chosen VFs' throughput rate limitations (Maximum through-
put). The granularity of the rate limitation is 1Mbits.
Ignore Frame Upon receiving packets, the packets go through a checksum valida-
Check Sequence tion process for the FCS field. If the validation fails, the received
(FCS) Errors packets are dropped. Using this feature, enables you to choose
whether or not to drop the frames in case the FCS is wrong and use
the FCS field for other info.
Sockets Direct Pro- Sockets Direct Protocol (SDP) is a byte-stream transport protocol
tocol (SDP) that provides TCP stream semantics. and utilizes InfiniBand's
advanced protocol offload capabilities.
Scalable Subnet The Scalable Subnet Administration (SSA) solves Subnet Admin-
Administration istrator (SA) scalability problems for Infiniband clusters. It distrib-
(SSA) utes the needed data to perform the
path-record-calculation needed for a node to connect to another
node, and caches these locally in the compute (client) nodes.
SSAa requires AF_IB address family support (3.12.28-4 kernel and
later).
SR-IOV in Con- Changed the Alias GUID support behavior in InfiniBand.
nectX-3 cards
LLR max retrans- Added LLR max retransmission rate as specified in Vendor Spe-
mission rate cific MAD V1.1, Table 110 - PortLLRStatistics MAD Description
ibdiagnet presents the LLR max_retransmission_rate counter as
part of the PM_INFO in db_csv file.
Experimental Added the following verbs:
Verbs • ibv_exp_create_res_domain
• ibv_exp_destroy_res_domain
• ibv_exp_query_intf
• ibv_exp_release_intf
Added the following interface families:
• ibv_exp_qp_burst_family
• ibv_exp_cq_family
2.4-1.0.4 Bug Fixes See “Bug Fixes History” on page 41.

2.4-1.0.0 mlx4_en net- Added support for Ethtool speed control and advertised link mode.
device Ethtool Added ethtool txvlan control for setting ON/OFF hardware TX
VLAN insertion: ethtool -k txvlan [on/off]
Ethtool report on port parameters improvements.
Ethernet TX packet rate improvements.
RoCE RoCE uses now all available EQs and not only the 3 legacy EQs.
InfiniBand IRQ affinity hints are now set when working in InfiniBand mode.
Virtualization VXLAN fixes and performance improvements.
libmlx4 & libmlx5 Improved message rate of short massages.
libmlx5 Added ConnectX®-4 device (4114) to the list of supported devices
(hca_table),
Storage Added iSER Target driver.
Ethernet net-device New adaptive interrupt moderation scheme to improve CPU utili-
zation.
RSS support of fragmented IP datagram.
Connect-IB Vir- Added Connect-IB Virtual Function to the list of supported
tual Function devices.
2.3-2.0.5 mlx5_core Added the following files under /sys/class/infiniband/
mlx5_0/mr_cache/:
• rel_timeout: Defines the minimum allowed time between
the last MR creation to the first MR released from the cache.
When rel_timeout = -1, MRs are not released from the
cache
• rel_imm: Triggers the immediate release of excess MRs from
the cache when set to 1. When all excess MRs are released
from the cache, rel_imm is reset back to 0.
Bug Fixes See “Bug Fixes History” on page 41.
2.3-2.0.1 Bug Fixes See “Bug Fixes History” on page 41.
2.3-2.0.0 Connect-IB Added Suspend to RAM (S3).
Reset Flow Added Enhanced Error Handling for PCI (EEH), a recovery strat-
egy for I/O errors that occur on the PCI bus.
Register Contigu- Added the option to ask for a specific address when the register
ous Pages memory is using contiguous page.
mlx5_core Moved the mr_cache subtree from debugfs to mlx5_ib while
preserving all its semantics.
InfiniBand Utilities Updated the ibutils package. Added to the ibdiagnet tool the "ibdi-
agnet2.mlnx_cntrs" option to enable reading of Mellanox diagnos-
tic counters.
Bug Fixes See “Bug Fixes History” on page 41.
Rev 3.4-1.0.0.0

2.3-1.0.1 Ethernet Added support for arbitrary UDP port for VXLAN.
From upstream 3.15-rc1 and onward, it is possible to use arbitrary
UDP port for VXLAN.
This feature requires firmware version 2.32.5100 or higher.
Additionally, the following kernel configuration option CON-
FIG_MLX4_EN_VXLAN=y must be enabled.
MLNX_OFED no longer changes the OS sysctl TCP parameters.
Added Explicit Congestion Notification (ECN) support
Added Flow Steering: A0 simplified steering support
Added RoCE v2 support
OpenSM Added Routing Chains support with Minhop/UPDN/FTree/DOR/
Torus-2QoS
Added double failover elimination.
When the Master SM is turned down for some reason, the Standby
SM takes ownership over the fabric and remains the Master SM
even when the old Master SM is brought up, to avoid any unneces-
sary re-registrations in the fabric.
To enable this feature, set the “master_sm_priority” parameter
to be greater than the “sm_priority” parameter in all SMs in the
fabric. Once the Standby SM becomes the Master SM, its priority
becomes equal to the “master_sm_priority”. So that addi-
tional SM handover is avoided. Default value of the mas-
ter_sm_priority is 14.
To disable this feature, set the “master_sm_priority” in
opensm.conf to 0.
Added credit-loop free unicast/multicast updn/ftree routing
Added multithreaded Minhop/UPDN/DOR routing
RoCE Added IP routable RoCE modes.
For further information, please refer to the MLNX_OFED User
Manual.
Installation Added apt-get installation support.

InfiniBand Added Secure host to enable the device to protect itself and the
Network subnet from malicious software.
Added User-Mode Memory Registration (UMR) to enable the
usage of RDMA operations and to scatter the data at the remote
side through the definition of appropriate memory keys on the
remote side.
Added On-Demand-Paging (ODP), a technique to alleviate much
of the shortcomings of memory registration.
Added Masked Atomics operation support
Added Checksum offload for packets without L4 header support
Added Memory re-registration to allow the user to change attri-
butes of the memory region.
Resiliency Added Reset Flow for ConnectX®-3 (+SR-IOV) support.
SR-IOV Added Virtual Guest Tagging (VGT+), an advanced mode of Vir-
tual Guest Tagging (VGT), in which a VF is allowed to tag its own
packets as in VGT, but is still subject to an administrative VLAN
trunk policy.
Ethtool Added Cable EEPROM reporting support
Disable/Enable ethernet RX VLAN tag striping offload via ethtool
128 Byte Completion Queue Entry (CQE)
Non-Linux Virtual Added Windows Virtual Machine over Linux KVM Hypervisor
Machines (SR-IOV with InfiniBand only) support
Rev 3.4-1.0.0.0

Rev 2.2-1.0.1 Reset Flow Reset Flow is not activated by default. It is controlled by the mlx-
4_core'internal_err_reset' module parameter.
mlnxofedinstall 32-bit libraries are no longer installed by default on 64-bit OS.
To install 32-bit libraries use the '--with-32bit' installation
parameter.
openibd Added pre/post start/stop scripts support.
For further information, please refer to section “openibd Script” in
the MLNX_OFED User Manual.
InfiniBand Core Asymmetric MSI-X vectors allocation for the SR-IOV hypervisor
and guest instead of allocating 4 default MSI-X vectors. The maxi-
mum number of MSI-X vectors is num_cpu for port
ConnectX®-3 has 1024 MSI-X vectors, 28 MSI-X vectors are
reserved.
• Physical Function - gets the number of MSI-X vectors accord-
ing to the pf_msix_table_size (multiple of 4 - 1) INI
parameter
• Virtual Functions – the remaining MSI-X vectors are spread
equally between all VFs, according to the num_vfs mlx-
4_core module parameter
Ethernet Ethernet VXLAN support for kernels 3.12.10 or higher
Power Management Quality of Service: when the traffic is active,
the Power Management QoS is enabled by disabling the CPU
states for maximum performance.
Ethernet PTP Hardware Clock support on kernels/OSes that sup-
port it
Verbs Added additional experimental verbs interface.
This interface exposes new features which are not integrated yet in
to the upstream libibverbs. The Experimental API is an extended
API therefor, it is backward compatible, meaning old application
are not required to be recompiled to use MLNX-OFED v2.2-1.0.1.
Performance Out of the box performance improvements:
• Use of affinity hints (based on NUMA node of the device) to
indicate the IRQ balancer daemon on the optimal IRQ affinity
• Improvement in buffers allocation schema (based on the hint
above)
• Improvement in the adaptive interrupt moderation algorithm

Rev 2.1-1.0.6 IB Core Added allocation success verification process to ib_alloc_device.
dapl dapl is recompiled with no FCA support.
openibd Added the ability to bring up child interfaces even if the parent's
ifcfg file is not configured.
libmlx4 Unmapped the hca_clock_page parameter from mlx4_uninit_con-
text.
scsi_transport_srp scsi_transport_srp cannot be cleared up when rport reconnecting
fails.
mlnxofedinstall Added support for the following parameters:
• '--umad-dev-na'
• '--without-<package>'
Content Packages The following packages were updated:
Updates • bupc to v2.2-407
• mstflint to v3.5.0-1.1.g76e4acf
• perftest to v2.0-0.76.gbf9a463
• hcoll to v2.0.472-1
• Openmpi to v1.6.5-440ad47
• dapl to v2.0.40
Rev 2.1-1.0.0 EoIB EoIB is supported only in SLES11SP2 and RHEL6.4.
eIPoIB eIPoIB is currently at GA level.
Connect-IB® Added the ability to resize CQs.
IPoIB Reusing DMA mapped SKB buffers: Performance improvements
when IOMMU is enabled.
mlnx_en Added reporting autonegotiation support.
Added Transmit Packet Steering (XPS) support.
Added reporting 56Gbit/s link speed support.
Added Low Latency Socket (LLS) support.
Added check for dma_mapping errors.
eIPoIB Added non-virtual environment support.
Rev 3.4-1.0.0.0

Rev 2.0-3.0.0 Operating Systems Additional OS support:
• SLES11SP3
• Fedora16, Fedora17
Hardware Added Connect-IB™ support
Installation Added ability to install MLNX_OFED with SR-IOV support.
Added Yum installation support
EoIB EoIB (at beta level) is supported only in SLES11SP2 and RHEL6.4
mlx4_core Modified module parameters to associate configuration values with
specific PCI devices identified by their bus/device/function value
format
mlx4_en Reusing DMA mapped buffers: major performance improvements
when IOMMU is enabled
Added Port level QoS support
IPoIB Reduced memory consumption
Limited the number TX and RX queues to 16
Default IPoIB mode is set to work in Datagram, except for Con-
nect-IB™ adapter card which uses IPoIB with Connected mode as
default.
Storage iSER (at GA level)
Rev 2.0-2.0.5b Virtualization SR-IOV for both Ethernet and InfiniBand (at Beta level)
Ethernet Network RoCE over SR-IOV (at Beta level)
eIPoIB to enable IPoIB in a Para-Virtualized environment (at
Alpha level)
Ethernet Performance Enhancements (NUMA related and others)
for 10G and 40G
Ethernet Time Stamping (at Beta level)
Flow Steering for Ethernet and InfiniBand. (at Beta level)
Raw Eth QPs:
• Checksum TX/RX
• Flow Steering
InfiniBand Net- Contiguous pages:
work • Internal memory allocation improvements
• Register shared memory
• Control objects (QPs, CQs)
Installation YUM update support
VMA OFED_VMA integration to a single branch
Storage iSER (at Beta level) and SRP
Operating Systems Errata Kernel upgrade support
API VERSION query API: library and headers
Counters 64bit wide counters (port xmit/recv data/packets unicast/mcast)
a. SSA is tested on SLES 12 only (x86-64 architecture).
b. SR-IOV, Ethernet Time Stamping and Flow Steering are ConnectX®-3 HCA capability.
Rev 3.4-1.0.0.0 API Change Log History
6 API Change Log History

Release Name Description
Rev 3.2-2.0.0.0 libibverbs • Support FCS scattering for Raw Packet QPs and WQs.
• Query: ibv_exp_query_device reports IBV_EX-
P_DEVICE_SCATTER_FCS when it is supported.
• Enablement of this feature is done in the creation:
1. For Raw Packet QPs: Set IBV_EXP_QP_CRE-
ATE_SCATTER_FCS in exp_create_flags.
2. For WQs: Set IBV_EXP_CREATE_WQ_FLAG_SCAT-
TER_FCS in flags of ibv_exp_wq_init_attr.
• Indication of L4 packet type on the receive side comple-
tions:
• Query: ibv_exp_query_device reports IBV_EX-
P_DEVICE_RX_TCP_UDP_PKT_TYPE when it is sup-
ported.
bv_exp_cq_family_flags was extended with two
flags IBV_EXP_CQ_RX_TCP_PACKET and IBV_EX-
P_CQ_RX_UDP_PACKET to support L4 packet type
when using poll_length_flags().
• Support CVLAN insertion for WQs:
• Query: IBV_EXP_RECEIVE_WQ_CVLAN_INSERTION
is set in ibv_exp_vlan_offloads when CVLAN
insertion is supported.
• Enablement: The ibv_exp_qp_burst_family was
extended to support CVLAN insetion:
1. send_pending_vlan: Put one message in the pro-
vider send queue and insert vlan_tci to header.
2. send_pending_inline_vlan: Put one inline message in
the provider send queue and insert vlan_tci to header.
3. send_pending_sg_list_vlan: Put one scatter-
gather(sg) message in the provider send queue and insert
vlan_tci to header.
Rev 3.4-1.0.0.0

3.2-1.0.1.1 libibverbs • Added API and primitives for Erasure Coding calcula-
tions.
• Verbs:
• Ibv_exp_alloc_ec_calc
• Ibv_exp_dealloc_ec_calc
• Ibv_exp_ec_encode_sync
• Ibv_exp_ec_decode_sync
• Ibv_exp_ec_encode_async
• Ibv_exp_ec_decode_async
• Ibv_exp_ec_encode_send
For further information, please refer to the manual page of
the verbs.
• Structs:
• Ibv_exp_ec_calc
• Ibv_exp_ec_mem
• Ibv_exp_ec_stripe
• Ibv_exp_ec_comp
• Added version 1 of the CQ family with support for:
• Multi-Packet RQ (also called striding RQ)
• Cvlan stripping offload
• Added enhanced masked-atomic device capability
• Added a flag to the create QP/WQ option to enable end of
RX message padding
Rev 3.1-1.0.3 libibverbs • Added ibv_exp_wq_family interface family (Sup-
ported only by ConnectX®-4)
• Added flag to the QP-burst family to enable Multi-Packet
WR
• Added return error statuses to the ibv_exp_query_intf
to notify that common-flags/family-flags are not sup-
ported.
• Added ibv_exp_query_gid_attr verb.
For further information, please refer to the manual page of
the verb.
Rev 3.0-1.0.0 libibverbs • Added the following new APIs:
• ibv_exp_create_res_domain - create resource domain
• ibv_exp_destroy_res_domain - destroy resource
domain
• ibv_exp_query_intf - query for family of verbs inter-
face for specific QP/CQ
• ibv_exp_release_intf - release the queried interface
• Updated the following APIs:
• ibv_exp_create_qp - Add resource-domain to the verb
parameters
• ibv_exp_create_cq - Add resource-domain to the verb
parameters
Rev 2.4-1.0.0 libibverbs Added the following verbs interfaces:
• ibv_create_flow
• ibv_destroy_flow
• ibv_exp_use_priv_env
• ibv_exp_setenv

Rev 2.3-1.0.1 libibverbs • ibv_exp_rereg_mr - Added new API for memory
region re-reintegration (For futher information, please
refer to MLNX_OFED User Manual)
• Added to the experimental API ibv_exp_post_send
the following opcodes:
• IBV_EXP_WR_EXT_MASKED_ATOMIC_CMP_AND_SWP
• IBV_EXP_WR_EXT_MASKED_-
ATOMIC_FETCH_AND_ADD
• IBV_EXP_WR_NOP
and these completion opcodes:
• IBV_EXP_WC_MASKED_COMP_SWAP
• IBV_EXP_WC_MASKED_FETCH_ADD
Rev 2.2-1.0.1 libibverbs The following verbs changed to align with upstream libib-
verbs:
• ibv_reg_mr - ibv_access_flags changed.
• ibv_post_send - opcodes and send flags
changed and wr fields removed (task, op, dc
and bind_mw)
• ibv_query_device - capability flags
changed.
• ibv_poll_cq - opcodes and wc flags changed.
• ibv_modify_qp - mask bits changed
• ibv_create_qp_ex - create_flags field
removed.
The following verbs removed to align with upstream libib-

verbs:
• ibv_bind_mw
• ibv_post_task
• ibv_query_values_ex
• ibv_query_device_ex
• ibv_poll_cq_ex
• ibv_reg_shared_mr_ex
• ibv_reg_shared_mr
• ibv_modify_cq
• ibv_create_cq_ex
• ibv_modify_qp_ex
Rev 3.4-1.0.0.0

Rev 2.2-1.0.1 Verbs Experimental API The following experimental verbs added (replacing the
removed extended verbs):
• ibv_exp_bind_mw
• ibv_exp_post_task
• ibv_exp_query_values
• ibv_exp_query_device
• ibv_exp_poll_cq
• ibv_exp_reg_shared_mr
• ibv_exp_modify_cq
• ibv_exp_create_cq
• ibv_exp_modify_qp
New experimental verbs:

• ibv_exp_arm_dct
• ibv_exp_query_port
• ibv_exp_create_flow
• ibv_exp_destroy_flow
• ibv_exp_post_send
• ibv_exp_reg_mr
• ibv_exp_get_provider_func
Rev 2.1-1.0.0 Dynamically Connected (DC) The following verbs were added:
• struct ibv_dct *ibv_exp_create_dct(struct
ibv_context *context, struct ibv_ex-
p_dct_init_attr *attr)
• int ibv_exp_destroy_dct(struct ibv_dct
*dct)
• int ibv_exp_query_dct(struct ibv_dct *dct,
struct ibv_exp_dct_attr *attr)
Verbs Extension API: • ibv_post_task
Verbs extension API defines • ibv_query_values_ex
OFA APIs extension scheme • ibv_query_device_ex
to detect ABI compatibility • ibv_create_flow
and enable backward and for- • ibv_destroy_flow
ward compatibility support. • ibv_poll_cq_ex
• ibv_reg_shared_mr_ex
• ibv_open_xrcd
• ibv_close_xrcd
• ibv_modify_cq
• ibv_create_srq_ex
• ibv_get_srq_num
• ibv_create_qp_ex
• ibv_create_cq_ex
• ibv_open_qp
• ibv_modify_qp_ex

Rev 2.1-1.0.0 Verbs Experimental API: • ibv_exp_create_qp
Verbs experimental API • ibv_exp_query_device
defines MLNX-OFED APIs • ibv_exp_create_dct
extension scheme which is • ibv_exp_destroy_dct
similar to the “Verbs exten- • ibv_exp_query_dct
sion API”. This extension
provides a way to introduce
new features before they are
integrated into the formal
OFA API and to the upstream
kernel and libs.
Rev 2.0-3.0.0 XRC The following verbs have become deprecated:
• struct ibv_xrc_domain *ibv_open_xrc_domain
• struct ibv_srq *ibv_create_xrc_srq
• int ibv_close_xrc_domain
• int ibv_create_xrc_rcv_qp
• int ibv_modify_xrc_rcv_qp
• int ibv_query_xrc_rcv_qp
• int ibv_reg_xrc_rcv_qp
• int ibv_unreg_xrc_rcv_qp
Rev 2.0-2.0.5 Libibverbs - Extended speeds • Missing the ext_active_speed attribute from the
struct ibv_port_attr
• Removed function ibv_ext_rate_to_int
• Added functions ibv_rate_to_mbps and
mbps_to_ibv_rate
Libibverbs - Raw QPs QP types IBV_QPT_RAW_PACKET and IBV_QPT_RAW_ETH
are not supported
Libibverbs - Contiguous • Added Contiguous pages support
pages • Added function ibv_reg_shared_mr
Libmverbs • The enumeration IBV_M_WR_CALC was renamed to
IBV_M_WR_CALC_SEND
• The enumeration IBV_M_WR_WRITE_WITH_IMM was
added
• In the structure ibv_m_send_wr, the union wr.send was
renamed to wr.calc_send and wr.rdma was added
• The enumerations IBV_M_WQE_CAP_CALC_RD-
MA_WRITE_WITH_IMM was added
• The following enumerations were renamed:
• From IBV_M_WQE_SQ_ENABLE_CAP to
IBV_M_WQE_CAP_SQ_ENABLE
• From IBV_M_WQE_RQ_ENABLE_CAP to
IBV_M_WQE_CAP_RQ_ENABLE
• From IBV_M_WQE_CQE_WAIT_CAP to IBV_M_WQE_-
CAP_CQE_WAIT
• From IBV_M_WQE_CALC_CAP to IBV_M_WQE_CAP_-
CALC_SEND

Mellanox OFED Linux Release Notes 3 4-1 0 0 0

Uploaded by

Copyright:

Available Formats

Mellanox OFED Linux Release Notes 3 4-1 0 0 0

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Mellanox OFED Linux Release Notes 3 4-1 0 0 0

Uploaded by

Copyright:

Available Formats

Mellanox OFED for Linux Release Notes

© Copyright 2016. Mellanox Technologies Ltd. All Rights Reserved.

All other trademarks are property of their respective owners.

For the most updated list of Mellanox trademarks, visit http://www.mellanox.com/page/trademarks

3.6.3 SRP Interop Known Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

Release Update History

Release Date Description

1.1 Content of Mellanox OFED for Linux

1.2 Supported Platforms and Operating Systems

Operating System Platform

1.2.1 Supported Hypervisors

1.2.2 Supported Non-Linux Virtual Machines

1.2.3 Tested Hypervisors in Paravirtualized and SR-IOV Environments

Tested Hypervisors HCAs Operating System

1.3 Hardware and Software Requirements

1.4 Supported HCAs Firmware Versions

HCA Recommended Firmware Rev. Additional Firmware Rev. Supported

1.5 Compatibility Matrix

1.6 RoCE Modes Matrix

2 Changes and New Features in Rev 3.4-1.0.0.0

HCAs Feature/Change Description

2.1 Future API Changes in MLNX_OFED

Feature Type Current API New API

3.1 Driver Installation/Loading/Unloading/Start Known Issues

Index Internal Reference Number: Description Workaround

Index Internal Reference Number: Description Workaround

3.1.2 Driver Unload Known Issues

Index Internal Reference Number: Description Workaround

3.1.3 Driver Start Known Issues

Index Internal Reference Number: Description Workaround

Index Internal Reference Number: Description Workaround

Index Internal Reference Number: Description Workaround

3.1.4 System Time Known Issues

Index Internal Reference Number: Description Workaround

3.1.5 UEFI Secure Boot Known Issues

Index Internal Reference Number: Description Workaround

[4671958.383506] Request for unknown

This error can be safely ignored as long as Secure

3.2 Performance Known Issues

Index Internal Reference Number: Description Workaround

3.3 HCAs Known Issues

3.4 Ethernet Network

Index Internal Reference Number: Description Workaround

kernel warning: mlx4_core 0000:07:00.0:

Index Internal Reference Number: Description Workaround

Index Internal Reference Number: Description Workaround

3.4.2 Port Type Management Known Issues

Index Internal Reference Number: Description Workaround

3.4.3 Flow Steering Known Issues

Index Internal Reference Number: Description Workaround

3.4.4 Quality of Service Known Issues

Index Internal Reference Number: Description Workaround

3.4.5 Ethernet Performance Counters Known Issues

Index Internal Reference Number: Description Workaround

3.5 InfiniBand Network

Index Internal Reference Number: Description Workaround

Index Internal Reference Number: Description Workaround

Where N is the IPoIB required interface index

Index Internal Reference Number: Description Workaround