Datagram vs Connected modes

The IPoIB driver supports two modes of operation: datagram and

connected. The mode is set and read through an interface's

/sys/class/net/<intf name>/mode file.

In datagram mode, the IB UD (Unreliable Datagram) transport is used

and so the interface MTU has is equal to the IB L2 MTU minus the

IPoIB encapsulation header (4 bytes). For example, in a typical IB

fabric with a 2K MTU, the IPoIB MTU will be 2048 - 4 = 2044 bytes.

In connected mode, the IB RC (Reliable Connected) transport is used.

Connected mode is to takes advantage of the connected nature of the

IB transport and allows an MTU up to the maximal IP packet size of

64K, which reduces the number of IP packets needed for handling

large UDP datagrams, TCP segments, etc and increases the performance

for large messages.

In connected mode, the interface's UD QP is still used for multicast

and communication with peers that don't support connected mode. In

this case, RX emulation of ICMP PMTU packets is used to cause the

networking stack to use the smaller UD MTU for these neighbours.

Stateless offloads

If the IB HW supports IPoIB stateless offloads, IPoIB advertises

TCP/IP checksum and/or Large Send (LSO) offloading capability to the

network stack.

Large Receive (LRO) offloading is also implemented and may be turned

on/off using ethtool calls. Currently LRO is supported only for

checksum offload capable devices.

Stateless offloads are supported only in datagram mode.

Interrupt moderation

If the underlying IB device supports CQ event moderation, one can

use ethtool to set interrupt mitigation parameters and thus reduce

the overhead incurred by handling interrupts. The main code path of

IPoIB doesn't use events for TX completion signaling so only RX

moderation is supported.

IP over InfiniBand: Connected Mode (RFC 4755)

http://ietf.org/rfc/rfc4755.txt