Proc. of ACM/IEEE SC 2003: High-Performance Networking and Computing Conference, November 2003.

Optimizing 10-Gigabit Ethernet for Networks of Workstations, Clusters, and Grids:

A Case Study







Wu-chun Feng,  Justin (Gus) Hurwitz,

Harvey
Newman,


Sylvain Ravot,

R. Les Cottrell,


Olivier Martin, Fabrizio Coccetti, Cheng Jin, Xiaoliang (David) Wei, and Steven Low
 
Los Alamos National Laboratory (LANL) California Institute of Technology (CalTech)
Computer & Computational Sciences Division Pasadena, CA 91125
Research and Development in Advanced Network Technology Charles C. Lauritsen Laboratory of High Energy Physics
Los Alamos, NM 87545  newman,ravot  @caltech.edu
 feng,ghurwitz  @lanl.gov Computer Science & Electrical Engineering Dept.
 chengjin,weixl,slow  @caltech.edu

European Organization for Nuclear Research (CERN) Stanford Linear Accelerator Center (SLAC)
Information Technology Division SLAC Computing Services
Geneva, Switzerland Menlo Park, CA 94025
 Olivier.Martin  @cern.ch  cottrell,cfabrizo  @SLAC.stanford.edu

Abstract 1. Introduction
This paper presents a case study of the 10-Gigabit Ether- Thirty years ago in a May 1973 memo, Robert Metcalfe

net (10GbE) adapter from Intel R . Specifically, with appropri- described the technology that would evolve into today’s ubiq-
ate optimizations to the configurations of the 10GbE adapter uitous Ethernet protocol. By 1974, Metcalfe and his col-
and TCP, we demonstrate that the 10GbE adapter can perform league, David Boggs, built their first Ethernet; and by 1975,
well in local-area, storage-area, system-area, and wide-area they demonstrated what was at the time a dazzling 2.94 Mb/s
networks. of throughput over the 10-Mb/s Ethernet medium. Since that
For local-area, storage-area, and system-area networks in time, Ethernet has proliferated and evolved tremendously and
support of networks of workstations, network-attached stor- has done so in virtual lockstep with the ubiquitous TCP/IP
age, and clusters, respectively, we can achieve over 7-Gb/s (Transmission Control Protocol / Internet Protocol) protocol
end-to-end throughput and 12- s end-to-end latency between suite which was started at Stanford University in the summer
applications running on Linux-based PCs. For the wide-area of 1973. Today’s Ethernet carries 99.99% of Internet pack-
network in support of grids, we broke the recently-set Inter- ets and bears little resemblance to the original Ethernet [11].
net2 Land Speed Record by 2.5 times by sustaining an end- About the only aspect of the original Ethernet that still re-
to-end TCP/IP throughput of 2.38 Gb/s between Sunnyvale, mains is its packet format.
California and Geneva, Switzerland (i.e., 10,037 kilometers) So, even though the recently ratified 10-Gigabit Ether-
to move over a terabyte of data in less than an hour. Thus, net (10GbE) standard differs from earlier Ethernet standards,
the above results indicate that 10GbE may be a cost-effective mainly with respect to operating only over fiber and only in
solution across a multitude of computing environments. full-duplex mode, it still remains Ethernet, and more impor-
tantly, does not obsolete current investments in network in-
frastructure. Furthermore, the 10GbE standard ensures inter-

operability not only with respect to existing Ethernet but also
This work was supported by the US DOE Office of Science through other networking technologies such as SONET (i.e., Ethernet
LANL contract W-7405-ENG-36 Caltech contract DE-FG03-92-ER40701, over SONET), thus paving the way for Ethernet’s expanded
and SLAC contract DE-AC03-76SF00515. Additional support was provided
by NSF through grant ANI-0230967, AFOSR through grant F49620-03-1- use in metropolitan-area networks (MANs) and wide-area net-
0119, and ARO through grant DAAD19-02-1-0283. works (WANs). Finally, while 10GbE is arguably intended
This paper is also available as the following LANL technical report: to ease migration to higher aggregate performance levels in
LA-UR 03-5728, July 2003.
institutional network-backbone infrastructures, the results in
SC’03, November 15-21, 2003, Phoenix, Arizona, USA this paper will demonstrate 10GbE’s versatility in a myriad of
Copyright 2003 ACM 1-58113-695-1/1/03/0011...$5.00 computing environments.
 P 8B/10B
C D M PCS
I M A 3.125Gbps
− SerDes
X A C
I/F XGM II

4x3.125Gbps
R X P P RX 10.3Gbps IN
Intel G C M Optics
XAUI
PCI−X Bus 82597EX X S A TX 10.3Gbps OUT
(8.5Gbps) S Optics
4x3.125Gbps
Intel R 1310nm Serial Optics
512K flash Intel R PRO/10GbE−LR

Figure 1. Architecture of the 10GbE Adapter

The remainder of the paper is organized as follows: Sec- data and descriptor transfers between the host memory and on-
tion 2 briefly describes the architecture of the Intel 10GbE chip memory while the latter provides a complete glueless in-
adapter. Section 3 presents the local-area network (LAN) terface to a 33/66-MHz, 32/64-bit PCI bus or a 33/66/100/133-
and system-area network (SAN) testing environments, exper- MHz, 32/64-bit PCI-X bus.
iments, and results and analysis, and Section 4 does the same As is already common practice with high-performance
for the wide-area network (WAN). Finally, we summarize and adapters such as Myricom’s Myrinet [2] and Quadrics’ Qs-
conclude in Section 5. Net [17], the 10GbE adapter frees up host-CPU cycles by per-
forming certain tasks (in silicon) on behalf of the host CPU.
2. Architecture of a 10GbE Adapter In contrast to the Myrinet and QsNet adapters, however, the
10GbE adapter focuses on host off-loading of certain TCP/IP

The recent arrival of the Intel R PRO/10GbE LRTM server tasks1 rather than on remote direct-memory access (RDMA)
adapter paves the way for 10GbE to become an all- and source routing. As a result, unlike Myrinet and Qs-
encompassing technology from LANs and SANs to MANs Net, the 10GbE adapter provides a general-purpose, TCP/IP-
and WANs. This first-generation 10GbE adapter consists of based solution to applications, a solution that does not require
three major components: Intel 82597EXTM 10GbE controller, any modification to application codes to achieve high perfor-
512-KB of flash memory, and Intel 1310-nm serial optics, as mance, e.g., as high as 7 Gb/s between end-host applications
shown in Figure 1. with an end-to-end latency as low as 12  s.
The 10GbE controller provides an Ethernet interface that As we will see later, achieving higher throughput will re-
delivers high performance by providing direct access to all quire either efficient offloading of network tasks from soft-
memory without using mapping registers, minimizing pro- ware to hardware (e.g., IETF’s RDMA-over-IP effort, known
grammed I/O (PIO) read access required to manage the de- as RDDP or remote direct data placement [19]) and/or sig-
vice, minimizing interrupts required to manage the device, nificantly faster machines with large memory bandwidth.
and off-loading the host CPU of simple tasks such as TCP Achieving substantially higher throughput, e.g., approaching
checksum calculations. Its implementation is in a single 10 Gb/s, will not be possible until the PCI-X hardware bottle-
chip and contains both the medium-access control (MAC) and neck in a PC is addressed. Currently, the peak bandwidth of a
physical (PHY) layer functions, as shown at the top of Fig- 133-MHz, 64-bit PCI-X bus in a PC is 8.5 Gb/s (see left-hand
ure 1. The PHY layer, to the right of the MAC layer in Fig- side of Figure 1), which is less than half the 20.6-Gb/s bidirec-
ure 1, consists of an 8B/10B physical coding sublayer and a tional data rate (see right-hand side of Figure 1) that the Intel
10-gigabit media independent interface (XGM II). To the left 10GbE adapter can support.
of the MAC layer is a direct-memory access (DMA) engine
and the “peripheral component interconnect extended” inter-
face (PCI-X I/F). The former handles the transmit and receive 1 Specifically, TCP & IP checksums and TCP segmentation.
3. LAN/SAN Tests PE2650 10GbE PE2650

In this section, we present our LAN/SAN experimental re-

sults and analysis. The results here show that we can achieve (a) Direct single flow
over 7 Gb/s of throughput and 12- s end-to-end latency with
TCP/IP.
FastIron
3.1. Testing Environments 1500
PE2650 PE2650
We evaluate the performance of the Intel 10GbE adapter in
three different LAN/SAN environments, as shown in Figure 2: 10GbE 10GbE
(b) Indirect single flow
(a) Direct single flow between two computers
connected back-to-back via a crossover cable,
(b) Indirect single flow
between two computers
through a Foundry R FastIronTM 1500 switch, PE2650 FastIron 10GbE
(c) Multiple flows through the Foundry FastIron 1500 1GbE
1500 switch,

where

the computers that host the 10GbE adapters are either

Dell R PowerEdgeTM 2650 (PE2650) servers or Dell Pow- P P P P P
E E E E E
erEdge 4600 (PE4600) servers. 2 2 4 4 4
(Recently, we have also conducted additional back-to-back 6 6 6 6 6
5 5 0 0 0
tests on computer systems, provided by Intel, with a slightly 0 0 0 0 0
faster CPU and front-side bus (FSB). Given that we only had
these systems for a few days, we merely use the results from (c) Multiple flows through the switch
these systems for anecdotal purposes as well as a “sanity-
check” on our more exhaustive tests on the Dell PowerEdge
servers.2) Figure 2. LAN/SAN Testing Environments
Each PE2650 contains dual 2.2-GHz Intel XeonTM CPUs
running on
a 400-MHz front-side bus (FSB), using a
ServerWorks R GC-LE chipset with 1 GB of memory and a streams from (or to) many hosts into a 10GbE stream to (or
dedicated 133-MHz PCI-X bus for the 10GbE adapter. The- from) a single host. The total backplane bandwidth (480 Gb/s)
oretically, this architectural configuration provides 25.6-Gb/s in the switch far exceeds the needs of our tests as each of the
CPU bandwidth, up to 25.6-Gb/s memory bandwidth, and 8.5- two 10GbE ports is limited to 8.5 Gb/s.
Gb/s network bandwidth via the PCI-X bus. From a software perspective, all the above hosts run current
Each PE4600 contains dual 2.4-GHz Intel Xeon CPUs run- installations of Debian Linux with customized kernel builds
ning on a 400-MHz FSB, using a ServerWorks GC-HE chipset and tuned TCP/IP stacks. Specific kernels that we used in-
with 1 GB of memory and a dedicated 100-MHz PCI-X bus clude 2.4.19, 2.4.19-ac4, 2.4.19-rmap15b, 2.4.20, and 2.5.44.
for the 10GbE adapter. This particular configuration provides Because the performance differences between these various
theoretical bandwidths of 25.6-Gb/s, 51.2-Gb/s, and 6.4-Gb/s kernel builds prove negligible, we do not report the running
for the CPU, memory, and PCI-X bus, respectively. kernel version in any of the results.
(The systems provided by Intel contain dual 2.66-GHz In-
tel Xeon CPUs running on a 533-MHz FSB, using Intel’s 3.2. Experiments
E7505 chipset with 2 GB of memory and a dedicated 100-
MHz PCI-X bus for the 10GbE adapter. This architecture In this paper, our experiments focus on the performance
provides theoretical bandwidths of 34-Gb/s, 25.6-Gb/s, and of bulk data transfer. We use two tools to measure network
6.4-Gb/s for the CPU, memory, and PCI-X bus, respectively.) throughput — NTTCP [16] and Iperf [8] — and note that the
In addition to the above hosts, we use a Foundry FastIron experimental results from these two tools correspond to an-
1500 switch for both our indirect single-flow and multi-flow other oft-used tool called netperf [14].
tests. In the latter case, the switch aggregates GbE and 10GbE NTTCP and IPerf work by measuring the time required
2 We also have even more promising (but again, preliminary) 10GbE re- to send a stream of data. Iperf measures the amount of data
sults on a 1.5-GHz Itanium-II system. sent over a consistent stream in a set time. NTTCP, a ttcp
variant, measures the time required to send a set number of Running in a LAN or SAN, we expect this product to be
fixed-size packets. In our tests, Iperf is well suited for mea- relatively small, even at 10GbE speeds. The initial latency
suring raw bandwidth while NTTCP is better suited for opti- numbers that we observed are 19  s running back-to-back
mizing the performance between the application and the net- and 25  s running through the Foundry switch. At full
work. As our goal is to maximize performance to the applica- 10GbE speed, this results in a maximum bandwidth-delay
tion, NTTCP provides more valuable data in these tests. We product of about 48 KB, well below the default window
therefore present primarily NTTCP data throughout the pa- setting of 64 KB. At observed speeds, the maximum product
per. (Typically, the performance difference between the two is is well under half of the default. In either case, these values
within 2-3%. In no case does Iperf yield results significantly are within the scope of the default maximum window settings.
contrary to those of NTTCP.)
To estimate the end-to-end latency between a pair of Stock TCP
10GbE adapters, we use NetPipe [15] to obtain an averaged We begin with single-flow experiments across a pair of un-
round-trip time over several single-byte, ping-pong tests and optimized (stock) Dell PE2650s using standard 1500-byte and
then divide by two. 9000-byte (jumboframe) maximum transfer units (MTUs). In
To measure the memory bandwidth of our Dell PowerEdge their stock (i.e., default) configurations, the dual-processor
systems, we use STREAM [10]. PE2650s have a standard maximum PCI-X burst transfer size
To estimate the CPU load across our throughput tests, we — controlled by the maximum memory read byte count (MM-
sample /proc/loadavg at five- to ten-second intervals. RBC) register — of 512 bytes and run a symmetric multi-
And finally, to better facilitate the analysis of data transfers, processing (SMP) kernel. In each single-flow experiment,
we make use of two tools, tcpdump [21] and MAGNET [6]. NTTCP transfers 32,768 packets ranging in size from 128
tcpdump is commonly available and used for analyzing pro- bytes to 16 KB at increments ranging in size from 32 to 128
tocols at the wire level. MAGNET is a publicly available tool bytes.
developed in part by the co-authors from Los Alamos Na- Figure 3 shows the baseline results. Using a larger MTU
tional Laboratory. MAGNET allowed us to trace and profile size produces 40-60% better throughput than the standard
the paths taken by individual packets through the TCP stack 1500-byte MTU. For 1500-byte MTUs, the CPU load is
with negligible effect on network performance. By observing approximately 0.9 on both the send and receive hosts while
a random sampling of packets, we were able to quantify how the CPU load is only 0.4 for 9000-byte MTUs. We observe
many packets take each possible path, the cost of each path, bandwidth peaks at 1.8 Gb/s with a 1500-byte MTU and 2.7
and the conditions necessary for a packet to take a “faster” Gb/s with a 9000-byte MTU.
path. (We note that this is just one of many possible uses for
MAGNET.)
4500
"1500MTU,SMP,512PCI"
4000 "9000MTU,SMP,512PCI"
3.3. Experimental Results
3500

This section presents an abridged account of the optimiza- 3000

Mbit/sec

tions that we implemented to achieve greater than 4 Gb/s of 2500

throughput for a single TCP/IP flow between a pair of low- 2000
end 2.2-GHz Dell PE2650s. For a more in-depth discussion 1500
of each optimization step, we refer the reader to [7]. 1000
We begin our experiments with a stock TCP stack. From 500
this starting point, we implement optimizations one by one
0
to improve network performance between two identical Dell 0 2048 4096 6144 8192 10240 12288 14336 16384
PE2650s connected via 10GbE, as shown in Figure 2(a). Payload Size (bytes)
The more common device and TCP optimizations result in
little to no performance gains. These optimizations include Figure 3. Throughput of Stock TCP: 1500- vs.
changing variables such as the device transmit queue lengths 9000-byte MTU
and the use of TCP timestamps.

3.3.1 Bandwidth Stock TCP + Increased PCI-X Burst Size

Next, we increase the PCI-X burst transfer size (i.e., MM-
Before commencing our formal testing, we tune the TCP RBC register) from 512 bytes to 4096 bytes. Although this
window sizes by calculating the ideal bandwidth-delay optimization only produces a marginal increase in throughput
product and setting the TCP window sizes accordingly [22]. for 1500-byte MTUs, it dramatically improves performance
with 9000-byte MTUs. The peak throughput increases to 4500
"1500MTU,UP,4096PCI,256kbuf,medres"
over 3.6 Gb/s, a throughput increase of 33% over the baseline 4000 "9000MTU,UP,4096PCI,256kbuf,medres"
case, while the average throughput increases to 2.63 Gb/s, an 3500
increase of 17%. The CPU load remains relatively unchanged
3000
from the baseline numbers reported above.

Mbit/sec
2500
2000
Stock TCP + Increased PCI-X Burst Size + Uniprocessor
At the present time, the P4 Xeon SMP architecture assigns 1500

each interrupt to a single CPU instead of processing them 1000

in a round-robin manner between CPUs. Consequently, 500
our next counterintuitive optimization is to replace the 0
SMP kernel with a uniprocessor (UP) kernel. This change 0 2048 4096 6144 8192 10240 12288 14336 16384
Payload Size (bytes)
further improves average throughput for 9000-byte MTUs by
approximately 10% to 2.9 Gb/s. For 1500-byte MTUs, the
average and maximum throughputs increase by about 25% Figure 4. Throughput of TCP with Oversized
and 20% to 2.0 Gb/s and 2.15 Gb/s, respectively. In addi- Windows and Increased PCI-X Burst Size Run-
tion, the CPU load was uniformly lower than in the SMP tests. ning on a Uniprocessor Kernel

TCP with Oversized Windows + Increased PCI-X Burst

Size + Uniprocessor 5000
"16000MTU,UP,4096PCI,256kbuf,medres"
Though we calculated that the default window sizes 4500 "8160MTU,UP,4096PCI,256kbuf,medres"
10GbE (Actual)
were much larger than the bandwidth-delay product, we 4000

improve throughput further by setting the window size to 3500

Quadrics (Theoretical)
be four times larger than the default setting (and roughly 3000
Mbit/sec

ten times larger than the actual bandwidth-delay product). 2500

Myrinet (Theoretical)
That is, we set the receive socket buffer to 256 KB in 2000
/proc/sys/net/ipv4/tcp rmem. With a 256-KB 1500
socket buffer, the peak bandwidth increases to 2.47 Gb/s with 1000 1GbE (Theoretical)

1500-byte MTUs and 3.9 Gb/s with 9000-byte MTUs, as 500

shown in Figure 4. A detailed discussion of this particular 0
0 2048 4096 6144 8192 10240 12288 14336 16384
optimization will be presented in Section 3.5.1.
Payload Size (bytes)

Tuning the MTU Size

We achieve even better performance with non-standard Figure 5. Throughput of Cumulative Optimiza-
MTU sizes. Figure 5 shows that the peak bandwidth achieved tions with Non-Standard MTUs
is 4.11 Gb/s with an 8160-byte MTU.3 This result is a direct
consequence of Linux’s memory-allocation system. Linux
allocates memory from pools of “power-of-2” sized blocks,
that larger MTUs, and consequently, larger block sizes are not
An 8160-byte MTU allows an entire packet, i.e., payload +
without consequences. Using larger blocks places far greater
TCP/IP headers + Ethernet headers, to fit in a single 8192-
stress on the kernel’s memory-allocation subsystem because
byte block whereas a 9000-byte MTU requires the kernel to
it is generally harder to find the contiguous pages required for
allocate a 16384-byte block, thus wasting roughly 7000 bytes.
the larger blocks.
The above discussion leads us to our next logical step —
(Note: As points of reference, Figure 5 also labels
using the largest MTU that the Intel 10GbE adapter can sup-
the theoretical maximum bandwidths for Gigabit Ethernet,
port, namely 16000 bytes. With a 16000-byte MTU, the peak
Myrinet [12, 13], and QsNet [17]. A more detailed discussion
throughput achieved is 4.09 Gb/s, virtually identical to the
of these interconnects versus 10-Gigabit Ethernet is presented
8160-byte MTU case. However, the average throughput with
in Section 3.5.4.)
the larger MTU is clearly much higher, as shown in Figure 5.
The surprisingly marginal increase in throughput is due to the
sender’s congestion window artificially limiting throughput, 3.3.2 Latency
as discussed in more detail in Section 3.5.1. It is worth noting Although our experiments did not focus on optimizations with
3 8160-byte MTUs can be used in conjunction with any hardware that sup- respect to latency, we are acutely aware that low latency is crit-
ports 9000-byte MTUs. ical for scientific applications such as global climate model-
ing [24]. Therefore, we report our preliminary latency results
here, demonstrate how end-to-end latency can be improved, 2.4e-05

and suggest other avenues for improvement. 2.2e-05

Our latency measurements, running NetPipe between a pair
2e-05
of 2.2-GHz Dell PowerEdge 2650s, produce 19- s end-to-end

Seconds
latency when the machines are connected back-to-back and 1.8e-05
25- s end-to-end latency when going through the Foundry Fa-
1.6e-05
stIron 1500 switch. As the payload size increases from one
byte to 1024 bytes, latencies increase linearly in a stepwise 1.4e-05
fashion, as shown in Figure 6. Over the entire range of pay-
loads, the end-to-end latency increases a total of 20% such that 1.2e-05
0 128 256 384 512 640 768 896 1024
the back-to-back latency is 23  s and the end-to-end latency Payload Size (bytes)
through the switch is 28  s.
To reduce these latency numbers even further, particularly
for latency-sensitive environments, we trivially shave off an Figure 6. End-to-End Latency in Test Configu-
additional 5  s (i.e., down to 14- s end-to-end latency) by ration
simply turning off a feature called interrupt coalescing (Fig-
ure 7). In our bandwidth tests and the above latency tests in 2.4e-05
Figure 6, we had configured the 10GbE adapters with a 5- s
interrupt delay. This delay is the period that the 10GbE card 2.2e-05

waits between receiving a packet and raising an interrupt to 2e-05

signal packet reception. Such a delay allows multiple packet Seconds
receptions to be coalesced into a single interrupt, thus reduc- 1.8e-05
ing the CPU load on the host at the expense of latency.
1.6e-05
From the perspective of the host system, newer versions
1.4e-05
of Linux (which we have yet to test) implement a New API
(NAPI) for network processing. This NAPI allows for better 1.2e-05
handling of network adapters (or network interface cards) that 0 128 256 384 512 640 768 896 1024
support hardware-based interrupt coalescing by coalescing the Payload Size (bytes)

software processing of packets from the network adapter’s

ring buffer. The older API queues each received packet sep-
Figure 7. End-to-End Latency without Interrupt
arately, regardless of whether multiple packets are received
Coalescing
in a single interrupt, resulting in wasted time in an interrupt
context to process each individual packet. In the NAPI, the
interrupt context only queues the fact that packets are ready
to be processed and schedules the packets to be received from 3.4. Anecdotal Results
the network interface card later, outside the scope of the in-
terrupt context. This approach provides two major benefits: In addition to the more thorough experimental results
(1) less time spent in an interrupt context and (2) more ef- above, we have preliminary results on machines that we had
ficient processing of packets, which ultimately decreases the very limited access to. The first set of these machines, pro-
load that the 10GbE card places on the receiving host. (In sys- vided by Intel and described briefly in Section 3.1, primar-
tems where the host CPU is a bottleneck, it would also result ily differ from the Dell PE2650s by having dual 2.66-GHz
in higher bandwidth.) CPUs and a 533-MHz FSB. The essentially “out-of-box” per-
Newer versions of Linux also support TCP Segmentation formance of the 10GbE adapters on these machines was 4.64-
Offload (TSO, also known as Large Send), another offload Gb/s in a back-to-back configuration. (It is worth noting that
feature supported by Intel’s 10GbE adapters. TSO allows the this performance required TCP timestamps to be disabled be-
transmitting system to use a large (64 KB) “virtual” MTU. cause enabling timestamps reduced throughput by approxi-
The 10GbE card then re-segments the payload into smaller mately 10%. The reason for this behavior is explained in
packets for transmission. As the NAPI does for receiving Section 3.5.2). Latency numbers between these machines ap-
systems, the implementation of TSO should reduce the CPU peared to be up to 2  s less than those seen between the Dell
load on transmitting systems, and in many cases, will increase PE2650s, indicating that end-to-end latencies could reach as
throughput. low as 12  s.
 We also have anecdotal 10GbE results for a 1-GHz quad- The effect can be even more severe with smaller
processor Itanium-II system. Specifically, multiple 1GbE bandwidth-delay products, as shown in Figure 8. Such a situ-
clients were aggregated through a 10GbE switch into a sin- ation can arise on either the send or receive side. We describe
gle 1-GHz quad-processor Itanium-II system with a 10GbE the send-side case as it is easier to understand — the theoreti-
adapter. Performing the same set of optimizations, as de- cal (or ideal) window size is a hard limit that the sender cannot
scribed above, on the Itanium-II system produces a unidirec- exceed (e.g., 26 KB in Figure 8). Because the congestion-
tional throughput of 7.2 Gb/s. control window must be MSS-aligned on the sender side, the
actual congestion-control window is only 18 KB, or 31% less
3.5. Discussion of LAN/SAN Results than the allowable 26-KB window.

This subsection presents a more detailed analysis of our

experiments, proposes ways to improve performance further,
~9K MSS
and puts our 10GbE results in context with respect to other
network interconnects.
Theoretical
(~26KB) ~9K MSS
3.5.1. LAN/SAN TCP Windows Best possible
or advertised
window
window
Back in Section 3.3, the “9000-byte MTU” throughput results due to MSS
~9K MSS
in Figure 3 showed a marked dip for payload sizes between
7436 and 8948 bytes (as well as a series of smaller, but higher
frequency, dips across all payload sizes). Even optimizing the
PCI-X burst transfer size and using a uniprocessor kernel did
not eliminate the marked dip; however, oversizing the TCP Figure 8. Ideal vs. MSS-allowed Window
windows did eliminate the marked dip, as shown in Figure 4.
Using tcpdump and by monitoring the kernel’s internal
In addition to the above inefficiencies, the full socket-buffer
state variables with MAGNET, we trace the causes of this be-
havior to inefficient window use by both the sender and re- memory (as allocated for window use by the kernel) on the re-
ceiver. Briefly, the throughput dips are a result of (1) a large ceive may never be available to be advertised as the window.
That is, although the window that the receiver advertises is a
Maximum Segment Size (MSS4 ) relative to the ideal window
size and (2) Linux’s TCP stack keeping both the advertised function of the available buffer memory, there is no require-
and congestion windows MSS-aligned.5 ment that this available buffer memory be a multiple of the
MSS. As a result, the amount of memory that is not a multiple
On the receive side, the actual advertised window is sig-
of the MSS is simply wasted.
nificantly smaller than the expected value of 48 KB, as cal-
culated in Section 3.3. This behavior is a consequence To further complicate matters, the MSS is not necessarily
of Linux’s implementation of the Silly Window Syndrome constant between communicating hosts, i.e., the sender’s MSS
(SWS) avoidance algorithm [3]. Because the advertised win- is no necessarily equal to the receiver’s MSS. Why is this an
dow is kept aligned with the MSS, it cannot be increased by issue? Consider a sender MSS of 8960 bytes and a receiver
small amounts.6 The larger that the MSS is relative to the MSS of 8948 bytes and let the receiver have 33,000 bytes of
advertised window, the harder it becomes to increase the ad- available socket memory.8 The receiver will then advertise a
vertised window.7 window of 5687:9);=<?<@>A > ? > CB=D-EFBHGJI=KML B-E=E bytes, or 19% less
On the sender side, performance is similarly limited be- than the available 33,000 bytes. With a sender MSS of 8960
cause the congestion window is kept aligned with the MSS [1]. bytes, the maximum size that the congestion window can be,
For instance, with a 19- s latency, the theoretical ideal win- due to the fact that?the A A congestion window must be kept MSS-
dow size for 10GbE is about 48 KB. With a 9000-byte MTU aligned, is 5N687:9);=O ? P @A =B=D=EQCGSRUTL DI-Q bytes, or 33% less than
>
(8948-byte MSS with options), this translates to about 5.5 the receiver’s advertised window and nearly 50% smaller than
packets per window. Thus, neither the sender nor the receiver the actual available socket memory of 33,000 bytes. If win-
can transfer 6 complete packets; both can do at best 5 packets. dow scaling is used (as is becoming more prevalent with the
This immediately attenuates the ideal data rate by nearly 17%. proliferation of gigabit-per-second networks in the wide-area
network), the situation is exacerbated even further because the
4 Loosely speaking, MSS = MTU – packet headers. accuracy of the window diminishes as the scaling factor in-
5 Linux is not unique in this behavior; it is shared by most modern TCPs.
6
 



   8 We

The
   window
! "#%$'&)()is aligned
" /.1 03242 by frequently observe such a situation, i.e., the sender using a larger
*,+-+ . This rounds the window down to MSS value than the receiver, in our tests. This is apparently a result of how
the nearest increment of MSS bytes. the receiver estimates the sender’s MSS and might well be an implementation
7 We note that this is actually an argument for not increasing the MTU size. bug.
creases. Even without scaling, in this example, the sender and bottleneck. Furthermore, by moving from 1500- to 9000-byte
receiver are both artificially limiting the bandwidth by a total MTUs, we typically expect a performance increase of 3x-6x
of nearly 50%. if the CPU were the bottleneck. Our results show an increase
To overcome the above problems, network researchers rou- of only 1.5x-2x.
tinely (and in many cases, blindly) increase the default socket It is possible that the inherent complexity of the TCP re-
buffer sizes even further until performance improves. How- ceive path (relative to the transmit path) results in a receive-
ever, this is a poor “band-aid” solution in general. There path bottleneck. In addition, while we have anecdotal evi-
should be no need to set the socket buffer to many times the dence that the ServerWorks GC-LE chipset is capable of sus-
ideal window size in any environment; in a WAN environ- taining better than 90% of the PCI-X bandwidth it has not
ment, setting the socket buffer too large can severely impact been confirmed. In short, both the TCP receive path and the
performance, as noted in Table 1. Furthermore, the low laten- actual PCI-X bus performance are potential bottlenecks. To
cies and large MSS in LAN/SAN environments are contrary evaluate both, we conduct multi-flow testing of the 10GbE
to this “conventional wisdom” of setting the window/buffer adapters through our Foundry FastIron 1500 switch. These
size so large. In addition, the above solution does not prevent tests allow us to aggregate nearly 16 Gb/s from multiple
the sender from being artificially limited by the congestion 1GbE-enabled hosts to one or two 10GbE-enabled hosts (or
window, as noted earlier. Better solutions might include the vice versa).
following: In the first set of tests, we transmit to (or from) a single
10GbE adapter. These tests identify bottlenecks in the receive
Modifying the SWS avoidance and congestion-window path, relative to the transmit path, by multiplexing the process-
algorithms to allow for fractional MSS increments when ing required for one path across several machines while keep-
the number of segments per window is small. ing the aggregated path to (or from) a single 10GbE-enabled
Making “better” (or at least, more conservative) adaptive Dell PE2650 constant. These results unexpectedly show that
calculations of the MSS on the receive side. the transmit and receive paths are of statistically equal perfor-
mance. Given the relative complexity of the receive path com-
Allowing the sender to incrementally decrease the MSS pared to the transmit path, we initially expect to see better per-
if the congestion window is locked in a steady state. formance when the 10GbE adapter is transmitting to multiple
hosts than when receiving from multiple hosts. Previous expe-
In summary, although larger MTUs tend to improve net- rience provides a likely explanation for this behavior. Packets
work throughput and reduce CPU load, they magnify prob- from multiple hosts are more likely to be received in frequent
lems that were not as apparent with the standard 1500-byte bursts than are packets from a single host, allowing the receive
MTU, particularly in a LAN/SAN environment. Specifi- path to benefit from interrupt coalescing, thereby increasing
cally, a large MSS relative to the ideal window size (as the receive-side bandwidth relative to transmit bandwidth.9
in a LAN/SAN) and TCP’s “enforcement” of MSS-aligned Multiplexing GbE flows across two 10GbE adapters on in-
windows results in lower-than-expected, highly-fluctuating dependent buses in a single machine yields results statistically
throughput. identical to those obtained using a single 10GbE adapter. We
(Interestingly, we first ran into these problems when work- can therefore rule out the PCI-X bus as a primary bottleneck.
ing with IP over the Quadrics interconnect. The problem man- In addition, this test also eliminates the 10GbE adapter as a
ifested itself to a lesser extent due to the lower data rates of primary bottleneck.
Quadrics QsNet (3.2 Gb/s). However, as latency decreases, Using the Dell PE4600s and Intel-provided dual 2.66-GHz
bandwidth increases, and perhaps most importantly, MTU systems, we determine that memory bandwidth is not a likely
size increases [9], this problem will only exacerbate itself fur- bottleneck either. The PE4600s use the GC-HE chipset, of-
ther.) fering a theoretical memory bandwidth of 51.2 Gb/s; the
STREAM [10] memory benchmark reports 12.8-Gb/s mem-
3.5.2. Analysis of Performance Limits ory bandwidth on these systems, nearly 50% better than that
of the Dell PE2650s. Despite this higher memory bandwidth,
Given that the hardware-based bottleneck in the Dell PE2650s
we observe no increase in network performance. There are,
is the PCI-X bus at 8.5 Gb/s, the peak throughput of 4.11 Gb/s
unfortunately, enough architectural differences between the
is only about half the rate that we expect.
PE2650 and PE4600 that further investigation is required.
In all of our experiments, the CPU load remains low
The Intel-provided systems, however, further confirm that
enough to indicate that the CPU is not a primary bottleneck.
memory bandwidth is not a likely bottleneck. STREAM re-
This is supported by the fact that disabling TCP timestamps
sults for the PE2650s and Intel-provided systems are within
on the PE2650s yields no increase in throughput; disabling
a few percent of each other. However, the Intel-provided sys-
timestamps gives the CPU more time for TCP processing and
should therefore yield greater throughput if the CPU were a 9 This confirms results in [4], albeit by very different means.
tems achieved 4.64 Gb/s with virtually no optimizations while The two prevailing approaches towards improving TCP/IP
the heavily optimized PE2650s only reached 4.11 Gb/s. This performance are TCP offload engines (TOEs) and RDMA
difference in performance, better than 13%, cannot be ac- over IP [19]. The former will offload the entire TCP stack into
counted for by differences in the memory bandwidth alone. the silicon of the network adapter while the latter will lever-
A more likely, but related, explanation is the change in front- age the use of a processor on the network adapter to run TCP
side bus (FSB) speed. Further investigation is needed, how- software and enable direct data placement from the network
ever, before making such a conclusion. adapter into application space (RDMA over IP), thus elimi-
All of the above results are supported by Linux’s packet nating excessive copying across the memory bus and virtually
generator. The packet generator bypasses the TCP/IP and eliminating processing load from the host CPU.
UDP/IP stacks entirely. It is a kernel-level loop that transmits It is our belief that TOE is not the best solution for a
pre-formed “dummy” UDP packets directly to the adapter general-purpose TCP. Many previous attempts at TOE engines
(that is, it is single-copy). We observe a maximum band- failed because design and implementation errors are virtu-
width of 5.5 Gb/s (8160-byte packets at approximately 88,400 ally impossible to change once they are cast in silicon (un-
packets/sec) on the PE2650s when using the packet generator. less an FPGA, field-programmable gate array, or other field-
This rate is maintained when additional load is placed on the upgradable processor is used). In a TOE, the TCP implemen-
CPU, indicating that the CPU is not a bottleneck. Because the tation is effectively hidden from the host CPU software, mak-
packet generator is single-copy (as opposed to the IP stack’s ing it difficult to interface with. Hardware design and produc-
triple-copy), memory bandwidth is not a limit to its perfor- tion costs are significantly larger than with so-called “dumb”
mance. Our observed TCP bandwidth, which is not single- adapters. Most importantly, the adapter must still transfer
copy, is about 75% of the packet generator bandwidth. It is data across the memory and I/O buses, introducing a poten-
reasonable to expect, however, that the TCP/IP stack would tial source of data errors, errors that a TOE has no way to
attenuate the packet generator’s performance by about 25%. detect or correct.
While this does not demonstratively rule out memory band- Our experience has shown that hardware can have design
width as a bottleneck, it does indicate the the observed perfor- defects that lead to data errors. In high-load environments,
mance is in line with what we should expect were the memory heat, high bit rates, or poor hardware designs often contribute
bandwidth not a bottleneck. in error rates far higher than predicted by hardware manufac-
Overall, we are left to believe that the host software’s abil- turers [20]. This is especially the case with “bleeding edge”
ity to move data between every component in the system is and other high-performance hardware. For instance, in our
likely the bottleneck. Given that the Linux kernel’s packet educated opinion, received TCP data should not be check-
generator reports a maximum total bandwidth of approxi- summed in the adapter; rather they must be computed once
mately 5.5 Gb/s on the PE2650s, this movement of data at- the data has reached the system’s main memory. Unfortu-
tenuates throughput by 3 Gb/s (i.e., 8.5 Gb/s - 5.5 Gb/s) and is nately, current proposals for TOEs perform checksums in the
the primary bottleneck toward achieving higher performance. adapter.
Rather than implement a complete TOE, we would like to
see an implementation of a TCP header-parsing engine, e.g.,
3.5.3. Breaking the Bottlenecks
a la ST [18]. Briefly, such an engine would use a hash ta-
To improve network performance, contemporary network ble of established sockets to transfer the payload of incoming
adapters provide various means for reducing the load on the packets directly into user memory. The headers would then be
host operating system and hardware. These methods, often- passed on to the kernel for normal processing. In the event of
times referred to as “offload” techniques, attempt to move net- out-of-order packets, or other TCP circumstances that cannot
working tasks from the host processor to the adapter. A few of be handled on a fast path, the adapter passes the entire packet
these techniques have been discussed above, e.g., TSO, check- on to the kernel for traditional processing.
sum offloading, and interrupt coalescing. While each of these Such an implementation requires a small amount of logic
techniques do offer measurable performance gains, their main and buffer memory on the adapter itself as well as a simple
benefit is in decreasing the load on the host CPU rather than interface for interacting with the kernel. Furthermore, this ap-
substantially improving throughput and end-to-end latency. proach keeps the TCP logic on the host while allowing the
However, in all of our experiments, we have seen that CPU adapter to transfer the payload. It also isolates TCP from
load is not the primary bottleneck. Thus, the host system’s hardware-dependent design decisions and allows for an easy
bandwidth is more likely I/O-limited than CPU-limited. So, upgrade, maintenance, and development path for the TCP.
the key to improving bandwidth is to either improve the host An additional possibility that we hope to see implemented
system’s ability to move data, decrease the amount of data that in the future is the placement of network adapters on the
needs to be moved, or decrease the number of times that the Memory Controller Hub (MCH), typically found on the
data needs to be moved across the memory bus [18, 19]. Northbridge. Intel’s Communication Streaming Architecture
(CSA) [5] is such an implementation for Gigabit Ethernet. 300% better than Myrinet, and over 200% better than Qs-
Placing the adapter on the MCH allows for the bypass of the Net. Finally, even when comparing our 10GbE TCP/IP perfor-
I/O bus. By eliminating the I/O bus, we eliminate both an mance numbers with the numbers from other interconnects’
important bottleneck and a source of data errors. In addition, specialized network software (e.g., GM and Elan3), we find
the adapter is better enabled to compute the checksum of the the 10GbE performance to be highly competitive — gener-
payload once placed in system memory. Such a computation ally better with respect to throughput and slightly worse with
would significantly improve the accuracy and reliability of the respect to latency.
checksums.
4. WAN Tests
3.5.4. Putting the 10GbE LAN/SAN Numbers in Perspec-
tive In this section, we provide a brief account of our wide-area
network tests, and more specifically, our Internet2 Land Speed
We now turn to discuss the actual performance that one can
Record effort back in February 2003.
expect out of Gigabit Ethernet, Myrinet, and even QsNet
(rather than the theoretical maximums shown in Figure 5) in
4.1. Environment
order to provide a better reference point for the 10GbE results.
Our extensive experience with 1GbE chipsets (e.g., In-
In this section, we present the performance of our 10GbE
tel’s e1000 line and Broadcom’s Tigon3) allows us to achieve
adapters across a WAN that stretched from Sunnyvale, Cal-
near line-speed performance with a 1500-byte MTU in a
ifornia to Geneva, Switzerland (i.e., 10,037 kilometers), as
LAN/SAN environment with most payload sizes. With ad-
shown in Figure 9. The WAN utilized a loaned Level3 OC-
ditional optimizations in a WAN environment, similar perfor-
192 POS (10 Gb/s) circuit from the Level3 PoP at Sunnyvale
mance can be achieved but with a 9000-byte MTU.10
to StarLight in Chicago and then traversed the transatlantic
For comparison to Myrinet, we report Myricom’s pub-
LHCnet OC-48 POS (2.5 Gb/s) circuit between Chicago and
lished performance numbers for their adapters [12, 13]. Using
Geneva.
their proprietary GM API, sustained unidirectional bandwidth
At Sunnyvale, the OC-192 POS circuit originated at a
is 1.984 Gb/s and bidirectional bandwidth is 3.912 Gb/s. Both
Cisco GSR 12406 router and was connected to a Juniper
of these numbers are within 3% of the 2-Gb/s unidirectional
T640 (NSF TeraGrid router) at Starlight in Chicago. The
hardware limit. The GM API provide latencies on the order of
TeraGrid router then crossed over to a Cisco 7609 router be-
6 to 7  s. To use this API, however, may oftentimes require
fore heading overseas to a Cisco 7606 router in Geneva. In
rewriting portions of legacy applications’ code.
traversing roughly halfway across the world, our WAN traffic
Myrinet provides a TCP/IP emulation layer to avoid this
crossed two different autonomous systems: AS75 (TeraGrid)
problem. The performance of this layer, however is notably
and AS503 (CERN).
less than that of the GM API. Bandwidth drops to 1.853 Gb/s,
At each end point, we had a dual 2.4-GHz Intel Xeon PC
and latencies skyrocket to over 30  s.
with 2 GB of memory and a dedicated 133-MHz PCI-X bus
Our experiences with Quadrics’ QsNet produced unidirec-
for the 10GbE adapter. Each node ran Linux 2.4.19 with
tional bandwidth and latency numbers of 2.456 Gb/s and 4.9
jumbo frames and optimized its buffer size to be approxi-
 s, respectively, using QsNet’s Elan3 API. As with Myrinet’s
mately the bandwidth-delay product, i.e.,
GM API, the Elan3 API may require application codes to
rewrite their network code, typically from a sockets API to echo ‘‘4096 87380 128388607’’ >
Elan3 API. To address this issue, Quadrics also has a highly /proc/sys/net/ipv4/tcp_rmem
echo ‘‘4096 65530 128388607’’ >
efficient implementation of TCP/IP that produces 2.240 Gb/s /proc/sys/net/ipv4/tcp_wmem
of bandwidth and under 30- s latency. For additional perfor- echo 128388607 > /proc/sys/net/core/wmem_max
mance results, see [17]. echo 128388607 > /proc/sys/net/core/rmem_max
In summary, when comparing TCP/IP performance across
/sbin/ifconfig eth1 txqueuelen 10000
all interconnect technologies, our established 10GbE through- /sbin/ifconfig eth1 mtu 9000
put number (4.11 Gb/s) is over 300% better than GbE, over
120% better than Myrinet, and over 80% than QsNet while our 4.2. Experiment & Result
established 10GbE latency number (19  s) is roughly 400%
better than GbE and 50% better than Myrinet and QsNet. Our
The additive increase, multiplicative decrease (AIMD) al-
preliminary 10GbE throughput number of 7.2 Gb/s on the
gorithm governs TCP’s bandwidth through a sender-side state
Itanium-II systems is nearly 700% better than GbE, nearly
variable called the congestion window. The AIMD algorithm
10 Internet2 Land Speed Record set on November 19, 2002: single-stream modifies the size of the congestion window according to the
TCP/IP of 923 Mb/s over a distance of 10,978 km. network conditions. Without any packet loss, the congestion
 Figure 9. 10-Gigabit Ethernet WAN Environment

window normally opens at a constant rate of one segment per Second, given that the bottleneck bandwidth is the transat-
round-trip time; but each time a congestion signal is received lantic LHCnet OC-48 POS at 2.5 Gb/s, achieving 2.38 Gb/s
(i.e., packet loss), the congestion window is halved. means that the connection operated at roughly 99% payload
However, as the bandwidth and latency increase, and efficiency. Third, the end-to-end WAN throughput is actu-
hence, increasing the bandwidth-delay product, the effect of a ally larger than what an application user typically sees in a
single packet loss is disastrous in these long fat networks with LAN/SAN environment. Fourth, our results smashed both the
gargantuan bandwidth-delay products. For example, in future single- and multi-stream Internet2 Land Speed Records by 2.5
scenarios, e.g., 10-Gb/s connection from end-to-end between times.
Geneva and Sunnyvale, Table 1 shows how long it takes to
recover from a packet loss and eventually return to the origi- 5. Conclusion
nal transmission rate (prior to the packet loss), assuming that
the congestion window size is equal to the bandwidth-delay With the current generation of SAN interconnects such as
product when the packet is lost. Myrinet and QsNet being theoretically hardware-capped at 2
To avoid this problem, one simply needs to reduce the Gb/s and 3.2 Gb/s, respectively, achieving over 4 Gb/s of end-
packet-loss rate. But how? In our environment, packet loss to-end throughput with 10GbE makes it a viable commod-
is due exclusively to congestion in the network, i.e., packets ity interconnect for SANs in addition to LANs. However,
are dropped when the number of unacknowledged packets ex- its Achilles’ heel is its 12- s (best-case) end-to-end latency,
ceeds the available capacity of the network. In order to reduce which is 1.7 times slower than Myrinet/GM (but over two
the packet-loss rate, we must “stop” the increase of the con- times faster than Myrinet/IP) and 2.4 times slower than Qs-
gestion window before it reaches a congested state. Because Net/Elan3 (but over two times faster than QsNet/IP). These
explicit control of the congestion-control window is not possi- performance differences can be attributed mainly to the host
ble, we turn to the flow-control window (TCP buffer sizing) to software.
implicitly cap the congestion-window size to the bandwidth- In recent tests on the dual 2.66-GHz CPUs with 533-
delay product of the wide-area network so that the network MHz FSB Intel E7505-based systems running Linux, we have
approaches congestion but avoids it altogether. achieved 4.64 Gb/s throughput “out of the box.” The great-
As a result, using only a single TCP/IP stream between est difference between these systems and the PE2650s is the
Sunnyvale and Geneva, we achieved an end-to-end through- FSB, which indicates that the CPU’s ability to move — but
put of 2.38 Gb/s over a distance of 10,037 kilometers. This not process — data, might be an important bottleneck. These
translates into moving a terabyte of data in less than one hour. tests have not yet been fully analyzed.
Why is this result so remarkable? First, it is well-known To continue this work, we are currently instrumenting the
that TCP end-to-end throughput is inversely proportional to Linux TCP stack with MAGNET to perform per-packet profil-
round-trip time; that is, the longer the round-trip time (in this ing and tracing of the stack’s control path. MAGNET allows
case, 180 ms, or approximately 10,000 times larger than the us to profile arbitrary sections of the stack with CPU-clock
round-trip time in the LAN/SAN), the lower the throughput. accuracy, while 10GbE stresses the stack with previously im-
 Path Bandwidth Assumption RTT (ms) MSS (bytes) Time to Recover
LAN 10 Gbps 1 1460 428 ms
Geneva - Chicago 10 Gb/s 120 1460 1 hr 42 min
Geneva - Chicago 10 Gb/s 120 8960 17 min
Geneva - Sunnyvale 10 Gb/s 180 1460 3 hr 51 min
Geneva - Sunnyvale 10 Gb/s 180 8960 38 min

Table 1. Time to Recover from a Single Packet Loss

possible loads. Analysis of this data is giving us an unprece- Doug Walsten from Cisco Systems for the 10GbE switch
dentedly high-resolution picture of the most expensive aspects support in Sunnyvale, and
of TCP processing overhead [4].
While a better understanding of current performance bot- Peter Kersten and John Szewc from Foundry Networks
tlenecks is essential, the authors’ past experience with Myrinet for providing Los Alamos National Laboratory with a
and Quadrics leads them to believe that an OS-bypass proto- 10GbE switch to run their LAN/SAN tests on.
col, like RDMA over IP, implemented over 10GbE would re- If any of the above infrastructure pieces would have fallen
sult in throughput approaching 8 Gb/s, end-to-end latencies through, no attempt at the Internet2 Land Speed Record would
below 10  s, and a CPU load approaching zero. However, be- have ever been made.
cause high-performance OS-bypass protocols require an on-
Last, but not least, we gratefully acknowledge the invalu-
board (programmable) network processor on the adapter, the able contributions and support of the following colleagues
10GbE adapter from Intel currently cannot support an OS- during our record-breaking effort on the Internet2 Land Speed
bypass protocol. Record: Julian Bunn and Suresh Singh of Caltech; Paolo
The availability of 10-Gigabit Ethernet provides a remark- Moroni and Daniel Davids of CERN; Edoardo Martelli of
able opportunity for network researchers in LANs, SANs,
CERN/DataTAG; Gary Buhrmaster of SLAC; and Eric Wei-
MANs, and even WANs in support of networks of worksta- gle and Adam Engelhart of Los Alamos National Labora-
tions, clusters, distributed clusters, and grids, respectively. tory. In addition, we wish to thank the Internet2 consortium
The unprecedented (commodity) performance offered by the
(http://www.internet2.edu) for creating a venue for “supernet-
Intel PRO/10GbE server adapter also enabled us to smash working” achievements.
the Internet2 Land Speed Record (http://lsr.internet2.edu) on
February 27, 2003, by sustaining 2.38 Gb/s across 10,037 km
between Sunnyvale, California and Geneva, Switzerland, i.e., References
23,888,060,000,000,000 meters-bits/sec.
[1] M. Allman, V. Paxson, and W. Stevens, “TCP Congestion Con-
trol,” RFC-2581, April 1999.
Acknowledgements [2] N. Boden, D. Cohen, R. Felderman, A. Kulawik, C. Seitz,
J. Seizovic, and W. Su, “Myrinet: A Gigabit-Per-Second Local
First and foremost, we would like to thank the Intel team Area Network,” IEEE Micro, Vol. 15, No. 1, January/February
— Patrick Connor, Caroline Larson, Peter Molnar, and Marc 1995.
Rillema of the LAN Access Division — for their tremendous [3] D. Clark, “Window and Acknowledgment Strategy in TCP,”
support of our research efforts and Eric Weigle for his assis- RFC-813, July 1982.
tance throughout this project. [4] D. Clark, V. Jacobson, J. Romkey, and H. Salwen, “An Analy-
With respect to the wide-area network, none of the research sis of TCP Processing Overhead,” IEEE Communications, Vol.
27, No. 6, June, 1989, pp. 23-29.
(e.g., Internet2 Land Speed Record) would have been possi-
[5] “Communication Streaming Architecture: Reducing the PCI
ble without the generous support and contributions of many Network Bottleneck,” Intel Whitepaper 252451-002. Available
people and institutions. From an infrastructure standpoint, at: http://www.intel.com/design/network/papers/252451.htm.
we thank Linda Winkler and Tom DeFanti from Argonne Na- [6] M. K. Gardner, W. Feng, M. Broxton, A Engelhart, and
tional Laboratory and the University of Illinois of Chicago, G. Hurwitz, “MAGNET: A Tool for Debugging, Analysis
respectively, for the use of a Juniper T640 TeraGrid router at and Reflection in Computing Systems,” Proceedings of the
Starlight in Chicago. Equipment-wise, in addition to Intel, we 3rd IEEE/ACM International Symposium on Cluster Comput-
thank ing and the Grid (CCGrid’2003), May 2003.
[7] G. Hurwitz, and W. Feng, “Initial End-to-End Performance
Paul Fernes from Level(3) Communications for the OC- Evaluation of 10-Gigabit Ethernet,” Proceedings of Hot Inter-
192 link from Sunnyvale to Chicago, connects 11 (HotI’03), August 2003.
 [8] “Iperf 1.6 - The TCP/UDP Bandwidth Measurement Tool,”
http://dast.nlanr.net/Projects/ Iperf/.
[9] M. Mathis, “Pushing Up the Internet MTU,” Presentation to
Miami Joint Techs, Feburary 2003.
[10] J. McCalpin, “STREAM: Sustainable Memory Bandwidth
in High-Performance Computers,” http://www.cs.virginia.edu/
stream/.
[11] R. Metcalfe and D. Boggs, “Ethernet: Distributed Packet
Switching for Local Computer Networks,” Communications
of the ACM, Vol. 19, No. 5, July 1976.
[12] “Myrinet Ethernet Emulation (TCP/IP & UDP/IP) Perfor-
mance,” http://www.myri.com/myrinet/performance/ ip.html.
[13] “Myrinet Performance Measurements,” http://www.myri.com/
myrinet/performance/index.html.
[14] “Netperf: Public Netperf Homepage,” http://www.netperf.org/.
[15] “NetPIPE,” http://www.scl.ameslab.gov/netpipe/.
[16] “NTTCP: New TTCP program,” http://www.leo.org/˜ el-
mar/nttcp/.
[17] F. Petrini, W. Feng, A. Hoisie, S. Coll, and E. Frachtenberg,
“The Quadrics Network: High-Performance Clustering Tech-
nology,” IEEE Micro, Vol. 22, No. 1, January/Feburary 2002.
[18] I. Philp and Y.-L. Liong, “The Scheduled Transfer (ST) Pro-
tocol,” 3rd International Workshop on Communication, Ar-
chitecture, and Applications for Network-Based Parallel Com-
puting (CANPC’99), Lecture Notes in Computer Science, Vol.
1602, January 1999.
[19] A. Romanow, and S. Bailey, “An Overview of RDMA over IP,”
Proceedings of the First International Workshop on Protocols
for Fast Long-Distance Networks (PFLDnet 2003), Feburary
2003.
[20] J. Stone, and C. Partridge, “When the CRC and TCP Check-
sum Disagree,” Proceedings of ACM SIGCOMM 2000, August
2000.
[21] “TCPDUMP Public Repository,” http://www.tcpdump.org.
[22] B. Tierney, “TCP Tuning Guide for Distributed Application
on Wide-Area Networks,” USENIX ;login:, Vol. 26, No. 1,
February 2001.
[23] S. Ubik, and P. Chimbal, “Achieving Reliable High Perfor-
mance in LFNs,” Proceedings of Trans-European-Research
and Education Networking Association Networking Confer-
ence (TERENA 2003), May 2003.
[24] W. Washington and C. Parkinson, An Introduction to Three-
Dimensional Climate Modeling, University Science Books,
1991.