Introduction to

PCI Express

Paolo Durante
(CERN EP-LBC)

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 1

Where can you find ?
PCI (Peripheral Component Interconnect) Express is a
popular standard for high-speed computer expansion
overseen by PCI-SIG
(Special Interest Group)
• PCIe interconnects can be present at all levels of your
DAQ chain…
• Readout boards
• Storage media
• Network interfaces
• Compute accelerators (GPUs, FPGAs…)
• …and may be even more so in the future
• Understanding your data acquisition system requires
(some) level of understanding of PCI Express

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 2

What is this presentation about?
• History and evolution of PCIe

• PCIe concepts

• PCIe layers

• PCIe performance

• PCIe future roadmap

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 3

PCI (“conventional PCI”)
• 1992
• Peripheral Component
Interconnect
• Parallel Interface
• Bandwidth
• 133 MB/s (~1.0 Gb/s)
(32-bit@33 MHz)
• 533 MB/s (~4.2 Gb/s)
(64-bit@66 MHz)
• Plug-and-Play
configuration (BARs)
18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 4
PCI example: ATLAS FILAR
• ~2003
• 4 optical channels
• 160 MB/s (1.28 Gb/s)
• S-LINK protocol
• 2 Altera FPGAs
• Burst-DMA over PCI
• 3rd Altera FPGA
• 64-bit@66MHz PCI

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 5

PCI-X (“Extended PCI”)
• 1998
• PCI compatible
• Hardware and software
• Half-duplex bidirectional
• Higher bus efficiency
• Split-responses
• Message Signaled Interrupts
• Bandwidth
• ≤ 1066 MB/s (~8.5 Gb/s)
(64-bit@133 MHz)
• 2133 MB/s (~17 Gb/s)
(PCI-X 266)
• 4266 MB/s (~34 Gb/s)
(PCI-X 533)

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 6

PCI-X example: CMS FEROL
• ~2011
• 4 SFP+ cages
• 1x 10 Gb/s Ethernet
• 3x SlinkXpress
• PCI-X interface to
legacy FE (Slink64)
• Altera FPGA
• Simplex TCP-IP

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 7

PCI Express (PCIe)
PCIe x16 | PCI | PCIe x8|PCI-X
• 2004
• PCI “inspired”
• software, topology
• Serial interface
• Full-duplex bidirectional
• Bandwidth
• x1: ≤2 GB/s (16 Gb/s)
(in each direction)
• x16: ≤32 GB/s(256 Gb/s)
(in each direction)
• Still evolving
• 1.0, 2.0, 3.0, 4.0, 5.0, 6.0…

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 8

PCIe example: ALICE C-RORC
• ~2014
• 3x QSFP
• 36 channels
• up to 6.6Gb/s/channel
• 2x DDR SO-DIMM
• XilinX FPGA
• PCIe Gen2 x8

• Also used by ATLAS

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 9
PCIe example: LHCb TELL40
• Introduced for LHC Run3
• Currently in production
• ≤ 48 duplex optical links
• GBT (3.2 Gb/s)
• WideBus (4.48 Gb/s)
• GWT (5.12 Gb/s)
• Altera Arria10 FPGA
• 110 Gb/s DMA
• PCIe 3.0 x16
• Also used by ALICE

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 10

PCIe example: ATLAS FELIX
• Introduced for LHC Run3
• ≤ 48 duplex optical links
• XilinX Ultrascale FPGA
• 2x DDR4 SO-DIMM
• PCIe 3.0 x16
• Wupper DMA
(Open Source)
• Also used by DUNE

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 11

What is this presentation about?
• History and evolution of PCIe

• PCIe concepts

• PCIe layers

• PCIe performance

• PCIe future roadmap

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 12

PCIe concepts – Packets
• Point-to-point connection
• “Serial” “bus” (fewer pins)
• Scalable link: x1, x2, x4, x8, x12, x16, x32
• Packet encapsulation
Packet

PCI Express PCI Express

Device A Device B

Packet

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 13

PCIe concepts – Root complex

• Connects the processor

and memory
subsystems to the PCIe
fabric via a Root Port
• Generates and
processes transactions
with Endpoints on
behalf of the processor

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 14

 PCIe concepts – Topology
Relative to root – up is towards, down is away

“UPSTREAM”

“DOWNSTREAM”

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 15

PCIe concepts – BDF
“geographical $ lspci -tv
-+-[0000:ff]-+-08.0 Intel Corporation Xeon ...
| +-08.3 Intel Corporation Xeon ...

addressing” |
|
+-08.4 Intel Corporation Xeon ...
+-09.0 Intel Corporation Xeon ...
| ...

• Bus : Device . Function

+-[0000:80]-+-00.0-[81]--
| +-01.0-[82]--
| +-02.0-[83]----00.0 Intel Corporation Xeon Phi coprocessor 31S1

• Form a hierarchy-
| +-03.0-[84]--
| +-03.2-[85]----00.0 Intel Corporation Xeon Phi coprocessor 31S1
| +-05.0 Intel Corporation Xeon E5/Core i7 Address Map, VTd_Misc, System Management
based address |
|
+-05.2 Intel Corporation Xeon E5/Core
\-05.4 Intel Corporation Xeon E5/Core
i7 Control Status and Global Errors
i7 I/O APIC

• Multiple logical +-[0000:7f]-+-08.0 Intel Corporation Xeon E5/Core

| +-08.3 Intel Corporation Xeon E5/Core
i7 QPI Link 0
i7 QPI Link Reut 0
| ...
“Functions” allowed on \-[0000:00]-+-00.0 Intel Corporation Xeon E5/Core i7 DMI2
+-01.0-[01]--
one physical device +-01.1-[02]--
+-02.0-[03]----00.0 Intel Corporation Xeon Phi coprocessor 31S1

• Bridges (PCI/PCI-X) +-03.0-[04]----00.0 Intel Corporation

+-05.0 Intel Corporation Xeon E5/Core
Xeon Phi coprocessor 31S1
i7 Address Map, VTd_Misc, System Management

form hierarchy +-05.2 Intel Corporation Xeon E5/Core

+-05.4 Intel Corporation Xeon E5/Core
i7 Control Status and Global Errors
i7 I/O APIC
+-11.0-[05]--+-00.0 Intel Corporation C602 chipset 4-Port SATA Storage Control Unit

• Switches (PCIe) form | \-00.3 Intel Corporation

+-1c.0-[06]----00.0 Intel Corporation
C600/X79 series chipset SMBus Controller 0
82574L Gigabit Network Connection

hierarchy ...

00:00.0 Host bridge: Intel Corporation Xeon E5/Core i7 DMI2 (rev 07)

On linux: $ man lspci

80:02.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2a (rev 07)
83:00.0 Co-processor: Intel Corporation Xeon Phi coprocessor 31S1 (rev 11)

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 16

Troubleshooting with lspci
• Device works but is “slow”
• Link speed
• Link width
• MaxPayloadSize
• Interrupts
• Error flags
• Look for bottlenecks upstream
• Device is “there” but driver fails to load
• Unreadable config space
• Unallocated BARs

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 20

PCIe concepts – Address spaces
• Address spaces
• Configuration
(Bus/Device/Function)
• Memory (64-bit)
• I/O (32-bit)

• Configuration space
• Base Address Registers
(BARs) (32/64-bit)
• Capabilities (linked list)

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 21

PCIe concepts – Memory & I/O
• Memory space maps cleanly to CPU semantics
• 32-bits of address space initially
• 64-bits introduced via Dual-Address Cycles (DAC)
• Extra period of address time on PCI/PCI-X
• 4DWORD header in PCI Express
• Burstable (= Multiple DWORDs)
• I/O space maps cleanly to CPU semantics
• 32-bits of address space
• Non-burstable

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 22

PCIe concepts - Bridges

Transparent Non-Transparent
• Single root (or SR-IOV) • Joins two independent
• Single address space topologies
• Multiple downstreams • One root on each side
(switch) • Each side has its own
• Downstreams appear in address space
the same topology • Needs translation table
• Addresses are passed • Fault tolerance,
through unchanged “networking”, HPC
18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 23
PCIe concepts – Bus address
This is actually not specific to PCIe, but a generic
reminder:
• Physical address: the address the CPU sends to the
memory controller
• Virtual address: an indirect address created by the
operating system, translated by the CPU to physical
• Bus address: an address understood by the devices
connected to a specific bus
• On Linux, see: pci_iomap(), remap_pfn_range(), …

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 24

PCIe concepts – Interrupts
• PCI pci_read_config_byte(dev,
PCI_INTERRUPT_PIN,
• INTx# &(...));
• x ∈ {A, B, C, D}
• Level sensitive pci_read_config_byte(dev,
PCI_INTERRUPT_LINE,
• Can be mapped to CPU &(...));
interrupt number
• PCIe pci_enable_msi(dev);
• “Virtual Wire”
emulation request_irq(dev->irq, my_isr,
• Assert_INTx code IRQF_SHARED, devname,
cookie);
• Deassert_INTx code
18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 25
PCIe concepts – MSI & MSI-X
• Based on messages (MWr)
• MSI uses one address with a
variable data value indicating
which “vector” is asserting
• ≤ 32 per device (in theory)
• MSI-X uses a table of
independent address and
data pairs for each “vector”
• ≤ 2048 per device (use affinity!)
• Vector: interrupt id

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 26

PCIe concepts – interr. coalescing

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 27

PCIe concepts – latency

Typical: ~1us

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 28

PCIe Gen1 (2003)
• Introduced at 2.5 GT/sec
• Also called 2.5 GHz, 2.5 Gb/s
• 100 MHz reference clock
• Eases synchronization between ends
• Can use Spread Spectrum Clocking to reduce EMI
• Optional, but nearly universal
• 8b/10b encoding used to provide DC balance and
reduce “runs” of 0s or 1s which make clock
recovery difficult
• Specification Revisions: 1.0, 1.0a, 1.1

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 29

PCIe Gen2 (2006)
• Speed doubled from 2.5 to 5 GT/sec
• Reference clock remains at 100 MHz
• Lower jitter clock sources required vs 2.5 GT/sec
• Generally higher quality clock generation/distribution
required
• 8b/10b encoding continues to be used
• Specification Revisions: 2.0, 2.1
• Devices choosing to implement a maximum rate of
2.5 GT/sec can still be fully 2.x compliant

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 30

PCIe Gen3 (2010)

2x5=?

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 31

PCIe Gen3 (2010)

2x5=8
• Speed “doubled” from 5 GT/sec
• More efficient encoding (20% → ~1%)
• 8 GT/sec electrical rate
• 10 GT/sec required significant cost and complexity in
channel, receiver design, etc.
• Reference clock remains at 100 MHz
• Backwards-compatible speed negotiation

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 32

PCIe Gen4 (2017)

2x8=?

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 33

PCIe Gen4 (2017)

2 x 8 = 16
• Speed doubled from 8 GT/sec
• Same 128b/130b encoding
• 16 GT/sec electrical rate
• Channel length: ≤ 10”/14”
• Retimer mandatory for longer channels
• More complex pre-amplification, equalization stages
• Reference clock remains at 100 MHz
• Backwards-compatible protocol negotiation
and CEM spec
18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 34
PCIe Gen5 (2019)

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 35

What is this presentation about?
• History and evolution of PCIe

• PCIe concepts

• PCIe layers

• PCIe performance

• PCIe future roadmap

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 36

PCIe – Protocol stack
PCI Express Device A PCI Express Device B

Application Layer Application Layer

PCI Express PCI Express

Logic Interface Logic Interface

Transaction Layer Transaction Layer

T R T R
Data Link Layer Data Link Layer
X X X X

Physical Layer Physical Layer

Link
18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 37
FPGA Hardened PCIe IP

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 38

PCIe – Transaction layer
• Four possible transaction types
• Memory Read | Memory Write
• Transfer data from or to a memory mapped location
• Address routing
• IO Read | IO Write
• Transfer data from or to an IO location (on a legacy endpoint)
• Address routing
• Config Read | Config Write
• Discover device capabilities, status, parameters
• ID routing (BDF)
• Messages
• Event signaling

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 39

PCIe – TLP structure
MaxPayloadSize (MPS)
Application Layer parameter limits and
dominates performance
Transmit order

STP Sequence Header Data Payload ECRC LCRC End

1B 2B 3-4DW 0-1024DW 1DW 1DW 1B

Created by Transaction Layer

Appended by Data Link Layer

Appended by Physical Layer

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 40

PCIe – Split transaction model
• Posted transaction
• Single TLP, no completion

• Non-posted transaction
• Split transaction model
• Requester initiates transaction (Requester ID + Tag)
• Requester and Completer IDs encode the sender BDF
• Completer executes transaction internally
• Completer creates completion transaction (Cpl/CplD)

• Bus efficiency of Read is different (lower) wrt Write

• Writes are posted while Reads are not
18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 41
PCIe – DMA transaction

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 42

PCIe – Data Link Layer
• ACK / NAK Packets
• Error handling mechanism
• Flow Control Packets (FCPs)
• Propagate credit allocation status
• Power Management Packets
• Vendor extensions
• E.g.: CAPI, CCIX (memory coherency)

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 43

PCIe – DLLP structure

Transmit order

SDP DLLP CRC End

1B 4B 2B 1B

Created by Data Link Layer

Appended by Physical Layer

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 44

PCIe – Flow control
• Credit-based
• Point-to-point (not end-to-end)
Available space

Transmitter TLP Receiver

VC buffer
Data Link Layer Data Link Layer

Flow Control DLLP (FCx)

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 45

PCIe – RAS/QoS features
• Data Integrity and Error Handling
• PCIe is RAS (Reliable, Available, Serviceable)
• Data integrity at
• link level (LCRC)
• end-to-end (ECRC, optional)
• Virtual channels (VCs) and traffic classes (TCs) to
support differentiated traffic or Quality of Service (QoS)
• In theory
• Ability to define levels of service for packets of different TCs
• 8 TCs and 8 VCs available
• In practice
• Rarely more than 1 VC and 1 TC are implemented

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 46

PCIe – Error handling

Correctable Uncorrectable
• Recovery happens • Fatal
• Platform-specific handling
automatically in DLL
• Non-fatal
• Performance is • Can be exposed to
degraded application layer and
handled explicitly
• Can and do cause system
deadlock / reset
• Example: LCRC error • Recovery mechanisms are
→ automatic DLL retry outside the spec
(there is no forward error correction) • Example: failover for HA

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 47

PCIe – ACK/NAK

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 48

 PCIe – Physical layer

PCI Express Device

Signal Link

Wire
…
Lane

PCI Express Device

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 49

PCIe – Ordered-Set Structure
Transmit order COM Identifier Identifier … Identifier

Six ordered sets are possible

• Training Sequences (TS1, TS2): 1 COM + 15 TS
• Used to de-skew between lanes
• SKIP: 1 COM + 3 SKP identifiers
• Used to recalibrate receiver clock
• Fast Training Sequence (FTS): 1 COM + 3 FTS
• Power management
• Electrical Idle (IDLE): 1 COM + 3 IDL
• Transmitted continuously when no data
• Electrical Idle Exit (EIEOS): 16 characters (since 2.0)
character: 8 unscrambled bits
18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 50
PCIe – Link training 37181 ns
37312 ns
(1881):
EP LTSSM State: RECOVERY.RCVRLOCK
RP PCI Express Link Status Register

37312 ns Negotiated Link Width: x8

37312 ns Slot Clock Config: System
TS TS TS
Reference Clock Used
Device Device 37949 ns EP LTSSM State: RECOVERY.RCVRCFG
38845 ns RP LTSSM State: RECOVERY.RCVRCFG
1 TS TS TS 2 41053 ns RP LTSSM State: RECOVERY.SPEED
41309 ns EP LTSSM State: RECOVERY.SPEED
43573 ns EP LTSSM State: RECOVERY.RCVRLOCK
43765 ns RP LTSSM State: RECOVERY.RCVRLOCK

• Lane polarity
43797 ns RP LTSSM State: REC_EQULZ.PHASE0
43825 ns RP LTSSM State: REC_EQULZ.PHASE1
44141 ns EP LTSSM State: REC_EQULZ.PHASE0

• Link width / ordering 44673 ns

44929 ns
EP LTSSM State: REC_EQULZ.PHASE1
RP LTSSM State: REC_EQULZ.DONE

• Link equalization
44949 ns RP LTSSM State: RECOVERY.RCVRLOCK
45209 ns EP LTSSM State: REC_EQULZ.DONE
45229 ns EP LTSSM State: RECOVERY.RCVRLOCK
• Dynamic equalization! 45425 ns EP LTSSM State: RECOVERY.RCVRCFG
45581 ns RP LTSSM State: RECOVERY.RCVRCFG

• Link speed 45925 ns

46073 ns
RP LTSSM State: RECOVERY.IDLE
EP LTSSM State: RECOVERY.IDLE

• ...
46169 ns EP LTSSM State: L0
46313 ns RP LTSSM State: L0
47824 ns Current Link Speed: 8.0GT/s

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 51

PCIe Link-Training State Machine
(LTSSM) DLL RESET Detect

Poll Link Training

• L0: active
• L0 standby, L1: lower
power, higher latency
• L2: cold standby, even Conf
lower power
• L3: power off

Power
L2 Recovery Link Re-Training
Management

L1 L0 L0s
18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 52
 PCIe – Framing (x1)
…
Transmit order (TIME) STP Framing Symbol
(Physical Layer)
Reserved bits
Sequence Number
(Data Link Layer)

TLP structure
… (Transaction Layer)

LCRC
(Data Link Layer)

END Framing Symbol

(Physical Layer)
…
18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 53
 PCIe – Framing (x4)
Transmit order (TIME) Lane order (SPACE)

Lane 0 Lane 1 Lane 2 Lane 3 (Lane-reversal

… … … … possible)
STP

… … … …

END
… … … …

Physical Layer
Data Link Layer
Transaction Layer

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 54

PCIe troubleshooting
Signal integrity – Environment

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 55

PCIe troubleshooting
Signal integrity – EQ presets

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 56

 PCIe CEM Spec – AIC form factors
Solder side
(A)
Component side • Standard Height
(B)
• 4.20” (106.7mm)
• Low Profile
• 2.536” (64.4mm)
Standard
Height
Full Length
• Half Length (e.g.“HHHL”)
Low Profile • 6.6” (167.65mm)
Half Length • Full Length (e.g. “FHFL”)
Single/Dual • 12.283” (312mm)
Width Power: up to 10W, 25W, 75W, 300W or 375W depending on
form factor & optional extra power connectors
18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 57
PCIe storage – More form factors
M.2
≤ 4 lanes

U.2
≤ 4 lanes

“ruler” (EDSFF, NGSFF) ≤ 8 lanes

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 58

PCIe CEM Spec – Power Cables
EPS receptacle

PCIe cable

GPU power

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 59

What is this presentation about?
• History and evolution of PCIe

• PCIe concepts

• PCIe layers

• PCIe performance

• PCIe future roadmap

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 60

PCIe – Theoretical data rates
64 64 64

32 32 32 32

16 16 16 16 16

8 8 8 8 8
GB/s

4 4 4 4 4

2 2 2 2

1 1 1

0,5 0,5
x1 (x2) x4 x8 x16 (x32)
LINK WIDTH

Gen 1.x (2.5 GT/s) Gen 2.x (5.0 GT/s) Gen 3.0 (8.0 GT/s) Gen 4.0 (16.0 GT/s)

• “Aggregate” bandwidth in both directions

• Considering 20% encoding overhead in 1.x and 2.x
18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 61
PCIe – Effective data rates
Theoretical bandwidth Packet efficiency

𝐿𝑎𝑛𝑒 𝑟𝑎𝑡𝑒 × 𝐿𝑎𝑛𝑒 𝑤𝑖𝑑𝑡ℎ 𝑀𝑃𝑆

•ρ= ×
𝐸𝑛𝑐𝑜𝑑𝑖𝑛𝑔 𝑀𝑃𝑆+𝐻𝑒𝑎𝑑𝑒𝑟𝑠
• Example: Gen2 x8, 128 Bytes MPS
128
• ρ = 40 × 0.8 × = 32 x 0.84 = 26.9 Gb/s
128+24
• Example: Gen3 x8, 128 Bytes MPS
128
• ρ = 64 × 0.98 × = 62.7 x 0.84 = 52.6 Gb/s
128+24
• Example: Gen3 x8, 256 Bytes MPS
256
• ρ = 64 × 0.98 × = 62.7 x 0.91 = 57 Gb/s
256+24

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 62

PCIe 3.0 x8 – DMA Performance

MPS = 128 Bytes MPS = 256 Bytes

10%

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 63

What is this presentation about?
• History and evolution of PCIe

• PCIe concepts

• PCIe layers

• PCIe performance

• PCIe future roadmap

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 64

PCIe scalability – As of 2020
• Intel Xeon Broadwell • Counting Northbridge lanes only
• PCIe Gen3 x 40 lanes • Excludes additional lanes from
• Intel Xeon Skylake / Cascade Lake Southbridge
• PCIe Gen3 x 48 lanes • More density possible using
• IBM Power 8 switches
• PCIe Gen3 x 48 lanes • Shared bandwidth
• AMD Epyc Threadripper
• PCIe Gen3 x 64 lanes
• AMD Epyc Naples • Mostly driven by storage market
• PCIe Gen3 x 128 lanes (!) (dense NVMe)
• IBM Power 9
• PCIe Gen3 x 64 lanes ➢ Plus OpenCAPI & NVLINK lanes
• PCIe Gen4 x 64 lanes
• AMD Epyc Rome / Ryzen (Zen2)
• PCIe Gen4 x 128-160 lanes (!!) ➢ Depending on XGMII setup

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 65

PCIe Gen4 – On paper

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 66

PCIe Gen4 – On silicon (IBM)
IBM Power AC922

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 67

PCIe Gen4 – On silicon (Mellanox)
Mellanox ConnectX®-5 Mellanox ConnectX®-6

Dual-Port 100 Gbit/s Single-Port 200 Gbit/s

LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 68

PCIe Gen4 – On silicon (AMD)
• Zen 2 (EPYC™ Rome) architecture

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 69

 PCIe Gen4 – On silicon (Xilinx)
• Xilinx Virtex Ultrascale+™ VU37P
Bifurcated (2x) Gen4x8 PCIe, ~208 Gbps

https://www.alpha-data.com/news.php?item=58

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 70

PCIe Gen4 – On silicon (Intel)

• No Gen4 CPU until

IceLake (2020?)
• No Gen4 FPGA until
AgileX (2020?)
• Gen4 SSDs starting to
sample now

https://www.tomshardware.com/news/intel-has-pcie-40-optane-
ssds-ready-but-nothing-to-plug-them-in-to
18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 71
PCIe Gen5 – On paper

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 72

PCIe Gen5 + coherency = CXL
(Compute eXpress Link)
Announced by Intel in March 2019

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 73

Conclusions
• PCI Express interconnects can be present at all
levels of your DAQ chain…
• …and may be even more so in the future
• Lindy Effect: “The longer a technology has been around,
the longer it's likely to stay around.”
• Our DAQ systems keep increasing their bandwidth
requirements
• PCIe has a track record of delivering “2x” improvements
• It is up to us to implement and interconnect our systems
in ways that actually deliver this performance

18/01/2019 ISOTDAQ 2020 - Introduction to PCIe 74