ISP & IXP Design

Philip Smith
APNIC 34
21st 31st August 2012

1
ISP & IXP Network Design
p PoP Topologies and Design
p Backbone Design
p Upstream Connectivity & Peering
p Addressing
p Routing Protocols
p Out of Band Management
p Operational Considerations
p Internet Exchange Points

2
Point of Presence
Topologies

3
PoP Topologies
p Core routers high speed trunk
connections
p Distribution routers and Access routers
high port density
p Border routers connections to other
providers
p Service routers hosting and servers
p Some functions might be handled by a
single router
4
PoP Design
p ModularDesign
p Aggregation Services separated according
to
n connection speed
n customer service
n contention ratio
n security considerations

5
 Modular PoP Design
Other ISPs
Web Cache
ISP Services Hosted Services &
(DNS, Mail, News, Datacentre
FTP, WWW)

Backbone link Backbone link

to another PoP to another PoP

Network
Core

Consumer cable,
Consumer xDSL and
DIal Access wireless Access

Business customer MetroE customer

aggregation layer aggregation layer
Network
Operations
Centre
Channelised circuits GigE fibre trunks
for leased line circuit delivery for MetroE circuit delivery
6
Modular Routing Protocol Design
p Modular IGP implementation
n IGP area per PoP
n Core routers in backbone area (Area 0/L2)
n Aggregation/summarisation where possible
into the core
p Modular iBGP implementation
n BGP route reflector cluster
n Core routers are the route-reflectors
n Remaining routers are clients & peer with
route-reflectors only
7
Point of Presence Design

8
PoP Modules
p Low Speed customer connections
n PSTN/ISDN dialup
n Low bandwidth needs
n Low revenue, large numbers

p Leased line customer connections

n E1/T1 speed range
n Delivery over channelised media
n Medium bandwidth needs
n Medium revenue, medium numbers

9
PoP Modules
p Broad Band customer connections
n xDSL, Cable and Wireless
n High bandwidth needs
n Low revenue, large numbers

p MetroE & Highband customer connections

n Trunk onto GigE or 10GigE of 10Mbps and
higher
n Channelised OC3/12 delivery of E3/T3 and
higher
n High bandwidth needs
n High revenue, low numbers

10
PoP Modules
p PoP Core
n Two dedicated routers
n High Speed interconnect
n Backbone Links ONLY
n Do not touch them!

p Border Network
n Dedicated border router to other ISPs
n The ISPs front door
n Transparent web caching?
n Two in backbone is minimum guarantee for
redundancy 11
PoP Modules
p ISP Services
n DNS (cache, secondary)
n News (still relevant?)
n Mail (POP3, Relay, Anti-virus/anti-spam)
n WWW (server, proxy, cache)

p Hosted Services/DataCentres
n Virtual Web, WWW (server, proxy, cache)
n Information/Content Services
n Electronic Commerce

12
PoP Modules
p Network Operations Centre
n Consider primary and backup locations
n Network monitoring
n Statistics and log gathering
n Direct but secure access

p Out of Band Management Network

n The ISP Network Safety Belt

13
 Low Speed Access Module

Web Cache

Access Network
Primary Rate
Gateway Routers
T1/E1
Access Servers
PSTN lines to
modem bank To Core Routers

PSTN lines to
built-in modems

TACACS+/Radius
proxy, DNS resolver,
Content 14
 Medium Speed Access Module

Aggregation Edge

Channelised T1/E1

64K and nx64K

circuits To Core Routers

Mixture of channelised
T1/E1, 56/64K and
nx64K circuits

15
High Speed Access Module

Aggregation Edge

Metro Ethernet

Channelised T3/E3
To Core Routers

Channelised OC3/OC12

16
 Broadband Access Module

Web Cache

DSLAM
Telephone Network BRAS
Access Network
Gateway Routers
IP, ATM

Cable RAS To Core Routers

The cable system

SSG, DHCP, TACACS+

or Radius Servers/Proxies,
DNS resolver, Content 17
ISP Services Module
To core routers

Service Network
Gateway Routers

WWW DNS Mail DNS

cache POP3 NEWS
secondary Relay cache

18
Hosted Services Module
To core routers

Hosted Network
Gateway Routers

vlan11 vlan12 vlan13 vlan14 vlan15 vlan16 vlan17

Customer 1 Customer 3 Customer 5 Customer 7

Customer 2 Customer 4 Customer 6

19
Border Module

To local IXP ISP1 ISP2

NB: router has no default route +
local AS routing table only

Network
Border Routers

To core routers

20
 NOC Module
Critical Services
To core routers Module
Corporate LAN
Out of Band
Hosted Network
Management Network Gateway Routers Firewall

2811/32async

Billing, Database
and Accounting
Systems
NetFlow TACACS+ SYSLOG Primary DNS
Analyser server server

Network Operations Centre Staff 21

 Out of Band Network

Out of Band
Management Network

Router
consoles
Terminal server
To the NOC
NetFlow
enabled
routers
NetFlow
Collector

Out of Band Ethernet

22
Backbone Network
Design

23
Backbone Design
p Routed Backbone
p Switched Backbone
n Virtually obsolete
p Point-to-point circuits
n nx64K, T1/E1, T3/E3, OC3, OC12, GigE, OC48,
10GigE, OC192, OC768
p ATM/Frame Relay service from telco
n T3, OC3, OC12, delivery
n Easily upgradeable bandwidth (CIR)
n Almost vanished in availability now
24
Distributed Network Design
p PoP design standardised
n operational scalability and simplicity
p ISP essential services distributed around
backbone
p NOC and backup NOC
p Redundant backbone links

25
 Distributed Network Design
Customer
ISP Services connections

Backup
Operations Centre POP Two

Customer Customer
connections connections

ISP Services

POP Three POP One

ISP Services
External External
connections connections
Operations Centre
26
Backbone Links
p ATM/Frame Relay
n Virtually disappeared due to overhead, extra
equipment, and shared with other customers
of the telco
n MPLS has replaced ATM & FR as the telco
favourite
p Leased Line/Circuit
n Most popular with backbone providers
n IP over Optics and Metro Ethernet very
common in many parts of the world

27
Long Distance Backbone Links
p These usually cost more
p Important to plan for the future
n This means at least two years ahead
n Stay in budget, stay realistic
n Unplanned emergency upgrades will be
disruptive without redundancy in the network
infrastructure

28
Long Distance Backbone Links
p Allow
sufficient capacity on alternative
paths for failure situations
n Sufficient can depend on the business strategy
n Sufficient can be as little as 20%
n Sufficient is usually over 50% as this offers
business continuity for customers in the case
of link failure
n Some businesses choose 0%
p Very short sighted, meaning they have no spare
capacity at all!!

29
Long Distance Links

POP Two
Long distance link

POP Three POP One

Alternative/Backup Path

30
Metropolitan Area Backbone Links
p Tend to be cheaper
n Circuit concentration
n Choose from multiple suppliers

p Think big
n More redundancy
n Less impact of upgrades
n Less impact of failures

31
Metropolitan Area Backbone Links

POP Two
Metropolitan Links

POP Three POP One

Metropolitan Links

Traditional Point to Point Links 32

Upstream Connectivity
and Peering

33
Transits
p Transit provider is another autonomous system
which is used to provide the local network with
access to other networks
n Might be local or regional only
n But more usually the whole Internet
p Transit providers need to be chosen wisely:
n Only one
p no redundancy
n Too many
p more difficult to load balance
p no economy of scale (costs more per Mbps)
p hard to provide service quality
p Recommendation: at least two, no more
than three
Common Mistakes
p ISPs sign up with too many transit providers
n Lots of small circuits (cost more per Mbps than larger
ones)
n Transit rates per Mbps reduce with increasing transit
bandwidth purchased
n Hard to implement reliable traffic engineering that
doesnt need daily fine tuning depending on customer
activities
p No diversity
n Chosen transit providers all reached over same satellite
or same submarine cable
n Chosen transit providers have poor onward transit and
peering
Peers
p A peer is another autonomous system with which
the local network has agreed to exchange locally
sourced routes and traffic
p Private peer
n Private link between two providers for the purpose of
interconnecting
p Public peer
n Internet Exchange Point, where providers meet and
freely decide who they will interconnect with
p Recommendation: peer as much as possible!
Common Mistakes
p Mistaking a transit providers Exchange
business for a no-cost public peering point
p Not working hard to get as much peering
as possible
n Physically near a peering point (IXP) but not
present at it
n (Transit is rarely cheaper than peering!!)

p Ignoring/avoiding
competitors because
they are competition
n Even though potentially valuable peering
partner to give customers a better experience
Private Interconnection
p Twoservice providers agree to
interconnect their networks
n They exchange prefixes they originate into the
routing system (usually their aggregated
address blocks)
n They share the cost of the infrastructure to
interconnect
p Typically each paying half the cost of the link (be it
circuit, satellite, microwave, fibre,)
p Connected to their respective peering routers

n Peering routers only carry domestic prefixes

38
Private Interconnection
Upstream
Upstream

PR ISP2
PR

ISP1

p PR = peering router
n Runs iBGP (internal) and eBGP (with peer)
n No default route
n No full BGP table
n Domestic prefixes only
p Peering router used for all private interconnects39
Public Interconnection
p Service
provider participates in an
Internet Exchange Point
n It exchanges prefixes it originates into the
routing system with the participants of the IXP
n It chooses who to peer with at the IXP
p Bi-lateral peering (like private interconnect)
p Multi-lateral peering (via IXPs route server)

n It provides the router at the IXP and provides

the connectivity from their PoP to the IXP
n The IXP router carries only domestic prefixes

40
Public Interconnection
Upstream
ISP6-PR
ISP5-PR

IXP ISP1-PR ISP1

ISP4-PR

ISP2-PR
ISP3-PR

p ISP1-PR = peering router of our ISP

n Runs iBGP (internal) and eBGP (with IXP peers)
n No default route
n No full BGP table
n Domestic prefixes only
p Physically located at the IXP 41
Public Interconnection
p The ISPs router IXP peering router needs careful
configuration:
n It is remote from the domestic backbone
n Should not originate any domestic prefixes
n (As well as no default route, no full BGP table)
n Filtering of BGP announcements from IXP peers (in and
out)

42
Upstream/Transit Connection
p Two scenarios:
n Transit provider is in the locality
p Which means bandwidth is cheap, plentiful, easy to
provision, and easily upgraded
n Transit provider is a long distance away
p Over undersea cable, satellite, long-haul cross
country fibre, etc
p Each
scenario has different considerations
which need to be accounted for

43
Local Transit Provider

AR ISP1
BR

Transit

p BR = ISPs Border Router

n Runs iBGP (internal) and eBGP (with transit)
n Either receives default route or the full BGP table from
upstream
n BGP policies are implemented here (depending on
connectivity)
n Packet filtering is implemented here (as required) 44
Distant Transit Provider

AR1

ISP1
Transit BR

AR2

p BR = ISPs Border Router

n Co-located in a co-lo centre (typical) or in the upstream providers
premises
n Runs iBGP with rest of ISP1 backbone
n Runs eBGP with transit provider router(s)
n Implements BGP policies, packet filtering, etc
n Does not originate any domestic prefixes
45
Distant Transit Provider
p Positioning
a router close to the Transit
Providers infrastructure is strongly
encouraged:
n Long haul circuits are expensive, so the router
allows the ISP to implement appropriate
filtering first
n Moves the buffering problem away from the
Transit provider
n Remote co-lo allows the ISP to choose another
transit provider and migrate connections with
minimum downtime
46
Distant Transit Provider
p Other points to consider:
n Does require remote hands support
n (Remote hands would plug or unplug cables,
power cycle equipment, replace equipment, etc
as instructed)
n Appropriate support contract from equipment
vendor(s)
n Sensible to consider two routers and two long-
haul links for redundancy

47
Summary
p Design considerations for:
n Private interconnects
p Simple private peering
n Public interconnects
p Router co-lo at an IXP
n Local transit provider
p Simple upstream interconnect
n Long distance transit provider
p Router remote co-lo at datacentre or Transit
premises

48
Addressing

49
Getting IPv4 & IPv6 address space
p Take part of upstream ISPs PA space
or
p Become a member of your Regional Internet
Registry and get your own allocation
n Require a plan for a year ahead
n General policies are outlined in RFC2050, more
specific details are on the individual RIR website
p There is no more IPv4 address space at IANA
n APNIC is now in its final /8 IPv4 delegation policy
n Limited IPv4 available
n IPv6 allocations are simple to get in most RIR regions

50
What about RFC1918 addressing?
p RFC1918 defines IPv4 addresses reserved for
private Internets
n Not to be used on Internet backbones
n http://www.ietf.org/rfc/rfc1918.txt
p Commonly used within end-user networks
n NAT used to translate from private internal to public
external addressing
n Allows the end-user network to migrate ISPs without a
major internal renumbering exercise
p Most ISPs filter RFC1918 addressing at their
network edge
n http://www.cymru.com/Documents/bogon-
list.html 51
What about RFC1918 addressing?
p There is a long list of well known problems:
n http://datatracker.ietf.org/doc/draft-ietf-grow-private-ip-sp-cores

p Including:
n False belief it conserves address space
n Adverse effects on Traceroute
n Effects on Path MTU Discovery
n Unexpected interactions with some NAT implementations
n Interactions with edge anti-spoofing techniques
n Peering using loopbacks
n Adverse DNS Interaction
n Serious Operational and Troubleshooting issues
n Security Issues
p false sense of security, defeating existing security 52
techniques
What about RFC1918 addressing?
p Infrastructure Security: not improved by using
private addressing
n Still can be attacked from inside, or from customers, or
by reflection techniques from the outside
p Troubleshooting: made an order of magnitude
harder
n No Internet view from routers
n Other ISPs cannot distinguish between down and broken
p Summary:
n ALWAYS use globally routable IP addressing for ISP
Infrastructure

53
Addressing Plans ISP
Infrastructure
p Address block for router loop-back interfaces
p Address block for infrastructure
n Per PoP or whole backbone
n Summarise between sites if it makes sense
n Allocate according to genuine requirements, not historic
classful boundaries
p Similar allocation policies should be used for IPv6
as well
n ISPs just get a substantially larger block (relatively) so
assignments within the backbone are easier to make

54
Addressing Plans Customer
p Customers are assigned address space
according to need
p Should not be reserved or assigned on a
per PoP basis
n ISP iBGP carries customer nets
n Aggregation not required and usually not
desirable

55
Addressing Plans (contd)
p Document infrastructure allocation
n Eases operation, debugging and management
p Document customer allocation
n Contained in iBGP
n Eases operation, debugging and management
n Submit network object to RIR Database

56
Routing Protocols

57
Routing Protocols
p IGP Interior Gateway Protocol
n carries infrastructure addresses, point-to-point
links
n examples are OSPF, ISIS,...

p EGP Exterior Gateway Protocol

n carries customer prefixes and Internet routes
n current EGP is BGP version 4

p No connection between IGP and EGP

58
Why Do We Need an IGP?
p ISP backbone scaling
n Hierarchy
n Modular infrastructure construction
n Limiting scope of failure
n Healing of infrastructure faults using dynamic
routing with fast convergence

59
Why Do We Need an EGP?
p Scaling to large network
n Hierarchy
n Limit scope of failure

p Policy
n Control reachability to prefixes
n Merge separate organizations
n Connect multiple IGPs

60
Interior versus Exterior Routing
Protocols
p Interior p Exterior
n Automatic neighbour n Specifically configured
discovery peers
n Generally trust your IGP n Connecting with outside
routers networks
n Prefixes go to all IGP n Set administrative
routers boundaries
n Binds routers in one AS n Binds ASs together
together

61
Interior versus Exterior Routing
Protocols
p Interior p Exterior
n Carries ISP n Carries customer
infrastructure addresses prefixes
only n Carries Internet
n ISPs aim to keep the prefixes
IGP small for efficiency n EGPs are independent
and scalability of ISP network topology

62
Hierarchy of Routing Protocols
Other ISPs
BGP4

BGP4
and OSPF/ISIS

BGP4 Static/BGP4

Customers
IXP
63
Routing Protocols:
Choosing an IGP
p OSPF and ISIS have very similar properties
p Which to choose?
n Choose which is appropriate for your operators
experience
n In most vendor releases, both OSPF and ISIS have
sufficient nerd knobs to tweak the IGPs behaviour
n OSPF runs on IP
n ISIS runs on infrastructure, alongside IP
n ISIS supports both IPv4 and IPv6
n OSPFv2 (IPv4) plus OPSFv3 (IPv6)

64
Routing Protocols:
IGP Recommendations
p Keep the IGP routing table as small as possible
n If you can count the routers and the point to point links
in the backbone, that total is the number of IGP entries
you should see
p IGP details:
n Should only have router loopbacks, backbone WAN
point-to-point link addresses, and network addresses of
any LANs having an IGP running on them
n Strongly recommended to use inter-router
authentication
n Use inter-area summarisation if possible

65
Routing Protocols:
More IGP recommendations
p To fine tune IGP table size more, consider:
n Using ip unnumbered on customer point-to-
point links saves carrying that /30 in IGP
p (If customer point-to-point /30 is required for
monitoring purposes, then put this in iBGP)
n Use contiguous addresses for backbone WAN
links in each area then summarise into
backbone area
n Dont summarise router loopback addresses
as iBGP needs those (for next-hop)
n Use iBGP for carrying anything which does not
contribute to the IGP Routing process
66
Routing Protocols:
iBGP Recommendations
p iBGP
should carry everything which
doesnt contribute to the IGP routing
process
n Internet routing table
n Customer assigned addresses
n Customer point-to-point links
n Dial network pools, passive LANs, etc

67
Routing Protocols:
More iBGP Recommendations
p Scalable iBGP features:
n Use neighbour authentication
n Use peer-groups to speed update process and
for configuration efficiency
n Use communities for ease of filtering
n Use route-reflector hierarchy
p Route reflector pair per PoP (overlaid clusters)

68
Out of Band Management

69
Out of Band Management
p Not optional!
p Allows access to network equipment in
times of failure
p Ensures quality of service to customers
n Minimises downtime
n Minimises repair time
n Eases diagnostics and debugging

70
Out of Band Management
p OoB Example Access server:
n modem attached to allow NOC dial in
n console ports of all network equipment
connected to serial ports
n LAN and/or WAN link connects to network
core, or via separate management link to NOC
p Full
remote control access under all
circumstances

71
Out of Band Network
Equipment Rack Equipment Rack

Router, switch
and ISP server
consoles

(Optional) Out of band

WAN link to other PoPs

Modem access
to PSTN for out of
band dialin

Ethernet
72
to the NOC
Out of Band Management
p OoB Example Statistics gathering:
n Routers are NetFlow and syslog enabled
n Management data is congestion/failure
sensitive
n Ensures management data integrity in case of
failure
p Full
remote information under all
circumstances

73
Test Laboratory

74
Test Laboratory
p Designed to look like a typical PoP
n Operated like a typical PoP
p Used to trial new services or new software
under realistic conditions
p Allows discovery and fixing of potential
problems before they are introduced to
the network

75
Test Laboratory
p Some ISPs dedicate equipment to the lab
p Other ISPs purchase ahead so that
todays lab equipment becomes
tomorrows PoP equipment
p Other ISPs use lab equipment for hot
spares in the event of hardware failure

76
Test Laboratory
p Cant afford a test lab?
n Set aside one spare router and server to trial
new services
n Never ever try out new hardware, software or
services on the live network
p Everymajor ISP in the US and Europe has
a test lab
n Its a serious consideration

77
 Operational
Considerations

78
Operational Considerations

Why design the worlds best

network when you have not
thought about what operational
good practices should be
implemented?
79
Operational Considerations
Maintenance
p Never
work on the live network, no matter
how trivial the modification may seem
n Establish maintenance periods which your
customers are aware of
p e.g. Tuesday 4-7am, Thursday 4-7am
p Never do maintenance on a Friday
n Unless you want to work all weekend cleaning
up
p Never do maintenance on a Monday
n Unless you want to work all weekend preparing
80
Operational Considerations
Support
p Differentiate
between customer support
and the Network Operations Centre
n Customer support fixes customer problems
n NOC deals with and fixes backbone and
Internet related problems
p Network Engineering team is last resort
n They design the next generation network,
improve the routing design, implement new
services, etc
n They do not and should not be doing support!

81
Operational Considerations
NOC Communications
p NOCshould know contact details for
equivalent NOCs in upstream providers
and peers

82
ISP Network Design
Summary

83
ISP Design Summary
p KEEP IT SIMPLE & STUPID ! (KISS)
p Simple is elegant is scalable
p Use Redundancy, Security, and
Technology to make life easier for yourself
p Above all, ensure quality of service for
your customers

84
 Why an Internet
Exchange Point?
Saving money, improving QoS,
Generating a local Internet
economy

85
Internet Exchange Point
Why peer?
p Consider a region with one ISP
n They provide internet connectivity to their customers
n They have one or two international connections
p Internet grows, another ISP sets up in
competition
n They provide internet connectivity to their customers
n They have one or two international connections
p How does traffic from customer of one ISP get to
customer of the other ISP?
n Via the international connections

86
Internet Exchange Point
Why peer?
p Yes, International Connections
n If satellite, RTT is around 550ms per hop
n So local traffic takes over 1s round trip

p International bandwidth
n Costs significantly more than domestic
bandwidth
n Congested with local traffic
n Wastes money, harms performance

87
Internet Exchange Point
Why peer?
p Solution:
n Two competing ISPs peer with each other
p Result:
n Both save money
n Local traffic stays local
n Better network performance, better QoS,
n More international bandwidth for expensive
international traffic
n Everyone is happy

88
Internet Exchange Point
Why peer?
p A third ISP enters the equation
n Becomes a significant player in the region
n Local and international traffic goes over their
international connections
p They agree to peer with the two other
ISPs
n To save money
n To keep local traffic local
n To improve network performance, QoS,

89
Internet Exchange Point
Why peer?
p Peering
means that the three ISPs have to
buy circuits between each other
n Works for three ISPs, but adding a fourth or a
fifth means this does not scale
p Solution:
n Internet Exchange Point

90
Internet Exchange Point
p Every
participant has to buy just one
whole circuit
n From their premises to the IXP
p Rather
than N-1 half circuits to connect to
the N-1 other ISPs
n 5 ISPs have to buy 4 half circuits = 2 whole
circuits already twice the cost of the IXP
connection

91
Internet Exchange Point
p Solution
n Every ISP participates in the IXP
n Cost is minimal one local circuit covers all domestic
traffic
n International circuits are used for just international
traffic and backing up domestic links in case the IXP
fails
p Result:
n Local traffic stays local
n QoS considerations for local traffic is not an issue
n RTTs are typically sub 10ms
n Customers enjoy the Internet experience
n Local Internet economy grows rapidly

92
Exchange Point Design

93
 Layer 2 Exchange
ISP 6 ISP 5 ISP 4

IXP Services:
IXP
Root & TLD DNS, Management
Routing Registry Network
Ethernet Switch
Looking Glass, etc

ISP 1 ISP 3
ISP 2

94
 Layer 2 Exchange
ISP 6 ISP 5 ISP 4

IXP Services:
IXP
Root & TLD DNS, Management
Routing Registry Network
Looking Glass, etc Ethernet Switches

ISP 1 ISP 3
ISP 2

95
Layer 2 Exchange
p Two switches for redundancy
p ISPs use dual routers for redundancy or
loadsharing
p Offer services for the common good
n Internet portals and search engines
n DNS Root & TLD, NTP servers
n Routing Registry and Looking Glass

96
Layer 2 Exchange
p Requires neutral IXP management
n Usually funded equally by IXP participants
n 24x7 cover, support, value add services

p Secure and neutral location

p Configuration
n IPv4 /24 and IPv6 /64 for IXP LAN
n ISPs require AS, basic IXP does not

97
Layer 2 Exchange
p Network Security Considerations
n LAN switch needs to be securely configured
n Management routers require TACACS+
authentication, vty security
n IXP services must be behind router(s) with
strong filters

98
Layer 3 IXP
p Layer 3 IXP is marketing concept used by
Transit ISPs
p Real Internet Exchange Points are only
Layer 2

99
 IXP Design
Considerations

100
Exchange Point Design
p The IXP Core is an Ethernet switch
p Has superseded all other types of network
devices for an IXP
n From the cheapest and smallest 12 or 24 port
10/100 switch
n To the largest 192 port 10GigEthernet switch

101
Exchange Point Design
p Each ISP participating in the IXP brings a
router to the IXP location
p Router needs:
n One Ethernet port to connect to IXP switch
n One WAN port to connect to the WAN media
leading back to the ISP backbone
n To be able to run BGP

102
Exchange Point Design
p IXP
switch located in one equipment rack
dedicated to IXP
n Also includes other IXP operational equipment
p Routers from participant ISPs located in
neighbouring/adjacent rack(s)
p Copper (UTP) connections made for
10Mbps, 100Mbps or 1Gbps connections
p Fibre used for 10Gbps, 40Gbps or
100Gbps connections

103
Peering
p Each participant needs to run BGP
n They need their own AS number
n Public ASN, NOT private ASN

p Eachparticipant configures external BGP

directly with the other participants in the
IXP
n Peering with all participants
or
n Peering with a subset of participants

104
Peering (more)
p Mandatory Multi-Lateral Peering (MMLP)
n Each participant is required to peer with every other
participant as part of their IXP membership
n Has no history of success the practice is strongly
discouraged
p Multi-Lateral Peering (MLP)
n Each participant peers with every other participant
(usually aided by a Route Server)
p Bi-Lateral Peering
n Participants set up peering with each other according to
their own requirements and business relationships
n This is the most common situation at IXPs today

105
Routing
p ISP border routers at the IXP generally should
NOT be configured with a default route or carry
the full Internet routing table
n Carrying default or full table means that this router and
the ISP network is open to abuse by non-peering IXP
members
n Correct configuration is only to carry routes offered to
IXP peers on the IXP peering router
p Note: Some ISPs offer transit across IX fabrics
n They do so at their own risk see above

106
Routing (more)
p ISP
border routers at the IXP should not
be configured to carry the IXP LAN
network within the IGP or iBGP
n Use next-hop-self BGP concept
p Dont
generate ISP prefix aggregates on
IXP peering router
n If connection from backbone to IXP router goes
down, normal BGP failover will then be
successful

107
Address Space
p Some IXPs use private addresses for the IX LAN
n Public address space means IXP network could be leaked
to Internet which may be undesirable
n Because most ISPs filter RFC1918 address space, this
avoids the problem
p Some IXPs use public addresses for the IX LAN
n Address space available from the RIRs
n IXP terms of participation often forbid the IX LAN to be
carried in the ISP member backbone

108
Services Offered
p Services
offered should not compete with
member ISPs (basic IXP)
n e.g. web hosting at an IXP is a bad idea unless
all members agree to it
p IXP
operations should make performance
and throughput statistics available to
members
n Use tools such as MRTG/Cacti to produce IX
throughput graphs for member (or public)
information

109
Services to Offer
p ccTLD DNS
n the country IXP could host the countrys top level DNS
n e.g. SE. TLD is hosted at Netnod IXes in Sweden
n Offer back up of other country ccTLD DNS
p Root server
n Anycast instances of I.root-servers.net, F.root-
servers.net etc are present at many IXes
p Usenet News
n Usenet News is high volume
n could save bandwidth to all IXP members

110
Services to Offer
p Route Collector
n Route collector shows the reachability
information available at the exchange
p Looking Glass
n One way of making the Route Collector routes
available for global view (e.g.
www.traceroute.org)
n Public or members only access

111
Services to Offer
p Content Redistribution/Caching
n For example, Akamised update distribution
service
p Network Time Protocol
n Locate a stratum 1 time source (GPS receiver,
atomic clock, etc) at IXP
p Routing Registry
n Used to register the routing policy of the IXP
membership

112
Conclusion
p IXPs are technically very simple to set up
p Little more than:
n An ethernet switch
n Neutral secure reliable location
n Consortium of members to operate it

p Political aspects can be more challenging:

n Competition between ISP members
n ownership or influence by outside parties

113