Hervey Allen

Carlos Vicente
Network Startup Resource Center

Registry Operations Curriculum

Network Performance Definitions
and Analysis
Network Performance Metrics

 Planning performance management


Metrics
Network

Systems

Services


Definitions
Planning
 What's the intention?
Baselining, Troubleshooting, Planning growth
Defend yourself from accusations -”it's the network!”

Who is the information for?
Administration, NOC, customers
How to structure and present the information

Reach: Can I measure everything?

Impact on devices (measurements and measuring)

Balance between amount of information and time to get
it
Metrics

Network performance metrics

 Channel capacity, nominal & effective
 Channel utilization
 Delay and jitter
 Packet loss and errors
Metrics

System performance metrics

• Availability
• Memory, CPU Utilization, load, I/O wait, etc.
Service performance metrics
• Wait time / Delay
• Availability
• How can I justify maintaining the service?
• Who is using it? How often?
• Economic value? Other value?
Common Network Performance
Measurements
 Relative to traffic:
Bits per second
Packets per second
Unicast vs. non-unicast packets
Errors

Dropped packets
Flows per second
Round trip time (RTT)
Jitter (variation between packet RTT)
Nominal Channel Capacity
 The maximum number of bits that can be transmitted for a
unit of time (eg: bits per second)
 Depends on:
Bandwidth of the physical medium
Cable
 Electromagnetic waves

Processing capacity for each transmission element

Efficiency of algorithms in use to access medium

Channel encoding and compression
Effective Channel Capacity
 Always a fraction of the nominal channel
capacity

Dependent on:
Additional overhead of protocols in each layer
Device limitations on both ends

Flow control algorithm efficiency, etc.
 For example: TCP
Channel Utilization

 What fraction of the nominal channel capacity is

actually in use
 Important!
Future planning
 What utilization growth rate am I seeing?
 For when should I plan on buying additional capacity?
 Where should I invest for my updates?
Problem resolution
 Where are my bottlenecks, etc.
95th Percentile
95 Percentile
th

 The smallest value that is larger than 95% of the values in

a given sample
 This means that 95% of the time the channel utilization is
equal to or less than this value
Or rather, the peaks are discarded from consideration
 Why is this important in networks?
Gives you an idea of the standard, sustained channel
utilization.
ISPsuse this measure to bill customers with “larger”
connections.
95th Percentile
Bits per second vs Packets p.s.
End-to-end Delay

The time required to transmit a packet along its entire path

Created by an application, handed over to the OS, passed
to a network card (NIC), encoded, transmitted over a
physical medium (copper, fibre, air), received by an
intermediate device (switch, router), analyzed, retransmitted
over another medium, etc.
The most common measurement uses ping for total round-
trip-time (RTT).
Historical Measurement of Delay
Types of Delay

Causes of end-to-end delay:

 Processor delays
 Buffer delays
 Transmission delays
 Propagation delays
Processing Delay
Required time to analyze a packet header and
decide where to send the packet (eg. a routing
decision)
- Inside a router this depends on the number of entries
in the routing table, the implementation of data
structures, hardware in use, etc.
This can include error verification /
checksumming (i.e. IPv4, IPv6 header
checksum)
Queuing Delay
Queuing Delay
 The time a packet is enqueued until it is
transmitted
 The number of packets waiting in the queue will
depend on traffic intensity and of the type of
traffic
 Router queue algorithms try to adapt delays to
specific preferences, or impose equal delay on
all traffic.
Transmission Delay
Transmission Delay
The time required to push all the bits in a
packet on the transmission medium in use
For N=Number of bits, S=Size of packet, d=delay
d = S/N
For example, to transmit 1024 bits using Fast
Ethernet (100Mbps):

d = 1024/1x10e8 = 10.24 micro seconds

Propagation Delay

• Once a bit is 'pushed' on to the transmission

medium, the time required for the bit to propagate
to the end of its physical trajectory
• The velocity of propagation of the circuit depends
mainly on the actual distance of the physical circuit
• In the majority of cases this is close to the speed
of light.
For d = distance, s = propagation velocity
PD = d/s
Transmission vs. Propagation

Can be confusing at first

Consider this example:
Two 100 Mbps circuits
- 1 km of optic fiber
- Via satellite with a distance of 30 km between the base
and the satellite

For two packets of the same size which will

have the larger transmission delay?
Propagation delay?
Packet Loss

Occurs due to the fact that buffers are not

infinite in size
- When a packet arrives to a buffer that is full the packet
is discarded.
- Packet loss, if it must be corrected, is resolved at
higher levels in the network stack (transport or
application layers)
- Loss correction using retransmission of packets can
cause yet more congestion if some type of (flow)
control is not used (to inform the source that it's
pointless to keep sending more packets at the present
time)
Jitter
Flow Control and Congestion
Flow Control and Congestion
• Limits the transmission amount (rate)
because the receiver cannot process
packets at the same rate that packets are
arriving.
• Limit the amount sent (transmission rate)
because of loss or delays in the circuit.
Controls in TCP

IP (Internet Protocol) implements service that

not connection oriented.
- There is no mechanism in IP to deal with packet
loss.
TCP (Transmission Control Protocol)
implements flow and congestion control.
- Only on the ends as the intermediate nodes at the
network level do not talk TCP
Congestion vs. Flow in TCP
Congestion vs. Flow in TCP
Flow: controlled by window size (RcvWindow),
which is sent by the receiving end.
Congestion: controlled by the value of the
congestion window (Congwin)
• Maintained independently by the sender
• This varies based on the detection of packets lost
- Timeout or receiving three ACKs repeated
• Behaviors:
- Additive Increments / Multiplicative Decrements
(AIMD)
- Slow Start
- React to timeout events
Different TCP Congestion Control
Algorithms
Questions?

?
Local Analysis
Questions?

As we know...
Before we blame the network, let's verify
whether the problem is ours.

What can go wrong locally?
 Hardware problems
 Excessive load (CPU, memory, I/O)

What's considered 'normal'?
 Use analysis tools frequently

Become familiar with the normal state and values
for your machine.
 It is essential to maintain history

SNMP agents and databases
Local Analysis
Linux Performance Analysis

Three main categories:

 Processes
 Processes that are executing (running)
 Processes that are waiting (sleeping)
 waiting their turn
 blocked

 Memory
 Real
 Virtual
 I/O (Input/Output)
 Storage
 Network
Key Indicators

Insufficent CPU
 Number of processes waiting to execute is always
high
 High CPU utilization (load avg.)
Insufficient memory
 Very little free memory
 Lots of swap activity (swap in, swap out)
Slow I/O
 Lots of blocked processes
 High number of block transfers
Local Analysis

Luckily, in Unix there are dozens of useful

tools that give us lots of useful information

about our machine

Some of the more well-known include:
 vmstat - tcpdump
 top - wireshark (ethereal)
 lsof - iptraf
 netstat - iperf
vmstat
Show periodic summary information about
processes, memory, pagin, I/O, CPU state, etc
vmstat <-options> <delay> <count>

# vmstat 2
# vmstat
procs 2
-----------memory---------- ---swap-- -----io---- --system-- ----cpu----
rprocs
b -----------memory----------
swpd free buff cache ---swap--
si so -----io----
bi bo --system--
in cs us----cpu----
sy id wa
r b swpd free buff cache
2 0 209648 25552 571332 2804876 si
0 so
0 bi
3 bo
4 in
3 cs us
3 15 11sy 73
id wa0
2 2 0 0209648
209648 24680
25552571332
5713322804900
2804876 00 00 0 3 444 4 273 379356 3161516116873 0 0
1 2 0 0209648
209648 25216
24680571336
5713322804904
2804900 00 00 6 0 1234
444 439
27346735
79356161610167468 0 0
1 1 0 0209648
209648 25212
25216571336
5713362804904
2804904 00 00 0 6 1234 439100282
22 159 46735 1716 21
10 62
74 00
1 0 209648 25212 571336 2804904
2 0 209648 25196 571348 2804912 0 0 0 0 0 0 500 270 82455 14 18 68 620 0
22 159 100282 17 21
2 0 209648 25196 571348 2804912
1 0 209648 25192 571348 2804912 0 0 0 0 0 0 500 243
272 27077480
82455161415186968 0 0
2 1 0 0209648
209648 25880
25192571360
5713482804916
2804912 00 00 0 0 444
272 255
24383619
77480161614156969 0 0
2 0 209648 25880 571360 2804916
2 0 209648 25872 571360 2804920 0 0 0 0 0 0 444 255 83619161618146669 0 0
178 220 90521
2 0 209648 25872 571360 2804920 0 0 0 178 220 90521 16 18 66 0
top

 Basic performance tool for Unix/Linux

environments
 Periodically show a list of system
performance statistics:
 CPU use
 RAM and SWAP memory usage
 Load average (cpu utilization)
 Information by process
top cont.

 Information by process (most relevant

columns shown):
 PID: Process ID
 USER: user running (owner) of the process
 %CPU: Percentage of CPU utilization by the process
since the last sample
 %MEM: Percentage of physical memory (RAM) used
by the process
 TIME: Total CPU time used by the process since it
was started
Load Average

 Average number of active processes in

the last 1, 5 and 15 minutes
 A simple yet useful measurement
 Depending on the machine the acceptable
range considered to be normal can vary:
 Multi-processor machines can handle more active
processes per unit of time (than single processor
machines)
top
Some useful interactive keyboard
commands for top
 f : Add or remove columns
 F : Specify which column to order by
 <,> : Move the column on which we order
 u : Specify a specific user
 k : Specify a process to kill (stop)
 d,s : Change the display update interval
netstat
Show us information about:
 Network connections
 Routing tables
 Interface (NIC) statistics
 Multicast group members
netstat
Some useful options
-n: Show addresses, ports and userids in numeric form
-r: Routing table
-s: Statistics by protocol
-i: Status of interfaces
-l: Listening sockets
--tcp, --udp: Specify the protocol
-A: Address family [inet | inet6 | unix | etc.]
-p: Show the name of each process for each port
-c: Show output/results continuously
netstat
Examples:
netstat -n --tcp -c #
# netstat
Active -n --tcp
Internet connections -cservers)
(w/o
Active
Proto Internet
Recv-Q connections
Send-Q (w/o servers)
Local Address Foreign Address State
tcp Proto
0 Recv-Q Send-Q Local Address
272 ::ffff:192.188.51.40:22 Foreign Address
::ffff:128.223.60.27:60968 State
ESTABLISHED
tcptcp 0 0 0272 ::ffff:192.188.51.40:22
::ffff:192.188.51.40:22 ::ffff:128.223.60.27:60968ESTABLISHED
::ffff:128.223.60.27:53219 ESTABLISHED
tcp 0 0 ::ffff:192.188.51.40:22 ::ffff:128.223.60.27:53219 ESTABLISHED
# netstat -lnp --tcp
netstat
Active -lnp
Internet --tcp
connections #
(only servers)
Active
Proto Internet
Recv-Q connections
Send-Q (only servers)
Local Address Foreign Address State PID/Program name
tcp Proto Recv-Q
0 Send-Q Local Address
0 0.0.0.0:199 Foreign Address
0.0.0.0:* State 11645/snmpd
LISTEN PID/Program name
tcp tcp
0 0 0 0.0.0.0:199
0 0.0.0.0:3306 0.0.0.0:* 0.0.0.0:* LISTEN LISTEN
1997/mysqld11645/snmpd
tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN 1997/mysqld

# netstat -ic
# netstat
Kernel -ic table
Interface
Kernel Interface
Iface MTU MettableRX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
Iface
eth0 MTU Met
1500 RX-OK RX-ERR
0 2155901 0 RX-DRP
0 RX-OVR
0 TX-OK TX-ERR
339116 0 TX-DRP
0 TX-OVR Flg
0 BMRU
loeth0 1500 0 0 2155901
16436 18200 00 00 00 339116
18200 00 00 0 0LRU
BMRU
lo
eth0 16436
1500 0 18200
0 2155905 0 0 0 0 00 18200
339117 0 0 0 0 0 LRU
0 BMRU
loeth0 1500 0 0 2155905
16436 18200 00 00 00 339117
18200 00 00 0 0LRU
BMRU
lo
eth0 16436
1500 0 18200
0 2155907 0 0 0 0 00 18200
339120 0 0 0 0 0 LRU
0 BMRU
loeth0 1500
16436 0 0 2155907
18200 0 0 0 0 00 339120
18200 0 0 0 0 0 BMRU
0 LRU
lo
eth0 16436 0 0 2155910
1500 18200 00 00 00 18200
339122 00 00 0 0BMRU
LRU
loeth0 1500
16436 0 0 2155910
18200 0 0 0 0 00 339122
18200 0 0 0 0 0 BMRU
0 LRU
lo
eth0 16436 0 0 2155913
1500 18200 00 00 00 18200
339124 00 00 0 0BMRU
LRU
eth0 1500 0 2155913 0 0 0 339124 0 0 0 BMRU
netstat cont.
Examples:
netstat –tcp –listening --program #
# netstat –tcp –listening --program
Active Internet connections (only servers)
Active
Proto Internet
Recv-Q connections
Send-Q (only servers) Foreign Address
Local Address State PID/Program name
Proto
tcp Recv-Q
0 Send-Q Local
0 *:5001 Address Foreign Address
*:* State
LISTEN PID/Program name
13598/iperf
tcp tcp 0 00 localhost:mysql
0 *:5001 *:* *:* LISTEN LISTEN5586/mysqld
13598/iperf
tcp tcp 0 00 *:www 0 localhost:mysql *:* *:* LISTEN LISTEN 5586/mysqld
7246/apache2
tcp tcp 0 0 0 *:www
0 t60-2.local:domain *:* *:* LISTENLISTEN 7246/apache2
5378/named
tcp tcp 0 00 t60-2.local:domain
0 t60-2.local:domain*:* *:* LISTEN LISTEN 5378/named
5378/named
tcp tcp 0 0 0 t60-2.local:domain
0 t60-2.local:domain *:* *:* LISTEN LISTEN 5378/named
5378/named
tcp tcp 0 0 0 t60-2.local:domain
0 localhost:domain *:* *:* LISTEN LISTEN 5378/named
5378/named
tcp tcp 0 00 localhost:ipp
0 localhost:domain *:* *:* LISTEN LISTEN 5378/named
5522/cupsd
tcp tcp 0 0 0 localhost:ipp
0 localhost:smtp *:* *:* LISTEN LISTEN 5522/cupsd
6772/exim4
tcp tcp 0 0 0 localhost:smtp
0 localhost:953 *:* *:* LISTEN LISTEN 6772/exim4
5378/named
tcp tcp 0 0
0 *:https 0 localhost:953 *:* *:* LISTEN LISTEN 5378/named
7246/apache2
tcp6 tcp 0 0 0 [::]:ftp
0 *:https [::]:**:* LISTENLISTEN 7185/proftpd
7246/apache2
tcp6
tcp6 0 0 0 [::]:ftp
0 [::]:domain [::]:*
[::]:* LISTEN
LISTEN 7185/proftpd
5378/named
tcp6 tcp6 0 0
0 [::]:ssh 0 [::]:domain [::]:* [::]:* LISTEN LISTEN 5378/named
5427/sshd
tcp6 tcp6
0 0 0[::]:3000 0 [::]:ssh [::]:* [::]:* LISTEN LISTEN 5427/sshd
17644/ntop
tcp6 tcp6 0 0 0 [::]:3000
0 ip6-localhost:953 [::]:* [::]:* LISTEN LISTEN 17644/ntop
5378/named
tcp6 tcp6 0 0
0 [::]:30050 ip6-localhost:953[::]:* [::]:* LISTEN LISTEN 5378/named
17644/ntop
tcp6 0 0 [::]:3005 [::]:* LISTEN 17644/ntop
netstat cont.
$ sudo netstat -atup
$ sudo
Active netstat
Internet -atup
connections (servers and established) (if run as root PID/Program name is included)
Active
Proto Internet
Recv-Q connections
Send-Q (servers and established)
Local Address (if run as root
Foreign Address PID/Program
State name is included)
PID/Program name
Proto
tcp Recv-Q
0 Send-Q Local
0 *:35586 Address Foreign
*:* Address State
LISTEN PID/Program
2540/ekpd name
tcp
tcp 0 0 0 *:35586
0 localhost:mysql *:*
*:* LISTEN
LISTEN 2540/ekpd
2776/mysqld
tcp
tcp 00 0 0*:www
localhost:mysql *:*
*:* LISTEN
LISTEN 2776/mysqld
14743/apache2
tcp
tcp 0 0 0 *:www
0 d229-231.uoregon:domain *:**:* LISTEN
LISTEN 14743/apache2
2616/named
tcp
tcp 0 0 0 d229-231.uoregon:domain
0 *:ftp *:*
*:* LISTEN
LISTEN 2616/named
3408/vsftpd
tcp
tcp 00 0 0localhost:domain
*:ftp *:*
*:* LISTEN
LISTEN 3408/vsftpd
2616/named
tcp
tcp 0 0 0 localhost:domain
0 *:ssh *:*
*:* LISTEN
LISTEN 2616/named
2675/sshd
tcp
tcp 0 0 0 *:ssh
0 localhost:ipp *:*
*:* LISTEN
LISTEN 2675/sshd
3853/cupsd
tcp
tcp 0 0 0 localhost:ipp
0 localhost:smtp *:*
*:* LISTEN
LISTEN 3853/cupsd
3225/exim4
tcp
tcp 00 0 0localhost:953
localhost:smtp *:*
*:* LISTEN
LISTEN 3225/exim4
2616/named
tcp
tcp 0 0 0 localhost:953
0 *:https *:*
*:* LISTEN
LISTEN 2616/named
14743/apache2
tcp
tcp6 0 0 0 *:https
0 [::]:domain *:*
[::]:* LISTEN
LISTEN 14743/apache2
2616/named
tcp6
tcp6 00 0 0[::]:ssh
[::]:domain [::]:*
[::]:* LISTEN
LISTEN 2616/named
2675/sshd
tcp6
tcp6 0 0 0 [::]:ssh
0 ip6-localhost:953 [::]:*
[::]:* LISTEN
LISTEN 2675/sshd
2616/named
tcp6
udp 0 0 0 ip6-localhost:953
0 *:50842 [::]:*
*:* LISTEN 2616/named
3828/avahi-daemon:
udp
udp 00 0 0localhost:snmp
*:50842 *:*
*:* 3828/avahi-daemon:
3368/snmpd
udp
udp 0 0 0 localhost:snmp
0 d229-231.uoregon:domain *:**:* 3368/snmpd
2616/named
udp
udp 0 0 0 d229-231.uoregon:domain
0 localhost:domain *:*
*:* 2616/named
2616/named
udp
udp 0 0 0 localhost:domain
0 *:bootpc *:*
*:* 2616/named
13237/dhclient
udp
udp 00 0 0*:mdns
*:bootpc *:*
*:* 13237/dhclient
3828/avahi-daemon:
udp
udp 0 0 0 *:mdns
0 d229-231.uoregon.ed:ntp *:**:* 3828/avahi-daemon:
3555/ntpd
udp
udp 0 0 0 d229-231.uoregon.ed:ntp
0 localhost:ntp *:*
*:* 3555/ntpd
3555/ntpd
udp
udp 00 0 0*:ntp
localhost:ntp *:*
*:* 3555/ntpd
3555/ntpd
udp
udp6 0 0 0 *:ntp
0 [::]:domain *:*
[::]:* 3555/ntpd
2616/named
udp6
udp6 0 0 0 [::]:domain [::]:*
0 fe80::213:2ff:fe1f::ntp [::]:* 2616/named
3555/ntpd
udp6
udp6 00 0 0ip6-localhost:ntp
fe80::213:2ff:fe1f::ntp[::]:*
[::]:* 3555/ntpd
3555/ntpd
udp6
udp6 0 0 0 ip6-localhost:ntp
0 [::]:ntp [::]:*
[::]:* 3555/ntpd
3555/ntpd
udp6 0 0 [::]:ntp [::]:* 3555/ntpd
lsof (LiSt of Open Files)

 lsof is particularly useful because in Unix

everything is a file: unix sockets, ip
sockets, directories, etc.
 Allows you to associate open files by:
-p: PID (Process ID)
-i : A network address (protocol:port)
-u: A user
lsof

 Example:
 First, using netstat -ln –tcp determine that port
6010 is open and waiting for a connection
(LISTEN)
##netstat
netstat-ln
-ln--tcp
--tcp
Active Internet connections (only servers)
Active
Proto Internet
Recv-Q connections
Send-Q (only servers)
Local Address Foreign Address State
Proto Recv-Q Send-Q Local Address Foreign Address State

tcp 0 0 127.0.0.1:6010 0.0.0.0:* LISTEN

tcp 0 0 127.0.0.1:6010 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:6011 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:6011 0.0.0.0:* LISTEN
lsof
Determine what process has the port (6010) open
and what other resources are being used:
# lsof -i tcp:6010
# lsof PID
COMMAND -i tcp:6010
USER FD TYPE DEVICE SIZE NODE NAME
COMMAND 10301
sshd PID root
USER FD TYPE53603
6u IPv4 DEVICE SIZE
TCPNODE NAME
localhost.localdomain:x11-ssh-offset (LISTEN)
sshd
sshd 10301root
10301 root 7u6u IPv6
IPv4 53604
53603 TCP[::1]:x11-ssh-offset
TCP localhost.localdomain:x11-ssh-offset
(LISTEN) (LISTEN)
sshd 10301 root 7u IPv6 53604 TCP [::1]:x11-ssh-offset (LISTEN)

# lsof -p 10301
# lsof -p
COMMAND PID10301
USER FD TYPE DEVICE SIZE NODE NAME
COMMAND10301
sshd PIDroot
USER cwdFD TYPE
DIR DEVICE
8,2 SIZE
4096 NODE
2 / NAME
sshd
sshd 10301 root
10301 root cwd
rtd DIR
DIR 8,2
8,2 4096
4096 2 2/ /
sshd
sshd 10301root
10301 root rtd
txt DIR
REG 8,2 379720
8,2 40961422643 2/usr/sbin/sshd
/
sshd
sshd 10301 root
10301 root txt
mem REG
REG 8,2
8,2 379720 1422643 /usr/sbin/sshd
32724 1437533 /usr/lib/libwrap.so.0.7.6
sshd
sshd 10301 root
10301 root mem
mem REG
REG 8,2
8,2 32724 1437533 /usr/lib/libwrap.so.0.7.6
15088 3080329 /lib/libutil-2.4.so
sshd
sshd 10301root
10301 root mem
mem REG
REG 8,2 75632
8,2 150881414093
3080329/usr/lib/libz.so.1.2.3
/lib/libutil-2.4.so
sshd
sshd 10301 root
10301 root mem
mem REG
REG 8,2
8,2 75632 1414093 /usr/lib/libz.so.1.2.3
96040 3080209 /lib/libnsl-2.4.so
sshd
sshd 10301root
10301 root mem
mem REG
REG 8,2 100208
8,2 960401414578
3080209/usr/lib/libgssapi_krb5.so.2.2
/lib/libnsl-2.4.so
sshd
sshd 10301 root
10301 root mem
mem REG
REG 8,2
8,2 100208 1414578 /usr/lib/libgssapi_krb5.so.2.2
11684 1414405 /usr/lib/libkrb5support.so.0.0
sshd
sshd 10301 root
10301 root mem
mem REG
REG 8,2
8,2 11684 1414405 /usr/lib/libkrb5support.so.0.0
10368 3080358 /lib/libsetrans.so.0
sshd
sshd 10301root
10301 root mem
mem REG
REG 8,2
8,2 103683080231
7972 3080358/lib/libcom_err.so.2.1
/lib/libsetrans.so.0
sshd
sshd 10301 root
10301 root mem
mem REG
REG 8,2
8,2 7972 3080231 /lib/libcom_err.so.2.1
30140 1420233 /usr/lib/libcrack.so.2.8.0
sshd
sshd 10301root
10301 root mem
mem REG
REG 8,2 11168
8,2 301403080399
1420233/lib/security/pam_succeed_if.so
/usr/lib/libcrack.so.2.8.0
sshd
... 10301 root mem REG 8,2 11168 3080399 /lib/security/pam_succeed_if.so
...
lsof cont.

What network services am I running?

# lsof -i
# lsof -iPID
COMMAND USER FD TYPE DEVICE SIZE NODE NAME
COMMAND
firefox PID
4429 USER 50u
hervey FD TYPE1875852
IPv4 DEVICE SIZE TCP
NODE192.168.179.139:56890-
NAME
>128.223.60.21:www (ESTABLISHED 50u
firefox 4429 hervey IPv4 1875852 TCP 192.168.179.139:56890-
>128.223.60.21:www
named 5378 (ESTABLISHED
bind 20u IPv6 13264 TCP *:domain (LISTEN)
named
named 5378
5378 bind 21u
bind 20u IPv6 13267
IPv4 13264 TCPlocalhost:domain
TCP *:domain (LISTEN) (LISTEN)
named
sshd 5378
5427 bind
root 21u
3u IPv4
IPv6 13267
13302 TCP localhost:domain
TCP *:ssh (LISTEN) (LISTEN)
sshd
cupsd 5427
5522 root
root 3u3u IPv61983466
IPv4 13302 TCPlocalhost:ipp
TCP *:ssh (LISTEN)(LISTEN)
cupsd
mysqld 5522
5586 root
mysql 10u3u IPv4
IPv4 1983466
13548 TCP localhost:ipp (LISTEN)
TCP localhost:mysql (LISTEN)
mysqld
snmpd 5586
6477 mysql
snmp 10u
8u IPv4 14633
IPv4 13548 TCPlocalhost:snmp
UDP localhost:mysql (LISTEN)
snmpd
exim4 6477Debian-exim
6772 snmp 3u8u IPv4 14675
IPv4 14633 UDPlocalhost:smtp
TCP localhost:snmp(LISTEN)
exim4
ntpd 6772
6859 Debian-exim
ntp 16u3u IPv4
IPv4 14675
14743 TCP localhost:smtp (LISTEN)
UDP *:ntp
ntpd
ntpd 6859
6859 ntp
ntp 16u
17u IPv4
IPv6 14743
14744 UDP *:ntp
UDP *:ntp
ntpd
ntpd 6859
6859 ntp
ntp 17u
18u IPv6
IPv6 14744
14746 UDP *:ntp
UDP [fe80::250:56ff:fec0:8]:ntp
ntpd
ntpd 6859
6859 ntp 19u
ntp 18u IPv6 14747
IPv6 14746 UDPip6-localhost:ntp
UDP [fe80::250:56ff:fec0:8]:ntp
ntpd
proftpd 6859
7185 ntp
proftpd 19u
1u IPv6
IPv6 14747
15718 UDP ip6-localhost:ntp
TCP *:ftp (LISTEN)
proftpd
apache2 7185
7246 proftpd
www-data 3u1u IPv6
IPv4 15718
15915 TCP *:ftp(LISTEN)
TCP *:www (LISTEN)
apache2
apache2 7246
7246 www-data
www-data 4u3u IPv4 15917
IPv4 15915 TCP*:https
TCP *:www (LISTEN)
(LISTEN)
apache2
... 7246 www-data 4u IPv4 15917 TCP *:https (LISTEN)
...
iperf 13598 root 3u IPv4 1996053 TCP *:5001 (LISTEN)
iperf
apache2 13598
27088 root
www-data 3u3u IPv4 1996053
IPv4 15915 TCP*:www
TCP *:5001 (LISTEN)
(LISTEN)
apache2 27088
apache2 27088 www-data
www-data 4u3u IPv4 15917
IPv4 15915 TCP*:https
TCP *:www (LISTEN)
(LISTEN)
apache2 27088 www-data 4u IPv4 15917 TCP *:https (LISTEN)
tcpdump


Show received packet headers by a given
interface. Optionally filter using boolean
expressions.

Allows you to write information to a file for
later analysis.

Requires administrator (root) privileges to
use since you must configure network
interfaces (NICs) to be in “promiscuous”
mode.
tcpdump

Some useful options:

-i : Specify the interface (ex: -i eth0)
-l : Make stdout line buffered (view as you
capture)
-v, -vv, -vvv: Display more information
-n : Don't convert addresses to names
(avoid DNS)
-nn : Don't translate port numbers
-w : Write raw packets to a file
-r : Read packets from a file created by '-w'
tcpdump

Boolean expressions:
 Using the 'AND', 'OR', 'NOT' operators
 Expressions consist of one, or more,
primtives, which consist of a qualifier and an
ID (name or number):
Expression ::= [NOT] <primitive> [ AND | OR | NOT <primitive> ...]
<primitive> ::= <qualifier> <name|number>
<qualifier> ::= <type> | <address> | <protocol>
<type> ::= host | net | port | port range
<address> ::= src | dst
<protocol> ::= ether | fddi | tr | wlan | ip | ip6 | arp | rarp | decnet | tcp | udp
tcpdump

Examples:
 Show all HTTP traffic that originates from
192.168.1.1
# tcpdump -lnXvvv port 80 and src host 192.168.1.1
# tcpdump -lnXvvv port 80 and src host 192.168.1.1

- Show all traffic originating from

192.168.1.1 except SSH

# tcpdump -lnXvvv src host 192.168.1.1 and not port 22

# tcpdump -lnXvvv src host 192.168.1.1 and not port 22
Wireshark


Wireshark is a graphical packet analyser
based on libpcap, the same library that
tcpdump utilizes for capturing and storing
packets

The graphical interface has some advant-
ages, including:
 Hierarchical visualization by protocol (drill-down)
 Follow a TCP “conversation” (Follow TCP Stream)
 Colors to distinguish traffic types
 Lots of statistics, graphs, etc.
Wireshark

 Wireshark is what came after Ethereal.


The combination of tcpdump and
wireshark can be quite powerful. For
example:

# tcpdump -i eth1 -A -s1500 -2 dump.log port 21

$ sudo wireshark -r dump.log
Wireshark
iptraf


Many measurable statistics and
functions
 By protocol/port
 By packet size
 Generates logs
 Utilizes DNS to translate addresses

Advantages
 Simplicity
 Menu-based (uses “curses”)
 Flexible configuration
iptraf

 You can run it periodically in the

background (-B)
 It allows you, for example, to run as a cron job
to periodically analyze logs.
Generate alarms



Save in a data base

Has a great name... “Interactive Colorful IP LAN
Monitor”

etc...
Example: iptraf -i eth1
iptraf –i eth0
Sample iptraf output from the above command:
iperf

 To measure network throughput between

two points

iperf has two modes, server and
client

Easy to use
 Great to help determine optimal TCP
parameters
 TCP window size (socket buffer)
 MTU maximum segment size
 See man iperf for more
iperf

 Using UDP you can generate packet loss

and jitter reports
 You can run multiple parallel sessions
using threads
 Supports IPv6
iperf parameters
Usage: iperf [-s|-c host] [options]
Usage: iperf [-s|-c[-v|--version]
iperf [-h|--help] host] [options]
iperf [-h|--help] [-v|--version]
:Client/Server
f,Client/Server:
--format [kmKM] format to report: Kbits, Mbits, KBytes, MBytes-
-f, --format
i, --interval # [kmKM] secondsformat between to periodic
report: Kbits,
bandwidth Mbits, KBytes, MBytes
reports-
-i, --interval
l, --len #[KM] # lengthseconds between
of buffer to read periodic bandwidth
or write (default 8 reports
KB)-
-m, -l,--print_mss
--len #[KM] length print TCPof buffer to readsegment
maximum or write (default
size (MTU8 KB)- TCP/IP header)
-p,m, --print_mss
--port # serverprint
portTCP maximum
to listen on/connectsegmentto size (MTU - TCP/IP header)-
-u,p,--udp
--port # server
use UDP port to listen
rather thanon/connect
TCP to-
-w,u,--window
--udp #[KM] use TCPUDP window
rather than sizeTCP-(socket buffer size)
-B,w,--bind
--window <host>#[KM] bindTCP window
to <host>, an size (socket
interface buffer size)-
or multicast address
-C,B,--compatibility
--bind <host>forbind useto <host>,
with older an interface
versions does or not
multicast address-
sent extra msgs
-M,C,--mss
--compatibility
# setforTCPusemaximum
with older versions
segmentdoes size not
(MTU sent extra
- 40 msgs-
bytes)
-N,M,--nodelay
--mss # setsetTCP
TCPnomaximum segment
delay, disabling Nagle'ssize Algorithm
(MTU - 40 bytes)-
-V,N,--IPv6Version
--nodelay set
SetTCP the no delay,to
domain disabling
IPv6 Nagle's Algorithm-
V, --IPv6Version Set the domain to IPv6-
Server specific:
:Server
-s, --serverspecific run in server mode
-U,s,--single_udp
--server run
run inin server mode- UDP mode
single threaded
-D,U,--daemon
--single_udp runthe
run in server
single threaded
as a daemon UDP mode-
D, --daemon run the server as a daemon-
Client specific:
:Client
-b, specific #[KM] for UDP, bandwidth to send at in bits/sec
--bandwidth
b, --bandwidth #[KM] 1 Mbit/sec,
(default for UDP, bandwidth
implies -u) to send at in bits/sec-
-c, --client <host> )default
run 1 inMbit/sec,
client mode, implies -u(
connecting to <host>
-d,>c, --client <host>
--dualtest Do arun in client mode,
bidirectional connecting to <host-
test simultaneously
-n,d,--num
--dualtest #[KM] Do a bidirectional
number test simultaneously-
of bytes to transmit (instead of -t)
-r,n, --num
--tradeoff #[KM]Do anumber of bytes
bidirectional testtoindividually
transmit (instead of -t)-
-t,r,--time
--tradeoff# Do ainbidirectional
time seconds totest individually-
transmit for (default 10 secs)
-F,t,--fileinput
--time <name> # time in seconds
input the data to to be
transmit for (default
transmitted 10 secs)-
from a file
-I, F, --fileinput <name>
--stdin input the input
datatheto data to be transmitted
be transmitted from stdin from a file-
-L,I,--listenport
--stdin # input port the data tobidirectional
to recieve be transmitted tests from
backstdin-
on
-P,L,--parallel
--listenport # # numberport to recieve bidirectional
of parallel tests back
client threads to runon-
-T,P,--ttl
--parallel
# # time-to-live,
numberfor ofmulticast
parallel client
(defaultthreads
1) to run-
T, --ttl # time-to-live, for multicast (default 1)-
iperf - TCP

iperf -s $
$ iperf -s
------------------------------------------------------------
------------------------------------------------------------
Server listening on TCP port 5001
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
[ 4] local 128.223.157.19 port 5001 connected with 201.249.107.39 port 39601
local 128.223.157.19 port 5001 connected with 201.249.107.39 port 39601 ]4 [
[ 4] 0.0-11.9 sec 608 KBytes 419 Kbits/sec
sec 608 KBytes 0.0-11.9 ]4 [Kbits/sec 419
------------------------------------------------------------
------------------------------------------------------------
# iperf -c nsrc.org
iperf -c nsrc.org #
------------------------------------------------------------
------------------------------------------------------------
Client connecting to nsrc.org, TCP port 5001
Client connecting to nsrc.org, TCP port 5001
TCP window size: 16.0 KByte (default)
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
[ 3] local 192.168.1.170 port 39601 connected with 128.223.157.19 port 5001
local 192.168.1.170 port 39601 connected with 128.223.157.19 port 5001 ]3 [
[ 3] 0.0-10.3 sec 608 KBytes 485 Kbits/sec
sec 608 KBytes 0.0-10.3 ]3 [Kbits/sec 485
iperf - UDP
iperf -c host1 -u -b100M #
# iperf -c host1 -u -b100M
------------------------------------------------------------
------------------------------------------------------------
Client connecting to nsdb, UDP port 5001
Client connecting
Sending 1470 to nsdb, UDP port 5001
byte datagrams
Sending
UDP buffer1470 byte 106
size: datagrams
KByte (default)
UDP buffer size: 106 KByte (default)
------------------------------------------------------------
[ ------------------------------------------------------------
3] local 128.223.60.27 port 39606 connected with 128.223.250.135 port 5001
local 128.223.60.27
[ 3] 0.0-10.0 sec port
114 39606 connected
MBytes with 128.223.250.135 port 5001 ]3 [
95.7 Mbits/sec
[ sec 11481377
3] Sent MBytes 95.7 Mbits/sec 0.0-10.0 ]3 [
datagrams
[ Sent 81377 datagrams
3] Server Report: ]3 [
:Server Report
[ 3] 0.0-10.0 sec ]3 [ 114 MBytes 95.7 Mbits/sec 0.184 ms 1/81378 (0.0012%)
sec 114 MBytes 95.7 Mbits/sec 0.184 ms 1/81378 (0.0012%) 0.0-10.0 ]3 [
$ iperf -s -u -i 1
iperf -s -u -i 1 $
------------------------------------------------------------
------------------------------------------------------------
Server listening on UDP port 5001
Server
Receiving listening
1470 byteondatagrams
UDP port 5001
Receiving
UDP 1470 byte
buffer size: 108datagrams
KByte (default)
UDP buffer size: 108 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
[ 3] local 128.223.250.135 port 5001 connected with 128.223.60.27 port 39606
[ local 128.223.250.135
3] 0.0- 1.0 sec 11.4 port 5001 connected
MBytes with 128.223.60.27
95.4 Mbits/sec 0.184 ms port
0/ 39606
8112 ]3 [
(0%)
[ sec
3] 11.4
1.0- MBytes
2.0 sec 95.4
11.4Mbits/sec 0.184
MBytes 95.7 ms
Mbits/sec0/ 0.177
8112 (0%)
ms 1.00/ -0.0 ]3 [
8141 (0%)
sec 11.4 MBytes 95.7 Mbits/sec 0.177 ms
[ 3] 2.0- 3.0 sec 11.4 MBytes 95.6 Mbits/sec 0.182 ms0/ 8141 (0%) 2.0 -1.0 ]3 [
0/ 8133 (0%)
sec 11.4 MBytes 95.6 Mbits/sec 0.182 ms
... 0/ 8133 (0%) 3.0 -2.0 ]3 [
[ ...
3] 8.0- 9.0 sec 11.4 MBytes 95.7 Mbits/sec 0.177 ms 0/ 8139 (0%)
sec 11.4 MBytes 95.7 Mbits/sec 0.177 ms
[ 3] 9.0-10.0 sec 11.4 MBytes 95.7 Mbits/sec 0.180 ms0/ 8139 (0%) 9.0
0/ -8.0 ]3 [
8137 (0%)
[ sec
3] 11.4 MBytes
0.0-10.0 sec 95.7114
Mbits/sec 0.180Mbits/sec
MBytes 95.7 ms 0/ 8137
0.184(0%)
ms 9.0-10.0
1/81378]3 [
(0.0012%)
sec 114 MBytes 95.7 Mbits/sec 0.184 ms 1/81378 (0.0012%) 0.0-10.0 ]3 [
Bibliography
• Monitoring Virtual Memory with vmstat
http://www.linuxjournal.com/article/8178
• How to use TCPDump
http://www.erg.abdn.ac.uk/users/alastair/tcpdump.html
• linux command tcpdump example
http://smartproteam.com/linux-tutorials/linux-command-tcpdump/
• simple usage of tcpdump
http://linux.byexamples.com/archives/283/simple-usage-of-tcpdump/
• TCPDUMP Command man page with examples
http://www.cyberciti.biz/howto/question/man/tcpdump-man-page-with-examples.php
• TCPDump Tutorial
http://inst.eecs.berkeley.edu/~ee122/fa06/projects/tcpdump-6up.pdf