Python Network Programming

1. Network Fundamentals
2. Client Programming

Slide Title Index

0. Introduction

Introduction 0-1
Support Files 0-2
Python Networking 0-3
This Course 0-4
Standard Library 0-5
Prerequisites 0-6

1. Network Fundamentals

Network Fundamentals 1-1

The Problem 1-2
Two Main Issues 1-3
Network Addressing 1-4
Standard Ports 1-5
Using netstat 1-6
Connections 1-7
Client/Server Concept 1-8
Request/Response Cycle 1-9
Using Telnet 1-10
Data Transport 1-11
Sockets 1-12
Socket Basics 1-13
Socket Types 1-14
Using a Socket 1-15
TCP Client 1-16
Exercise 1.1 1-17
Server Implementation 1-18
TCP Server 1-19
Exercise 1.2 1-27
Advanced Sockets 1-28
Partial Reads/Writes 1-29
Sending All Data 1-31
End of Data 1-32
Data Reassembly 1-33
Timeouts 1-34
Non-blocking Sockets 1-35
Socket Options 1-36
Sockets as Files 1-37
Exercise 1.3 1-39
Odds and Ends 1-40
UDP : Datagrams 1-41
UDP Server 1-42
UDP Client 1-43
Unix Domain Sockets 1-44
Raw Sockets 1-45
Sockets and Concurrency 1-46 1 SAX Parsing 3-11
 Section 0

Introduction

Python Networking

• Network programming is a major use of Python

• Python standard library has wide support for
network protocols, data encoding/decoding, and
other things you need to make it work

• Writing network programs in Python tends to be

substantially easier than in C/C++

2
 This Course
• This course focuses on the essential details of
network programming that all Python
programmers should probably know

• Low-level programming with sockets

• High-level client modules
• How to deal with common data encodings
• Simple web programming (HTTP)
• Simple distributed computing

3
 Standard Library
• We will only cover modules supported by the
Python standard library
• These come with Python by default
• Keep in mind, much more functionality can be
found in third-party modules

• Will give links to notable third-party libraries as

appropriate

Prerequisites

• You should already know Python basics

• However, you don't need to be an expert on all
of its advanced features (in fact, none of the code
to be written is highly sophisticated)

• You should have some prior knowledge of

systems programming and network concepts

4
 Section 1

Network Fundamentals

The Problem
• Communication between computers

Network

• It's just sending/receiving bits

5
 Two Main Issues

• Addressing
• Specifying a remote computer and service
• Data transport
• Moving bits back and forth

Network Addressing
• Machines have a hostname and IP address
• Programs/services have port numbers
foo.bar.com
205.172.13.4

port 4521 www.python.org

Network 82.94.237.218

port 80

6
 Standard Ports
• Ports for common services are preassigned
21 FTP
22 SSH
23 Telnet
25 SMTP (Mail)
80 HTTP (Web)
110 POP3 (Mail)
119 NNTP (News)
443 HTTPS (web)

• Other port numbers may just be randomly

assigned to programs by the operating system

Using netstat
• Use 'netstat' to view active network connections
shell % netstat -a
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address
tcp 0 0 *:imaps *:*
tcp 0 0 *:pop3s *:*
tcp 0 0 localhost:mysql *:*
tcp 0 0 *:pop3 *:*
tcp 0 0 *:imap2 *:*
tcp 0 0 *:8880 *:*
tcp 0 0 *:www *:*
tcp 0 0 192.168.119.139:domain *:*
tcp 0 0 localhost:domain *:*
tcp 0 0 *:ssh *:*
...

• Note: Must execute from the command shell on

both Unix and Windows

7
 Connections
• Each endpoint of a network connection is always
represented by a host and port #

• In Python you write it out as a tuple (host,port)

("www.python.org",80)
("205.172.13.4",443)

• Inwrite,
almost all of the network programs you’ll
you use this convention to specify a
network address

Client/Server Concept
• Each endpoint is a running program
• Servers wait for incoming connections and
provide a service (e.g., web, mail, etc.)

• Clients make connections to servers

Client Server
www.bar.com
205.172.13.4

browser web Port 80

8
 Request/Response Cycle
• Most network programs use a request/
response model based on messages
• Client sends a request message (e.g., HTTP)
GET /index.html HTTP/1.0

• Server sends back a response message

HTTP/1.0 200 OK
Content-type: text/html
Content-length: 48823

<HTML>
...

• The exact format depends on the application

Using Telnet
• As a debugging aid, telnet can be used to
directly communicate with many services
telnet hostname portnum

• Example:
shell % telnet www.python.org 80
Trying 82.94.237.218...
Connected to www.python.org.
type this Escape character is '^]'.
and press GET /index.html HTTP/1.0
return a few
times
HTTP/1.1 200 OK
Date: Mon, 31 Mar 2008 13:34:03 GMT
Server: Apache/2.2.3 (Debian) DAV/2 SVN/1.4.2
mod_ssl/2.2.3 OpenSSL/0.9.8c
...

9
 Data Transport
• There are two basic types of communication
• Streams (TCP): Computers establish a
connection with each other and read/write data
in a continuous stream of bytes---like a file. This
is the most common.

• Datagrams (UDP): Computers send discrete

packets (or messages) to each other. Each
packet contains a collection of bytes, but each
packet is separate and self-contained.

Sockets
• Programming abstraction for network code
• Socket: A communication endpoint
socket socket
network

• Supported by socket library module

• Allows connections to be made and data to be
transmitted in either direction

1
0
 Socket Basics
• To create a socket
import socket
s = socket.socket(addr_family, type)

• Address families
socket.AF_INET Internet protocol (IPv4)
socket.AF_INET6 Internet protocol (IPv6)

• Socket types
socket.SOCK_STREAM Connection based stream (TCP)
socket.SOCK_DGRAM Datagrams (UDP)

• Example:
from socket import *
s = socket(AF_INET,SOCK_STREAM)

Socket Types
• Almost all code will use one of following
from socket import *

s = socket(AF_INET, SOCK_STREAM)
s = socket(AF_INET, SOCK_DGRAM)

• Most common case:TCP connection

s = socket(AF_INET, SOCK_STREAM)

10
 Using a Socket
• Creating a socket is only the first step
s = socket(AF_INET, SOCK_STREAM)

• Further use depends on application

• Server
• Listen for incoming connections
• Client
• Make an outgoing connection

TCP Client
• How to make an outgoing connection
from socket import *
s = socket(AF_INET,SOCK_STREAM)
s.connect(("www.python.org",80)) # Connect
s.send("GET /index.html HTTP/1.0\n\n") # Send request
data = s.recv(10000) # Get response
s.close()

• s.connect(addr) makes a connection

s.connect(("www.python.org",80))

• Once connected, use send(),recv() to

transmit and receive data

• close() shuts down the connection

11
 Exercise 1.1

Time : 10 Minutes

Server Implementation

• Network servers are a bit more tricky

• Must listen for incoming connections on a
well-known port number
• Typically run forever in a server-loop
• May have to service multiple clients

12
 TCP Server
• A simple server
from socket import *
s = socket(AF_INET,SOCK_STREAM)
s.bind(("",9000))
s.listen(5)
while True:
c,a = s.accept()
print "Received connection from", a
c.send("Hello %s\n" % a[0])
c.close()

• Send a message back to a client

% telnet localhost 9000
Connected to localhost.
Escape character is '^]'.
Hello 127.0.0.1 Server message
Connection closed by foreign host.
%

TCP Server
• Address binding
from socket import *
s = socket(AF_INET,SOCK_STREAM)
binds the socket to
s.bind(("",9000))
s.listen(5) a specific address
while True:
c,a = s.accept()
print "Received connection from", a
c.send("Hello %s\n" % a[0])
c.close()

• Addressing binds to localhost

s.bind(("",9000))
s.bind(("localhost",9000))
If system has multiple
s.bind(("192.168.2.1",9000))
IP addresses, can bind
s.bind(("104.21.4.2",9000))
to a specific address

Copyright (C) 2010, http://www.dabeaz.com 1- 20

13
 TCP Server
• Start listening for connections
from socket import *
s = socket(AF_INET,SOCK_STREAM)
s.bind(("",9000)) Tells operating system to
s.listen(5) start listening for
while True:
c,a = s.accept()
connections on the socket
print "Received connection from", a
c.send("Hello %s\n" % a[0])
c.close()

• s.listen(backlog)
• backlog is # of pending connections to allow
• Note: not related to max number of clients

TCP Server
• Accepting a new connection
from socket import *
s = socket(AF_INET,SOCK_STREAM)
s.bind(("",9000))
s.listen(5)
while True:
c,a = s.accept() Accept a new client connection
print "Received connection from", a
c.send("Hello %s\n" % a[0])
c.close()

• s.accept() blocks until connection received

• Server sleeps if nothing is happening

14
 TCP Server
• Client socket and address
from socket import *
s = socket(AF_INET,SOCK_STREAM)
s.bind(("",9000))
s.listen(5)
Accept returns a pair (client_socket,addr)
while True:
c,a = s.accept()
print "Received connection from", a
c.send("Hello %s\n" % a[0])
c.close()

<socket._socketobject ("104.23.11.4",27743)
object at 0x3be30>
This is the network/port
This is a new socket address of the client that
that's used for data connected

TCP Server
• Sending data
from socket import *
s = socket(AF_INET,SOCK_STREAM)
s.bind(("",9000))
s.listen(5)
while True:
c,a = s.accept()
print "Received connection from", a
c.send("Hello %s\n" % a[0]) Send data to client
c.close()

Note: Use the client socket for

transmitting data.The server
socket is only used for
accepting new connections.

15
 TCP Server
• Closing the connection
from socket import *
s = socket(AF_INET,SOCK_STREAM)
s.bind(("",9000))
s.listen(5)
while True:
c,a = s.accept()
print "Received connection from", a
c.send("Hello %s\n" % a[0])
c.close() Close client connection

• Note: Server can keep client connection alive

as long as it wants

• Can repeatedly receive/send data

TCP Server
• Waiting for the next connection
from socket import *
s = socket(AF_INET,SOCK_STREAM)
s.bind(("",9000))
s.listen(5)
while True:
c,a = s.accept() Wait for next connection
print "Received connection from", a
c.send("Hello %s\n" % a[0])
c.close()

• Original server socket is reused to listen for

more connections

• Server runs forever in a loop like this

16
 Exercise 1.2

Time : 20 Minutes

Advanced Sockets
• Socket programming is often a mess
• Huge number of options
• Many corner cases
• Many failure modes/reliability issues
• Will briefly cover a few critical issues

17
 Partial Reads/Writes
• Be aware that reading/writing to a socket
may involve partial data transfer

• send() returns actual bytes sent

• recv() length is only a maximum limit
>>> len(data)
1000000
>>> s.send(data)
37722 Sent partial data
>>>

>>> data = s.recv(10000)

>>> len(data)
6420 Received less than max
>>>

Partial Reads/Writes
• Be aware that for TCP, the data stream is
continuous---no concept of records, etc.
# Client
...
s.send(data)
s.send(moredata)
...

# Server This recv() may return data

... from both of the sends
data = s.recv(maxsize)
...
combined or less data than
even the first send

• A lot depends on OS buffers, network

bandwidth, congestion, etc.

18
 Sending All Data
• To wait until all data is sent, use sendall()
s.sendall(data)

• Blocks until all data is transmitted

• For most normal applications, this is what
you should use

• Exception :You don’t use this if networking is

mixed in with other kinds of processing
(e.g., screen updates, multitasking, etc.)

End of Data
• How to tell if there is no more data?
• recv() will return empty string
>>> s.recv(1000)
''
>>>

• This means that the other end of the

connection has been closed (no more sends)

19
 Data Reassembly
• Receivers often need to reassemble
messages from a series of small chunks

• Here is a programming template for that

fragments = [] # List of chunks
while not done:
chunk = s.recv(maxsize) # Get a chunk
if not chunk:
break # EOF. No more data
fragments.append(chunk)

# Reassemble the message

message = "".join(fragments)

• Don't use string concat (+=). It's slow.

Timeouts
• Most socket operations block indefinitely
• Can set an optional timeout
s = socket(AF_INET, SOCK_STREAM)
...
s.settimeout(5.0) # Timeout of 5 seconds
...

• Will get a timeout exception

>>> s.recv(1000)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
socket.timeout: timed out
>>>

• Disabling timeouts
s.settimeout(None)

20
 Non-blocking Sockets
• Instead of timeouts, can set non-blocking
>>> s.setblocking(False)

• Future send(),recv() operations will raise an

exception if the operation would have blocked
>>> s.setblocking(False)
>>> s.recv(1000) No data available
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
socket.error: (35, 'Resource temporarily unavailable')
>>> s.recv(1000) Data arrived
'Hello World\n'
>>>

• Sometimes used for polling

Socket Options
• Sockets have a large number of parameters
• Can be set using s.setsockopt()
• Example: Reusing the port number
>>> s.bind(("",9000))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<string>", line 1, in bind
socket.error: (48, 'Address already in use')
>>> s.setsockopt(socket.SOL_SOCKET,
... socket.SO_REUSEADDR, 1)
>>> s.bind(("",9000))
>>>

• Consult reference for more options

21
 Sockets as Files
• Sometimes it is easier to work with sockets
represented as a "file" object
f = s.makefile()

• This will wrap a socket with a file-like API

f.read()
f.readline()
f.write()
f.writelines()
for line in f:
...
f.close()

22
 Sockets as Files
• Commentary : From personal experience,
putting a file-like layer over a socket rarely
works as well in practice as it sounds in theory.
• Tricky resource management (must manage
both the socket and file independently)

• It's easy to write programs that mysteriously

"freeze up" or don't operate quite like you
would expect.

23