Message-Passing Programming

Parallel Programming with

Message- Passing Interface (MPI) Paradigm
An Introduction Each processor in a message-passing program
runs a sub-program
written in a conventional sequential language
WW Grid
Rajkumar Buyya all variables are private
communicate via special subroutine calls
Grid Computing and Distributed Systems (GRIDS) Lab.
The University of Melbourne M M M Memory

Melbourne, Australia
P P P Processors
www.gridbus.org

Interconnection Network

Messages Messages

Messages are packets of Access:

data moving between sub- Each sub-program needs to be connected to a message passing
programs system
The message passing system Addressing:
has to be told the following Messages need to have addresses to be sent to
information Reception:
Sending processor
It is important that the receiving process is capable of dealing
Source location with the messages it is sent
Data type
Data length
A message passing system is similar to:
Post-office, Phone line, Fax, E-mail, etc
Receiving processor(s)
Destination location Message Types:
Destination size Point-to-Point, Collective, Synchronous (telephone)/Asynchronous
(Postal)

Point-to-Point Communication Point-to-Point variations

Simplest form of message passing Synchronous Sends

One process sends a message to another provide information about the completion of the
message
Several variations on how sending a message e.g. fax machines
can interact with execution of the sub- Asynchronous Sends
program Only know when the message has left
e.g. post cards
Blocking operations
only return from the call when operation has completed
Non-blocking operations
return straight away - can test/wait later for
completion

1
 Collective Communications MPI Routines – C and Fortran

Collective communication routines are higher Environment Management

level routines involving several processes at a Point- to- Point Communication
time Collective Communication
Can be built out of point-to-point Process Group Management
communications
Communicators
Barriers
synchronise processes
Derived Type
Broadcast Virtual Topologies
one-to-many communication Miscellaneous Routines
Reduction operations
combine data from several processes to produce a single
(usually) result

Environment Management Routines Point- to- Point Communication

Process Group Management

Collective Communication Routines
Routines

2
 Communicators Routines Derived Type Routines

Virtual Topologies Routines Miscellaneous Routines

MPI Messages MPI Basic Datatypes - C

A message contains a number of elements MPI datatype C datatype

MPI_C AR signe cha
of some particular datatype MPI_S ORT signe sho t in
MPI datatypes MPI_I T signe int
MPI_L NG signe lon int
Basic Types MPI_U SIGN D_CH R unsig ed c ar
Derived types MPI_U SIGN D_SH RT unsig ed s ort int
MPI_U SIGN D unsig ed i t
Derived types can be built up from basic MPI_U SIGN D_LO G unsig ed l ng i t
types MPI_F OAT float
MPI_D UBLE doubl
C types are different from Fortran types MPI_L NG_D UBLE long doubl
MPI_B TE
MPI_P CKED

3
 Point-to-Point Communication Standard Send
Completes once the message has been sent
Communication between two processes Note: it may or may not have been received
Source process sends message to Programs should obey the following rules:
destination process It should not assume the send will complete before the
Communication takes place within a receive begins - can lead to deadlock
It should not assume the send will complete after the
communicator receive begins - can lead to non-determinism
Destination process is identified by its rank processes should be eager readers - they should guarantee
to receive all messages sent to them - else network
in the communicator overload
MPI provides four communication modes Can be implemented as either a buffered
for sending messages send or synchronous send
standard, synchronous, buffered, and ready
Only one mode for receiving

Standard Send (cont.) Standard Blocking Receive

MPI_Send(void *buf, int count, MPI_Datatype datatype, int dest,
int tag, MPI_Comm comm) Note: all sends so far have been blocking (but this
buf the address of the data to be sent only makes a difference for synchronous sends)
count the number of elements of datatype buf contains
Completes when message received
datatype the MPI datatype
MPI_Recv(buf, count, datatype, source, tag, comm,
dest rank of destination in communicator comm status)
tag a marker used to distinguish different message types source - rank of source process in communicator comm
comm the communicator shared by sender and receiver status - returns information about message
ierror the fortran return value of the send
Synchronous Blocking Message-Passing
processes synchronise
sender process specifies the synchronous mode
blocking - both processes wait until transaction completed

Synchronous Send Standard/Blocked Send/Receive

Completes when the message has been

received
Effect is to synchronise the sender and
receiver
Deadlocks if no receiver
Safer than standard send but may be
slower
MPI_Ssend(void *buf, int count, MPI_Datatype
datatype, int dest, int tag, MPI_Comm comm)
All parameters as for standard send
Fortran equivalent as usual (plus ierror)

4
 Buffered Send Ready Send
To be used on the sender only if he knows the
Guarantees to complete immediately matching receive has been posted
Copies message to buffer if necessary May improve performance on particular
implementation
To use buffered mode the user must Completes immediately
explicitly attach buffer space Guaranteed to succeed if receive is already posted
MPI_Bsend(void *buf, int count, MPI_Datatype Outcome is undefined in no receive posted
datatype, int dest, int tag, MPI_Comm comm) Requires careful attention to messaging patterns
MPI_Buffer_attach(void *buf, int size) MPI_Rsend(buf, count, datatype, dest, tag, comm)
process 0 process 1
Only one buffer can de attached at any one non-blocking receive with tag 0

time synchronous send with tag 1 blocking receive with tag 1

Buffers can be detached ready send with tag 0

test non-blocking receive
MPI_Buffer_detach(void *buf, int size)

Point-to-Point Rules Non Blocking Message Passing

Message Order Preservation

messages do not overtake each other
true even for non-synchronous sends
i.e. if process a posts two sends and process posts matching
receives then they will complete in the order they were
sent
Progress
It is not possible for a matching send and receive pair to
remain permanently outstanding.
It is possible for a third process to match one of the pair

5
This document was created with Win2PDF available at http://www.daneprairie.com.
The unregistered version of Win2PDF is for evaluation or non-commercial use only.