An Overview of Cloud & Distributed Computing -

Programming Paradigms

VCV.Rao
Centre for Development of Advanced Computing (C-DAC),

Pune University Campus

National Workshop on Big Data Analytics (BiDA2014) at CRRao AIMSCS

August 22-24, 2014,
jointly with CMSD, U of Hyderabad, & Computer Society of India.
VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

VCV. Rao, Ph.D

Associate Director /HoD
HPC Frontier technologies Exploration (HPC-FTE) Group
Centre for Development of Advanced Computing (C-DAC)
Pune University Campus,
GaneshKhind, Pune
Maharashtra State, India
Email : vcvrao@cdac.in
CDAC is a R&D Institute of Department of Electronics and Information Technology (DeitY) ,
Ministry of Communications & Information Technology (MCIT) );
Government of India

VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

Distributed and Cloud Computing Prog. & Software Environments An Overview

Lecture Outline

Following topics will be discussed

An Overview of Distributed Computing Architectures
(HPC Systems High Performance Computing, & High
Throughput Computing, Grid Computing & Utility
Computing) and Cloud Computing
Programming on Cloud and Distributed Computing
Systems
Opportunities for Applications

VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

References & Source for Presentation

The Presentation is prepared based on Authors experience on
various research projects on Cloud & Distributed Computing as
well as references given in this presentation.
Source :
Text Books, Research Articles, Web Sites as indicated in many
slides and References of this presentation

Courtesy :
Authors research work as indicated in Text Books, Research
Articles, Web Sites as indicated in many slides and References of
this presentation

VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

C-DAC , in brief

Centre for Development of Advanced Computing (C-DAC)

is the

premier R&D organization of the Department of

Electronics and Information Technology (DeitY), Ministry of

Communications & Information Technology (MCIT) for

carrying out R&D in IT, Electronics and associated areas.

The setting up in 1988 - C-DAC has been undertaking

building of multiple generations of Supercomputer starting

from PARAM with 1 GF in 1988.

VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

Distributed System Models & Enabling Technologies

Scalable Computing over the Internet

The New Computing Paradigms

Computing Paradigm Distinctions

Centralized Computing

Parallel Computing

Distributed Computing

Cloud Computing
Source : Distributed and Cloud Computing Book, From Parallel Processing to the Internet of Things, by
Kai Hwang, Geoffrey C. Fox, Jack J. Dongarra, Morgan & Kaufmann, Publishers 2012
Source : References given in the presentation

VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

Distributed System Models and Enabling Technologies

Scalable Computing over the Internet

High-Performance Computing

Supercomputers (massively parallel processors or MPPs)

are

gradually

replaced

by

clusters

of

cooperative

computers out of desire to share computing resources).

The Cluster is often a collection

of homogeneous

computer nodes that are physically connected in close

range to one another.
VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

Distributed System Models & Enabling Technologies

Scalable Computing over the Internet

High-Throughput Computing (HTC)

Peer-to-peer (P2P) networks are formed for distributed file

sharing and content delivery applications.

Change will take place on HPC Paradigms to an HTC

paradigms (High-flux computing). The applications for highflux computing is the Internet searches and web services by
million or more users simultaneously.
VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

Distributed Comp. & Software Environments

Parallel and Distributed Programming Paradigms
Consider a distributed computing system consisting of a set of network nodes
or workers. For Parallelization of application on Distributed Cloud computing,
below important issues should be addressed.

Partitioning
Computation Partitioning

Data Partitioning
Mapping
Synchronization
Communication
Scheduling
VCV.Rao. C-DAC, Pune

Source : Distributed and Cloud Computing Book, From Parallel Processing to

the Internet of Things, by
Kai Hwang, Geoffrey C. Fox, Jack J. Dongarra, Morgan & Kaufmann,
Publishers 2012
Source : References given in the presentation
National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014
9

Emerging Parallel Processing Platforms: Challenges

MIMD : Parallel Processing Platforms Software & Hardware

Scalability Vs. Single System Image

Distributed
Computer
Systems

Better Performance

Clusters

SMPs

Scalability Vs. Single System Image

MPPs

Future Cluster
Accelerators

Distributed
System

Single-System Image
Overlapped design space of
Clusters, MPPs, SMPs, and
Distributed Computer systems

MPP

CC-NUMA

Present
Cluster

SMP
UP

Single System Image

Source : Distributed and Cloud Computing Book, From Parallel Processing to the Internet of Things,
By Kai Hwang, Geoffrey C. Fox, Jack J. Dongarra, Morgan & Kaufmann, Publishers 2012
Source : References given in the presentation
VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

10

Shared Pool of Computing Applications

Shared Pool of Computing
Resources: Processors,
Memory, Disks.

Guarantee at least one

workstation to many
individuals (when active)

Deliver large % of collective

resources to few individuals at
any one time.

Source : Distributed and Cloud Computing Book, From Parallel Processing to the Internet of Things, by
Kai Hwang, Geoffrey C. Fox, Jack J. Dongarra, Morgan & Kaufmann, Publishers 2012
Source : References given in the presentation
VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

11

Emerging Parallel Processing Platforms: Challenges

Current Parallel Processing Platforms
Cray XT4, XT5; IBM Systems
CrayX1EVector Processing

One-to-All Broadcast

One-to-All Personalized
Communication
All-to-All Broadcast

Cluster of Multi-Core Systems

PARAM Series

All-to-All personalized
Communication
Circular Shift

ANUPAM

Reduction

ANURAG

Prefix Sum

3D torus Topology

NETWORK
Multi Core Node 0
CPU0

CPU1

CPU2

Memory

Multi Core Node 1

CPU3

CPU0

CPU1

CPU2

CPU3

Memory

Cluster of Multi Core Processor Systems

VCV.Rao. C-DAC, Pune

Source : References

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

12

Emerging Parallel Processing Platforms: Clusters

Future/ Idealized Cluster Architecture
Programming Environment : GLSL(OpenGL Shading Language);
HLSL(high level shader language); Direct X, Cg, OpenCL, OpenMP
Programming Environment
(Java, C, Fortran, MPI, PVM)

top-level switches (288 ports)

Connects to 12 nodes

node-level switches (24 ports)

Web Windows Other Subsystems

User Interface (Database, OLTP)
NVIDIA-Kepler-CUDA

Single System Image Infrastructure

Availability Infrastructure
AMD Fire Stream

User
Application

OS

OS

Node

Node

MPI

OS

TCP
IP

Node

Intel Xeom-Phi

Driver
NIC

Hardware

Gigabit Commodity Interconnect/SAN

VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

C-DAC RC-FPGA

13

Distributed System Models and Enabling Technologies

High
(100%)
Grid
Cluster

System availability

MPP
Cloud

NUMA

P2P
network

SMP

Low
(0)
Small

System size (# processor cores)

Large (106)

Estimated system availability by system size of common configurations in 2010.

Source : Distributed and Cloud Computing Book, From Parallel Processing to the Internet of Things,
by Kai Hwang, Geoffrey C. Fox, Jack J. Dongarra, Morgann & Kaufmann, Publishers 2012

VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

14

Distributed System Models and Enabling Technologies

Scalability (No. of processors or

cores in a system)

107
106

P2P

105

Cluster

104
Cloud

103
NUMA

Grid

102
10

SMP

1
1

10

102

103

104

105

106

107

System scalability versus multiplicity of OS images

Source : Distributed and Cloud Computing Book, From Parallel Processing to the Internet of Things,
by Kai Hwang, Geoffrey C. Fox, Jack J. Dongarra, Morgann & Kaufmann, Publishers 2012

VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

15

Applications in Grid Computing

Classification of Grid applications
Grid aware applications
Multi-disciplinary Applications

All applications which

make use of coupled
computational resources
that are not available at a
SINGLE SITE

SAME

Grid Services: A Layered Grid Architecture

Coordinating multiple resources:
ubiquitous infrastructure services, appspecific distributed services
Sharing single resources: negotiating
access, controlling use
Talking to things: communication
(Internet protocols) & security

Application

Collective

Application

Resource

Connectivity

Controlling things locally: Access

to, & control of, resources

Transport

Internet
Fabric

Source : References & The Globus CoG Home Page. http://www.globus.org/cog.

Globus. http://www.globus.org. http://www.teragrid.org/
The NASA Information Power Grid Home Page. http://www.ipg.nasa.gov. http://www.cdac.in/
VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

Internet Protocol Architecture

Meta Applications

Link

16

Applications in Grid Computing

Compute hardware
Intel/Linux Clusters, Alpha SMP clusters,
POWER4 cluster,
Large-scale storage systems
hundreds of terabytes for secondary storage
Very high-speed network backbone
bandwidth for rich interaction and tight
coupling
Grid middleware
Globus, data management,
Next-generation applications

Grid Systems

Grids are distributed systems that enable

the sharing, selection, and aggregation of
resources distributed across "multiple"
administrative domains based on
availability, capability, performance, cost,
and quality-of-service requirements

Categories of Grid Applications

First-in-time, or an on-demand
computing
Data-Intensive Computing
Collaborative Computing
High Throughput Computing
Distributed Supercomputing

Comp. Grid

Distributed Super
Computing

High Throughput

Data Grid
On Demand

Service Grid
Grid Taxonomy

Collaborative
Multi-Media

Source : References & The Globus CoG Home Page. http://www.globus.org/cog.

Globus. http://www.globus.org. http://www.teragrid.org/
The NASA Information Power Grid Home Page. http://www.ipg.nasa.gov. http://www.cdac.in/
VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

17

Utility Computing
Utility computing is the packaging of Computing resources such as computation
and storage, as a metered service similar to a traditional Public Utility(such as
electricity, water, natural gas or telephone network ).
More related to cloud computing
Amazon Web Service (AWS)
Applications, storage, computing power and
Elastics Computer Cloud (EC2)
network
Simple Storage Service (S3)
Requires cloud like infrastructure
Pay by the drink model
EMC cloud Storage
Similar to electric service at home
Microsoft Azure
Pay for extra resources when needed
Google App Engine
To handle expected surge in demand
Unanticipated surges in demand
Better economics

A utility computing service is one in which customers receive computing resources

from a service provider (hardware and/or software) and pay by the drink, much
as you do for your electric service at home
Source : References & The Globus CoG Home Page. http://www.globus.org/cog.
The NASA Information Power Grid Home Page. http://www.ipg.nasa.gov. http://www.cdac.in/
Raj Kumar Buyya,. Bubendorfer (Eds.), Market Oriented Grid and Utility Computing, John Wiley & Sons, 2009.
VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

18

Distributed System Models & Enabling Technologies

The trend towards Utility Computing: The vision of computer utilities in
modern distributed computing systems.
Web services
Data centers
Utility computing

Technology

Service

computing

convergence

HTC in business and HPC

in scientific applications

Grid computing
P2P computing
Cloud computing

Computing paradigms
Attributes/capabilities

Ubiquitous: Reliable and scalable

Automatic: Dynamic and discovery
Composable: QoS, SLA, etc.

Source : Distributed and Cloud Computing Book, From Parallel Processing to the Internet of Things, by
Kai Hwang, Geoffrey C. Fox, Jack J. Dongarra, Morgan & Kaufmann, Publishers 2012
Source : References given in the presentation
Raj Kumar Buyya,. Bubendorfer (Eds.), Market Oriented Grid and Utility Computing, John Wiley & Sons, 2009.
VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

19

Distributed and Cloud Computing

Disparate nodes

HTC Systems

HPC Systems

High speed

File sharing
Distributed control

Homogenous nodes

P2P network

Clusters of MPPs

Centralized control

Disparate clusters

Geographically sparse

Computational and data grids

Service-oriented architecture (SOA)

RFID and sensors

Virtualization

Web 2.0 services

Internet of Things

Internet clouds

Evolutionary trend toward parallel distributed, and cloud computing with clusters,
MPPs, P2P network and grids and clouds, web services, and the Internet of Things.
Source : Distributed and Cloud Computing Book, From Parallel Processing to the Internet of Things, by
Kai Hwang, Geoffrey C. Fox, Jack J. Dongarra, Morgan & Kaufmann, Publishers 2012
Source : References given in the presentation
VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

20

Extreme-Scale Computing Challenges

HPC Accelerators
Low Power
Consumption
GPU
(Nvidia, AMD)

FPGA

MIC
(Xilinx, Altera) (Intel Xeon Phi)

Effort to port and optimize code for a accelerator type

Speed versus Efforts : Speedup over CPU vs Time to develop

for a particular accelerator type
Application Topologies /Network

Courtesy & Source : Reference : Intel, AMD, NVIDIA, Xlinx, Altera;

VCV.Rao. C-DAC, Pune

David Bader (References - 176-179)

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

21

Accelerator Comp.: Prog. Env. : Energy Efficiency

Shared Address Space Programming (Offload, Native, Host)
OpenMP, Intel TBB, Cilk Plus, Pthreads

Shared Address Space Programming (Only Offload)

OpenMP , Intel TBB, Pthreads - CUDA

Message Passing Programming (MPI)

(Offload MIC Offload /Host Offload)
MPI - (Symmetric & Co-processor /Host)

Message Passing Programming (MPI)

(Offload MIC Offload /Host Offload)
MPI - (Symmetric & Co-processor /Host)

Hybrid Programming
(MPI OpenMP, MPI Cilk Plus
MPI-Intel TBB, MPI-Pthreads)

Application

Host

Coprocessor

Application

Host

Coprocessor

Hybrid Programming (MPI CUDA)

Source : References & http://www.intel.com/; http://www.nvidia.com, http://www.adm.com, http://www.cdac.in

VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

22

Applications

Software Threading

Multi-to-May-Core Systems UMA & NUMA Systems

Hybrid Computing

Prog.API - Multi-Core Systems with Devices

Very-high
level

Computational
Science

Data
Informatics

Information
Technology

Efficient, Deterministic, Declarative, Restrictive

Expressiveness based language

High level

Parallel Prog. Languages (OpenMP, PGAS, Intel TBB,

Cilk Plus, OpenACC, CUDA)

Low-level

Low-level APIs (MPI, Pthreads, OpenCL, Verilog)

Very
low-level

Machine code, Assembly

Heterogeneous
Hardware
Coprocessors
GPUs
FPGAs
Source : NVIDIA, AMD, SGI, Intel, IBM Alter, Xilinux & References
VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

23

Distributed System Models & Enabling Technologies

Cloud Computing over the Internet

Paid services

Hardware

Software

Internet cloud
User

Storage

Network

Submit requests
Service

Virtualized resources from data centres to form a Internet cloud,

provisioned with hardware, software, storage, network, and
services for paid users to run their applications
Source : Distributed and Cloud Computing Book, From Parallel Processing to the Internet of Things, by
Kai Hwang, Geoffrey C. Fox, Jack J. Dongarra, Morgan & Kaufmann, Publishers 2012
Source : References given in the presentation
VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

24

Distributed System Models & Enabling Technologies

Cloud Computing over the Internet

Features of Cloud & Grid Platforms
Cloud Capabilities & Platform
Features
Traditional Features Common to
Grids and Clouds
Data Features and Databases
Programming and Runtime Support

Paid
services

User
Submit

Cloud Platforms Capabilities

requests
Should offer cost-effective utility
computing with the elasticity to scaleup and down in power
Commercial Clouds offer different
capabilities
Commonly termed as Platform as a
Service (PaaS) Ex : Azure
Commonly termed as Infrastructure
as a service (Iaas) Ex : Amazon

Hardware

Software

Internet cloud
Storage

Network
Service

Source : Distributed and Cloud Computing Book, From Parallel Processing to the Internet of Things, by
Kai Hwang, Geoffrey C. Fox, Jack J. Dongarra, Morgan & Kaufmann, Publishers 2012
Source : References given in the presentation
VCV.Rao. C-DAC, Pune
National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014
25

Cloud Computing and Service Models

Public, Private and Hybrid Clouds

A typical public cloud

Data center

Cloud service
queues

Server cluster (VMs)

Platform fronted
(web service API)
To users or other
public cloud over the
Internet

Cloud storage

Amazon
AWS

Microsoft
Azure

IBM Blue
Cloud

The Internet
A hybrid cloud

Private cloud
(IBM RC2)

Public

Salesforce
Force.com

Google App
engine

...
An
Intranet

Cloud users

Public, private and hybrid clouds illustrated by functional architecture and

connectivity of representative clouds
Source : Distributed and Cloud Computing Book, From Parallel Processing to the Internet of Things, by
Kai Hwang, Geoffrey C. Fox, Jack J. Dongarra, Morgan & Kaufmann, Publishers 2012
Source : References given in the presentation
VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

26

Cloud Computing and Service Models

Data-Center Networking Structure
Internet

Internet
BR

Data Center
Layer 3

BR

AR

AR

LB

LB

AR

AR

Layer 2

Key:
BR = L3 border router
AR = L3 access router
S = L2 switch
LB = Load balancer
A = Rack of servers

A single layer 2 domain

Standard data-center networking for the cloud to access the Internet.
Source : Distributed and Cloud Computing Book, From Parallel Processing to the Internet of Things, by
Kai Hwang, Geoffrey C. Fox, Jack J. Dongarra, Morgan & Kaufmann, Publishers 2012
VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

27

Cloud Programming & Software Environment

Parallel and Distributed Programming Paradigms
Parallel Computing and Programming Paradigms
MapReduce, Twister and Iterative MapReduce
Hadoop Library from Apache
Dryad and DryadLINQ from Microsoft
Sawzail and Pig Latin High-Level Languages
Mapping Apps. to Parallel & Distributed Systems

Monitor & Manage

Resources

Programming on Amazon AWS and Micro Azure

Programming on Amazon EC2
Amazon Simple Storage Service (S3)
Amazon Elastic Block Store (EBS) and SimpleDB
Microsoft Azure Programming Support

HPC/HTC System
Infrastructure

Access
Services

Service Consumers

Cloud
Administrator

IT Cloud

Component
Library

Programming Support Google App Engine

Programming on Google App Engine
Google File System (GFS)
BigTable , Googles NOSQL System
Chubby, Googles Distributed Lock Service

Component Vendors /
Software Publishers

Publish & Update

Components

Source : Distributed and Cloud Computing Book, From Parallel Processing to the Internet of Things, by
Kai Hwang, Geoffrey C. Fox, Jack J. Dongarra, Morgan & Kaufmann, Publishers 2012
Source : References given in the presentation
VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

28

Distributed System Models & Virtualization

Technologies for Network-based systems

Virtual Networking and Virtualization Middleware

Virtual Machines (VMs) - X86 Architecture Guest OS
Between the VMs and host-platform, one need to deploy
a middleware layer called a virtual machine monitor
(VMM)
A native VM installed with the use of a VMM called a
hypervisor in privileged node.
VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

29

Cloud Computing : Storage Virtualization

Virtual
Servers

Virtual
Storage

Logical
Representation

Virtual
Network

Virtual
Applications
Middleware

Virtual
Clients

Storage Virtualization is
Technology that makes one set of resources look and feel
like another set of resources, preferably with more
desirable characteristics
A logical representation of resources not constrained by
Physical limitations

Virtualization

- Hides some of the complexity

- Adds or integrates new function with existing services
- Can be nested or applied to multiple layers of a system

Physical Resources

Programming APIs Integrating with Schedulers & Resource Utilization

Threading on Multi-core CPUs & GPUs

Source : Distributed and Cloud Computing Book, From Parallel Processing to the Internet of Things,
By Kai Hwang, Geoffrey C. Fox, Jack J. Dongarra, Morgan & Kaufmann, Publishers 2012
Source : References given in the presentation
VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

30

Cloud Storage and Virtualization

Storage
Cloud storage is of two types : Object based & Block based
Block storage is like a raw disk space on which file system can be
created.
For both the storage types, backend can be anything like local attached
disks, nas, das, iscsi based or fiber channel based storage boxes.
In Cloud the use of these storage :
Mostly Object storage is used for holding Virtual Machine/Virtual
Instances templates & ISOs
This type can also be used to store the app/user related files.
Block storage serves the persistent disk space for Virtual
Machines/Virtual Instances.

The cloud giants, amazon also provides these options. S3

(Simple Storage Service; Object based storage) and elastic storage (block
based storage).
Source : Distributed and Cloud Computing Book, From Parallel Processing to the Internet of Things,
By Kai Hwang, Geoffrey C. Fox, Jack J. Dongarra, Morgan & Kaufmann, Publishers 2012
Source : References given in the presentation
VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

31

Cloud Storage and Virtualization

Storage
Cloud storage is of two types : Object based & Block based
Block storage is like a raw disk space on which file system can be created.
For both the storage types, backend can be anything like local attached disks, nas, das,
iscsi based or fiber channel based storage boxes.

In Cloud the use of these storage :

Mostly Object storage is used for holding Virtual Machine/Virtual Instances
templates & ISOs
This type can also be used to store the app/user related files.
Block storage serves the persistent disk space for Virtual Machines/Virtual
Instances.

The cloud giants, amazon also provides these options. S3

(Simple Storage Service; Object based storage) and elastic storage (block
based storage).
HPC Storage - Out-of-Core Algorithms Berkeley DB, Mango NO SQL,
HDF5.0 NetCDF
Source : Distributed and Cloud Computing Book, From Parallel Processing to the Internet of Things,
By Kai Hwang, Geoffrey C. Fox, Jack J. Dongarra, Morgan & Kaufmann, Publishers 2012
Source : References given in the presentation
VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

32

Cloud Storage and Virtualization

Storage : The opensource cloud middle-wares (OpenStack &

eucalyptus) For Openstack based implementation, refer these URLS:
High level overview of Object based (Swift) and block based
storage (Cinder)
http://www.openstack.org/software/openstack-storage/ [www.openstack.org]

Refer Storage Concepts in this url:

http://docs.openstack.org/admin-guide-cloud/content//ch_gettin>

cal-architecture [docs.openstack.org]
Swift (object based storage) concepts
http://docs.openstack.org/developer/swift/overview_architecture.html

[docs.openstack.org]
Info on cinder (block based storage) concepts & APIs
http://docs.openstack.org/developer/cinder [docs.openstack.org]

Service; Object based storage) and elastic storage (block based

storage).

Source : Distributed and Cloud Computing Book, From Parallel Processing to the Internet of Things,
By Kai Hwang, Geoffrey C. Fox, Jack J. Dongarra, Morgan & Kaufmann, Publishers 2012
Source : References given in the presentation
VCV.Rao. C-DAC, Pune
National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014
33

Cloud Storage and Virtualization

Storage : Eucalyptus cloud based implementation, refer these urls :

Overview of object based storage (Walrus) in Eucalyptus clouds
: http://www.eucalyptus.com/blog/2013/03/18/eucalyptus-walrus-storageconsiderations [http://www.eucalyptus.com]

few slides on block storage (elastic block storage) in eucalyptus

cloud
: http://www.slideshare.net/eucalyptus/eucalyptus-elastic-block-storage-ebsin-devtest-scenarios
[www.slideshare.net]

APIs & documentation

: https://github.com/eucalyptus/eucalyptus/wiki/Storage [github.com]

Storage types in Eucalyptus

: http://www.eucalyptus.com/blog/2012/10/03/cloud-storage-types
[www.eucalyptus.com]
Source : Distributed and Cloud Computing Book, From Parallel Processing to the Internet of Things,
By Kai Hwang, Geoffrey C. Fox, Jack J. Dongarra, Morgan & Kaufmann, Publishers 2012
Source : References given in the presentation
VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

34

Programing on Cloud & Distributed Computing

Parallel and Distributed Programming Models & Tool Sets.
Model

Description

PVM

PVM is a software package that permits a heterogeneous collection

of serial, parallel and vector computers which are connected to a
network to appear as one large computing resource

MPI

A library of subprograms that can be called from C or FORTRAN

to write parallel program running on distributed computer systems

MapReduce

A web programming model for scalable data processing on large

clusters over large data sets, or in web search operations

Hadoop

A software library to write and run large user applications on vast

sets in business applications (http://hadoop .apache.org/core)

VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

35

Programing on Cloud & Distributed Computing

Parallel and Distributed Programming Models & Tool Sets.
Model

Description

Features

PVM

PVM is a software package that

permits a heterogeneous collection
of serial, parallel and vector
computers which are connected to a
network to appear as one large
computing resource

Parallel Virtual Machine (PVM) uses the message

passing model to allow programmers to exploit
distributed computing across a wide variety of
computer types, including MPPs..PVM transparently
handles all message routing, data conversion, and
task scheduling across a network of incompatible
computer architectures.

MPI

A library of subprograms that can be

called from C or FORTRAN to write
parallel program running on
distributed computer systems

Specify synchronous or asynchronous point-to-point

and collective communication commands and I/O
operations in user programs for message-passing
execution

MapReduce

A web programming model for

scalable data processing on large
clusters over large data sets, or in
web search operations

Map function generates a set of intermediate

key/value pairs; Reduce function merges all
intermediate values with same key

Hadoop

A software library to write and run

large user applications on vast sets
in business applications
(http://hadoop .apache.org/core)

A scalable, economical, efficient, and reliable tool for

providing users with easy access of commercial
clusters

VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

36

Cloud Prog. & Software Environments

Parallel and Distributed Programming Paradigms
Motivation :Simplicity of writing parallel programs in an
important metric for parallel and distributed programming
paradigms. Motivations behind parallel and distributed
programming models are :
1. To improve productivity of programmers
2. To decrease programs time to market
3. To leverage underlying resources more efficiently
4. To increase system throughput, and
5. To Support higher levels of abstraction
Source : Distributed and Cloud Computing Book, From Parallel Processing to the Internet of Things, by
Kai Hwang, Geoffrey C. Fox, Jack J. Dongarra, Morgan & Kaufmann, Publishers 2012
Source : References given in the presentation
VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

37

Cloud Prog. & Software Environments

Parallel and Distributed Programming Paradigms
MapReduce : The software framework abstracts the data
flow of running a parallel program on a distributed
computing system by providing users with two interfaces
in the form of two interfaces : Map and Reduce
Users can override these two functions to interact with
and manipulate the data flow of running their program.
In MapReduce framework, the value part of the data
(key, value), is the actual data, and the key part is only
used by the MapReduce controller to control the data flow.
Source : Distributed and Cloud Computing Book, From Parallel Processing to the Internet of Things, by
Kai Hwang, Geoffrey C. Fox, Jack J. Dongarra, Morgan & Kaufmann, Publishers 2012
Source : References given in the presentation
VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

38

Cloud Prog. & Software Environments

Parallel and Distributed Programming Paradigms

Definition of MapReduce : The MapReduce software

framework provides an abstraction layer with the data flow
and flow of control to users, and hide the implementation
of all data flow steps such as
data partitioning
mapping
Synchronization
Communication

Scheduling
Source : Distributed and Cloud Computing Book, From Parallel Processing to the Internet of Things, by
Kai Hwang, Geoffrey C. Fox, Jack J. Dongarra, Morgan & Kaufmann, Publishers 2012
Source : References given in the presentation
VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

39

Cloud Prog. & Software Environments

Parallel and Distributed Programming Paradigms

Recently promoted proposed Parallel and distributed

Programming Paradigms
1. MapReduce

2. Hadoop
3. Dryad

Used for Information Retrieval Applications

Traditional Parallel Computing Models : MPI
Source : Distributed and Cloud Computing Book, From Parallel Processing to the Internet of Things, by
Kai Hwang, Geoffrey C. Fox, Jack J. Dongarra, Morgan & Kaufmann, Publishers 2012
Source : References given in the presentation
VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

40

Programing on Cloud and

Distributed Computing
Programming Enviornment

VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

41

A long history of research efforts in message passing

P4
Chameleon
Parmacs
TCGMSG
CHIMP
NX (Intel i860, Paragon)
PVM (Got PVM Now ! -Good for Distributed Computing)
MPI (Got MPI now !. - Good for Large Multi-Processing)

Evaluate the needs of your application then choose (PVM or MPI)

MPI is a standard, has a steeper learning curve and doesnt have a standard way
to start tasks. MPICH does have an mpirun command
If building a new scalable, production code, should use MPI (widely supported
now)
Source : MPI References

VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

42

Message Passing Architecture (MPI) Model

Message-Passing Programming Paradigm : Processors are connected
using a message passing interconnection network.

COMMUNICATION
NETWORK

P
M

On most Parallel Systems, the processes involved in the execution of a

parallel program are identified by a sequence of non-negative integers.
If there are p processes executing a program, they will have ranks 0,
1,2, ., p-1.
Source : MPI References
VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

43

The Minimal Set of MPI Routines

The MPI library contains over 125 routines
But fully functional message-passing programs can be written
using only the following 6 MPI routines
MPI_Init

Initializes MPI

MPI_Finalize

Terminates MPI

MPI_Comm_size

Determines the number of processes

MPI_Comm_rank

Determines the label of the calling process

MPI_Send

Sends a message

MPI_Recv

Receives a message

All 6 functions return MPI_SUCCESS upon successful completion,

otherwise return an implementation-defined error code

All MPI routines, data-types and constants are prefixed by MPI_

All of them are defined in mpi.h ( for C/C++)
VCV.Rao. C-DAC, Pune

Source : MPI References

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

44

Starting and Terminating the MPI Library

1.#include <mpi.h>
2.main(int argc, char *argv[ ] )

3. {
4. MPI_Init(&argc, &argv );
5. // do some work
6. MPI_Finalize( );
7. }
Both MPI_Init and MPI_Finalize must be called by all
processes

Command line should be processed only after MPI_Init

No MPI function may be called after MPI_Finalize
VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

45

An Overview Message Passing : Parallel Virtual Machine (PVM)

Parallel Virtual Machine (PVM) uses the message

passing model to allow programmers to exploit
distributed computing across a wide variety of computer
types, including MPPs.
When a programmer wishes to exploit a collection of
networked computers, they may have to contend with
several different types of heterogeneity:
architecture
data format
computational speed
machine load and
network load.
Source : PVM References
VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

46

An Overview Message Passing : Parallel Virtual Machine (PVM)

PVM in Nutshell
Each host (could be an MPP or SMP) runs a PVMD
A collection of PVMDs define a virtual machine
Once configured, tasks can be started (spawned), killed,
signaled from a console
Basic message passing
Performance is OK, but API Semantics limit optimizations
PVM Strengths : interoperability; fault tolerance;
heterogeneity; resource control; dynamic model
Source : PVM References
VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

47

Cloud Prog. & Software Environments

Parallel and Distributed Programming Paradigms
MapReduce Actual Data and Control Flow :
The Main Responsibility of the MapReduce Framework :
Partitioning function
Map worker
Partitioning
function

Partitioning
function

Partitioning
function

Partitioning
function

Regions

Reduce worker

Use of MapReduce partitioning function to link the Map and Reduce workers.
Source : Distributed and Cloud Computing Book, From Parallel Processing to the Internet of Things,
By Kai Hwang, Geoffrey C. Fox, Jack J. Dongarra, Morgan & Kaufmann, Publishers 2012
Source : References given in the presentation
VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

48

Cloud Prog. & Software Environments

Parallel and Distributed Programming Paradigms
MapReduce, Twister, and Iterative MapReduce
User interfaces

Input
Files

Map
Reduce
Function Flow of data Function

Output
Files

Abstraction
layer

Control flow
MapReduce
software
framework

MapReduce Library
Controller

MapReduce framework. Input data flows through the Map and Reduce functions
to generate the output result under the control flow using MapReduce software
library. Special user interfaces are used to access the Map and Reduce resources.
Source : Distributed and Cloud Computing Book, From Parallel Processing to the Internet of Things, by
Kai Hwang, Geoffrey C. Fox, Jack J. Dongarra, Morgan & Kaufmann, Publishers 2012
Source : References given in the presentation
VCV.Rao. C-DAC, Pune
National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014
49

Cloud Prog. & Software Environments : Parallel and Distributed Programming Paradigms
Map worker
Input split

Map worker
Input split

Map worker
Input split

K3:v k5:v

K1:v k2:v k1:v k3:v K1:v k2:v k3:v k2:v

K3:v

K3:v k5:v

K1:v

Map function

M
K4:v k4:v K3:v
k4:v

Sort and group

Sort and group

Sort and group

K1:v k2:v k3:v

K1:v k2:v k3:v

K3:v k4:v

Partitioning function
R1
R2
k3:v
k5:v k1:v

Partitioning function
R1
R2
k4:v k3:v

Partitioning function
R1

R2
k2:v k3:v

Data flow implementation

of many functions in the
Map workers and in the
Reduce workers through
multiple sequences of
partitioning, combining,
synchronization and
communication, sorting
and grouping, and reduce
operations.

k1:v

Input file
partitioning

Combiner

Partitioning

Synchronization
communication
R1

R2
k2:v k3:v k3:v k4:v k3:v

Sort and group

k2:v k3:v k3:v k4:v k3:v
R

k1:v k5:v k1:v

Sort and group

Sorting and grouping

k1:v, v, v k5:v
R

Output file

Output file

Reduce worker

Reduce worker

Reduce
Output

Source : Distributed and Cloud Computing Book, From Parallel Processing to the Internet of Things,
By Kai Hwang, Geoffrey C. Fox, Jack J. Dongarra, Morgan & Kaufmann, Publishers 2012
Source : References given in the presentation
VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

50

Cloud Prog. & Software Environments

Parallel and Distributed Programming Paradigms
MapReduce Actual Data and Control Flow :
User program
(2) Fork

Controller
(2) Fork

(2) Fork

User program
Master
(3) Assign
map task

(3) Assign
reduce task

Worker

User program

Split 2

(4) Read

(5) Read
User program

(9) Communications

Worker
(12) Write
(11) Reduce

Worker
User program

Split 3

File 1

Output
files

User program

Worker

File 1

Split 1

Input
files

User program
Worker

Control flow implementation of the MapReduce functionalities in Map workers and Reduce workers
(running user programs) from input files to the output files under the control of the master user program.
Source : Distributed and Cloud Computing Book, From Parallel Processing to the Internet of Things,
By Kai Hwang, Geoffrey C. Fox, Jack J. Dongarra, Morgan & Kaufmann, Publishers 2012
Source : References given in the presentation
VCV.Rao. C-DAC, Pune
National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014
51

Cloud Prog. & Software Environments

Parallel and Distributed Programming Paradigms
MapReduce Actual Data and Control Flow :
The Main Responsibility of the MapReduce Framework :
Summary of the following steps :
How to describe the map workers. ?

Reading the Input data (data distribution)

Each Map worker reads its corresponding portion of the
input data, namely the input data split and sends it to its
Map function.
Although a map worker may run more than one Map
function, which means it has been assigned more than one
input data split, each worker is usually assigned one inout
split only
VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

52

Cloud Prog. & Software Environments

Parallel and Distributed Programming Paradigms
MapReduce Logical Dataflow
The input data to the Map function is in the form of a
(key, value) pair.
The output data from the Map function is structured as
key, value) pairs called intermediate (key value) pairs.
The user-defined Map function processes each input
(key, value) pair and produces a number of (zeros, one
or more) intermediate (key, value) pairs.
The goal is to process all input (key, value) pairs to the
Map functions in parallel.
Source : Distributed and Cloud Computing Book, From Parallel Processing to the Internet of Things, by
Kai Hwang, Geoffrey C. Fox, Jack J. Dongarra, Morgan & Kaufmann, Publishers 2012
Source : References given in the presentation
VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

53

Cloud Prog. & Software Environments

Parallel and Distributed Programming Paradigms
MapReduce Logical Dataflow
The Reduce function receives the intermediate (key, value)
pairs in the form of a group of intermediate values associated
with one intermediate key (key, [set of values] ).
The MapReduce framework forms these groups by first
sorting the intermediate (key, value) pairs and then grouping
values with the same key.

Data is sorted to simplify the grouping process.

The Reduce function processes each (key, [set of
values] ) group and produces a set of (key, value) pairs as
output.
VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

54

Cloud Prog. & Software Environments

Parallel and Distributed Programming Paradigms
MapReduce Actual Data and Control Flow :
The Main Responsibility of the MapReduce Framework : Run a users
program on a distributed computing system. Therefore, the MapReduce
framework meticulously handles all

data partitioning
mapping
Synchronization

Communication and
Scheduling
Details of such data flows are required to address overheads and
performance issues from scalability point of view.
Source : Distributed and Cloud Computing Book, From Parallel Processing to the Internet of Things,
By Kai Hwang, Geoffrey C. Fox, Jack J. Dongarra, Morgan & Kaufmann, Publishers 2012
Source : References given in the presentation
VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

55

Cloud Prog. & Software Environments

Parallel and Distributed Programming Paradigms
MapReduce Actual Data and Control Flow :
The Main Responsibility of the MapReduce Framework :
Summary of the following steps :
1. Data Partitioning : The MapReduce library splits the input
data (files) already stored in GFS into M pieces that also
correspond to the number of map tasks.

Source : Distributed and Cloud Computing Book, From Parallel Processing to the Internet of Things,
By Kai Hwang, Geoffrey C. Fox, Jack J. Dongarra, Morgan & Kaufmann, Publishers 2012
Source : References given in the presentation
VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

56

Cloud Prog. & Software Environments

Parallel and Distributed Programming Paradigms
MapReduce Actual Data and Control Flow :
The Main Responsibility of the MapReduce Framework :
Summary of the following steps :
2. Computation Partitioning : This is implicitly handled (in
the MapReduce framework) by obliging users to write
their programs in the form of the Map and Reduce
functions. Therefore, the MapReduce library only
generates copies of a user program (e.g., by a fork system
call) containing the Map and Reduce functions, distributes
them, and starts them up on a number of available
computation engines.
Source : Distributed and Cloud Computing Book, From Parallel Processing to the Internet of Things,
By Kai Hwang, Geoffrey C. Fox, Jack J. Dongarra, Morgan & Kaufmann, Publishers 2012
Source : References given in the presentation
VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

57

Cloud Prog. & Software Environments

Parallel and Distributed Programming Paradigms
MapReduce Actual Data and Control Flow :
The Main Responsibility of the MapReduce Framework :
Summary of the following steps :
3. Determining the master and workers : The MapReduce
architecture is based on a master-worker model.
Therefore, one of the copies of the user program
becomes the master and the rest become workers. The
master picks idle workers, and assigns the Map and
Reduce tasks to them.
Source : Distributed and Cloud Computing Book, From Parallel Processing to the Internet of Things,
By Kai Hwang, Geoffrey C. Fox, Jack J. Dongarra, Morgan & Kaufmann, Publishers 2012
Source : References given in the presentation
VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

58

Cloud Prog. & Software Environments

Parallel and Distributed Programming Paradigms
MapReduce Actual Data and Control Flow :
The Main Responsibility of the MapReduce Framework :
Summary of the following steps :
3. Determining the master and workers : The MapReduce
architecture is based on a master-worker model.
What is Compute Engine ?
A map/reduce worker is typically a computation engine such
as a cluster node to run map/reduce tasks by executing
Map/Reduce functions.
Source : Distributed and Cloud Computing Book, From Parallel Processing to the Internet of Things,
By Kai Hwang, Geoffrey C. Fox, Jack J. Dongarra, Morgan & Kaufmann, Publishers 2012
Source : References given in the presentation
VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

59

Programing on Cloud and

Distributed Computing
Application Challenges

VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

60

Massive Data Computing - BIG DATA ANALYTICS

Distributed Commuting - Prog. Paradigms

Where are Opportunities ?

Application-oriented Opportunities Through
Artificial Intelligence Algorithms
High Performance Comp. for massive graphs
Streaming
Visualization of large Data Sets
Heterogeneous Systems under CLOUD Comp.
Combined use of Many-core (PARAM YUVA-II) (A
combination of Co-processors & GPU Accelerators)
Power-aware Computing Energy Efficient
VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

61

Graphs are pervasive in large-scale data analysis

Graphs are spared through-out in large-scale data analysis

Source of massive data : Petascale simulations, internet, scientific applications

Challenges : Data Size, heterogeneity, uncertainty, data quality.: Discover
emergent communities, model spread of Information

Astro-Physics
Problem : Outlier Detection of
Observations
Challenges: Massive Data Sets
Variation - timewise
Graph Problems: Data Mining
Alg. Association & Clusters;
Data Storage handling

Bio-Informatics
Problem : Drug Design
/Protein/Seq. Analysis
Challenges: Massive Data
Sets Data Heterogeneity
Graph Problems: Data
Mining Alg. Clustering; Fast
Query Searching Alg.

Bio-Informatics
Problem : Discover emergent
communities, Real-time Information
spread & Knowledge recovery
/Challenges: Massive Data Sets Data
Heterogeneity New analytics
Graph Problems: Data Mining Alg.
Clustering Shortest Path, Small /Large
Queries; Data Movement Irregular
Access

Source : References & David Bader s talk on Massive Scale Graph Analytics in HIPC-2012 India by George Bader
:Georgia Tech College of Computing
VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

62

Social Networks Analysis & Pattern Recognition

Massive Data Computing - BIG DATA ANALYTICS
Application: Data Mining Large Scale Weather Simulation
Ocean and Land Temperature (Jan 1982)

Incremental Improvements :
Few Cores to Many Cores
Research Goals:

NPP
Pressure
Precipitation

Find global climate patterns of

interest to Earth Scientists

Parallel Data Mining

Global snapshots of values for a

number of variables on land surfaces
or water.

NPP
Pressure

Precipitation

SST

SST

Latitude

grid cell

Longitude

Data Mining

Time

zone

A key interest is finding

connections between the
ocean and the land. Large
Computing Power

Source - Reference : Vipin Kumar; Discovery of Patterns in the Global Climate System using Data Mining -

VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

63

Many Core- Killer Apps of Tomorrow Parallel Alg. Design

Computer
Vision
Face
Body
Tracking Detection

(Financial)
Analytics

Physical
Simulation

Rendering
Global
Illumination

CFD

Face,
Cloth

Rigid
Body

Portfolio
Mgmt

Data Mining

Option
Pricing

Machine Cluster/ Text

learning Classify Index

Media
Synthesis

SVM
Classification

SVM
Training

PDE

Collision
detection

Level Set

Particle
Filtering

Filter/
transform

Fast Marching
Method

LCP

NLP

FIMI

IPM
(LP, QP)

K-Means

Monte Carlo

Index
Bench

Source : Intel
Krylov Iterative
Solvers (PCG)

Direct Solver
(Cholesky)

Basic matrix primitives

(dense/sparse, structured/unstructured)
VCV.Rao. C-DAC, Pune

Basic Iterative Solver

(Jacobi, GS, SOR)

Non-Convex
Method

Basic geometry primitives

(partitioning structures, primitive tests)

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

64

Massive Data Computing - BIG DATA ANALYTICS

Distributed Commuting - Prog. Paradigms

Where are Opportunities ?

Opportunity 1 : Use of Massive Graph
Computations in High Performance Computing
Interactions: Vertices (No.s); Edges (Type of
Interactions); Time Wise variation; Non re-use
Non-Locality of Data : Distributed cache based systems
Massive Threads based models may be required
Data Movement Storage Energy Efficiency
A Focused research is required to design and implement new
class of algorithms based on Graph Computations

Source : References & David Bader s talk on Massive Scale Graph Analytics in HIPC-2012 India by George Bader
:Georgia Tech College of Computing
VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

65

Massive Data Computing - BIG DATA ANALYTICS

Distributed Commuting - Prog. Paradigms

Opportunity 1 : Use of Massive Graph

Computations in High Performance Computing
O(Billion) vertices, O(Trillion) edges,
1 Million updates/sec
Challenge
Maintain Analytics, update quickly
Connected Components
Agglomerative Clustering
Problem :
Irregular execution
Bandwidth and latency bound
Irregular memory access
Low-to-no reuse

Source : References & David Bader s talk on Massive Scale Graph Analytics in HIPC-2012 India by George Bader
:Georgia Tech College of Computing
VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

66

Massive Data Computing - BIG DATA ANALYTICS

Distributed Commuting - Prog. Paradigms

Where are Opportunities ?

Opportunity 2 : Use of Graph Data Bases
Efficient Data Structures Few Billion vertices of
Graph Face Book- contains nodes and relationships;
Nodes contain properties (key-value pairs);
Relationships are named, directed and always have a
start and end node; Relationships can also contain
properties
Data Movement Storage Energy Efficiency
A Focused research is required to implement Graph based
Computations (FOCUS on GRAPH DATABASES)

Source : References & David Bader s talk on Massive Scale Graph Analytics in HIPC-2012 India by George Bader
:Georgia Tech College of Computing

VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

67

Modeling Data with Graphs

Going Beyond Social Networks
Data Instance
Graphs are suitable for
capturing arbitrary relations
between the various
elements.

Graph Instance

Element
Elements Attributes
Relation Between
Two Elements

Vertex
Vertex Label
Edge

Type Of Relation

Edge Label

Relation between
a Set of Elements

Hyper Edge

Provide enormous flexibility for modeling the underlying data as they allow
the modeler to decide on what the elements should be and the type of
relations to be modeled
Graph Partitioning Software METICS /ParMETIS /HMETIS : http://www.cs.umn.edu/~karypis

VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

68

Modeling Data with Graphs

Graph Classification

Approach

Graphs with multi-dimensional labels

Stream graphs
phone-network connections
Hypergraphs
compact representation of set relations
Benchmarks and real-life test cases!

Graph
Databases

Discover Frequent
Sub-graphs

Transform Graphs
in Feature
Representation

Select Discriminating
Features

Learn a Classification
Model

Source : References & Nishith-Pathak, Analyzing Information Flow in Social Networks: Communities, Topics, Cognition and Influence, Doctor
of Philosophy Thesis, Department of Computer Science, University of Minnesota, Minneapolis, March 2012
Graph Partitioning Software METICS /ParMETIS /HMETIS : http://www.cs.umn.edu/~karypis

VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

69

Massive Data Computing - BIG DATA ANALYTICS

Distributed Commuting - Prog. Paradigms

Lake Superior: Coarse Mesh

Lake Superior: Mesh

Partitioning (by METIS) Lake Superior: Refined Mesh
Source : Graph Partitioning Software METICS /ParMETIS /HMETIS : http://www.cs.umn.edu/~karypis
Graph Partitioning Software METICS /ParMETIS /HMETIS : http://www.cs.umn.edu/~kumar

VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

70

Massive Data Computing - BIG DATA ANALYTICS

Distributed Commuting - Prog. Paradigms
Approaches for obtaining Coarsening graphs
The process of finding a maximal matching and contracting the
graph to obtain the next level coarser graph is important.

Coarsening graphs are obtained using maximal matchings Heavy edge

matching works very well. Significantly reduces the edge-weight left in the
graph. It requires very little refinement effort.
Source : Graph Partitioning Software METICS /ParMETIS /HMETIS : http://www.cs.umn.edu/~karypis
Graph Partitioning Software METICS /ParMETIS /HMETIS : http://www.cs.umn.edu/~kumar

VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

71

Modeling Data with Graphs

HyperGraph Computations
Multi-level graph Partitioning Algorithms

(b) Groups Determined by Ede-Coarsening

Various Ways of matching the vertices in the hypergraph connecting they induce
Source : Graph Partitioning Software METICS /ParMETIS /HMETIS : http://www.cs.umn.edu/~karypis
Graph Partitioning Software METICS /ParMETIS /HMETIS : http://www.cs.umn.edu/~kumar

VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

72

Massive Data Computing - BIG DATA ANALYTICS

Distributed Commuting - Prog. Paradigms

Where are Opportunities ?

Opportunity 3: Quick Visualization Data
New tech. for Massive Graph Data
Scientific Computing Billion of Vertices with multiple
weights proportional to Floating Point Data
Graph Analytics for Discrete Data Million to few
Billion Vertices
Use od Data Mining & Artificial Intelligence Alg.
Context-based or Situation-aware based Computing Coprocessors & Accelerators

Source : References & David Bader s talk on Massive Scale Graph Analytics in HIPC-2012 India by George Bader
:Georgia Tech College of Computing

VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

73

Massive Data Computing - BIG DATA ANALYTICS

Distributed Commuting - Prog. Paradigms

Where are Opportunities ?

Opportunity 4: Combination of using

Heterogeneous Cloud Computing framework with
Many Core Systems
Solving Data analysis
Combination of Grid Computing & Cloud Computing
Combination of Hadoop & Parallel Programming for
Massively Multi-threading for SNA & BIG DATA
New Algorithms for Storage APIs for Data Science

Source : References & David Bader s talk on Massive Scale Graph Analytics in HIPC-2012 India by George Bader
:Georgia Tech College of Computing
VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

74

Massive Data Computing - BIG DATA ANALYTICS

Distributed Commuting - Prog. Paradigms

Where are Opportunities ?

Opportunity 5: Energy Efficiency Computing
Power aware Computing Methodologies (Calculate Power in
Mill-watts and Performance of Application Kernels)
Consumption of Power-in-Milliwatts : Understand Floating
Point or Memory Intensive Computing
Consumption of Power in Milliwatts: Movement of Data
Across different Levels of Storage
New Algorithms for less power consumption for SNA and
BIG-DATA

Source : References & David Bader s talk on Massive Scale Graph Analytics in HIPC-2012 India by George Bader
:Georgia Tech College of Computing

VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

75

Massive Data Computing - BIG DATA ANALYTICS

Distributed Commuting - Prog. Paradigms

To work on each Opportunity

What is Required ?
Graph Algorithms - Large Scale Data Bases Graph - Data Mining Association & Clustering
Use Graph Data Bases (Irregular Computations
/Irregular Memory Access)
BIG Data Analytics Scientific Comp./ NonScientific Comp.) Data Parallelism Approach

Source : References & David Bader s talk on Massive Scale Graph Analytics in HIPC-2012 India by George Bader
:Georgia Tech College of Computing
VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

76

Massive Data Computing - BIG DATA ANALYTICS

Distributed Commuting - Prog. Paradigms

To work on each Opportunity

What is Required ?
Storage Data Access in real time & Data
Management - Unstructured /Irregular Data
Storage HPC Storage (HDF5, Berkeley DB,
PLFS, netCDF, MPI-IO, SWIFT)
Hadoop implementation of MapReduce

Source : References & David Bader s talk on Massive Scale Graph Analytics in HIPC-2012 India by George Bader
:Georgia Tech College of Computing
VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

77

Cloud and Distributed System Technologies

Conclusions
An Overview of Distributed Computing and Cloud
Computing
Programming on Cloud and Distributed Computing
Systems
Opportunities for Applications on Disturbed Computing

VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

78

References
1.
2.

3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.

Distributed and Cloud Computing Book, From Parallel Processing to the Internet of Things, by Kai
Hwang, Geoffrey C. Fox, Jack J. Dongarra, Morgann & Kaufmann, Publishers 2012
Amazon EC2 and S3, Elastic Compute Cloud (EC2) and Simple Scalable Strorage (S3),
http://en.wikipedia.org/wiki/Amazon_Elastic_Compute_Cloud and
http://spatten_presentations.s3.amazonaws.com/s3-pm-rails.pdf
F. Berman, G. Fox, T. Hey (Eds.), Grid Computing, Wiley, 2003.
K. Birman, Reliable Distributed Systems: Technologies, Web Services, and Applications, SpringerVerlag, 2005.
G. Boss, et al., Cloud Computing-The BlueCloud Project.
www.ibm.com/developerworks/webspehere/zones/hipods/, October 2007.
J. Dongarra, et al. (Eds.), Source Book of Parallel Computing, Morgan Kaufmann, San Francisco, 2003.
V. K. Garg, Elements of Distributed Computing, Wiley-IEEE Press, 2002.
K. Hwang, Advanced Computer Architecture: Parallelsim, Scalability, Programming, McGraw-Hill,
1993.
K. Hwang, Z. Xu, Scalable Parallel Computing, McGraw-Hill, 1998.
NVIDIA Corp. Kepler: NVIDIAs Next-Generation CUDA Compute Architecture, White paper, 2013.
I. Taylor, From P2P to Web Sertives and grids, Springer-Verlag, London 2005.
Twister, Open Source Software for Iterative MapReduce, http://www.iterativemapreduce.org/.
Wikipedia, Internet of Things, http://en.wikipedia.org/wiki/Internet_of_Things, June 2010.
Wikipedia, CUDA, http://en.wikipedia.org/wiki/CUDA, March 2011.
Wikipedia. TOP500, http://en.wikipedia.org/wiki/TOP500, February 2011.

VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

79

References
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.

29.

D. Bader, R. Pennington, Cluster computing applications, Int. J. High Perform. Comput. (May) (2001).
R. Buyya (Ed.), High-Performance Cluster Computing. Vols. 1 and 2, Prentice Hall, New Jersey, 1999.
J. Dongarra, Survey of present and future supercomputer architectures and their interconnects, in:
International Supercomputer Conference, Heidelberg, Germany 2004.
Wikipedia, CUDA. http://en.wikipedia.org/wiki/CUDA, 2011, (accessed 19.02.2011).
R. Buyya, J. Broberg, A. Goscinski (Eds.), Cloud Computing; Principles and Paradigms, Wiley Press,
New York, 2011.
K. Hwang, D. Li, Trusted cloud computing with secure resources and data coloring, IEEE Internet
Comput., (September/October) (2010) 30-39.
D. Meyer, et. al., Parallax: Virtual disks for virtual machines, in: Proceedings of EuroSys, 2008.
L. Shri, H. Chen, J. Sun, vCUDA: GPU accelerated high performance computing in virtual machines, in:
Proceedings of the IEEE International Symposium on Parallel and Distributed Processing, 2009.
J. Smith, R. Nair, The architecture of virtual machines, IEEE Comput., (May) (2005).
J. Smith, R. Nair, Virtual Machines: Versatile Platforms for Systems and Processes, Morgan Kaufmann,
2006.
R. Ublig, et al. Intel virtualization technology, IEEE Comput., (May) (2005).
VMware (white paper). Understanding Full Virtualization, Paravirtualization, and Hardware Assist,
www.vmware.com/files/pdf/VMware_paravirtualization.pdf.
J. Walters, et al., A Comparison of virtualization technologies for HPC, in: Proceedings of Advanced
Information Networking and Applications (AINA), 2008.
K. Aberer, Z. Depotovic, Managing trust in a peer-to-peer information system, in: ACM CIKM
International Conference on Information and Knowledge Management, 2001.
Amazon EC2 and S3, Elastic Compute Cloud (EC2) and Simple Scalable Storage (S3).
http://spatten_presentations.s3amazonaws.com/s3-on-rails.pdf

VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

80

References
30.
31.
32.
33.

34.
35.
36.

37.
38.
39.
40.
41.
42.

M. Armbrust, A. Fox, R. Griffifth, et al., Above the Clouds: A Berkeley View of Cloud Computing, Technical
Report No. UCB/EECS-2009-28, University of California at Berkeley, 10 February 2009
I. Arutyun, et al., Open circus: a globalcloud computing testbed, IEEE Comput. Mag. (2010) 35-43.
G. Boss, P. Mllladi, et al., Cloud Computing: the bluecloud project.www.ibn.com/developerworks/
websphere/zones/hipods/, 2007.
R. Buyya, CS Yeo, S. Venugopal, Market-oriented cloud computing: vision, hype, and reality for delivering IT
services as a computing utilities, in: Proceedings of the 10th IEEE International Conference on High
Performance Computing and Communications (HPCC), Dalian, China, 25-27 September 2008.
I. Foster, The grid: computing without bounds, Sci. Am. 288 (4) (2003) 78-85.
V. Jinesh, Cloud Architectures, White paper, Amazon. http://aws.amazon.com/about-aws/whatsnew/2008/07/16/cloud-architectures-white-paper/.
W. Norman, M. Paton, T. de Aragao, et al., Optimizing utility in cloud computing through autonomous
workload execution, in: Bulletin of the IEEE Computer Society Technical Committee on Data Engineering,
2009.
S. Roschke, F. Cheng, C. Meinel, Intrusion detection in the cloud, in: IEEE International Conference on
Dependable. Autonomic, and Secure Computing (DASC 09), 13 December 2009.
J. Rittinghouse, J. Ransome, Cloud Computing: Implementation, Management, and Security, CRC Publishers
2010.
Salesforce.com, http://en/wikipedia.org/wikiSalesforce.com/,2010.
VMware, Inc., Migrating Virtual Machines with Zero Downtime, www.vmware.com/, 2010 (accessed 07).
Wikipedia Cloud computing, http://en.wikipedia.org/wiki/Cloud_computing, 2010 (accessed 26.01.10)
Wikipedia, Data center, http://en.wikipedia.org/wiki/Data_center, 2010 (accessed 26.01.10)

VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

81

References
43.
44.
45.

46.
47.
48.
49.
50.
51.
52.
53.

54.
55.
56.
57.

M. Hadley, Web Application Description Language (WADL), W3C Member Submission,

www.w3.org/Submission/wadl/, 2009 (accessed 18.10.10).
I. Foster, H. Kishimoto, A. Savva, et al., The open grid services architecture version 1.5, Open Grid Forum,
GFD.80, www.ogf.org/documents/GFD.80.pdf, 2006.
G. Fox, D. Gannon, A Survey of the Role and Use of Web Services and Service Oriented Architectures in
Scientific/Technical Grids, http://grids.ucs.indiana.edu/ptliupages/publications/ReviewofServices and
Workflow-IU-Aug2006B.pdf, 2006 (accessed 16.10.10)
Apache ActiveMQ open source messaging system, http://activemq-apache.org/.
NaradaBrokering open source content distribution infrastructure, ww.naradabrokering.org/.
Amazon Simple Queue Service (Amazon SQS), http://aws.amazon.com/sqs/.
Microsoft Azure Queues, http://msdn.microsoft.com/en-us/windowsazure/ff635854.aspx.
N. Wilins-Diehr, Special issue: Science gateways common community interfaces to grid resources Concurr.
Comput. Pract, Exper. 19 (6) (2007) 743-749.
D. Thain, T. Tannenbaum, M. Livny, Distributed computing in practice: the condor experience, Concurr, Pract.
Exper. 17 (2-4) (2005) 323-356.
Open Grid Computing Environments web site, www.collab-pgce.org (accessed 18.10.10).
E. Deelman, D. Gannon, M. Shields, I. Taylor, Workflows and e-Science: an overview of workflow system
features and capabilities, Future Generation Comp. Syst. 25 (5) (2009) 528-540, doi: http://dx.doi.org/
10.1016/j.future.2008.06.012
R. Allen, Workflow: An Introduction, Workflow Handbook. Workflow Management Coalition, 2001.
Kepler Open Xource Scientific Workflow System, http://kepler-project.org.
H. Gadgil, G. Fox, S. Pallickara, M. Pierce, Managing grid messaging middleware, in: Challenges of Large
Applications in Distributed Engironments (CLADE), 2006, pp. 83-91.
Condor home page, www.cs.wisc.edu/condor/.

VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

82

References
58.
59.
60.
61.
62.
63.
64.
65.
66.
67.
68.
69.

70.
71.
72.
73.

Microsoft, Project Trident: A Scientific Workflow Workbench, http//research.microsoft.com/enus/collaboration/tools/trident.aspx, 2010.

Zend PHP Company, The Simple Cloud API for Storage, Queues and Table, http://www.simplecloug.org/
home, 2010.
NOSQL Movement, Wikipedia list of resources, http://en.wikipedia.org/wiki/NosQL, 2010.
NoSQL Link Archive, LIST OF NOSQL DATABASES, http://nosql-database.org/, 2010.
Amazon, Welcome to Amazon SimpleDB, http://docs.amazonwebservices.com/AmazonSimpleDB/latest/
DevelperGuide/index.html, 2010.
Apache, The CouchDB document-oriented database project, http://couchdb.apache.org/, 2010.
Pig! Platform for analyzing large data sets, http://hadoop.apache.org/pig/, 2010.
A. Grama, G Karypis, V. Kumar, A. Gupta, Introduction to Parallel Computing, second ed., Addison Wesley,
2003.
S. Hariri, M. Parashar, Tools and Environments for Parallel and Distributed Computing, Series on Parallel and
Distributed Computing, Wiley, 2004, ISBN: 978-0471332886.
L. Silva, Rajkumar. Buyya, Parallel Programming Models and paradigms, (2007).
G. Fox, MPI and MapReduce, in: Clusters, Clouds, and Grids for Scientific Computing CCGSC, Flat Rock, NC,
http://grids.ucs.indiana.edu/ptliupages/presentations/CCGSC-Sept8-2010.pptx, 8 September 2010.
J. Ekanayake, X. Qiu T. Gunarathne, S. Beason, G. Fox, High Performance Parallel Computing with Clouds and
Cloud Technologies, Cloud Computing and Software Services: Theory and Technologies, CRC Press (Taylor
and Francis), 2010.
Wikipedia, MapReduce, http://en.wikipedia.org/wiki/MapReduce, 2010 (accessed 06.11.10).
R. Lammel, Googles MapReduce programming model Revisited, Sci. Comput. Prog. 68 (3) (2007) 208-237.
S. Ghemawat, H. Gobioff, S. Leung, The Google File System, in: 19th ACM Symposium on Operating Systems
Principles, 2003, pp.20-43.
Google, Introduction to Parallel Programming and MapReduce, http://code.ggogle.com/edu/parallel/
mapreduce-tutorial,html.2010.
VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

83

References
74.
75.
76.
77.
78.

79.
80.
81.
82.
83.
84.
85.
86.
87.
88.

89.

90.

SALSA Group, Iterative MapReduce, http://www,iterativemapreduce.org/, 2010.

T. White, Hadoop: The Definitive Guide, Second ed., Yahoo Press, 2010.
Apache, HDFS Overview, http://hadoop.apache.org/hdfs/, 2010.
Apache! Pig! (part of Hadoop), http://pig.apache.org/hdfs/, 2010.
G. C. Fox, A. Ho, E, Chan, W. Wang, Measured characteristics of distributed and cloud computing
infrastructure for message-based collaboration applications, in: Proceedings of the 2009 International
Symposium on Collaborative Technologies and Systems, IEEE Computer Society, 2009, pp 465-467.
Eucalyptus LLC, White Papers. http://www.eucalyptus.com/whitepapers.
Nimbus, Cloud computing for science, http://www.nimbusproject.org, 2010.
Nimbus, Frequently Asked Questions, http://www.nimbusproject.org/docs/current/faq.html, 2010.
Amazon, Amazon Elastic Compute Cloud (Amazon EC2). http://aws.amazon.com/ec2.
Sector/Sphere, High Performance Distributed File System and Parallel Data Processing Engine.
http://sector.sourceforge.net
Open Stack, Open Source, Open Standards Cloud, http://openstack.org/index.php, 2010.
SALSA Group, Catalog of Cloud Material, http://salsaphc.indiana.edu/content/cloud-materials, 2010.
Microsoft Research, Cloud Futures Workshop, http://research.microsoft.com/en-us/events/
cloudfutures2010/default.aspx, 2010.
P. Chaganti, Cloud computing with Amazon Web Services, Part 1: Introduction When its smarter to rent
than to buy, http://www.ibm.com/developerworks/architecture/library/ar-cloudaes1/, 2008.
Cloud computing with Amazon Web Services, Part 2: Storage in the cloud with Amazon Simple Storage
Service (S3) Reliable, flexible, and inexpensive storage and retrieval of your data, http://www.ibm.com/
developerworks/architecture/library/ar-cloudaws2/, 2008.
P. Changanti, Cloud computing with Amazon Web Services, Part 3: Servers on demand with EC2, http://
www.ibm.com/developerworks/architecture/library/ar-cloudaws3/, 2008.
G. Fox, S. Bae, J. Ekanayake, X. Qiu, H. Yuan, Parallel Data Mining from Multicore to Cloudy Grids, book
chapter of High Speed and Large Scale Scientific Computing, IOS Press, Amsterdam, 2009
http://grids.ucs.indiana.edu/ptilupages/publications.CetratoWriteupJune11-09. pdf.
VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

84

References
91.

W. Allcock, J. Bresnahan, R. Kettimuthu, et al., The globus stripped gridFTP framework and server, in:
Proceedings of the ACM/IEEE Conference on Supercomputing, 2005.
92. R. Aydt, D. Gunter, W. Smith, et al., A Grid Monioring Architecture, Global Grid Forum Performance
Working Group, 2002.
93. Raj Kumar Buyya,. Bubendorfer (Eds.), Market Oriented Grid and Utility Computing, John Wiley &
Sons, 2009.
94. Dongarra, I. Fister, G. Fox, et al., Sourcebook of Parallel Computing, Morgan Kaufman Publishers,
2002.
95. L. Ferreira, et al., Grid Computing in Research and Education,
(http://www.redbooks.ibm.com/abstracts/sg246895.html?Open )
96. L. Ferreira, et al., Grid Computing in Research and Education,
(http://www.redbooks.ibm.com/abstracts/sg246649.html?Open )
97. I. Foster, C. Kesselman, S. Tuecke, The anatomy of the grid: enabling scalable virtual organizations, Int.
J High. Perform. Comput. Appl. 15 (3) (2001) 200.
98. I. Foster, Globus toolkit version 4: software for service-oriented systems, J. Comput. Sci. Technol, 21
(4) (2006) 513-520.
99. H. Jin. Challenges of grid computing, Advances in Web-Age Information Management. Lecture Notes
in Computer Science, 3739 (2005) 25-31.
100. I Taylor, From P2P to Web Services and Grids, Springer-Verlag, London, 2005.
101. D. Thain, T. Tannenbaum, M. Livny, Distributed computing in practice: the condor experience,
Concurrency. Comput. Pract. Exp., (2005) 323-356.
102. Wikipedia, Grid Computing. http://een.wikipedia.org/wiki/Grid_computing, (accessed 26.04.11)

VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

85

References
103. S. Androutsellis, D. Spoinellis, A survey of P2P content distribution technologies. ACM,. Comput. Surv.
(December) (2004).
104. J. Buford, H. Yu, E. Lua, P2P Networking and Applications, Morgan Kayfmann, December 2008. Also
www. P2pna.com.
105. X. Cheng, J. Liu, NetTube: exploring social networks for peer-to-peer short video sharing, in:
Proceedings of IEEE Inforcom, March 2009.
106. P. B. Godfrey, S. Shenker, I. Stoica, Minimizing churn in distributed systems. in Proceedings of ACM
SIGCOMM, 2006.
107. K. Ross, D. Rubenstein, Peer-to-peer systems in: IEEE Inforcom, Hong Kong, 2004, (Tutorial slides).
108. M. Ambrust, A. Fox R. Giffith, et al., Above the Clouds: A Berkeley View of Cloud Computing, Technical
Report No. UCB/CCES-2009-28, University of California at Berkeley, 10 February 2009.
109. J. Ekanayke, T. Gunarathne, J. Qiu, Cloud technologies for bioinformatics applications, in: IEEE
Transactions on Parallel and Distributed Systems, accepted to appear, http://grids.ucs.indiana.edu
/ptliupages/publications/BioCloud_TPDS_Journal_Jan4_2010.pdf, 2011)
110. S. Garfinkel, Commodity grid computing with Amazons S3 and EC2, Login 32 (1) (2007) 7-13
111. Grid 5000 and ALLADIN-G5K: An infrastructure distributed in 9 sites around France, for research in
large-scale parallel and distributed systems. http://www.grid5000.fr/mediawiki/index.php/Grid5000:
Home, (accessed 20.11.10).
112. R. Grossman, Y. Gu, M. Sabala, et al., The open cloud testbed: Supporting open source cloud
computing systems based on large scale high performance, in: A. Doulamis, et al., (Eds.),
DynamicNetwork Services, Springer, Berlin Heidelberg, 2010, pp. 89-97.
113. J. Rittinghouse, J. Ransome, Cloud Computing: Implementation, Management and Security, CRC
Publisher, 2010.

VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

86

References

114. H. Song, Exploring Facebook and Twitter Technologies for P2P Social Networking, in: EE 657 Final Project
Report, University of Sothern California, May 2010.
115. E. Walker, Benchmarking Amazon EC2 for high-performance scientific computing, Login 33 (5) (2008) page
18-23.
116. M. Weng, A Multimedia Social Networking Community for Mobile Devices, Tisch School of The Arts, New
York University, 2007.
117. M. Yigitbasi, A. losup, D. Epema, C-Meter: a framework for performance analysis of computing clouds, in:
International Workshop on Cloud Computing, May 2009.
118. PVM Parallel Virtual Machine A Users Guide and Tutorial For Networked Parallel Computing Al Geist,
Adam Beguelin, Jack Dongarra, Weicheng Jiang, Robert Mancheck, Vaidy Sunderam, The MIT Press
Cambrdge, Massachusetts, London England, 1994
119. pvmhome http://www.csm.ornl.gov/pvm/pvm_home.htmloo
120. David Browning, Embarrassingly Parallel Benchmark under PVM, Computer Science Corp., NASA Ames
Research Labs.
121. J.Dongara et al PVM- experiences, current status and Future Directions, ORNL- Tenessee.
122. Building Linux Clusters O-Reilly
123. A Performance Comparison of DSM, PVM, and MPI, Paul Werstein, Mark Pethick, Zhiyi Huang, University
of Otago, Dunedin, New Zealand , 2003

VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

87

References
124. Andrews, Grogory R. (2000), Foundations of Multithreaded, Parallel, and Distributed
Programming, Boston, MA : Addison-Wesley
125. Butenhof, David R (1997), Programming with POSIX Threads , Boston, MA : Addison Wesley
Professional
126. Culler, David E., Jaswinder Pal Singh (1999),
Parallel Computer Architecture - A
Hardware/Software Approach , San Francsico, CA : Morgan Kaufmann
127. Grama Ananth, Anshul Gupts, George Karypis and Vipin Kumar (2003), Introduction to Parallel
computing, Boston, MA : Addison-Wesley
128. Intel Corporation, (2003), Intel Hyper-Threading Technology, Technical User's Guide, Santa Clara
CA : Intel Corporation Available at : http://www.intel.com
129. Shameem Akhter, Jason Roberts (April 2006), Multi-Core Programming - Increasing Performance
through Software Multi-threading , Intel PRESS, Intel Corporation,
130. Bradford Nichols, Dick Buttlar and Jacqueline Proulx Farrell (1996), Pthread Programming O'Reilly
and Associates, Newton, MA 02164,
131. James Reinders, Intel Threading Building Blocks (2007) , OREILLY series
132. Laurence T Yang & Minyi Guo (Editors), (2006) High Performance Computing - Paradigm and
Infrastructure Wiley Series on Parallel and Distributed computing, Albert Y. Zomaya, Series Editor
133. Intel Threading Methodology ; Principles and Practices Version 2.0 copy right (March 2003), Intel
Corporation
134. William Gropp, Ewing Lusk, Rajeev Thakur (1999), Using MPI-2, Advanced Features of the
Message-Passing Interface, The MIT Press..
135. Pacheco S. Peter, (1992), Parallel Programming with MPI, , University of Sanfrancisco, Morgan
Kaufman Publishers, Inc., Sanfrancisco, California
136. Kai Hwang, Zhiwei Xu, (1998),
Scalable Parallel Computing (Technology Architecture
Programming), McGraw Hill New York.
137. Michael J. Quinn (2004), Parallel Programming in C with MPI and OpenMP McGraw-Hill
International Editions, Computer Science Series, McGraw-Hill, Inc. Newyork
138. Andrews, Grogory R. (2000),
Foundations of Multithreaded, Parallel, and Distributed
Progrmaming, Boston, MA : Addison-Wesley

VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

88

References
139. SunSoft. Solaris multithreaded programming guide. SunSoft Press, Mountainview, CA, (1996),
Zomaya, editor. Parallel and Distributed Computing Handbook. McGraw-Hill,
140. Chandra, Rohit, Leonardo Dagum, Dave Kohr, Dror Maydan, Jeff McDonald, and Ramesh Menon,
(2001),Parallel Programming in OpenMP San Fracncisco Moraan Kaufmann
141. S.Kieriman, D.Shah, and B.Smaalders (1995), Programming with Threads, SunSoft Press,
Mountainview, CA. 1995
142. Mattson Tim, (2002), Nuts and Bolts of multi-threaded Programming Santa Clara, CA : Intel
Corporation, Available at : http://www.intel.com
143. I. Foster (1995, Designing and Building Parallel Programs ; Concepts and tools for Parallel
Software Engineering, Addison-Wesley (1995
144. J.Dongarra, I.S. Duff, D. Sorensen, and H.V.Vorst (1999), Numerical Linear Algebra for High
Performance Computers (Software, Environments, Tools) SIAM, 1999
145. OpenMP C and C++ Application Program Interface, Version 1.0". (1998), OpenMP Architecture
Review Board. October 1998
146. D. A. Lewine. Posix Programmer's Guide: (1991), Writing Portable Unix Programs with the Posix.
1 Standard. O'Reilly & Associates, 1991
147. Emery D. Berger, Kathryn S McKinley, Robert D Blumofe, Paul R.Wilson, Hoard : A Scalable
Memory Allocator for Multi-threaded Applications ; The Ninth International Conference on
Architectural Support for Programming Languages and Operating Systems (ASPLOS-IX).
Cambridge, MA, November (2000). Web site URL : http://www.hoard.org/
148. Marc Snir, Steve Otto, Steyen Huss-Lederman, David Walker and Jack Dongarra, (1998) MPI-The
Complete Reference: Volume 1, The MPI Core, second edition [MCMPI-07].
149. William Gropp, Steven Huss-Lederman, Andrew Lumsdaine, Ewing Lusk, Bill Nitzberg, William
Saphir, and Marc Snir (1998) MPI-The Complete Reference: Volume 2, The MPI-2 Extensions
150. A. Zomaya, editor. Parallel and Distributed Computing Handbook. McGraw-Hill, (1996)
151. OpenMP C and C++ Application Program Interface, Version 2.5 (May 2005), From the OpenMP
web site, URL : http://www.openmp.org/
152. Stokes, Jon 2002 Introduction to Multithreading, Super-threading and Hyper threading Ars
Technica, October (2002)
153. http://www.cdac.in/opecg2009/

VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

89

References
154.Stokes, Jon 2002 Introduction to Multithreading, Super-threading and Hyper threading Ars Technica,
October (2002)
155.Andrews Gregory R. 2000, Foundations of Multi-threaded, Parallel and Distributed Programming,
Boston MA : Addison Wesley (2000)
156.Deborah T. Marr , Frank Binns, David L. Hill, Glenn Hinton, David A Koufaty, J . Alan Miller, Michael
Upton, Hyperthreading, Technology Architecture and Microarchitecture, Intel (2000-01)
157.http://www.erc.msstate.edu/mpi/
158.http://www.arc.unm.edu/workshop/mpi/mpi.html
159.http://www.mcs.anl.gov/mpi/mpich
160.The MPI home page, with links to specifications for MPI-1 and MPI-2 standards :
http://www.mpiforum.org
161.Hybrid Programming Working Group Proposals, Argonne National Laboratory, Chiacago (2007-2008)
162.TRAC Link : https://svn.mpi-forum.org/trac/mpi-form-web/wiki/MPI3Hybrid
163.Threads and MPI Software, Intel Software Products and Services 2008 - 2009
164.Sun MPI 3.0 Guide November 2007
165.Treating threads as MPI processes thru Registration/deregistration Intel Software Products and
Services 2008 - 2009
166.Intel MPI library 3.2 - http://www.hearne.com.au/products/Intelcluster/edition/mpi/663/
167. http://www.erc.msstate.edu/mpi/
168. http://www.arc.unm.edu/workshop/mpi/mpi.html
169. http://www.mcs.anl.gov/mpi/mpich
167. The MPI home page, with links to specifications for MPI-1 and MPI-2 standards :
http://www.mpiforum.org
168. Hybrid Programming Working Group Proposals, Argonne National Laboratory, Chiacago (2007-2008)
169. TRAC Link : https://svn.mpi-forum.org/trac/mpi-form-web/wiki/MPI3Hybrid
170. Threads and MPI Software, Intel Software Products and Services 2008 - 2009
171. Sun MPI 3.0 Guide November 2007

VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

90

References
172. Treating threads as MPI processes thru Registration/deregistration Intel Software Products and
Services 2008 - 2009
173. Intel MPI library 3.2 - http://www.hearne.com.au/products/Intelcluster/edition/mpi/663/
174. http://www.cdac.in/opecg2009/
175. PGI Compilers http://www.pgi.com
176. David A. Bader, Massive-Scale Graph Analytics, Georgia Tech College of Computing, Department of
Computational Science and Engineering, HiPC-2012, Pune, India
177. D.A. Bader, G. Cong, and J. Feo, On the Architectural Requirements for Efficient Execution of Graph
Algorithms, The 34th International Conference on Parallel Processing (ICPP 2005), pp. 547-556,
Georg Sverdrups House, University of Oslo, Norway, June 14-17, 2005.
178. D.A. Bader and K. Madduri, Design and Implementation of the HPCS Graph Analysis Benchmark on
Symmetric Multiprocessors, The 12th International Conference on High Performance Computing
(HiPC 2005), D.A. Bader et al., (eds.), Springer-Verlag LNCS 3769, 465-476, Goa, India, December
2005.
179. Kamesh Madduri, David A. Bader, Jonathan W. Berry, Joseph R. Crobak, and Bruce A. Hendrickson,
Multithreaded Algorithms for Processing Massive Graphs, in D.A. Bader, editor, Petascale
Computing: Algorithms and Applications, Chapman & Hall / CRC Press, Chapter 12, 2007.
180. Cloud Applications Architectures Building Applications and Infrastructure in the Cloud George
Reese O-REILLY June 2011
181. Cloud Computing Bible Barrie Sosinsky, Wiley India Ltd, 2011
182. Cloud Computing Explained John Rhoton, British Library Cataloguing in Publication Data, 2009
183. Distributed and Cloud Computing Book, From Parallel Processing to the Internet of Things, by Kai
Hwang, Geoffrey C. Fox, Jack J. Dongarra, Morgann & Kaufmann, Publishers 2012
180. Cloud Computing Infrastructure http://www.ibm.com
181. Cloud Computing Virtualization Made Easy http://www.cisco.com/
182. Cloud Computing Infrastructure http://www.softlyaer.com
183. Microsoft Cloud OS : http://www.microsoft.com
184. Amazon EC2 Cloud Discover Windows Azure Storage http://www.windowsazure.com

VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

91

References
185.
186.
187.
188.
189.
190.
191.
192.
193.
194.
195.

196.
197.
198.

199.
200.
201.
202.
203.
204.
205.

Programming Windows Azure http://www.amazon.com

App Engine Google Cloud Platform https://cloud.google.com/appengine
Google App Engine Google Developers; https://developers.google.com/appengine/
Oracle NoSQL Database Architecture & Implementation http://www.oracle.com/NOSQLDatabase
NoSQL - MangoDB www.mongodb.com
Cloud Computing Meghdoot : http://www.cdac.in
High level overview of Object based (Swift) and block based storage http://www.openstack.org/software/openstack-storage/
Storage types in Eucalyptus : http://www.eucalyptus.com/
Graph Partitioning Software METICS /ParMETIS /HMETIS : http://www.cs.umn.edu/~karypis
Graph Partitioning Software METICS /ParMETIS /HMETIS http://www.cs.umn.edu/~kumar
Jie Tang, Jimeng Sun, Chi Wang, and Zi Yang. 2009. Social influence analysis in large-scale networks. In Proceedings of the
15th ACM SIGKDD international conference on Knowledge discovery and data mining (KDD 09). ACM, New York, NY, USA,
807-816
Nishith-Pathak, Analyzing Information Flow in Social Networks: Communities, Topics, Cognition and Influence, Doctor of
Philosophy Thesis, Department of Computer Science, University of Minnesota, Minneapolis, March 2012
Prof. Jaideep Srivastava ; http:/www.cs.umn.edu/~jaideep
G.Karypis, V. Kumar, and Vipin Kumar, Multilevel k-way partitioning scheme for irregular graphs, Jorunal of Parallel and
Distributed Computing, 48:96-129, 1998
The Anatomy of the Grid Enabling Scalable Virtual Organizations by Ian Foster, Carl Kesselman & etc..
Foster, I. Internet Computing and the Emerging Grid. Nature Web Matters, 2000.
http://www.nature.com/nature/webmatters/grid/grid.html
Foster, I. and Karonis, N. A Grid-Enabled MPI: Message Passing in Heterogeneous Distributed Computing Systems. In Proc.
SC'98, 1998.
The Globus CoG Home Page. http://www.globus.org/cog.
Globus. http://www.globus.org.
The NASA Information Power Grid Home Page. http://www.ipg.nasa.gov.
TeraGrid; http://www.teragrid.org/

VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

92

Thank You
Any Questions ?

VCV.Rao. C-DAC, Pune

National Workshop- BIDA-2014 at CRRAO AIMSCS @ UoH August 22-24, 2014

93