Dav LVC-2

GL Applied Data Science Program
adukejyme1990@gmail.com
DEWPNZAQ4Y Network Analysis
This file is meant for personal use by adukejyme1990@gmail.com only.

Sharing or publishing the contents in part or full is liable for legal action.
Caroline Uhler (MIT) Network Analysis Great Learning 1 / 25
Overview
Overview of this week / module:
Data collection and visualization for exploratory data analysis
Network analysis
Unsupervised learning - clustering

DEWPNZAQ4Y
Overview of this lecture:
Examples of networks and representing networks
Summary statistics of a network
Centrality measures - finding important nodes in a network

Network
A network (or graph) G is a collection of nodes (or vertices) V

connected by links (or edges) E . The network is denoted by G = (V , E ).
DEWPNZAQ4Y

Network
A network (or graph) G is a collection of nodes (or vertices) V

connected by links (or edges) E . The network is denoted by G = (V , E ).
Network research:
In recent years network research witnessed a big change:
DEWPNZAQ4Y
From study of a single graph on 10-100 nodes to the statistical
properties of large networks on millions of nodes
Characterize the structure of networks
Identify important nodes / edges in a network
Identify missing links in a network

Examples of networks
Network Vertex Edge

World Wide Web web page hyperlink
Internet computer network protocol
interaction
power grid generating station transmission line
adukejyme1990@gmail.com / substation
DEWPNZAQ4Y
friendship network person friendship
gene regulatory network gene regulatory effect
neural network neuron synapse
transportation airport direct flight
Netflix person / movie rating
Different kinds of networks
simple network: undirected network with at most one edge between
any pair of vertices and no self-loops
e.g. Internet, power grid, telephone network
multigraph: self-loops and multiple links between vertices possible

e.g. neural network, road network
directed network: i → j does not imply j → i

DEWPNZAQ4Y
e.g. World Wide Web, food web, citation network
weighted network: with edge weights or vertex attributes

e.g. transportation networks
bipartite network: edges between but not within classes

e.g. recommender systems such as Netflix
hypergraph: generalized ‘edges’ for interaction between > 2 nodes

e.g. protein-protein interaction network
Large networks look like hairballs
DEWPNZAQ4Y

Representation of a network
Two common representations of a network G = (V , E ):
adjacency list
undirected graph 1 − 2 − 3: E = {{1, 2}, {2, 3}}
directed graph 1 → 2 ← 3: E = {(1, 2), (3, 2)}
adjacency matrix of size n × n (where n = |V |) with
DEWPNZAQ4Y

1 if (i, j) ∈ E
Aij =
0 otherwise
For weighted graph, Aij can be non-binary
How does the adjacency matrix of an undirected graph look like? How to
count
Thisthe
filenumber
is meantoffor
friends or suggest
personal new friends in a social network?
use by adukejyme1990@gmail.com only.
Representation of a network
DEWPNZAQ4Y

Quantitative measures of networks
Some quantitative measures of networks to describe structural patterns of

a network and to compare networks:
connected components
DEWPNZAQ4Y
degree distribution
diameter and average path length
homophily or assortative mixing

Connected Components
Connected component: set of nodes that are reachable from one another
DEWPNZAQ4Y

Connected Components
Connected component: set of nodes that are reachable from one another
5
Many networks consist of one large component and many small ones
1e+06
Number of components
1e+04
DEWPNZAQ4Y
1e+02 1e+00
1e+00 1e+02 1e+04 1e+06 1e+08

Component size
Figure 3. Component size distribution. The fraction of components with a given component size
Component
onThis
a log-log issize
filescale.
meantdistribution
for personal
Most vertices
in the
use
(99.91%) are
2011 Facebook
bylargest
in the
network on a log-log
component. only.
scale. Most vertices (99.91%) are in the largest component.
Caroline[48].
algorithm UhlerWe
(MIT) Network
find that the average distance Analysis pairs of users was 4.7 for Great
between Learning
Facebook 10 / 25
users and
Degree distribution of the Internet
Degree of a node: number of edges connected to a node
DEWPNZAQ4Y

Degree distribution of the Internet
Degree of a node: number of edges connected to a node
Many networks show a power-law degree distribution (i.e., distribution
that is linear in log-log plot)
DEWPNZAQ4Y
Figures from Chapter 8 in “Networks: An Introduction” by

This Newman
M.E.J. file is meant for personal use by adukejyme1990@gmail.com only.
(2010)
Degree distribution of Facebook network
1e−01
1e−04 1e−03 1e−02 1e−01 1e+00

1e−03
Fraction
Fraction
1e−05
DEWPNZAQ4Y
Global Glo
1e−07
U.S. U.S
1 5 50 500 5000 1 5
Degree
(a)
This file is meant for personal
From “The Anatomy ofuse by adukejyme1990@gmail.com
the Facebook only.
Social Graph” by Ugander et al. (2011)
Figure
Caroline Uhler (MIT) 1. Degree Network
distribution
Analysis p . (a) The fraction of users with
Great Learning 12 / 25de
Diameter of a graph
Let dij denote the length of the geodesic path (or shortest path)
between node i and j
The diameter of a network is the largest distance between any two

nodes in the network:
adukejyme1990@gmail.com diameter = max dij
DEWPNZAQ4Y i,j∈V

Diameter of a graph
Let dij denote the length of the geodesic path (or shortest path)
between node i and j
The diameter of a network is the largest distance between any two

nodes in the network:
adukejyme1990@gmail.com diameter = max dij
DEWPNZAQ4Y i,j∈V
If network is not connected, one often computes the diameter in the

largest component.
Algorithms for finding shortest paths: breadth-first search for

unweighted graph, Dijkstra’s algorithm for weighted graphs

Small-world and 6 degrees of separation
Concept of 6 degrees of separation was made famous by sociologist

Stanley Milgram and his study “The Small World Problem” (1967)
In his experiment participants from a particular town were asked to

get a letter to a particular person in a different town by passing it
from acquaintance to acquaintance.
DEWPNZAQ4Y
18 out of 96 letters made it in an average of 5.9 steps, suggesting
that the diameter of the social network in the US is 6
Any reasons why we should take the conclusion of 6 degrees of

separation with a grain of salt?

Diameter of Facebook (2011)
Percentage of pairs within hop distance

1.0
Global
U.S.
0.6 0.8
0.4
DEWPNZAQ4Y
0.0 0.2
0 2 4 6 8 10
hop distance
This file is The

e 2. Diameter. meant for personal use by adukejyme1990@gmail.com only.
From “The Anatomy of the Facebook Social Graph” by Ugander et al. (2011)
neighborhood function N (h) showing the percentage of user pairs tha
Sharing
h hops of each or publishing
other. the contents
The average in part orusers
distance between full on
is liable for legal
Facebook action.
in May 2011 was 4
Homophily
Homophily (or assortative mixing): tendency of people to associate with
others that are similar
DEWPNZAQ4Y

From “The Anatomy of the Facebook Social Graph” by Ugander et al. (2011)
Find important nodes in a network
Centrality measure: A measure that captures importance of a node’s
position in the network; there are many different centrality measures:
DEWPNZAQ4Y

degree centrality
Simple and intuitive: individuals with more connections have more
influence and more access to information.
Does not capture “cascade of effects”: importance better captured by
having connections to important nodes
DEWPNZAQ4Y

degree centrality
DEWPNZAQ4Y
eigenvector centrality
score that is proportional to the sum of the score of all neighbors is
captured by largest eigenvector of adjacency matrix
builds the foundation for Google’s PageRank algorithm

degree centrality
DEWPNZAQ4Y
closeness centrality
tracks how close a node is to any other node

degree centrality
DEWPNZAQ4Y
closeness centrality
tracks how close a node is to any other node
betweenness centrality
measures the extent to which a node lies on paths between other nodes
Which centrality measure to use
Choice of centrality measure depends on application!
DEWPNZAQ4Y

Which centrality measure to use
Choice of centrality measure depends on application!
In a friendship network:
high degree centrality: most popular person

DEWPNZAQ4Y
high eigenvector centrality: most popular person that is friends with
popular people
high closeness centrality: person that could best inform the group
high betweenness centrality: person whose removal could best break

the network apart

Case study: CAVIAR (criminal network in Montreal)
Data based on 11 wiretap warrants from 1994-1996 → 11 periods
Mandate of CAVIAR project: Seize drugs, arrests only in period 11

11 seizures total with monetary losses for traffickers of $32 mio
phase 4: 1 seizure $ 2.5mio, 300kg of marijuana
phase 6: 3 seizures $ 1.3mio, 2 x 15kg of marijuana, 1 x 2 kg of
cocaine
DEWPNZAQ4Y
phase 8: 1 seizure $ 0.4mio, 9kg of cocaine
phase 9: 2 seizures $ 4.3mio, 2kg of cocaine + 1 x 500kg marijuana
phase 11: 2 seizures $ 1.3mio, 12kg of cocaine + 11kg of cocaine
Unique opportunity to study changes in the structure of a criminal network

This file isby
in upheaval meant
policeforforces
personal use by adukejyme1990@gmail.com only.
network consists of 110 (numbered) players: 1-82 are traffickers,
83-110 are non-traffickers (financial investors, accountants, owners of
various importation businesses, etc.)
initially, investigation targeted Daniel Serero, alleged mastermind of
drug network in downtown Montreal
initially marijuana
Project was imported to Canada from Morocco
Caviar
after first (The
DEWPNZAQ4Y seizurepolice bias
in phase 4, and the scope
traffickers of law-
reoriented to cocaine import
enforcement
from Colombia, transitingmonitoring)
through the United States
45
40
35
# de participants
30
25
20
15
10
5
0
1 2 3 4 5 6 7 8 9 10 11
This file is meant for personal use byPhase

adukejyme1990@gmail.com only.
Phase 3
Phase 6
DEWPNZAQ4Y
Phase 8 Phase 11

Role of the different actors:
Daniel Serero (node 1): mastermind of the network
Pierre Perlini (node 3): principal lieutenant of Serero (executes his
instructions)
Ernesto Morales (node 12): principal organizer of the cocaine import,
intermediary between the Colombians and the Serero organization
DEWPNZAQ4Y

Optional: Additional thoughts - Criminal networks
Given a social network and k criminal suspects, how to determine

other suspects?
Same question is extremely important in biology: given certain genes

that are known to cause a certain disease, determine other candidate
DEWPNZAQ4Y
genes (e.g. based on protein-protein interaction network for
determining autism genes: http://dx.doi.org/10.1101/057828)
How do we identify nodes that are “between” a given set of seed

nodes?

Optional: Steiner trees
Determine a small subnetwork that contains the given suspects / genes
and connects these nodes
Steiner tree:
shortest subnetwork that contains a given set of nodes
NP-complete problem
there exist polynomial time approximations
DEWPNZAQ4Y
⇒ use collection of approximate Steiner trees for further analysis:

autism interactome / criminal interactome
For genomics applications, see:
http://fraenkel-nsf.csbi.mit.edu/steinernet/tutorial.html
⇒ compute nodes with high betweenness centrality in interactome to

obtain candidate genes / suspects
References
Chapters 1 - 10 (but mostly chapters 6 - 8) in

M. E. J. Newman. Networks: An Introduction. 2010.
For an analysis of the Facebook network:

J. Ugander, B. Karrer, L. Backstrom and C. Marlow. The Anatomy of
DEWPNZAQ4Y
the Facebook Social Graph. 2011.
For more information on the CAVIAR network:

C. Morselli. Inside Criminal Networks (Springer, New York).
Chapter 6: Law-enforcement disruption of a drug-importation
network. 2009.

Dav LVC-2

Uploaded by

Copyright:

Available Formats

Dav LVC-2

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Dav LVC-2

Uploaded by

Copyright:

Available Formats

GL Applied Data Science Program

This file is meant for personal use by adukejyme1990@gmail.com only.

Overview of this week / module:

Data collection and visualization for exploratory data analysis

Unsupervised learning - clustering

Overview of this lecture:

Examples of networks and representing networks

Summary statistics of a network

Centrality measures - finding important nodes in a network

A network (or graph) G is a collection of nodes (or vertices) V

This file is meant for personal use by adukejyme1990@gmail.com only.

A network (or graph) G is a collection of nodes (or vertices) V

This file is meant for personal use by adukejyme1990@gmail.com only.

Network Vertex Edge

multigraph: self-loops and multiple links between vertices possible

directed network: i → j does not imply j → i

weighted network: with edge weights or vertex attributes

bipartite network: edges between but not within classes

hypergraph: generalized ‘edges’ for interaction between > 2 nodes

This file is meant for personal use by adukejyme1990@gmail.com only.

Two common representations of a network G = (V , E ):

For weighted graph, Aij can be non-binary

This file is meant for personal use by adukejyme1990@gmail.com only.

Some quantitative measures of networks to describe structural patterns of

diameter and average path length

homophily or assortative mixing

This file is meant for personal use by adukejyme1990@gmail.com only.

This file is meant for personal use by adukejyme1990@gmail.com only.

1e+00 1e+02 1e+04 1e+06 1e+08

This file is meant for personal use by adukejyme1990@gmail.com only.

Figures from Chapter 8 in “Networks: An Introduction” by

1e−04 1e−03 1e−02 1e−01 1e+00

The diameter of a network is the largest distance between any two

This file is meant for personal use by adukejyme1990@gmail.com only.

The diameter of a network is the largest distance between any two

If network is not connected, one often computes the diameter in the

Algorithms for finding shortest paths: breadth-first search for

This file is meant for personal use by adukejyme1990@gmail.com only.

Concept of 6 degrees of separation was made famous by sociologist

In his experiment participants from a particular town were asked to

Any reasons why we should take the conclusion of 6 degrees of

This file is meant for personal use by adukejyme1990@gmail.com only.

Percentage of pairs within hop distance

This file is The

This file is meant for personal use by adukejyme1990@gmail.com only.

This file is meant for personal use by adukejyme1990@gmail.com only.

This file is meant for personal use by adukejyme1990@gmail.com only.

This file is meant for personal use by adukejyme1990@gmail.com only.

This file is meant for personal use by adukejyme1990@gmail.com only.

Choice of centrality measure depends on application!

This file is meant for personal use by adukejyme1990@gmail.com only.

Choice of centrality measure depends on application!

high degree centrality: most popular person

high betweenness centrality: person whose removal could best break

This file is meant for personal use by adukejyme1990@gmail.com only.

Data based on 11 wiretap warrants from 1994-1996 → 11 periods

Mandate of CAVIAR project: Seize drugs, arrests only in period 11

Unique opportunity to study changes in the structure of a criminal network