Data Science

Lecture # 3
 Step 1 – Acquiring Data

• By the end of this discussion, you will be able to:

• List techniques and technologies to access and retrieve the data you
need
• Describe an example scenario that accesses data from a variety of
sources using different technologies

Note: All Images are taken from edx.org

Step 1 – Acquiring Data

3
 Where’s the Data?
• Identify suitable data related to problem
• Acquire all available data
• Leaving a small amount of data can lead to incorrect conclusions
• Data comes
• From many places i.e. Local & Remote
• In many varieties i.e. Structured & Unstructured
• In many different velocities i.e. Streaming speed of data

4
Where’s the Data?

5
Where’s the Data?

• A lot of data exists in

relational databases like
structured data coming
from organizations
• SQL is used to access
data

6
Where’s the Data?

• Data can also exist in

files such as text files
and Excel spreadsheets
• Scripting languages are
used to get data from
files like Python, VBA,
JavaScript, Perl, PHP, R,
Octa, MATLAB etc

7
Where’s the Data?

• An increasingly popular
way to get data is from
websites
• Common formats are
XML, JSON etc.
• Many websites host web
services to access their
data e.g. REST &
WebSocket

8
 Where’s the Data?

• REST stands for Representational State Transfer and it is an

approach for implementing webs services with performance,
scalability and maintainability in mind

• WebSocket services allow realtime notifications from the

websites

9
Where’s the Data?

• NoSQL storage systems

are increasingly used to
manage variety of data
• Examples are Cassandra,
mongoDB and HBASE
• They provide APIs to
allow users to access the
data

10
A Real Example

11
Summary

12
 Step 2A – Exploring Data

• By the end of this discussion, you will be able to:

• Explain the importance of exploring data

• Identify methods to perform preliminary analysis of your data

13
Step 2A – Exploring Data

14
 Step 2A – Exploring Data

• After getting data you might be tempted to immediately build

models to analyze the data
• We must resist this temptation
• Perform preliminary investigation to gain better understanding
of specific characteristics of data
• We’ll be looking for correlations, general trends, outliers
15
Temptation: The desire to do something, especially something wrong or unwise
Why Explore?
• Correlation
graphs explore
dependencies
between variables
• General trends
show how data is
progressing over
time
• Outliers show
data points that
are distant from
other data points
16
Why Explore?
• Summary statistics
provide numerical
values to describe data
• Mean & median are
measures of location of
specific values
• Mode is the value that
occurs most frequently
• Range and standard
deviation are measures
of spread in data
17
Visualize Data
• Heat map show hot spots
• Histogram show data
distribution (unusual
dispersion)
• Boxplots also show data
distribution
• Lines graphs show
change of value over time
• Scatter plots show
correlation between two
variables 18
 Step 2B – Pre-Processing Data

• By the end of this discussion, you will be able to:

• Identify some problems with real world data

• Describe what is needed to transform raw data to data that can be used
for analysis

19
 Step 2B – Pre-Processing Data

Clean: To address data quality issues

Transform: To make it suitable for analysis 20
 Real World Data is Messy!

• Inconsistent values: Customer with two different addresses

• Duplicate records: Customer recorded at two different locations
• Missing values: Missing customer age
• Invalid data: Invalid step code e.g. 6 digit zip code
• Outliers: Due to sensor failure values are much higher or lower
than expected for a period of time

Outliers: Things situated away or detached from main system 21

 Addressing Data Quality Issues
• Remove data with missing values
• Merge duplicate records
• Generate best estimate for invalid values
• Remove outliers

• To address these issues Domain Knowledge is required

• Keep record of changes you made
22
 Getting Data in Shape
• The second part is to manipulate the clean data into a format
needed for analysis called data manipulation, data pre-
processing, data wrangling or data munging
• Some operations in data munging
• Scaling
• Transformation
• Feature selection
• Dimensionality reduction
• Data manipulation
23
Scaling
• Scaling involves
changing range of
values such as from 0
to 1
• E.g. magnitude of
weight value is much
greater than
magnitude of height
value
• Scaling both values
between 0 and 1 will
equalize contributions
24
Transformation

• It reduces noise and

variability
• Aggregation is one type
of transformation
which results data in
less variability used in
long term analysis
• E.g. daily sales figures
transform into weekly
or monthly sales figures 25
Feature Selection

• It removes redundant
features, combining
features and creating
new features
• If two features are very
correlated, one can be
removed

26
Dimensionality Reduction
• It is useful when dataset
has large number of
dimensions
• It involves finding
smaller subset of
dimensions that
capture most of the
variation in the data
• E.g. principal
component analysis
27
Data Manipulation

• Raw data often has to

be manipulated to be in
the correct format for
the analysis
• It involves creating
groups and capturing
mean, range and
standard deviation for
each group

28
Summary
• Data preparation is very
important part of data
science process
• Here we spend most of
our time
• It can be tedious but is a
crucial step
• Don’t get good results if
we don’t put time and
effort, no matters how
sophisticated techniques
we use for analysis 29
 Step 3 – Analyze Data

• By the end of this discussion, you will be able to:

• Describe what is involved in applying an analysis technique to your data

• List three basic analysis techniques

30
Step 3 – Analyze Data

31
Step 3 – Analyze Data

32
 Categories of Analysis Techniques

• There are different types of problems so there are different types

of analysis techniques. The main techniques are
• Classification
• Regression
• Clustering
• Association analysis
• Graph analysis
33
Classification

• Goal is to predict the

category of input data
• An example is
predicting the weather
as sunny, rainy, windy
and cloudy
• Another example is to
identify handwritten
digits as being one of 10
categories i.e. 0 to 9 34
Regression

• When our model has to

predict a numeric value
then it becomes
regression problem
• An example is to
predict price of stock
over time
• Another example is to
estimate weekly sales
of a new product 35
Clustering

• The goal is to organize

similar items into
groups
• An example is to group
company’s customers
as seniors, teenagers
and adults
• Another example is to
determining different
weather groups like
rainy, cold or snowy 36
Association Analysis

• The goal is to find rules

to capture associations
between items or events
• Common example is
market basket analysis
to understand customer
purchasing behavior
• E.g. banking customer
with CD also interested
in other investments
• Diaper-bear example 37
Graph Analytics
• When data have lot of
entities and connections
like social networks, we
use graph analytics
• E.g. exploring the
spread of disease by
analyzing doctor’s
record
• Identification of security
threats by monitoring
social media, email etc 38
Modeling
• Modeling starts with
selecting one of these
techniques
• Construct the model
using prepared data
• To validate model, apply
it to new data samples
• Divide prepared data
into set of data for
constructing model and
reserve some for 39
evaluating the model
How to Evaluate Each Model?

• For classification and

regression we’ll have the
correct output for each
sample in our data
• Comparing the correct
output and predicted
output by the model
provides a way to
evaluate the model
40
How to Evaluate Each Model?
• The groups from
clustering should be
examined to see if they
make sense for our
application
• E.g. do the customer
segments reflect your
customer base?
• Are they helpful for use
in our targeted
marketing campaigns?
41
How to Evaluate Each Model?

• In this case some

investigations will be
needed to see if the
results are correct
• E.g. network traffic
delays needs to be
investigated to see if
what our model predicts
is actually happening?
42
Determine Next Steps

43
Summary

44
 Step 4 – Reporting Insights

• By the end of this discussion, you will be able to:

• Determine what to present in reporting your findings

• Identify techniques to communicate your results

45
Step 4 – Reporting Insights

46
What to Present?

• Look at results and

decide what to present
• Means determining
what part of analysis is
more important to our
company?
• Our findings determine
what the next step
should be
47
 What to Present?

• All findings must be presented so that informed decisions can

be made
• If your conclusions later found to be wrong your credibility
could be seriously damaged
• Better to tell a complete and true story, even if it isn’t very
clean, then to try finesse things and make them sound more
clear than they really are

48
How to Present?

• Visualization is an
important tool in
presenting results
• Scatter plots, line
graphs etc are
effective ways to
represent your
results visually
• We have tables
with details for
deeper analysis 49
Visualization Tools

50
 Step 5 – Turning Insights into Action

• By the end of this discussion, you will be able to:

• Explain what turning insights into action means

• Connect your results with your business question

51
Step 5 – Turning Insights into Action

52
Step 5 – Turning Insights into Action

• We bring together large

datasets to find
actionable insights to
help answer scientific or
commercial question

53
 Questions
• Business questions
• Is there something wrong in our process?
• Is there data that should be added to our application to make is more
accurate?
• Science questions
• Where the benefits from a drug trial statistically significant?
• What is the rate of deforestation? Can you predict how much forest will
remain in 15 years?

54
Implementation

• Now we’ve to figure out

how to implement the
actions
• How should it be
automated, if it can be?
• Stakeholders need to be
identified and get
involved in this change
55
Implementation

• We need to monitor and

measure the impact of
the action on the
process
• Be sure to think about
what data you should
collect during and after
the change to properly
evaluate its impact
56
Determine Next Steps

• Big data and data

science are only useful if
the insights can be
turned into actions and
actions should be
carefully defined and
evaluated

57