Data Visualization in
Data Science
Maloy Manna
biguru.wordpress.com
linkedin.com/in/maloy
twitter.com/itsmaloy
Synopsis
Having data is not enough. Adding context to data is essential to understand the
data, find patterns and engage audiences. Data visualization is a key element of data
science, the interdisciplinary field which deals with finding insights from data.
In this webinar, we explore the roles of data visualization at different stages of
the data science process, and why it is essential.
We also look at how data is encoded visually with shape, size, color and other
variables and also the basic principles of visual encoding can be applied to build
better visualizations.
We cover narratives, types of bias and maps.
Finally we look at how various tools both open source and off-the-shelf
software thats used in data science to build effective data visualizations.
Speaker profile
Maloy Manna
Project Manager - Engineering
AXA Data Innovation Lab
Over 14 years experience building data driven products and services
Previous organizations: Thomson Reuters, Saama, Infosys, TCS
biguru.wordpress.com
linkedin.com/in/maloy
twitter.com/itsmaloy
Contents
Defining Data visualization
Data science process
Data visualization
Visual encoding of data
Narrative structures
Dataviz Technology & Tools
Defining Data visualization
Visual display of quantitative information
Mapping data to visual elements
Encoding data with size, shape, color...
Storytelling / narrative elements
Defining Data Visualization
Exploratory
Find insights
Conversation between data and you
Explanatory
Present insights
Data science project life-cycle
Acquire data
Prepare data
Analysis &
Modeling
Evaluation &
Interpretation
Deployment
Operations &
Optimization
Data science process
EDA:
Exploratory
Data Analysis
Data Wrangling
Exploratory
Explanatory
Data Visualization
Source: Computational Information Design | Ben Fry
Exploratory data visualization
Data analysis approaches:
Classical:
Problem > Data > Model > Analysis > Conclusions
EDA: [Exploratory Data Analysis]
Problem > Data > Analysis > Model > Conclusions
Bayesian:
Problem > Data > Model > Prior distribution > Analysis > Conclusions
EDA = approach, not a set of techniques
Exploratory data visualization
Statistical approaches:
Quantitative
Hypothesis testing
Analysis of variance (ANOVA)
Point estimates and confidence intervals
Least squares regression
Graphical
Scatter plots
Histograms
Probability plots
Residual plots
Box plots
Block plots
Exploratory data visualization
Graphical
Scatter plots
Histograms
Probability plots
Residual plots
Box plots
Block plots
Exploratory data visualization
Graphical analysis procedures:
Testing assumptions
Model selection
Model validation
Estimator selection
Relationship identification
Factor effect determination
Outlier detection
MUST USE for deriving insights from data
Exploratory data analysis
Anscombe's quartet
N=11
Mean of X = 9.0
Mean of Y = 7.5
Intercept = 3
Slope = 0.5
Residual standard deviation = 1.237
Correlation = 0.816
Exploratory data analysis
Explanatory data visualization
Design
Engineering
Journalism
Explanatory data visualization
Visualization is both an art and science
Harry Beck's subway map of London
Visual encoding of data
Data Types
Quantitative
Continuous, Discrete
Categorical
Nominal, Ordered, Interval
Visual encoding of data
Categorical scales and graph design
Visual encoding of data
Bandwidth of our senses: [Tor Norretranders]
Visual encoding of data
Data visual display elements
Position x
Position y
Retinal variables
Size, Orientation (ordered data)
Color Hue, Shape (nominal data)
Animation
Visual encoding of data
Ranking visual display elements (framework):
1.
2.
Position along a common-scale e.g. scatter plots
Position on identical but non-aligned scales
E.g. multiple scatter plots
3. Length e.g. bar chart
4. Angle & Slope e.g. pie-chart
5. Area e.g. bubbles
6.
7.
Volume, density & color saturation e.g. heat-map
Color hue e.g. highlights
Ref. Graphical Perception & graphical methods for analyzing scientific data William
Cleveland & Robert McGill (1985)
Design principles
Choose the right type of chart
Trends / Change over time Line charts
Distributions Histograms
Summary Information Table
Relationships Scatter Plots
Get it right in black & white (before adding color)
Prefer 2D to 3D for statistical charts
Use color to highlight
Avoid rainbow palette
Avoid chartjunk : less is more
Try to have a high data-ink ratio
Design principles
Choose the right type of chart
Ranking
Time-series
Correlation
Nominal comparison
Deviation
Narrative structures
Data Journalism
Traditional journalism
Data journalism
Data around narrative
Narrative around data
Linear flow
Complex, often non-linear flow
Physical static media
Online interactive media
Narrative structures
Narrative structures
Narrative structures
Bias (and ethics: Dont lie with data)
Bar-charts must have a zero-baseline
Present data in its context
Narrative structures
Bias: Misleading with data
Selective presentation with line-charts
Author Bias
Data Bias
Reader Bias
Narrative structures
Bias and Errors (statistics):
Selection bias e.g. in sampling
Omitted-variable bias
Errors:
Hypothesis testing
Null Hypothesis = default/no-effect state
Null Hypothesis H0
Valid
Invalid
Reject
Type I error
False positive
Correct inference
True positive
Accept
Correct inference
True negative
Type II error
False negative
Narrative structures
Storytelling:
Visual narratives have moved from author-driven to viewerdriven with use of highly interactive media for data visualization
Author-driven
Viewer-driven
Author driven
Viewer driven
Strong ordering
Exploratory
Heavy messaging
Ability to ask questions
Need for clarity and speed
Build own story
DataViz Technologies & Tools
Off-the-shelf:
Tableau, Qlikview
Tools:
Predefined charts: Raw, Chartio, Plotly
Google fusion tables, Excel, Gephi
Code & Javascript libraries:
R ggplot2, ggvis, rCharts + shiny(interactive apps)
Python matplotlib,
D3.js, Dimple.js, Leaflet, Rickshaw (use JSON data)
Linux gnuplot
DataViz Technologies & Tools
Tableau data viz
DataViz Technologies & Tools
Chart in R ggplot2
References
Visual display of Quantitative Information: Edward Tufte http://goo.gl/qb5ej
Exploratory Data Analysis: John Tukey http://goo.gl/tV57HP
Data Science Life cycle : Maloy Manna
http://www.datasciencecentral.com/profiles/blogs/the-data-science-project-lifecycle
Selecting right graph for your message: Stephen Few
www.perceptualedge.com/articles/ie/the_right_graph.pdf
Practical rules for using color in charts: Stephen Few
www.perceptualedge.com/articles/visual.../rules_for_using_color.pdf
OpenIntro Statistics: https://www.openintro.org/stat/
Misleading with statistics: Eric Portelance
https://medium.com/i-data/misleading-with-statistics-c63780efa928
Computational Information Design: Ben Fry
http://benfry.com/phd/dissertation-050312b-acrobat.pdf
