Introduction to Statistical Learning
(ISLR 2.1)
Yingbo Li
Clemson University
MATH 8050
Yingbo Li (Clemson) Intro to Statistical Learning MATH 8050 1 / 16
Outline
1 Why Estimate f
2 How to Estimate f
3 Trade-Off: Prediction Accuracy and Model Interpretability
4 Supervised vs Unsupervised Learning
5 Regression vs Classification
Yingbo Li (Clemson) Intro to Statistical Learning MATH 8050 2 / 16
Why Estimate f
The Advertising data
For n = 200 different markets
Sales: sales of the product in this market (Y )
TV: advertising budget for TV (X1 )
Radio: advertising budget for radio (X2 )
Newspaper: advertising budget for newspaper (X3 )
Yingbo Li (Clemson) Intro to Statistical Learning MATH 8050 3 / 16
Why Estimate f
We believe that there is a relationship between Y and X
Y : output variable, response
X = (X1 , X2 , X3 ): input variables, predictors
25
25
25
20
20
20
Sales
Sales
Sales
15
15
15
10
10
10
5
5
0 50 100 200 300 0 10 20 30 40 50 0 20 40 60 80 100
TV Radio Newspaper
Yingbo Li (Clemson) Intro to Statistical Learning MATH 8050 4 / 16
Why Estimate f
Model the relationship between Y and X
The regression function
Y = f (X) +
f : unknown function
: random error with mean zero, i.e., E() = 0.
In the Advertising example:
f (X1 , X2 , X3 ) = E(Y | X1 , X2 , X3 )
Statistical learning, and this course, are all about how to estimate f .
Why?
Prediction.
Inference.
Yingbo Li (Clemson) Intro to Statistical Learning MATH 8050 5 / 16
Why Estimate f
Prediction
If we can get a good estimate for f, we can make accurate predictions for
the response Y , based on a new value of X.
For a new market, given three media budgets, what’s the sales?
Just want to predict sales, not to know which media is more
important.
Suppose our estimate for f is fˆ, the output Y for input X is predicted as
Ŷ = fˆ(X)
Mean square error
E(Y − Ŷ )2 = E[f (X) − fˆ(X)]2 + V ()
| {z } |{z}
Reducible Irreducible
Yingbo Li (Clemson) Intro to Statistical Learning MATH 8050 6 / 16
Why Estimate f
Inference
We are often interested in understanding the relationship between that Y
and each of X1 , . . . , Xp . For example
1 Which predictors actually affect the response?
2 Is the relationship positive or negative?
3 Is the relationship a simple linear one or is it more complicated?
How much impact does TV budges have on the sales.
Which media generate the biggest boost in sales?
Yingbo Li (Clemson) Intro to Statistical Learning MATH 8050 7 / 16
How to Estimate f
How to estimate f
Use the training data and a statistical method to estimate f .
We have observed a set of training data
{(x1 , y1 ), (x2 , y2 ), . . . , (xn , yn )},
where each xi = (xi,1 , xi,2 , . . . , xi,p )0 , and yi is a scalar.
Statistical learning methods:
I parametric
I non-parametric
Yingbo Li (Clemson) Intro to Statistical Learning MATH 8050 8 / 16
How to Estimate f
Income vs Education, Seniority
Incom
e
y
rit
Ye
io
a
n
rs
Se
of
Ed
uc
ati
on
Yingbo Li (Clemson) Intro to Statistical Learning MATH 8050 9 / 16
How to Estimate f
Parametric methods
Reduces the problem of estimating f down to one of estimating a (finite)
set of parameters. A two-step model based approach:
1 Come up with a model (some functional form assumption about f ).
The most common example is a linear model.
f (X) = β0 + β1 X1 + β2 X2 + · · · + βp Xp
I We only need to estimate p + 1 parameters β0 , β1 , . . . , βp .
I Although it is almost never correct, a linear model often serves as a
good and interpretable approximation to the unknown true f (X).
2 Use the training data to fit the model.
Estimate the unknown parameters such as β̂0 , β̂1 , . . . , β̂p .
I The most common approach is ordinary least squares (OLS).
I We will see later that there are other superior approaches.
Yingbo Li (Clemson) Intro to Statistical Learning MATH 8050 10 / 16
How to Estimate f
A linear model fˆL (X) = β0 + β1 X gives a reasonable fit here.
A quadratic model fˆQ (X) = β0 + β1 X + β2 X 2 fits slightly better.
Yingbo Li (Clemson) Intro to Statistical Learning MATH 8050 11 / 16
How to Estimate f
A linear regression fit to the Income data
Incom
e
ity
or
Ye
ni
ars
Se
of
Ed
uc
ati
on
Income = β0 + β1 × Education + β2 × Seniority
Yingbo Li (Clemson) Intro to Statistical Learning MATH 8050 12 / 16
How to Estimate f
Non-parametric methods
They do not make explicit assumptions about the functional form of f .
Advantages: accurately fit a wider range of possible shapes of f .
Disadvantages: require large n to obtain an accurate estimate.
Incom
Incom
e
e
ity
ity
or
Ye
or
Ye
ni
a
ni
a rs
Se
rs
Se
of of
E Edu
du ca
ca tio
tio n
n
A smooth thin-plate spline fit: A rough thin-plate spline fit:
flexible overfitting
Yingbo Li (Clemson) Intro to Statistical Learning MATH 8050 13 / 16
Trade-Off: Prediction Accuracy and Model Interpretability
Some trade-offs
Prediction accuracy vs model interpretability
Linear models are easy to interpret; thin-plate splines are not.
Good fit vs over-fit
A model that overfits the training data may not predict well.
High
Subset Selection
Lasso
Least Squares
Interpretability
Generalized Additive Models
Trees
Bagging, Boosting
Support Vector Machines
Low
Low High
Flexibility
Yingbo Li (Clemson) Intro to Statistical Learning MATH 8050 14 / 16
Supervised vs Unsupervised Learning
Supervised vs unsupervised learning
Supervised learning: both X and Y are available
Unsupervised learning: only X is available; there is no Y .
I Example: market segmentation where we try to divide potential
customers into groups based on their characteristics.
I A common approach is clustering.
12
8
10
8
6
X2
X2
6
4
4
2
2
0 2 4 6 8 10 12 0 2 4 6
X1 X1
Yingbo Li (Clemson) Intro to Statistical Learning MATH 8050 15 / 16
Regression vs Classification
Regression vs classification
Regression: Y is continuous (quantitative).
I Predicting the value of the Dow in 6 months.
I Predicting the value of a given house based on various inputs.
Classification: Y is categorical (qualitative).
I Will the Dow be up (U) or down (D) in 6 months?
I Is this email a SPAM or not?
Yingbo Li (Clemson) Intro to Statistical Learning MATH 8050 16 / 16
