5.4 MLBasics Estimators

Deep Learning Srihari
Machine Learning Basics:

Estimators, Bias and Variance
Sargur N. Srihari
srihari@cedar.buffalo.edu
This is part of lecture slides on Deep Learning:

http://www.cedar.buffalo.edu/~srihari/CSE676
1
Srihari
Topics in Basics of ML
Deep Learning
1.  Learning Algorithms

2.  Capacity, Overfitting and Underfitting
3.  Hyperparameters and Validation Sets
4.  Estimators, Bias and Variance
5.  Maximum Likelihood Estimation
6.  Bayesian Statistics
7.  Supervised Learning Algorithms
8.  Unsupervised Learning Algorithms
9.  Stochastic Gradient Descent
10. Building a Machine Learning Algorithm
11. Challenges Motivating Deep Learning 2
Topics in Estimators, Bias, Variance

0. Statistical tools useful for generalization
1.  Point estimation
2.  Bias
3.  Variance and Standard Error
4.  Bias-Variance tradeoff to minimize MSE
5.  Consistency
3
Statistics provides tools for ML

•  The field of statistics provides many tools to
achieve the ML goal of solving a task not only
on the training set but also to generalize
•  Foundational concepts such as
–  Parameter estimation
–  Bias
–  Variance
•  They characterize notions of generalization,
over- and under-fitting
4
Point Estimation
•  Point Estimation is the attempt to provide the
single best prediction of some quantity of
interest
–  Quantity of interest can be:
•  A single parameter
•  A vector of parameters
–  E.g., weights in linear regression
•  A whole function
5
Point estimator or Statistic

•  To distinguish estimates of parameters from
their true value, a point estimate of a parameter
θ is represented by θ̂
•  Let {x(1), x(2),..x(m)} be m independent and
identically distributed data points
–  Then a point estimator or statistic is any function of
the data
θ̂m = g(x (1),...x (m) )
•  Thus a statistic is any function of the data
•  It need not be close to the true θ
–  A good estimator is a function whose output is close
to the true underlying θ that generated the data 6
Function Estimation
•  Point estimation can also refer to estimation of
relationship between input and target variables
–  Referred to as function estimation
•  Here we predict a variable y given input x
–  We assume f(x) is the relationship between x and y
•  We may assume y=f(x)+ε
–  Where ε stands for a part of y not predictable from x
–  We are interested in approximating f with a model fˆ
•  Function estimation is same as estimating a parameter θ
–  where fˆ is a point estimator in function space
•  Ex: in polynomial regression we are either estimating a
parameter w or estimating a function mapping from x to y
Properties of Point Estimators

•  Most commonly studied properties of point
estimators are:
1.  Bias
2.  Variance
•  They inform us about the estimators
8
1. Bias of an estimator
•  The bias of an estimator θ̂m = g(x (1),...x (m)) for
parameter θ is defined as
( )
bias θ̂m = E ⎡⎣θ̂m ⎤⎦ − θ
•  The estimator is unbiased if bias( θ̂m )=0

–  which implies that E ⎡⎣θ̂m ⎤⎦ = θ
•  An estimator is asymptotically unbiased if
( )
limm→∞ bias θ̂m = 0
9
Examples of Estimator Bias

•  We look at common estimators of the following
parameters to determine whether there is bias:
–  Bernoulli distribution: mean θ
–  Gaussian distribution: mean µ
–  Gaussian distribution: variance σ2
10
Estimator of Bernoulli mean

•  Bernoulli distribution for binary variable x ε{0,1}
with mean θ has the form P(x;θ) = θx (1 − θ)1−x
•  Estimator for θ given samples {x(1),..x(m)} is θ̂ = m1 ∑ x
m
(i )
m
i =1
•  To determine whether this estimator is biased

determine bias(θ̂ ) = E ⎡⎣θ̂ ⎤⎦ − θ
m m
⎡ 1 m (i ) ⎤
= E ⎢ ∑x ⎥ − θ
⎣ m i −1 ⎦
m
1
= ∑ E ⎡⎣x (i ) ⎤⎦ − θ
m i =1
1 m 1
( )
= ∑ ∑ x (i )θx (1 − θ)(1−x ) − θ
m i =1 x (i ) =0
(i ) (i )
1 m
= ∑ (θ) − θ = θ − θ = 0
m i =1
–  Since bias( θ̂ )=0 we say that the estimator is unbiased

m
Estimator of Gaussian mean

•  Samples {x(1),..x(m)} are independently and
identically distributed according to p(x(i))=N(x(i);µ,σ2)
–  Sample mean is an estimator of the mean parameter
1 m (i )
µ̂m = ∑ x
m i =1
–  To determine bias of the sample mean:
–  Thus the sample mean is an unbiased estimator of the

Gaussian mean
Estimator for Gaussian variance

•  The sample variance is ( 1 m (i )
)
2
σ̂ = ∑ x − µ̂m
2
m
m i =1
•  We are interested in computing
bias( σ̂m2 ) =E( σ̂m2 ) - σ2
•  We begin by evaluating à
•  σ̂ 2
Thus the bias of m is –σ2/m
•  Thus the sample variance is a biased estimator
•  The unbiased sample variance estimator is
1 m (i )
( )
2
σ̂ =
2
m ∑ x − µ̂m
m − 1 i =1
13
2. Variance and Standard Error

•  Another property of an estimator:
–  How much we expect the estimator to vary as a
function of the data sample
•  Just as we computed the expectation of the
estimator to determine its bias, we can compute
its variance
•  The variance of an estimator is simply Var(θ̂ )
where the random variable is the training set
•  The square root of the the variance is called the
standard error, denoted SE(θ̂)
14
Importance of Standard Error

•  It measures how we would expect the estimate
to vary as we obtain different samples from the
same distribution
•  The standard error of the mean is given by
⎡ 1 m (i ) ⎤ σ
( )
SE µ̂m = Var ⎢ ∑ x ⎥ =
⎣ m i =1 ⎦ m
–  where σ2 is the true variance of the samples x(i)

–  Standard error often estimated using estimate of σ
•  Although not unbiased, approximation is reasonable
–  The standard deviation is less of an underestimate than variance
Standard Error in Machine Learning

•  We often estimate generalization error by
computing error on the test set
–  No of samples in the test set determine its accuracy
–  Since mean will be normally distributed, (according
to Central Limit Theorem), we can compute
probability that true expectation falls in any chosen
interval
•  Ex: 95% confidence interval centered on mean µ̂m is
(µ̂ m ( ) ( ))
− 1.96SE µ̂m , µ̂m + 1.96SE µ̂m
•  ML algorithm A is better than ML algorithm B if

–  upperbound of A is less than lower bound of B
Confidence Intervals for error
95% confidence intervals for error estimate
17
Trading-off Bias and Variance

•  Bias and Variance measure two different
sources of error of an estimator
•  Bias measures the expected deviation from the
true value of the function or parameter
•  Variance provides a measure of the expected
deviation that any particular sampling of the
data is likely to cause
18
Negotiating between bias - tradeoff

•  How to choose between two algorithms, one
with a large bias and another with a large
variance?
–  Most common approach is to use cross-validation

–  Alternatively we can minimize Mean Squared Error
which incorporates both bias and variance
19
Mean Squared Error

•  Mean Squared Error of an estimate is
MSE = E ⎡⎢( θ̂ − θ ) ⎤⎥
2
⎣m
⎦
=Bias ( θ̂ ) + Var ( θ̂ )
2
m m
•  Minimizing the MSE keeps both bias and

variance in check
As capacity increases, bias (dotted )
tends to decrease and variance (dashed)
tends to increase
20
Underfit-Overfit : Bias-Variance
Relationship of bias-variance to capacity is similar to
underfitting and overfitting relationship to capacity
Bias-Variance to capacity Model complexity to capacity
Both have a U-shaped curve of generalization

Error as a function of capacity
21
Consistency
•  So far we have discussed behavior of an
estimator for a fixed training set size
•  We are also interested with the behavior of the
estimator as training set grows
•  As the no. of data points m in the training set
grows, we would like our point estimates to
converge to the true value of the parameters:
plimm→∞θ̂m = θ
–  Symbol plim indicates convergence in probability
Weak and Strong Consistency

•  plimm→∞θ̂m = θ means that
For any ε > 0, P(| θ̂m − θ |> ε) → 0 as m → ∞
•  It is also known as weak consistency
•  Implies almost sure convergence of θ̂ to θ
•  Strong consistency refers to almost sure convergence
of a sequence of random variables x(1),x(2),… to a
value x occurs when
p(limm→∞x (m) = x) = 1
•  Consistency ensures that the bias induced by the
estimator decreases with m
23

5.4 MLBasics Estimators

Uploaded by

Copyright:

Available Formats

5.4 MLBasics Estimators

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

5.4 MLBasics Estimators

Uploaded by

Copyright:

Available Formats

Deep Learning Srihari

Machine Learning Basics:

This is part of lecture slides on Deep Learning:

1. Learning Algorithms

Topics in Estimators, Bias, Variance

Statistics provides tools for ML

Point estimator or Statistic

Properties of Point Estimators

• The estimator is unbiased if bias( θ̂m )=0

Examples of Estimator Bias

Estimator of Bernoulli mean

• To determine whether this estimator is biased

– Since bias( θ̂ )=0 we say that the estimator is unbiased

Estimator of Gaussian mean

– Thus the sample mean is an unbiased estimator of the

Estimator for Gaussian variance

2. Variance and Standard Error

Importance of Standard Error

– where σ2 is the true variance of the samples x(i)

Standard Error in Machine Learning

• ML algorithm A is better than ML algorithm B if

Confidence Intervals for error

95% confidence intervals for error estimate

Trading-off Bias and Variance

Negotiating between bias - tradeoff

– Most common approach is to use cross-validation

Mean Squared Error

• Minimizing the MSE keeps both bias and

Bias-Variance to capacity Model complexity to capacity

Both have a U-shaped curve of generalization

Weak and Strong Consistency

You might also like

1.  Learning Algorithms

•  The estimator is unbiased if bias( θ̂m )=0

•  To determine whether this estimator is biased

–  Since bias( θ̂ )=0 we say that the estimator is unbiased

–  Thus the sample mean is an unbiased estimator of the

–  where σ2 is the true variance of the samples x(i)

•  ML algorithm A is better than ML algorithm B if

–  Most common approach is to use cross-validation

•  Minimizing the MSE keeps both bias and