Machine Learning

Unit-5: Regularization

Rohit Kumar
(Asistant Professor)

P. K. Roy Memorial College, Dhanbad

August 1, 2023
Outline

1 Underfitting & Overfitting

pkrmc

Rohit Kumar(Asistant Professor) P. K. Roy Memorial College, Dhanbad

Machine Learning
Underfitting & Overfitting

Bias ! What does it mean for you?

Any hint! Do you remember the Data Science lecture of last
semester?

I said that a model has a low bias if it predicts well the labels
of the training data.
If the model makes many mistakes on the training data, we
say that the model has a high bias or that the model underfits.

Underfitting
So, underfitting is the inability of the model to predict well the
labels of the data it was trained on.
pkrmc

Rohit Kumar(Asistant Professor) P. K. Roy Memorial College, Dhanbad

Machine Learning
Underfitting & Overfitting (cont...)

What are the reasons behind the underfitting?

your model is too simple for the data
(for example a linear model can often underfit);
the features you engineered are not informative enough.
pkrmc

Rohit Kumar(Asistant Professor) P. K. Roy Memorial College, Dhanbad

Machine Learning
Underfitting & Overfitting

Behavior of different interpolating polynomial

pkrmc

Rohit Kumar(Asistant Professor) P. K. Roy Memorial College, Dhanbad

Machine Learning
 Behavior of 1st order interpolating polynomial

pkrmc

Rohit Kumar(Asistant Professor) P. K. Roy Memorial College, Dhanbad

Machine Learning
 Behavior of 3rd order (Spline) interpolating polynomial

pkrmc

Rohit Kumar(Asistant Professor) P. K. Roy Memorial College, Dhanbad

Machine Learning
 Behavior of 3rd order (PCHIP) interpolating polynomial

pkrmc

Rohit Kumar(Asistant Professor) P. K. Roy Memorial College, Dhanbad

Machine Learning
 Let’s say you want to predict whether a patient has
cancer, and the features you have are height, blood
pressure, and heart rate.
What will be your opinion in this case?

pkrmc

Rohit Kumar(Asistant Professor) P. K. Roy Memorial College, Dhanbad

Machine Learning
Underfitting (cont...)

Let’s say you want to predict whether a patient has

cancer, and the features you have are height, blood
pressure, and heart rate.
These three features are clearly not good predictors for
cancer so our model will not be able to learn a meaningful
relationship between these features and the label.

The solution to the problem of underfitting is to try a

more complex model or to engineer features with higher
predictive power.
pkrmc

Rohit Kumar(Asistant Professor) P. K. Roy Memorial College, Dhanbad

Machine Learning
Overfitting

Overfitting is another problem a model can exhibit.

The model that overfits predicts very well the training data
but poorly the data from at least one of the two hold-out sets.

pkrmc

Rohit Kumar(Asistant Professor) P. K. Roy Memorial College, Dhanbad

Machine Learning
 Several reasons can lead to overfitting, the most
important of which are:
1 your model is too complex for the data (for example a
very tall decision tree or a very deep or wide neural
network often overfit);

2 you have too many features but a small number of

training examples.

pkrmc

Rohit Kumar(Asistant Professor) P. K. Roy Memorial College, Dhanbad

Machine Learning
Overfitting(cont...)

Another name for the problem of overfitting!

"The problem of high variance"

This term comes from statistics.
The variance is an error of the model due to its sensitivity to
small fluctuations in the training set.
It means that if your training data was sampled differently, the
learning would result in a significantly different model.
Which is why the model that overfits performs poorly on the
test data.
Here it is to be noted that test and training data are sampled from
pkrmc
the dataset independently of one another.
Rohit Kumar(Asistant Professor) P. K. Roy Memorial College, Dhanbad
Machine Learning
Overfitting(cont...)

Several solutions to the problem of overfitting are possible:

Try a simpler model (linear instead of polynomial regression,
or SVM with a linear kernel instead of RBF, a neural network
with fewer layers/units).
Reduce the dimensionality of examples in the dataset.
Add more training data, if possible.
Regularize the model.
Regularization is the most widely used approach to
prevent overfitting.
pkrmc

Rohit Kumar(Asistant Professor) P. K. Roy Memorial College, Dhanbad

Machine Learning
Regularization

Regularization
Regularization encompasses the methods that force the
learning algorithm to build a less complex model.
In practice, that often leads to slightly higher bias but
significantly reduces the variance.
In the literature, this problem is known as the Bias-Variance
tradeoff.
The two most widely used types of regularization are called
L1-regularization and L2-regularization.
The idea is quite simple. To create a regularized model, we modify
the objective function by adding a penalizing term whose value is
pkrmc
higher when the model is more complex.
Rohit Kumar(Asistant Professor) P. K. Roy Memorial College, Dhanbad
Machine Learning
Regularization(cont...)

pkrmc

Rohit Kumar(Asistant Professor) P. K. Roy Memorial College, Dhanbad

Machine Learning