Chapter 02
Introduction to
deep learning
Woo Youn Kim
Professor of Chemistry &
Graduate School of Data Science, KAIST
CEO, HITS
Contents
• Linear regression
• Linear classification
• Concept of deep learning
• Multi-layer perceptron & Non-linearity
• Forward propagation for prediction
• Backward propagation for training
LAIDD: Chapter 3 (linear regression and classification) of the Lecture series on “QSAR(Quantitative Structure
Activity Relationship)” (struct 2
Statistical modeling
hypothesis
observations
• Statistical modeling aims to obtain a hypothesis with training samples (observations) {𝐗, 𝐘}, where
we consider that samples are independent and identically distributed (i.i.d.) random variables
sampled from the joint distribution 𝑝(𝐗, 𝐘).
• Statistical inference is a procedure of inferring a new output 𝒚∗ of a new input 𝒙∗ by using the
obtained hypothesis. 3
Learning algorithms
• Reinforcement learning: to learn a decision (policy) from experience by interacting with environments
• Supervised learning: to learn a hypothesis that best explains the input-label relationship
• Unsupervised learning: to learn features, which are also called as representations or latent variables
• Semi-supervised learning: obtaining a hypothesis by using both labeled and unlabeled data
• Self-supervised learning:
4
Supervised learning
Solubility
Toxicity
Drug efficacy
.
.
.
Input data Output data
Feature extraction (L)
Structure (X) Property (Y)
Y = f(X); f = neural networks
• Regression: predicting values
• Classification: predicting active or inactive
5
Linear regression
6
Regression
Let us start with a talk about a few examples of supervised learning problems.
X Y
hypothesis
Molecular structure Energy
Molecular structure Solubility
Molecular structure Biological activity
observations
𝑋
When the target variable that we’re trying to predict is continuous, we apply a
regression method to it.
7
Regression (linear hypothesis)
𝑊: parameters or weights
𝑏 : bias
The figure illustrates how linear regression works by fitting different lines
(hypotheses) to a given dataset. It visualizes the equation H(x)=Wx+b, where
W is the weight (or slope), and b is the bias (intercept). The goal is to find the
best-fitting line that minimizes the error between the predicted values and the
actual data points.
8
Cost function
How fit the line to our (training) data numerically?
9
Cost minimization
10
Simplified hypothesis
To simplify the calculation, we ignore the bias term in the hypothesis and thus in the cost function.
11
What cost looks like?
• 𝑾 = 𝟏, 𝒄𝒐𝒔𝒕 𝑾 = 𝟎
1 2 2
( 1∗1−1 + 1∗2−2 + 1 ∗ 3 − 3 2)
3
• 𝑾 = 𝟎, 𝒄𝒐𝒔𝒕 𝑾 = 𝟒. 𝟔𝟕
1 2 2
( 0∗1−1 + 0∗2−2 + 0 ∗ 3 − 3 2)
3
...
This graph shows the cost function Cost(W) with respect
to different values of W, which represents the weights in
a linear regression model.
12
Gradient descent algorithm
• Minimize the cost function using gradient descent algorithm
• Start with an initial guess
- Start at 0,0 (or any other value)
- Keep changing W and b a little bit to try and reduce cost(W, b)
• Each time you change the parameters, you select the gradient
which reduces cost(W, b) the most possible
a (= DW) : updating rate (or learning rate)
- the larger, the faster
- but may cause ill-convergence
• Repeat until you converge to a local minimum (i.e., the gradient = 0)
13
Least Mean Squares (LMS)
The resulting equation is called the LMS update rule (LMS stands for “least mean squares”)
14
Convex function
Note that the optimization problem here has only
one global, and no other local optima.
Thus, gradient descent always converges to the
the global minimum unless the learning rate α is
too large.
cost(W, b)
15
Gradient descent algorithm
• Batch gradient descent
Repeat until convergence {
scan through the entire training set before taking a single step:
computationally expensive: suppose you have million samples!
• Stochastic gradient descent
(also incremental gradient descent)
Loop {
for i=1 to m, {
}
}
start making progress right away, and continue to make progress with each example it looks at.
much faster convergence than the batch gradient descent.
16
Gaussian noise
Suppose that the target variables and the inputs are related via the equation.
where 𝜖 𝑖
is an error term that captures either unmodeled effects or random noise.
If 𝜖 𝑖 is independently and identically distributed according to a Gaussian distribution (also
called a Normal distribution) with mean zero and some variance 𝜎 2 ,
Thus,
The notation “𝑝 𝑦 𝑖 |𝑥 𝑖 ; 𝜃 ” indicates that this is the distribution of 𝑦 𝑖
given 𝑥 𝑖
and parameterized by 𝜃.
17
Likelihood
Given a data set, 𝑋 𝑥 𝑖
, and a ML model 𝜃, what is the distribution of target, Y 𝑦 𝑖
?
The probability of the data is given by p(Y|X; 𝜃), which is viewed a function of Y for a fixed value of 𝜃.
It can be instead called the likelihood function.
What is a reasonable way of choosing our best guess of the parameters 𝜃 or training the machine
learning model?
18
Maximum likelihood
The principal of maximum likelihood says that we should choose 𝜃 so as to make
the data as highly probable as possible.
That is, we should choose 𝜃 to maximize 𝐿 𝜃 .
19
Maximum likelihood
Instead of maximizing 𝐿 𝜃 , we can also maximize any strictly increasing function of 𝐿 𝜃 .
For example, maximize the log likelihood ℓ 𝜃 :
Note that the final 𝜽 has no
dependence on 𝝈.
Maximizing ℓ 𝜃 gives the same answer as minimizing
which is the original LMS cost function.
20
Classification
21
Classification
Classification problem: the values y we now want to predict take on only a small number of discrete
values.
Binary classification problem: y can take on only two values, 0 and 1.
0 or - : the negative class,
1 or + : the positive class,
22
Decision boundary
For 2D, the dimension of θ will be one.
Thus, the decision boundary will be a
linear line.
23
Problem of linear regression
Let’s try to predict y given z (= θTx) using the linear regression algorithm
It performs very poorly!
Simply it doesn’t make sense for hθ(x) to take values larger than 1 or smaller than 0
when we know that y ∈ {0, 1}.
24
Logistic function
change hθ(x) as
logistic function or the sigmoid function.
✓ It becomes 0.5 as a data point is on the decision
boundary (z = 0).
✓ It goes to 1 as z → ∞ and to 0 as z → −∞.
✓ It is always bounded between 0 and 1.
25
Logistic function
A useful property of the derivative of the sigmoid function,
26
Maximum likelihood
Assuming that
More compactly
27
Maximum likelihood
For m training examples, the likelihood of the parameters
Maximizing via the gradient ascent.
The corresponding log likelihood
for the maximization
28
Cost function
The log likelihood ➔ one has to maximize it
Cost function = -log likelihood ➔ one has to minimize it
𝑦 (𝑖) : true value in the training data ➔ true probablity of being in class 1
ℎ 𝑥 𝑖 : predicted value by the classifier ➔ predicted probability of being in class 1
1 − 𝑦 (𝑖) : true probablity of being in class 2
1 − ℎ 𝑥 𝑖 : predicted probability of being in class 2
29
Cost function
Two important properties:
1. Non-negative because 0 ≤ 𝑝𝑟𝑜𝑏 ≤ 1
2. For all training data (m), if ℎ 𝑥 𝑖
is close to 𝑦 (𝑖) , the cost goes to zero.
if 𝑦 (𝑖) = 0, ℎ 𝑥 𝑖
→0: 1st term = 0 and ln1=0 in the 2nd term; therefore Cost ➔0
if 𝑦 (𝑖) = 1, ℎ 𝑥 𝑖 →1: ln1=0 in the 1st term and 2nd term =0; therefore Cost ➔0
Indeed, it can play a role as a cost function.
30
Maximum likelihood
Let us take a derivative to derive the stochastic gradient ascent rule:
The stochastic gradient ascent rule
31
Maximum likelihood
The stochastic gradient ascent rule
Difference between the true and predicted values
The derivative becomes zero as the difference becomes zero.
In other words, the classifier perfectly predicts the class of input data!
32
Cross entropy
Gibbs entropy formula
VS
Entropy in information theory
It is also called the cross entropy
33
Softmax classification
34
Multinomial classification
We need 3 decision boundaries!
For 2D, 3 lines:
35
Softmax function
The softmax function If k = 2 like the binary classification (y = 0 or 1),
Prob. for k = 1
Prob. for k = 2
➔Probability of having the k-th label. The result is the same with the logistic function
for binary classification!!
Generalization of the logistic function for multinomial problems
36
Softmax classification
The hypothesis function
The softmax regression
Dog Cat Rabbit
Probability of being in each class
One has to determine the parameters {𝜃} with a given training set ➔ cost function
37
One-hot encoding
1
Dog = 0
0
0
Cat = 1
0
0
Rabbit = 0
1
• Multiple classes can be represented with a vector having elements whose number is the same
with the number of classes.
• This is called one-hot encoding.
• In this example, we have three classes and thus a vector with 3 elements.
• We take the maximum value and let it the unit value, resulting in only a single non-zero
element (in this example, the first element)
• Therefore, the input data is labeled the class 1.
38
Cost function
As the generalization of the logistic regression, we use the cross entropy as a cost function.
element-wise product
where
The element-wise product gives this result.
39
Concept of deep learning
40
Concept of machine learning
“easy-for-a-human, difficult-for-a-machine” tasks,
often referred to as pattern recognition.
• This image explains the concept of machine learning, specifically focusing on tasks that are
“easy for a human, difficult for a machine.” These tasks typically involve pattern recognition,
such as distinguishing between cats and dogs from labeled photos.
• On the right side of the image, there are labeled pictures of a cat and a dog. This visual
representation demonstrates how machines learn to recognize patterns, like identifying
different animals, by being trained with labeled data (photos with categories).
41
Why deep learning?
Classical ML (or shallow learning)
hand-crafted features by experts
Deep learning
automated feature extraction by AI
42
Neuron & synapse
Electrical and chemical signal transmission
Digital signal: on-off
threshold
Input Output
Analog signal transmission
Logistic (sigmoid) function 43
Neural network
Hebbian principle (1940s)
a neuro-scientific theory claiming that an increase in synaptic efficacy arises from a presynaptic cell’s
repeated and persistent stimulation of a postsynaptic cell. “Cells that fire together wire together.”
Complexity of human brain
• 1011 neurons
• each neuron has ~7,000 synaptic connections
• 3 years old: 1015 synapses
• adult: 1014~5X1014 synapses
➔ complex functions
• One of the key functions of a neural network is its ability to learn.
• A neural network is a complex adaptive system; it can change its internal structure based on the
information flowing through it by adjusting the amplitude of signals (weight).
44
Perception
“easy-for-a-human, difficult-for-a-machine” tasks, often referred to as pattern recognition.
Unique signal pattern
45
Perceptron
Invented in 1957 by Frank Rosenblatt at the Cornell Aeronautical Laboratory, a perceptron is the
simplest neural network possible: a computational model of a single neuron.
46
Principle of perceptron
0 if 𝑓 𝒘 ∙ 𝒙 < threshold
output = ቊ
1 if 𝑓 𝒘 ∙ 𝒙 ≥ threshold
inputs processor single output 𝒘 ∙ 𝒙 = 𝑤𝑗 𝑥𝑗
𝑗
Step 1: receive inputs Step 2: weight inputs Step 3: sum inputs Step 4: generate output
Input 0: x1 = 12 x1 × w1 = 12 × 0.5 = 6 Output = sign(sum)
Input 1: x2 = 4 x2 × w2 = 4 × -1 = -4 Sum = 6 + -4 = 2 = sign(2) = +1
47
Bias
0 if 𝑓 𝒘 ∙ 𝒙 < threshold 0 if 𝑓 𝒘 ∙ 𝒙 + 𝑏 < 0
output = ቊ output = ቊ
1 if 𝑓 𝒘 ∙ 𝒙 ≥ threshold 1 if 𝑓 𝒘 ∙ 𝒙 + 𝑏 ≥ 0
move the threshold inside the perceptron
and then make it a learnable parameter
a measure of how easy it is to get the perceptron to output 1.
or a measure of how easy it is to get the perceptron to fire.
48
Linear classification
By varying the weights and the bias, we can get different models of decision-making.
Learning ➔ determining appropriate weights and bias
to get an optimal output with a given training and test data set 49
Logic gate
✓ AND Gate
x1 x2 y
0 0 0
1 0 0
0 1 0
1 1 1
0 if 𝒘 ∙ 𝒙 + 𝑏 < 0
output = ቊ
1 if 𝒘 ∙ 𝒙 + 𝑏 ≥ 0
50
Logic gate
✓ Logic gate - OR Gate
x1 x2 y
0 0 0
1 0 1
0 1 1
1 1 1
0 if 𝒘 ∙ 𝒙 + 𝑏 < 0 the same slope
output = ቊ
1 if 𝒘 ∙ 𝒙 + 𝑏 ≥ 0 but a different bias value
51
Logic gate
✓ Logic gate- XOR Gate
x1 x2 y
0 0 0
1 0 1
0 1 1
1 1 0
x2
XOR Gate
0 if 𝒘 ∙ 𝒙 + 𝑏 < 0
output = ቊ
1 if 𝒘 ∙ 𝒙 + 𝑏 ≥ 0
x1
Non-linear expression with w and x 52
Drawback of a single perceptron
The single perceptron approach has one major drawback; it can only learn linearly separable
functions. Take XOR, a relatively simple function, and notice that it can’t be classified by a linear
separator.
But what if we make the model with more perceptrons? If one perceptron can solve OR and one
perceptron can solve NOT AND, then two perceptrons combined can solve XOR.
➔ Many perceptrons to go beyond linearity
53
Multilayer perceptron
& non-linearity
54
Multilayer perceptron (MLP)
MLP, a.k.a a forward-feed network
𝒙 = {𝑥1, 𝑥2}
𝒙 𝒘1 𝒘2
ℎ𝑗 = 𝑓(𝒘𝑗1 ∙ 𝒙 + 𝑏1)
𝒚
𝑦 = 𝑓(𝒘2 ∙ 𝒉 + 𝑏2)
𝑏1 𝑏2
Marvin Minsky 1969, founder of MIT AI lab
no way to train the neural network ➔ the first winter of AI 55
Nonlinearity
The nonlinear activation functions can
be expanded as
𝑢2
𝜎 𝑢 ≈ 𝜎0 + 𝜎1 ⋅ 𝑢 + 𝜎2 ⋅
2
The nonlinearity comes from both the activation
functions and many perceptrons
56
Universal approximation
57
Universal Approximation Theorem
• Why does neural networks work so well?
• The answer is, MLPs can approximate any non-linear function which is called the universal approximation
theorem. https://en.wikipedia.org/wiki/Universal_approximation_theorem
58
Universal approximation
One hidden-layer approximation of nonlinear functions: x2, sinx, |x|, and H(x)
Good bases for spanning a nonlinear function space 59
Supervised learning
Solubility
Toxicity
Drug efficacy
.
.
.
Input data Output data
Feature extraction (L)
Structure (X) Property (Y)
Y = f(X); f = neural networks
• Regression: predicting values
• Classification: predicting active or inactive
60
Why deeper?
Theorem. For any given multivariate polynomial and any tolerance ε > 0, there exists a neural
network of fixed finite size N (independent of ε) that approximates the polynomial to accuracy
better than ε. Furthermore, N is bounded by the complexity of the polynomial, scaling as the
number of multiplications required times a factor that is typically slightly larger than 4.
https://arxiv.org/pdf/1608.08225v4.pdf
61
Why deeper?
• ILSVRC (ImageNet Large Scale Visual Recognition Challenge) Winners
62
Forward propagation
for prediction
63
Prediction
Feed-forward MLP (or DNN)
x y
64
Example: forward propagation
65
Example: forward propagation
66
Example: forward propagation
67
Example: forward propagation
68
Back propagation
for training
69
Loss functions
Loss function = Cost function = Objective function = Error function
Regression
1 2
L2 loss function (or MSE or LMS) = 𝑦𝑖 − ℎ(𝑥𝑖 )
2𝑚
𝑖
Classification
Cross entropy = − 𝑦𝑖 ln ℎ 𝑥𝑖
𝑖
For binary classification, 𝐿 = −(𝑦𝑖 ln ℎ 𝑥𝑖 + (1 − 𝑦𝑖)ln(1 − ℎ 𝑥𝑖 )
70
How to train?
x y
(𝑖+1) (𝑖)
ℎ𝑖 = 𝑓(σ𝑗 𝒘𝑖𝑗ℎ𝑗 + 𝑏 (𝑖) )
One has to determine {w, b} so as to minimize the error, E(ypred, ytrue)
Minsky (1969): no one on earth had found a viable way to train MLPs good enough
to learn such simple functions (Winter of NN1 until 1986) 71
Stochastic gradient descent
Back propagation
1974, 1982 by Paul Werbos,
1986 by Hinton, rediscovery
72
Back propagation
73
Back propagation
74
Back propagation
75
Back propagation
76
Back propagation
77
Back propagation
78
Back propagation
79
Vanishing gradient
s(x) tanh(x)
Successive multiplication of the derivatives of the activation function ➔ zero update value
80
Vanishing gradient
Winter of neural network 2, CIFAR
Canadian Institute For Advanced Research
The vanishing gradient problem occurs during the training of deep neural networks,
particularly in very deep networks or when using activation functions like the sigmoid
or tanh. It happens when the gradients, which are used to update the weights of the
network during backpropagation, become extremely small as they propagate through
the layers. As a result, the early layers in the network receive negligible updates,
making it difficult for the network to learn effectively, slowing or even halting training.
This problem can be mitigated by using activation functions like ReLU and its variants
or by employing techniques like batch normalization.
81
Various cost functions
ReLU = Rectified Linear Unit
Variants of ReLU to improve its problem that has no
learning effect if the input has a negative value (zero
derivative): a.k.a. dying ReLU
• Leaky ReLU: very large negative output
• ELU: softer than Leaky ReLU (good for noise
handling) but more computational costs
Nair and Hinton, ICML (2010)
Go deeper
Maxout: piecewise linear function
(each perceptron has different values)
82
New terms
• Supervised learning
• Unsupervised learning
• Reinforcement learning
• Hypothesis
• Least Mean Square (LMS)
• Maximum Likelihood Estimate (MLE)
• Regression
• Gradient descent
• Classification
• Decision boundary
• Logistic or sigmoid function
• Cost function
• Softmax classification
• Cross entropy
• One-hot encoding
83
New terms
• Neuron & synapse
• Perceptron
• Activation function: sigmoid, tanh, ReLU, etc
• Multilayer perceptron (MLP) = Deep neural network (DNN) = Artificial
neural network (ANN)
• Nonlinearity
• Universal approximation
• Loss function = Cost function = Error function
• Feed forward
• Back propagation
84
