Machine Learning Lecture-Notes

Mathematical Foundations
of Machine Learning
Lectures on YouTube:
https://www.youtube.com/@mathtalent
Seongjai Kim
Department of Mathematics and Statistics

Mississippi State University
Mississippi State, MS 39762 USA
Email: skim@math.msstate.edu
Updated: April 30, 2023

Seongjai Kim, Professor of Mathematics, Department of Mathematics and Statistics, Mississippi
State University, Mississippi State, MS 39762 USA. Email: skim@math.msstate.edu.
Prologue
In organizing this lecture note, I am indebted by the following:
• S. R ASCHKA AND V. M IRJALILI, Python Machine Learning, 3rd Ed., 2019

[62].
• (Lecture note) http://fa.bianp.net/teaching/2018/eecs227at/, Dr. Fabian Pe-
dregosa, UC Berkeley
• (Lecture note) Introduction To Machine Learning, Prof. David Sontag,
MIT & NYU
• (Lecture note) Mathematical Foundations of Machine Learning, Dr. Justin
Romberg, Geoigia Tech
This lecture note will grow up as time marches; various core algorithms,
useful techniques, and interesting examples would be soon incorporated.
Seongjai Kim
April 30, 2023
iii
iv
Contents
Title ii
Prologue iii
Table of Contents ix
1 Introduction 1
1.1. Why and What in Machine Learning (ML)? . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.1. Inference problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.2. Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1.3. Machine learning examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2. Three Different Types of ML . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.2.1. Supervised learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.2.2. Unsupervised learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.3. A Machine Learning Modelcode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Exercises for Chapter 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2 Python Basics 17
2.1. Why Python? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.2. Python in 30 Minutes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.2.1. Python essentials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.2.2. Frequently used Python rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.2.3. Looping and functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.3. Zeros of Polynomials in Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.4. Python Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
3 Simple Machine Learning Algorithms for Classification 37

3.1. Binary Classifiers – Artificial Neurons . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
3.2. The Perceptron Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
3.2.1. The perceptron: A formal definition . . . . . . . . . . . . . . . . . . . . . . . . . 40
3.2.2. The perceptron learning rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.2.3. Problems with the perceptron algorithm . . . . . . . . . . . . . . . . . . . . . . . 44
3.3. Adaline: ADAptive LInear NEuron . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
3.3.1. Minimizing the cost function with the gradient descent method . . . . . . . . . 48
v
vi Contents
3.3.2. Convergence and optimization issues with the gradient descent method . . . . 49
3.3.3. Feature scaling and stochastic gradient descent . . . . . . . . . . . . . . . . . . 50
4 Gradient-based Methods for Optimization 55

4.1. Gradient Descent Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
4.1.1. The gradient descent method in 1D . . . . . . . . . . . . . . . . . . . . . . . . . . 59
4.1.2. The full gradient descent algorithm . . . . . . . . . . . . . . . . . . . . . . . . . 61
4.1.3. Surrogate minimization: A unifying principle . . . . . . . . . . . . . . . . . . . . 65
4.2. Newton’s Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
4.2.1. Derivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
4.2.2. Hessian and principal curvatures . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
4.3. Quasi-Newton Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
4.4. The Stochastic Gradient Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
4.5. The Levenberg–Marquardt Algorithm, for Nonlinear Least-Squares Problems . . . . . 81
4.5.1. The gradient descent method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
4.5.2. The Gauss-Newton method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
4.5.3. The Levenberg-Marquardt algorithm . . . . . . . . . . . . . . . . . . . . . . . . 84
5 Popular Machine Learning Classifiers 89

5.1. Logistic Sigmoid Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
5.1.1. The standard logistic sigmoid function . . . . . . . . . . . . . . . . . . . . . . . . 91
5.1.2. The logit function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
5.2. Classification via Logistic Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
5.2.1. The logistic cost function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
5.2.2. Gradient descent learning for logistic regression . . . . . . . . . . . . . . . . . . 99
5.2.3. Regularization: bias-variance tradeoff . . . . . . . . . . . . . . . . . . . . . . . . 100
5.3. Support Vector Machine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
5.3.1. Linear SVM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
5.3.2. The method of Lagrange multipliers . . . . . . . . . . . . . . . . . . . . . . . . . 104
5.3.3. Karush-Kuhn-Tucker conditions and Complementary slackness . . . . . . . . . 107
5.3.4. The inseparable case: Soft-margin classification . . . . . . . . . . . . . . . . . . 112
5.3.5. Nonlinear SVM and kernel trick . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
5.3.6. Solving the dual problem with SMO . . . . . . . . . . . . . . . . . . . . . . . . . 120
5.4. Decision Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
5.4.1. Decision tree objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
5.4.2. Random forests: Multiple decision trees . . . . . . . . . . . . . . . . . . . . . . . 127
5.5. k-Nearest Neighbors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
6 Data Preprocessing in Machine Learning 133

Contents vii
6.1. General Remarks on Data Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . 134

6.2. Dealing with Missing Data & Categorical Data . . . . . . . . . . . . . . . . . . . . . . . 136
6.2.1. Handling missing data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
6.2.2. Handling categorical data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
6.3. Feature Scaling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
6.4. Feature Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
6.4.1. Selecting meaningful variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
6.4.2. Sequential backward selection (SBS) . . . . . . . . . . . . . . . . . . . . . . . . . 142
6.4.3. Ridge regression vs. LASSO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
6.5. Feature Importance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
7 Feature Extraction: Data Compression 149

7.1. Principal Component Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
7.1.1. Computation of principal components . . . . . . . . . . . . . . . . . . . . . . . . 151
7.1.2. Dimensionality reduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
7.1.3. Explained variance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
7.2. Singular Value Decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
7.2.1. Interpretation of the SVD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
7.2.2. Properties of the SVD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
7.2.3. Computation of the SVD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
7.2.4. Application of the SVD to image compression . . . . . . . . . . . . . . . . . . . . 170
7.3. Linear Discriminant Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
7.3.1. Fisher’s LDA (classifier): two classes . . . . . . . . . . . . . . . . . . . . . . . . . 173
7.3.2. Fisher’s LDA: the optimum projection . . . . . . . . . . . . . . . . . . . . . . . . 175
7.3.3. LDA for multiple classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
7.3.4. The LDA: dimensionality reduction . . . . . . . . . . . . . . . . . . . . . . . . . . 186
7.4. Kernel Principal Component Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
7.4.1. Principal components of the kernel PCA . . . . . . . . . . . . . . . . . . . . . . . 190
7.4.2. Computation of the kernel PCA . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
8 Cluster Analysis 199

8.1. Basics for Cluster Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
8.1.1. Quality of clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
8.1.2. Types of clusters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
8.1.3. Types of clustering and Objective functions . . . . . . . . . . . . . . . . . . . . . 208
8.2. K-Means and K-Medoids Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
8.2.1. The (basic) K-Means clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
8.2.2. Bisecting K-Means algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
8.2.3. The K-Medoids algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
8.2.4. CLARA and CLARANS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222
viii Contents
8.3. Hierarchical Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224

8.3.1. Basics of AGNES and DIANA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224
8.3.2. AGNES: Agglomerative clustering . . . . . . . . . . . . . . . . . . . . . . . . . . 226
8.4. DBSCAN: Density-based Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
8.5. Cluster Validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
8.5.1. Basics of cluster validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
8.5.2. Internal and external measures of cluster validity . . . . . . . . . . . . . . . . . 241
8.6. Self-Organizing Maps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247
8.6.1. Basics of the SOM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247
8.6.2. Kohonen SOM networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251
8.6.3. The SOM algorithm and its interpretation . . . . . . . . . . . . . . . . . . . . . 256
9 Neural Networks and Deep Learning 261

9.1. Basics for Deep Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262
9.2. Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266
9.2.1. Sigmoid neural networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268
9.2.2. A simple network to classify handwritten digits . . . . . . . . . . . . . . . . . . 270
9.3. Back-Propagation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278
9.3.1. Notations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278
9.3.2. The cost function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280
9.3.3. The four fundamental equations behind the back-propagation . . . . . . . . . . 281
9.4. Deep Learning: Convolutional Neural Networks . . . . . . . . . . . . . . . . . . . . . . 285
9.4.1. Introducing convolutional networks . . . . . . . . . . . . . . . . . . . . . . . . . 286
9.4.2. CNNs, in practice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292
10 Data Mining 297

10.1.Introduction to Data Mining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298
10.2.Vectors and Matrices in Data Mining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302
10.2.1. Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302
10.2.2. Data compression: Low rank approximation . . . . . . . . . . . . . . . . . . . . 306
10.3.Text Mining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 310
10.3.1. Vector space model: Preprocessing and query matching . . . . . . . . . . . . . . 311
10.3.2. Latent Semantic Indexing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317
10.4.Eigenvalue Methods in Data Mining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 320
10.4.1. Pagerank . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 321
10.4.2. The Google matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325
10.4.3. Solving the Pagerank equation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327
Exercises for Chapter 10 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330
11 Quadratic Programming 331

Contents ix
11.1.Equality Constrained Quadratic Programming . . . . . . . . . . . . . . . . . . . . . . . 332

11.2.Direct Solution for the KKT System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337
11.2.1. Symmetric factorization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337
11.2.2. Range-space approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 339
11.2.3. Null-space approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 341
11.3.Linear Iterative Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 342
11.3.1. Convergence theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343
11.3.2. Graph theory: Estimation of the spectral radius . . . . . . . . . . . . . . . . . . 343
11.3.3. Eigenvalue locus theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345
11.3.4. Regular splitting and M-matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . 347
11.4.Iterative Solution of the KKT System . . . . . . . . . . . . . . . . . . . . . . . . . . . . 348
11.4.1. Krylov subspace methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 348
11.4.2. The transforming range-space iteration . . . . . . . . . . . . . . . . . . . . . . . 349
11.5.Active Set Strategies for Convex QP Problems . . . . . . . . . . . . . . . . . . . . . . . 350
11.5.1. Primal active set strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 351
11.6.Interior-point Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353
11.7.Logarithmic Barriers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 355
P Projects 361
P.1. mCLESS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 362
P.1.1. Review: Simple classifiers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363
P.1.2. The mCLESS classifier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 364
P.1.3. Feature expansion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 368
P.2. Noise-Removal and Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 373
P.3. Gaussian Sailing to Overcome Local Minima Problems . . . . . . . . . . . . . . . . . . 379
P.4. Quasi-Newton Methods Using Partial Information of the Hessian . . . . . . . . . . . . 381
P.5. Effective Preprocessing Technique for Filling Missing Data . . . . . . . . . . . . . . . . 383
Bibliography 385
Index 391
x Contents
1
C HAPTER
Introduction
What are we “learning” in Machine Learning (ML)?
This is a hard question to which we can only give a somewhat fuzzy answer.
But at a high enough level of abstraction, there are two answers:
• Algorithms, which solve some kinds of inference problems

• Models for datasets.
These answers are so abstract that they are probably completely unsatis-
fying. But let’s (start to) clear things up, by looking at some particular
examples of “inference" and “modeling" problems.
Contents of Chapter 1
1.1. Why and What in Machine Learning (ML)? . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2. Three Different Types of ML . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3. A Machine Learning Modelcode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1
2 Chapter 1. Introduction
1.1. Why and What in Machine Learning (ML)?

1.1.1. Inference problems
Definition 1.1. Statistical inference is the process of using data
analysis to deduce properties of an underlying probability distribution.
Inferential statistical analysis infers properties of a population, for ex-
ample by testing hypotheses and deriving estimates.
Loosely speaking, inference problems take in data, then output

some kind of decision or estimate. The output of a statistical infer-
ence is a statistical proposition; here are some common forms of statistical
proposition.
• a point estimate
• an interval estimate
• a credible interval
• rejection of a hypothesis
• classification or clustering of the data points into discrete groups
Example 1.2. Inference algorithms can answer the following.
(a) Does this image have a tree in it?
(b) What words are in this picture?
(c) If I give you a recording of somebody speaking, can you produce text of
what they are saying?
1.1. Why and What in Machine Learning (ML)? 3
Remark 1.3. What does a machine learning algorithm do?

Machine learning algorithms are not algorithms for performing in-
ference. Rather, they are algorithms for building inference algo-
rithms from examples. An inference algorithm takes a piece of data
and outputs a decision (or a probability distribution over the decision
space).
1.1.2. Modeling
A second type of problem associated with machine learning (ML) might
be roughly described as:
Given a dataset, how can I succinctly describe it (in a quantita-

tive, mathematical manner)?
One example is regression analysis. Most models can be broken into two
categories:
• Geometric models. The general problem is that we have example

data points
x 1 , x 2 , · · · , x N ∈ RD
and we want to find some kind of geometric structure that (ap-
proximately) describes them.
Here is an example: given a set of vectors, what (low dimensional) sub-
space comes closest to containing them?
• Probabilistic models. The basic task here is to find a probability
distribution that describes the dataset {xn }.
The classical name for this problem is density estimation – given
samples of a random variable, estimate its probability density function
(pdf). This gets extremely tricky in high dimensions (large values of
D) or when there are dependencies between the data points. Key to
solving these problems is choosing the right way to describe your
probability model.
Note: In both cases above, having a concise model can go a tremen-

dous way towards analyzing the data.
• As a rule, if you have a simple and accurate model, this
is tremendously helping in solving inference problems, be-
cause there are fewer parameters to consider and/or esti-
mate.
• The categories can either overlap with or complement each
other. It is often the case that the same model can be interpreted
as a geometric model or a probabilistic model.
1.1.3. Machine learning examples

• Classification: from data to discrete classes
– Spam filtering
– Object detection (e.g., face)
– Weather prediction (e.g., rain, snow, sun)
• Regression: predicting a numeric value
– Stock market
– Weather prediction (e.g., Temperature)
• Ranking: comparing items
– Web search (keywords)
– Given an image, find similar images
• Collaborative Filtering (e.g., Recommendation systems)
• Clustering: discovering structure in data
– Clustering points or images
– Clustering web search results
• Embedding: visualizing data
– Embedding images, words
• Structured prediction: from data to discrete classes
– Speech/image recognition
– Natural language processing
1.2. Three Different Types of ML 5
1.2. Three Different Types of ML

Example 1.4. Three different types of ML:
• Supervised learning: classification, regression

• Unsupervised learning: clustering
• Reinforcement learning: chess engine
Figure 1.1: Three different types of ML (by methods)

1.2.1. Supervised learning
Assumption. Given a data set {(xi , yi )}, ∃ a relation f : X → Y .

Supervised learning:
(
Given : Training set {(xi , yi ) | i = 1, · · · , N }
(1.1)
Find : fb : X → Y , a good approximation to f
Figure 1.2: Supervised learning and prediction.
Figure 1.3: Classification and regression.

Example 1.5. Regression

• Dataset: Consider a simple dataset: 10 points generated from a sin func-
tion, with noise
• Wanted: Find a regression model

f :X=R→Y =R (1.2)
• Question: How should we pick the hypothesis space, the set of possible
functions f ?
– Often, there exist infinitely many hypotheses
– Let’s select f from PM , polynomials of degree ≤ M .
– Which one is best?
• Error: misfit
– We measure the error using a loss function L(y, yb).

– For regression, a common choice is squared loss: for each (xi , yi ),
L(yi , f (xi )) = ((yi − f (xi ))2 .
– The empirical loss of the function f applied to the training data is

then:
N N
def 1 1 X
X
EM S == L(yi , f (xi )) = (yi − f (xi ))2 , (1.3)
N i=1 N i=1
which is the mean square error.
Figure 1.4: Learning curve.

Key Idea 1.6. Training and test performance. Assume that each
training and test example–label pair (x, y) is drawn independently at
random from the same (but unknown) population of examples and labels.
Represent this population as a probability distribution p(x, y), so that:
(xi , yi ) ∼ p(x, y).
• Then, given a loss function L:

N
1 X
– Empirical (training) loss = L(yi , f (xi )).
N i=1
(Also called the empirical risk, R(f,
b DN ).)
– Expected (test) loss = E(x,y)∼p {L(y, f (x))}.
(Also called the risk R(f ).)
• Ideally, learning chooses the hypothesis that minimizes the
risk.
– But this is impossible to compute!
– The empirical risk is a good (unbiased) estimate of the risk
(by linearity of expectation).
• The principle of empirical risk minimization reads
f ∗ (DN ) = arg min R(f,

b DN ). (1.4)
f ∈F
Remark 1.7. Key Issues in Machine Learning

• How do we choose a hypothesis space?
– Often we use prior knowledge to guide this choice
– The ability to answer to the next two questions also affects the
choice
• How can we gauge the accuracy of a hypothesis on unseen test-
ing data?
– The previous example showed that choosing the hypothesis which
simply minimizes training set error (i.e., empirical risk mini-
mization) is not optimal.
– This question is a main topic in learning theory.
• How do we find the best hypothesis?
– This is an algorithmic question, at the intersection of mathemat-
ics, computer science, and optimization research.
Proposition 1.8. Occam’s Razor Principle (a.k.a. Law of parsimony):
“One should not increase, beyond what is necessary,

the number of entities required to explain anything.”
• William of Occam: A monk living in the 14-th century, England

• When many solutions are available for a given problem, we should
select the simplest one.
• But what do we mean by simple?
• We will use prior knowledge of the problem to solve to define what
is a simple solution (Example of a prior: smoothness).
1.2.2. Unsupervised learning

Note:
• In supervised learning, we know the right answer beforehand when
we train our model, and in reinforcement learning, we define a mea-
sure of reward for particular actions by the agent.
• In unsupervised learning, however, we are dealing with unlabeled
data or data of unknown structure. Using unsupervised learn-
ing techniques, we are able to explore the structure of our data
to extract meaningful information without the guidance of a
known outcome variable or reward function.
• Clustering is an exploratory data analysis technique that allows
us to organize a pile of information into meaningful subgroups
(clusters) without having any prior knowledge of their group mem-
berships.
Figure 1.5: Clustering.

1.3. A Machine Learning Modelcode

A code for machine learning can start with the following machine learn-
ing modelcode. You may copy-and-paste the scripts to run.
Machine_Learning_Model.py
1 import numpy as np; import pandas as pd
2 import seaborn as sbn; import matplotlib.pyplot as plt
3 import time
4 from sklearn.model_selection import train_test_split
5 from sklearn import datasets; #print(dir(datasets))
6 np.set_printoptions(suppress=True)
7
8 #=====================================================================
9 # DATA: Read & Preprocessing
10 # load_iris, load_wine, load_breast_cancer, ...
11 #=====================================================================
12 data_read = datasets.load_iris(); #print(data_read.keys())
13
14 X = data_read.data
15 y = data_read.target
16 dataname = data_read.filename
17 targets = data_read.target_names
18 features = data_read.feature_names
19
20 print('X.shape=',X.shape, 'y.shape=',y.shape)
21 #---------------------------------------------------------------------
22 # SETTING
23 #---------------------------------------------------------------------
24 N,d = X.shape; labelset=set(y)
25 nclass=len(labelset);
26 print('N,d,nclass=',N,d,nclass)
27
28 rtrain = 0.7e0; run = 100

29 rtest = 1-rtrain
30
31 #=====================================================================
32 # CLASSIFICATION
33 #=====================================================================
34 btime = time.time()
35 Acc = np.zeros([run,1])
36 ##from sklearn.neighbors import KNeighborsClassifier
37 ##clf = KNeighborsClassifier(5)
38 from myCLF import myCLF ## My classifier
1.3. A Machine Learning Modelcode 13
39
40 for it in range(run):
41 Xtrain, Xtest, ytrain, ytest = train_test_split(
42 X, y, test_size=rtest, random_state=it, stratify = y)
43 ##clf.fit(Xtrain, ytrain);
44 clf = myCLF(Xtrain,ytrain); clf.fit(); ## My classifier
45 Acc[it] = clf.score(Xtest, ytest)
46
47 #-----------------------------------------------
48 # Print: Accuracy && E-time
49 #-----------------------------------------------
50 etime = time.time()-btime
51 print(' %s: Acc.(mean,std) = (%.2f,%.2f)%%; Average E-time= %.5f'
52 %(dataname,np.mean(Acc)*100,np.std(Acc)*100,etime/run))
53
54 #=====================================================================
55 # Scikit-learn Classifiers, for Comparisions
56 #=====================================================================
57 exec(open("sklearn_classifiers.py").read())
sklearn_classifiers.py
1 #=====================================================================
2 # Required: X, y, [dataname, run]
3 print('========= Scikit-learn Classifiers, for Comparisions =========')
4 #=====================================================================
5 from sklearn.preprocessing import StandardScaler
6 from sklearn.datasets import make_moons, make_circles, make_classification
7 from sklearn.neural_network import MLPClassifier
8 from sklearn.neighbors import KNeighborsClassifier
9 from sklearn.linear_model import LogisticRegression
10 from sklearn.svm import SVC
11 from sklearn.gaussian_process import GaussianProcessClassifier
12 from sklearn.gaussian_process.kernels import RBF
13 from sklearn.tree import DecisionTreeClassifier
14 from sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier
15 from sklearn.naive_bayes import GaussianNB
16 from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis
17 #from sklearn.inspection import DecisionBoundaryDisplay
18
19 #-----------------------------------------------
20 classifiers = [
21 LogisticRegression(max_iter = 1000),
22 KNeighborsClassifier(5),
23 SVC(kernel="linear", C=0.5),
24 SVC(gamma=2, C=1),
25 RandomForestClassifier(max_depth=5, n_estimators=50, max_features=1),
26 MLPClassifier(alpha=1, max_iter=1000),
27 AdaBoostClassifier(),
28 GaussianNB(),
29 QuadraticDiscriminantAnalysis(),
30 GaussianProcessClassifier(),
31 ]
32 names = [
33 "Logistic Regr",
34 "KNeighbors-5 ",
35 "Linear SVM ",
36 "RBF SVM ",
37 "Random Forest",
38 "Deep-NN ",
39 "AdaBoost ",
40 "Naive Bayes ",
41 "QDA ",
42 "Gaussian Proc",
43 ]
44 #-----------------------------------------------
45 if dataname is None: dataname = 'No-dataname';
46 if run is None: run = 100;
47
48 #===============================================
49 import os; acc_max=0
50 for name, clf in zip(names, classifiers):
51 Acc = np.zeros([run,1])
52 btime = time.time()
53
54 for it in range(run):
55 Xtrain, Xtest, ytrain, ytest = train_test_split(
56 X, y, test_size=rtest, random_state=it, stratify = y)
57
58 clf.fit(Xtrain, ytrain);
59 Acc[it] = clf.score(Xtest, ytest)
60
61 etime = time.time()-btime
62 accmean = np.mean(Acc)*100
63 print('%s: %s: Acc.(mean,std) = (%.2f,%.2f)%%; E-time= %.5f'
64 %(os.path.basename(dataname),name,accmean,np.std(Acc)*100,etime/run))
65 if accmean>acc_max:
66 acc_max= accmean; algname = name
67 print('sklearn classifiers max: %s= %.2f' %(algname,acc_max))
1.3. A Machine Learning Modelcode 15
Exercises for Chapter 1
1.1. The code in Section 1.3 will run without requiring any other implementation of yours,
if (1) uncomment lines 36-37 and 43 and (2) comment lines 38 and 44.
(a) Save the code written in two files, by copy-and-paste.

(b) Install all imported packages to run the code.
(c) Report the results.
Installation: If you are on Ubuntu, you may begin with

Install-Python-packages
1 sudo apt update
2 sudo apt install python3 -y
3 sudo apt install python3-pip -y
4 rehash
5 sudo pip3 install numpy scipy matplotlib sympy -y
6 sudo pip3 install sklearn seaborn pandas -y
2
C HAPTER
Python Basics
2.1. Why Python? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.2. Python in 30 Minutes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.3. Zeros of Polynomials in Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.4. Python Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
17
18 Chapter 2. Python Basics
2.1. Why Python?

Note: A good programming language must be easy to learn and use
and flexible and reliable.
Advantages of Python
Python has the following characteristics.
• Easy to learn and use
• Flexible and reliable
• Extensively used in Data Science
• Handy for Web Development purposes
• Having Vast Libraries support
• Among the fastest-growing programming languages in the tech
industry
Disadvantage of Python
Python is an interpreted and dynamically-typed language. The line-by-
line execution of code, built with a high flexibility, most likely leads to
slow execution. Python is slower than Matlab that is slower than C.
Remark 2.1. Speed up Python Programs.

• Use numpy and scipy for all mathematical operations.
• Always use a C library wherever possible.
• You yourself may create and import your own C-module into Python.
If you extend Python with pieces of compiled C-code, then the re-
sulting code is easily 100× faster than Python. Best choice!
• Cython: It is designed as a C-extension for Python, which is
developed for users not familiar with C. For Cyphon implemetation,
see e.g. https://www.youtube.com/watch?v=JKMkhARcwdU, one of
Simon Funke’s YouTube videos.
2.1. Why Python? 19
• The library numpy is designed for a Matlab-like implementation.

• Python can be used as a convenient desktop calculator.
– First, set a startup environment
– Use Python as a desktop calculator
∼/.python_startup.py
1 #.bashrc: export PYTHONSTARTUP=~/.python_startup.py
2 #.cshrc: setenv PYTHONSTARTUP ~/.python_startup.py
3 #---------------------------------------------------
4 print("\t^[[1;33m~/.python_startup.py")
5
6 import numpy as np
7 import numpy.linalg as la
8 import matplotlib.pyplot as plt
9 print("\tnp = numpy; la = numpy.linalg; plt = matplotlib.pyplot")
10
11 from numpy import zeros,ones,abs,sqrt,sum,pi,sin,cos,tan,exp,log

12 print("\tzeros,ones,abs,sqrt,sum,pi,sin,cos,tan,exp,log")
13
14 import random, sympy

15 from sympy import symbols,solve #x=symbols('x'); solve(x**2-4)
16 print("\trandom, sympy: x=symbols('x'); solve(x**2-4)")
17
18 print("\t^[[1;37mTo see details: dir() or dir(np)^[[m")
Figure 2.1: Python startup.

2.2. Python in 30 Minutes

Key Features of Python
• Python is a simple, readable, open source programming language
which is easy to learn.
• It is an interpreted language, not a compiled language.
• In Python, variables are untyped; i.e., there is no need to de-
fine/declare the data type of a variable before using it.
• Python supports the object-oriented programming model.
• It is platform-independent and easily extensible and embeddable.
• It has a huge standard library with lots of modules and packages.
• Python is a high level language as it is easy to use because of simple
syntax, powerful because of its rich libraries and extremely versatile.
Programming Features
• Python does not support pointers.
• Python codes are stored with .py extension.
• Indentation: Python uses indentation to define a block of code.
– A code block (body of a function, loop, etc.) starts with indenta-
tion and ends with the first unindented line.
– The amount of indentation is up to the user, but it must be consis-
tent throughout that block.
• Comments:
– The hash (#) symbol is used to start writing a comment.
– Multi-line comments: Python uses triple quotes, either ”’ or """.
2.2. Python in 30 Minutes 21
2.2.1. Python essentials

• Sequence datatypes: list, tuple, string
– [list]: defined using square brackets (and commas)
>>> li = ["abc", 14, 4.34, 23]
– (tuple): defined using parentheses (and commas)
>>> tu = (23, (4,5), ’a’, 4.1, -7)
– "string": defined using quotes (", ’, or """)
>>> st = ’Hello World’
>>> st = "Hello World"
>>> st = """This is a multi-line string
. . . that uses triple quotes."""
• Retrieving elements
>>> li[0]
’abc’
>>> tu[1],tu[2],tu[-2]
((4, 5), ’a’, 4.1)
>>> st[25:36]
’ng\nthat use’
• Slicing
>>> tu[1:4] # be aware
((4, 5), ’a’, 4.1)
• The + and ∗ operators
>>> [1, 2, 3]+[4, 5, 6,7]
[1, 2, 3, 4, 5, 6, 7]
>>> "Hello" + " " + ’World’
Hello World
>>> (1,2,3)*3
(1, 2, 3, 1, 2, 3, 1, 2, 3)
• Reference semantics
>>> a = [1, 2, 3]
>>> b = a
>>> a.append(4)
>>> b
[1, 2, 3, 4]
Be aware with copying lists and numpy arrays!
• numpy, range, and iteration
>>> range(8)
[0, 1, 2, 3, 4, 5, 6, 7]
>>> import numpy as np
>>> for k in range(np.size(li)):
... li[k]
. . . <Enter>
’abc’
14
4.34
23
• numpy array and deepcopy
>>> from copy import deepcopy
>>> A = np.array([1,2,3])
>>> B = A
>>> C = deepcopy(A)
>>> A *= 4
>>> B
array([ 4, 8, 12])
>>> C
array([1, 2, 3])
2.2.2. Frequently used Python rules

frequently_used_rules.py
1 ## Multi-line statement
2 a = 1 + 2 + 3 + 4 + 5 +\
3 6 + 7 + 8 + 9 + 10
4 b = (1 + 2 + 3 + 4 + 5 +
5 6 + 7 + 8 + 9 + 10) #inside (), [], or {}
6 print(a,b)
7 # Output: 55 55
8
9 ## Multiple statements in a single line using ";"

10 a = 1; b = 2; c = 3
11
12 ## Docstrings in Python
13 def double(num):
14 """Function to double the value"""
15 return 2*num
16 print(double.__doc__)
17 # Output: Function to double the value
18
19 ## Assigning multiple values to multiple variables

20 a, b, c = 1, 2, "Hello"
21 ## Swap
22 b, c = c, b
23 print(a,b,c)
24 # Output: 1 Hello 2
25
26 ## Data types in Python

27 a = 5; b = 2.1
28 print("type of (a,b)", type(a), type(b))
29 # Output: type of (a,b) <class 'int'> <class 'float'>
30
31 ## Python Set: 'set' object is not subscriptable

32 a = {5,2,3,1,4}; b = {1,2,2,3,3,3}
33 print("a=",a,"b=",b)
34 # Output: a= {1, 2, 3, 4, 5} b= {1, 2, 3}
35
36 ## Python Dictionary
37 d = {'key1':'value1', 'Seth':22, 'Alex':21}
38 print(d['key1'],d['Alex'],d['Seth'])
39 # Output: value1 21 22
40
41 ## Output Formatting
42 x = 5.1; y = 10
43 print('x = %d and y = %d' %(x,y))
44 print('x = %f and y = %d' %(x,y))
45 print('x = {} and y = {}'.format(x,y))
46 print('x = {1} and y = {0}'.format(x,y))
47 # Output: x = 5 and y = 10
48 # x = 5.100000 and y = 10
49 # x = 5.1 and y = 10
50 # x = 10 and y = 5.1
51
52 print("x=",x,"y=",y, sep="#",end="&\n")
53 # Output: x=#5.1#y=#10&
54
55 ## Python Input
56 C = input('Enter any: ')
57 print(C)
58 # Output: Enter any: Starkville
59 # Starkville
2.2.3. Looping and functions

Example 2.2. Compose a Python function which returns cubes of natural
numbers.
Solution.
get_cubes.py
1 def get_cubes(num):
2 cubes = []
3 for i in range(1,num+1):
4 value = i**3
5 cubes.append(value)
6 return cubes
7
8 if __name__ == '__main__':
9 num = input('Enter a natural number: ')
10 cubes = get_cubes(int(num))
11 print(cubes)
Remark 2.3. get_cubes.py

• Lines 8-11 are added for the function to be called directly. That is,
[Fri Jul.22] python get_cubes.py
Enter a natural number: 6
[1, 8, 27, 64, 125, 216]
• When get_cubes is called from another function, the last four lines
will not be executed.
call_get_cubes.py
1 from get_cubes import *
2
3 cubes = get_cubes(8)
4 print(cubes)
Execusion
1 [Fri Jul.22] python call_get_cubes.py
2 [1, 8, 27, 64, 125, 216, 343, 512]
2.3. Zeros of Polynomials in Python

In this section, a code in Python will be implemented to find zeros of poly-
nomials; we will compare the Python code with a Matlab code.
Recall: Let’s begin with recalling how to find zeros of polynomials.

• When the Newton’s method is applied for an approximate zero of
P (x), the iteration reads
P (xn )
xn+1 = xn − . (2.1)
P 0 (xn )
Thus both P (x) and P 0 (x) must be evaluated in each iteration.
• The derivative P 0 (x) can be evaluated by using the Horner’s
method with the same efficiency. Indeed,
– The remainder theorem says
P (x) = (x − xn )Q(x) + P (xn ). (2.2)
– Its derivative reads
P 0 (x) = Q(x) + (x − xn )Q0 (x). (2.3)
– Thus
P 0 (xn ) = Q(xn ). (2.4)
That is, the evaluation of Q at xn becomes the desired quantity
P 0 (xn ).
Note: The Horner’s method is also called the synthetic division.

2.3. Zeros of Polynomials in Python 27
Example 2.4. Let P (x) = x4 − 4x3 + 7x2 − 5x − 2. Use the Newton’s method
and the Horner’s method to implement a code and find an approximate zero
of P near 3.
Solution. First, let’s try to use built-in functions.
zeros_of_poly_built_in.py
2
3 coeff = [1, -4, 7, -5, -2]

4 P = np.poly1d(coeff)
5 Pder = np.polyder(P)
6
7 print(P)
8 print(Pder)
9 print(np.roots(P))
10 print(P(3), Pder(3))
Output
1 4 3 2
2 1 x - 4 x + 7 x - 5 x - 2
3 3 2
4 4 x - 12 x + 14 x - 5
5 [ 2. +0.j 1.1378411+1.52731225j 1.1378411-1.52731225j -0.2756822+0.j ]
6 19 37
Now, we implement a code for Newton-Horner method to find an approxi-

mate zero of P near 3.
Zeros-Polynomials-Newton-Horner.py
1 def horner(A,x0):
2 """ input: A = [a_n,...,a_1,a_0]
3 output: p,d = P(x0),DP(x0) = horner(A,x0) """
4 n = len(A)
5 p = A[0]; d = 0
6
7 for i in range(1,n):
8 d = p + x0*d
9 p = A[i] +x0*p
10 return p,d
11
12 def newton_horner(A,x0,tol,itmax):
13 """ input: A = [a_n,...,a_1,a_0]
14 output: x: P(x)=0 """
15 x=x0
16 for it in range(1,itmax+1):
17 p,d = horner(A,x)
18 h = -p/d;
19 x = x + h;
20 if(abs(h)<tol): break
21 return x,it
22
23 if __name__ == '__main__':
24 coeff = [1, -4, 7, -5, -2]; x0 = 3
25 tol = 10**(-12); itmax = 1000
26 x,it =newton_horner(coeff,x0,tol,itmax)
27 print("newton_horner: x0=%g; x=%g, in %d iterations" %(x0,x,it))
Execution
1 [Sat Jul.23] python Zeros-Polynomials-Newton-Horner.py
2 newton_horner: x0=3; x=2, in 7 iterations
Note: The above Python code must be compared with the Matlab code
below.
horner.m
1 function [p,d] = horner(A,x0)
2 % input: A = [a_0,a_1,...,a_n]
3 % output: p=P(x0), d=P'(x0)
4
5 n = size(A(:),1);
6 p = A(n); d=0;
7
8 for i = n-1:-1:1
9 d = p + x0*d;
10 p = A(i) +x0*p;
11 end
newton_horner.m
1 function [x,it] = newton_horner(A,x0,tol,itmax)
2 % input: A = [a_0,a_1,...,a_n]; x0: initial for P(x)=0
3 % outpue: x: P(x)=0
4
5 x = x0;
6 for it=1:itmax
7 [p,d] = horner(A,x);
8 h = -p/d;
9 x = x + h;
10 if(abs(h)<tol), break; end
11 end
2.3. Zeros of Polynomials in Python 29
Call_newton_horner.m
1 a = [-2 -5 7 -4 1];
2 x0=3;
3 tol = 10^-12; itmax=1000;
4 [x,it] = newton_horner(a,x0,tol,itmax);
5 fprintf(" newton_horner: x0=%g; x=%g, in %d iterations\n",x0,x,it)
6 Output: newton_horner: x0=3; x=2, in 7 iterations
Observation 2.5.
Python programming is as easy and simple as Matlab programming.
• In particular, numpy is developed for Matlab-like implementation,
with enhanced convenience.
• Python uses classes for object-oriented programming.
• Furthermore, Python is an open source (free) programming lan-
guage, which explains why Python is fastest-growing in use.
2.4. Python Classes
Remark 2.6. Classes are a key concept in the so-called object-

oriented programming (OOP). Classes provide a means of
bundling data and functionality together.
• A class is a user-defined template or prototype from which real-
world objects are created.
• A class tells us what data an object should have, what are the ini-
tial/default values of the data, and what methods are associated
with the object to take actions on the objects using their data.
• An object is an instance of a class, and creating an object from a
class is called instantiation.
In the following, we would build a simple class, as Dr. Xu did in [82, Ap-
pendix B.5]; you will learn how to initiate, refine, and use classes.
2.4. Python Classes 31
Polynomial_01.py
1 class Polynomial():
2 """A class of polynomials"""
3
4 def __init__(self,coefficient):
5 """Initialize coefficient attribute of a polynomial."""
6 self.coeff = coefficient
7
8 def degree(self):
9 """Find the degree of a polynomial"""
10 return len(self.coeff)-1
11
12 if __name__ == '__main__':
13 p2 = Polynomial([1,2,3])
14 print(p2.coeff) # a variable; output: [1, 2, 3]
15 print(p2.degree()) # a method; output: 2
• Lines 1-2: define a class called Polynomial with a docstring.

– The parentheses in the class definition are empty because we cre-
ate this class from scratch.
• Lines 4-10: define two functions, __init__() and degree(). A function
in a class is called a method.
– The __init__() method is a special method for initialization; it is
called the __init__() constructor.
– The self parameter is required and must come first before the
other parameters in each method.
– Whenever we make an object from the class, we need to provide
arguments for parameters, except for self.
– The variable self.coeff (prefixed with self) is available to every
method and is accessible by any object created from the class.
– Variables prefixed with self are called attributes.
• Line 13: The line p2 = Polynomial([1,2,3]) creates an object p2 (a
polynomial x2 + 2x + 3), by passing the coefficient list [1,2,3].
– When Python reads this line, it calls the method __init__() in the
class Polynomial and creates the object named p2 that represents
this particular polynomial x2 + 2x + 3.
Refinement of the Polynomial class

Polynomial_02.py
3
4 count = 0 #a class attribute

5
6 def __init__(self): #constructor

8 self.coeff = [1]
9 Polynomial.count += 1
10
11 def __del__(self): #destructor

12 """Delete a polynomial object"""
13 Polynomial.count -= 1
14 print('Destructor called, a Polynomial object deleted.')
15
16 def degree(self):
19
20 def evaluate(self,x):
21 """Evaluate a polynomial."""
22
23 n = self.degree()
24 eval = []
25 for xi in x:
26 p = self.coeff[0] #Horner's method
27 for k in range(1,n+1):
28 p = self.coeff[k]+ xi*p
29 eval.append(p)
30 return eval
31
32 if __name__ == '__main__':
33 poly1 = Polynomial()
34 print('poly1, default coefficients:', poly1.coeff)
35 poly1.coeff = [1,2,-3]
36 print('poly1, coefficients after reset:', poly1.coeff)
37 print('poly1, degree:', poly1.degree())
38
39 poly2 = Polynomial()
40 poly2.coeff = [1,2,3,4,-5]
41 print('poly2, coefficients after reset:', poly2.coeff)
42 print('poly2, degree:', poly2.degree())
43
44 print('number of created polynomials:', Polynomial.count)

45 del poly1
46 print('number of polynomials after a deletion:', Polynomial.count)
47
48 print('poly2.evaluate([-1,0,1,2]):',poly2.evaluate([-1,0,1,2]))
• Line 4: The variable count is a class attribute of Polynomial.

– A class attribute is a variable that belongs to a class but not a
particular object.
– All objects of the class share this same variable (the class at-
tribute).
• Line 8: initializes the class attribute self.coeff.
– Every object or class attribute in a class needs an initial value.
– One can set a default value for an object attribute in the
__init__() constructor, and we then do not have to include a pa-
rameter for that attribute in the constructor. See Lines 32 and
38.
• Lines 11-14: define the __del__() method, the destructor, for the
deletion of objects.
• Lines 20-30: define another method called evaluate, which uses the
Horner’s method. See Example 2.4, p.27.
Execution
1 [Tue Jan.24] python Polynomial_02.py
2 poly1, default coefficients: [1]
3 poly1, coefficients after reset: [1, 2, -3]
4 poly1, degree: 2
5 poly2, coefficients after reset: [1, 2, 3, 4, -5]
6 poly2, degree: 4
7 number of created polynomials: 2
8 Destructor called, a Polynomial object deleted.
9 number of polynomials after a deletion: 1
10 poly2.evaluate([-1,0,1,2]): [-7, -5, 5, 47]
11 Destructor called, a Polynomial object deleted.
Inheritance
Note: If we want to write a class that is just a specialized version of
another class, we do not need to write the class from scratch.
• We call the specialized class a child class and the other general
class a parent class.
• The child class can inherit all the attributes and methods form the
parent class; it can also define its own special attributes and meth-
ods or even overrides methods of the parent class.
Classes.py
3
6 self.coeff = coefficient
7
8 def degree(self):
11
12 class Quadratic(Polynomial):
13 """A class of quadratic polynomial"""
14
16 """Initialize the coefficient attributes ."""
17 super().__init__(coefficient)
18 self.power_decrease = 1
19
20 def roots(self):
21 a,b,c = self.coeff
22 if self.power_decrease != 1:
23 a,c = c,a
24 discriminant = b**2-4*a*c
25 r1 = (-b+discriminant**0.5)/(2*a)
26 r2 = (-b-discriminant**0.5)/(2*a)
27 return [r1,r2]
28
29 def degree(self):
30 return 2
• Line 12: We must include the name of the parent class in the paren-
theses of the definition of the child class (to indicate the parent-child
relation for inheritance).
• Line 17: The super() function is to give an child object all the at-
tributes defined in the parent class.
• Line 18: An additional child class attribute self.power_decrease is
initialized.
• Lines 20-27: define a new method called roots.
• Lines 29-30: The method degree() overrides the parent’s method.
call_Quadratic.py
1 from Classes import *
2
3 quad1 = Quadratic([2,-3,1])
4 print('quad1, roots:',quad1.roots())
5 quad1.power_decrease = 0
6 print('roots when power_decrease = 0:',quad1.roots())
7 # Output: quad1, roots: [1.0, 0.5]
8 # roots when power_decrease = 0: [2.0, 1.0]
Python Keywords
Keywords are some predefined and reserved words in python.
Ex.: and, or, not, if, else, elif, for, while, break, del, ...
You should use Python for the following problems.

2.1. Use nested for loops to assign entries of a 5 × 5 matrix A such that A[i, j] = ij.
2.2. The variable d is initially equal to 1. Use a while loop to keep dividing d by 2 until
d < 10−6 .
(a) Determine how many divisions are made.

(b) Verify your result by algebraic derivation.
Note: A while loop has not been considered in the lecture. However, you can figure it out
easily by yourself.
2.3. Write a function that takes as input a list of values and returns the largest value. Do
this without using the Python max() function; you should combine a for loop and an
if statement.
(a) Produce a random list of size 10-20 to verify your function.
2.4. Let P4 (x) = 2x4 − 5x3 − 11x2 + 20x + 10. Solve the following.
(a) Plot P4 over the interval [−3, 4].

(b) Find all zeros of P4 , modifying Zeros-Polynomials-Newton-Horner.py, p.27.
(c) Add markers for the zeros to the plot.
(d) Find all roots of P40 (x) = 0.
(e) Add markers for the zeros of P40 to the plot.
Do not use built-in functions, for (b) and (d).

Hint : For plotting, you may import: “import matplotlib.pyplot as plt” then use
plt.plot(). You will see the Python plotting is quite similar to Matlab plotting.
3
C HAPTER
Simple Machine Learning Algorithms

for Classification
In this chapter, we will make use of one of the first algorithmically described
machine learning algorithms for classification, the perceptron and adap-
tive linear neurons (adaline). We will start by implementing a perceptron
step by step in Python and training it to classify different flower species in
the Iris dataset.
3.1. Binary Classifiers – Artificial Neurons . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
3.2. The Perceptron Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
3.3. Adaline: ADAptive LInear NEuron . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
37
38 Chapter 3. Simple Machine Learning Algorithms for Classification
3.1. Binary Classifiers – Artificial Neurons

Definition 3.1. A binary classifier is a function which can decide
whether or not an input vector belongs to some specific class (e.g.,
spam/ham).
• Binary classification often refers to those classification tasks that
have two class labels. (two-class classification)
• It is a type of linear classifier, i.e. a classification algorithm that
makes its predictions based on a linear predictor function
combining a set of weights with the feature vector.
• Linear classifiers are artificial neurons.
Remark 3.2. Neurons are interconnected nerve cells that are involved
in the processing and transmitting of chemical and electrical signals.
Such a nerve cell can be described as a simple logic gate with binary
outputs;
• multiple signals arrive at the dendrites,
• they are integrated into the cell body,
• and if the accumulated signal exceeds a certain threshold, an output
signal is generated that will be passed on by the axon.
Figure 3.1: A schematic description of a neuron.

3.1. Binary Classifiers – Artificial Neurons 39
Linear classifiers
As artificial neurons, they have the following characteristics:
• Inputs are feature values: x
• Each feature has a weight: w
• Weighted sum (integration) is the activation
X
activationw (x) = wj x j = w · x (3.1)
j
• Decision/output: If the activation is

Positive ⇒ class 1
Negative ⇒ class 2
Unknowns, in ML:

Training : w
Prediction : activationw (x)
Examples:
• Perceptron
• Adaline (ADAptive LInear NEuron)
• Support Vector Machine (SVM) ⇒ nonlinear decision boundaries, too
3.2. The Perceptron Algorithm

The perceptron is a binary classifier of supervised learning.
• 1957: Perceptron algorithm is invented by Frank Rosenblatt, Cornell
Aeronautical Laboratory
– Built on work of Hebbs (1949)
– Improved by Widrow-Hoff (1960): Adaline
• 1960: Perceptron Mark 1 Computer – hardware implementation

• 1970’s: Learning methods for two-layer neural networks
3.2.1. The perceptron: A formal definition

Definition 3.3. We can pose the perceptron as a binary classifier,
in which we refer to our two classes as 1 (positive class) and −1 (negative
class) for simplicity.
• Input values: x = (x1 , x2 , · · · , xm )T
• Weight vector: w = (w1 , w2 , · · · , wm )T
• Net input: z = w1 x1 + w2 x2 + · · · + wm xm
• Activation function: φ(z), defined by

1 if z ≥ θ,
φ(z) = (3.2)
−1 otherwise,
where θ is a threshold.
For simplicity, we can bring the threshold θ in (3.2) to the left side of the
equation; define a weight-zero as w0 = −θ and reformulate as

1 if z ≥ 0,
φ(z) = z = wT x = w0 + w1 x1 + · · · + wm xm . (3.3)
−1 otherwise,
In the ML literature, the variable w0 is called the bias.

The equation w0 + w1 x1 + · · · + wm xm = 0 represents a hyperplane
in Rm , while w0 decides the intercept.
3.2. The Perceptron Algorithm 41
3.2.2. The perceptron learning rule

The whole idea behind the Rosenblatt’s thresholded perceptron model
is to use a reductionist approach to mimic how a single neuron in the brain
works: it either fires or it doesn’t.
Summary 3.4. Thus, Rosenblatt’s initial perceptron rule is fairly sim-
ple and can be summarized by the following steps:
1. Initialize the weights to 0 or small random numbers.
2. For each training sample x(i) ,
(a) Compute the output value yb(i) (:= φ(wT x(i) )).
(b) Update the weights.
The update of the weight vector w can be more formally written as:
w := w + ∆w, ∆w = η (y (i) − yb(i) ) x(i) , (3.4)
where η is the learning rate, 0 < η < 1, y (i) is the true class label of the
i-th training sample, and yb(i) denotes the predicted class label.
Remark 3.5. A simple thought experiment for the perceptron

learning rule:
• Let the perceptron predict the class label correctly. Then y (i) −b
y (i) = 0
so that the weights remain unchanged.
• Let the perceptron make a wrong prediction. Then
(i) (i)
∆wj = η (y (i) − yb(i) ) xj = ±2 η xj
so that the weight wj is pushed towards the direction of the positive

or negative target class, respectively.
Note: It is important to note that convergence of the perceptron

is only guaranteed if the two classes are linearly separable and the
learning rate is sufficiently small. If the two classes can’t be sep-
arated by a linear decision boundary, we can set a maximum number
of passes over the training dataset (epochs) and/or a threshold for the
number of tolerated misclassifications.
Definition 3.6. (Linearly separable dataset). A dataset {(x(i) , y (i) )}
is linearly separable if there exist w
b and γ such that
y (i) w
b T x(i) ≥ γ > 0, ∀ i, (3.5)
where γ is called the margin.
Figure 3.2: Linearly separable dataset.
Definition 3.7. (More formal/traditional definition). Let X and

Y be two sets of points in an m-dimensional Euclidean space. Then
X and Y are linearly separable if there exist m + 1 real numbers
m
X
w1 , w2 , · · · , wm , k such that every point x ∈ X satisfies wj xj > k and
j=1
m
X
every point y ∈ Y satisfies wj yj < k.
j=1
Theorem 3.8. Assume the data set D = {(x(i) , y (i) )} is linearly separa-
ble with margin γ, i.e.,
∃ w,
b kwk
b = 1, y (i) w
b T x(i) ≥ γ > 0, ∀ i. (3.6)
Suppose that kx(i) k ≤ R, ∀ i, for some R > 0. Then, the maximum num-
ber of mistakes made by the perceptron algorithm is bounded by R2 /γ 2 .
Proof. Assume the perceptron algorithm makes yet a mistake for (x(`) , y (`) ).
Then
kw(`+1) k2 = kw(`) + η (y (`) − yb(`) )x(`) k2
T
= kw(`) k2 + kη (y (`) − yb(`) )x(`) k2 + 2η (y (`) − yb(`) )w(`) x(`) (3.7)
≤ kw(`) k2 + kη (y (`) − yb(`) )x(`) k2 ≤ kw(`) k2 + (2η R)2 ,
where we have used

T
(y (`) − yb(`) )w(`) x(`) ≤ 0. (3.8)
(See Exercise 1.) The inequality (3.7) implies
kw(`) k2 ≤ ` · (2η R)2 . (3.9)
(Here we have used kw(0) k = 0.) On the other hand,
b T w(`+1) = w
w b T w(`) + η (y (`) − yb(`) )w
b T x(`) ≥ w
b T w(`) + 2η γ,
which implies
b T w(`) ≥ ` · (2η γ)
w (3.10)
and therefore
kw(`) k2 ≥ `2 · (2η γ)2 . (3.11)
It follows from (3.9) and (3.11) that ` ≤ R2 /γ 2 .
Properties of the perceptron algorithm: For a linearly separable

training dataset,
• Convergence: The perceptron will converge.
• Separability: Some weights get the training set perfectly correct.
3.2.3. Problems with the perceptron algorithm

Inseparable Datasets
• If the data is inseparable (due to noise, for example), there is no
guarantee for convergence or accuracy.
• Averaged perceptron is an algorithmic modification that helps with
the issue.
– Average the weight vectors, across all or a last part of iterations
Note: Frequently the training data is linearly separable! Why?

• For example, when the number of data points is much smaller than
the number of features.
– Perceptron can significantly overfit the data.
– An averaged perceptron may help with this issue, too.
Definition 3.9. Hold-out Method: Hold-out is when you split up your

dataset into a ‘train’ and ‘test’ set. The training set is what the model is
trained on, and the test set is used to see how well that model performs
on unseen data.
Optimal Separator?
Quesiton. Which of these linear separators is optimal?
Figure 3.3
Example 3.10. Support Vector Machine (Cortes & Vapnik, 1995) chooses
the linear separator with the largest margin.
Figure 3.4
We will consider the SVM in Section 5.3.

How Multi-class Classification?
Figure 3.5: Classification for three classes.
One-versus-all (one-versus-all) classification

Learning: learn 3 classifiers
• − vs {◦, +} ⇒ weights w−
• + vs {◦, −} ⇒ weights w+
• ◦ vs {+, −} ⇒ weights w◦
Prediction: for a new data sample

x,
Figure 3.6: Three weights: w− , w+ , and
w◦ . yb = arg max φ(wiT x).
i∈{−,+,◦}
OVA (OVR) is readily applicable for classification of general n classes, n ≥

2.
3.3. Adaline: ADAptive LInear NEuron 47
3.3. Adaline: ADAptive LInear NEuron

• (Widrow & Hoff, 1960)
• Weights are updated based on linear activation: e.g.,
φ(wT x) = wT x.
That is, φ is the identity function.

• Adaline algorithm is particularly interesting because it illustrates the
key concept of defining and minimizing continuous cost functions,
which will lay the groundwork for understanding more ad-
vanced machine learning algorithms for classification, such as lo-
gistic regression and support vector machines, as well as regression
models.
• Continuous cost functions allow the ML optimization to incorporate
advanced mathematical techniques such as calculus.
Figure 3.7: Perceptron vs. Adaline

3.3.1. Minimizing the cost function with the gradient de-

scent method
Key Idea 3.11. One of the key ingredients of supervised machine learn-
ing algorithms is to define an objective function that is to be optimized
during the learning process. This objective function is often a cost func-
tion that we want to minimize.
Definition 3.12. In the case of Adaline, we can define the cost function
J to learn the weights as the Sum of Squared Errors (SSE) between
the calculated outcomes and the true class labels:
1 X (i) T (i)
2
J (w) = y − φ(w x ) . (3.12)
2 i
Algorithm 3.13. The Gradient Descent Method uses −∇J (w) for
the search direction (update direction):
w := w + ∆w = w − η∇J (w), (3.13)
where η > 0 is the step length (learning rate).
Computation of ∇J (w) and ∆w :

The partial derivatives of the cost function J read
∂J (w) X
(i) T (i)

(i)
=− y − φ(w x ) xj . (3.14)
∂wj i
(Here we have used φ = I.) Thus

X
(i)
∆w = −η∇J (w) = η y − φ(w x ) x(i) .
T (i)
(3.15)
i
Note: The above Adaline rule is compared with the perceptron rule
(3.4):
w := w + ∆w, ∆w = η (y (i) − yb(i) ) x(i) .
3.3.2. Convergence and optimization issues with the gra-

dient descent method
• Depending on choices of certain algorithmic parameters, the gradi-
ent descent method may fail to converge to the the global minimizer.
• Data characteristics often determines both successability and speed
of convergence; data preprocessing operations may improve conver-
gence.
• For large-scale data, the gradient descent method is computationally
expensive; a popular alternative is the stochastic gradient descent
method.
♦ Hyperparameters
Definition 3.14. In ML, a hyperparameter is a parameter whose
value is set before the learning process begins. Thus it is an algorithmic
parameter. Examples are
• The learning rate (η)
• The number of maximum epochs/iterations (n_iter)
Figure 3.8: Well-chosen learning rate vs. a large learning rate
Hyperparameters must be selected to optimize the learning process:

• to converge fast to the global minimizer,
• avoiding overfit.
3.3.3. Feature scaling and stochastic gradient descent

Definition 3.15. Feature Scaling Preprocessing:
The gradient descent is one of the many algorithms that benefit from
feature scaling. Here, we will consider a feature scaling method called
standardization, which gives each feature of the data the property of a
standard normal distribution.
• For example, to standardize the j-th feature, we simply need to sub-
tract the sample mean µj from every training sample and divide it by
its standard deviation σj :
(i)
(i) xj − µj
x
ej = . (3.16)
σj
Then,
(i)
{e
xj | i = 1, 2, · · · , n} ∼ N (0, 1). (3.17)
Figure 3.9: Standardization, which is one of data normalization techniques.
The gradient descent method has a tendency to converge faster

with the standardized data.
Stochastic gradient descent method

Note: Earlier, we learned how to minimize a cost function with negative
gradients that are calculated from the whole training set; this is why
this approach is sometimes also referred to as batch gradient descent.
• Now imagine we have a very large dataset with millions of data points.
• Then, running with the gradient descent method can be computation-
ally quite expensive, because we need to reevaluate the whole training
dataset each time we take one step towards the global minimum.
• A popular alternative to the batch gradient descent algorithm is the
stochastic gradient descent (SGD).
Algorithm 3.16. The SGD method updates the weights incrementally

for each training sample:
Given a training set D = {(x(i) , y (i) ) | i = 1, 2, · · · , n}

1. For i = 1, 2, · · · , n (3.18)
w = w + η y (i) − φ(wT x(i) ) x(i) ;
2. If not convergent, shuffle D and goto 1;
• The SGD method updates the weights based on a single training ex-
ample.
• The SGD method typically reaches convergence much faster be-
cause of the more frequent weight updates.
• Since each search direction is calculated based on a single training
example, the error surface is smoother (not noisier) than in the gra-
dient descent method; the SGD method can escape shallow local
minima more readily.
• To obtain accurate results via the SGD method, it is important to
present it with data in a random order, which may prevent cy-
cles with epochs.
• In the SGD method, the learning rate η is often set adaptively, de-
creasing over iteration k. For example, ηk = c1 /(k + c2 ).
♦ Mini-batch learning
Definition 3.17. A compromise between the batch gradient descent and
the SGD is the so-called mini-batch learning. Mini-batch learning can
be understood as applying batch gradient descent to smaller subsets of
the training data – for example, 32 samples at a time.
The advantage over batch gradient descent is that convergence is reached

faster via mini-batches because of the more frequent weight updates.
Furthermore, mini-batch learning allows us to replace the for-loop over the
training samples in stochastic gradient descent by vectorized operations
(vectorization), which can further improve the computational efficiency of
our learning algorithm.
3.1. Verify (3.8).

Hint : We assumed that the parameter w(`) gave a mistake on x(`) . For example, let
T
w(`) x(`) ≥ 0. Then we must have (y (`) − yb(`) ) < 0. Why?
3.2. Experiment all the examples on pp. 38–51, Python Machine Learning, 3rd Ed.. Through
the examples, you will learn
(a) Gradient descent rule for Adaline,
(b) Feature scaling techniques, and
(c) Stochastic gradient descent rule for Adaline.
To get the Iris dataset, you have to use some lines on as earlier pages from 31.
3.3. Perturb the dataset (X) by a random Gaussian noise Gσ of an observable σ (so as for
Gσ (X) not to be linearly separable) and do the examples in Exercise 3.2 again.
4
C HAPTER
Gradient-based Methods for

Optimization
Optimization is the branch of research-and-development that aims to solve

the problem of finding the elements which maximize or minimize a given
real-valued function, while respecting constraints. Many problems in engi-
neering and machine learning can be cast as optimization problems, which
explains the growing importance of the field. An optimization problem is
the problem of finding the best solution from all feasible solutions.
In this chapter, we will discuss details about
• Gradient descent method,
• Newton’s method, and
• Their variants.
4.1. Gradient Descent Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
4.2. Newton’s Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
4.3. Quasi-Newton Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
4.4. The Stochastic Gradient Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
4.5. The Levenberg–Marquardt Algorithm, for Nonlinear Least-Squares Problems . . . . . 81
55
56 Chapter 4. Gradient-based Methods for Optimization
4.1. Gradient Descent Method
The first method that we will de-

scribe is one of the oldest meth-
ods in optimization: gradient de-
scent method, a.k.a steepest de-
scent method. The method was
suggested by Augustin-Louis Cauchy
in 1847 [47]. He was a French math-
ematician and physicist who made
pioneering contributions to mathe-
matical analysis. Motivated by the
need to solve “large" quadratic prob-
lems (6 variables) that arise in As-
tronomy, he invented the method of
gradient descent. Today, this method
is used to comfortably solve problems
with thousands of variables.
Figure 4.1: Augustin-Louis Cauchy
Problem 4.1. (Optimization Problem). Let Ω ⊂ Rd , d ≥ 1. Given a

real-valued function f : Ω → R, the general problem of finding the value
that minimizes f is formulated as follows.
min f (x). (4.1)

x∈Ω
In this context, f is the objective function (sometimes referred to as

loss function or cost function). Ω ⊂ Rd is the domain of the function
(also known as the constraint set).
4.1. Gradient Descent Method 57
Example 4.2. (Rosenbrock function). For example, the Rosenbrock

function in the two-dimensional (2D) space is defined as1
f (x, y) = (1 − x)2 + 100 (y − x2 )2 . (4.2)
Figure 4.2: Plots of the Rosenbrock function f (x, y) = (1 − x)2 + 100 (y − x2 )2 .
Note: The Rosenbrock function is commonly used when evaluating the

performance of an optimization algorithm; because
• its minimizer x = np.array([1.,1.]) is found in curved valley, and
so minimizing the function is non-trivial, and
• the Rosenbrock function is included in the scipy.optimize pack-
age (as rosen), as well as its gradient (rosen_der) and its Hessian
(rosen_hess).
1
The Rosenbrock function in 3D is given as f (x, y, z) = [(1 − x)2 + 100 (y − x2 )2 ] + [(1 − y)2 + 100 (z − y 2 )2 ],
which has exactly one minimum at (1, 1, 1). Similarly, one can define the Rosenbrock function in gen-
eral N -dimensional spaces, for N ≥ 4, by adding one more component for each enlarged dimension.
N
X −1
(1 − xi )2 + 100(xi+1 − x2i )2 , where x = [x1 , x2 , · · · , xN ] ∈ RN . See Wikipedia

That is, f (x) =
i=1
(https://en.wikipedia.org/wiki/Rosenbrock_function) for details.
Remark 4.3. (Gradient). The gradient ∇f is a vector (a direction to

move) that is
• pointing in the direction of greatest increase of the function, and
• zero (∇f = 0) at local maxima or local minima.
The goal of the gradient descent method is to address directly the process
of minimizing the function f , using the fact that −∇f (x) is the direction of
steepest descent of f at x. Given an initial point x0 , we move it to the
direction of −∇f (x0 ) so as to get a smaller function value. That is,
x1 = x0 − γ ∇f (x0 ) ⇒ f (x1 ) < f (x0 ).
We repeat this process till reaching at a desirable minimum. Thus the

method is formulated as follows.
Algorithm 4.4. (Gradient descent method). Given an initial point

x0 , find iterates xn+1 recursively using
xn+1 = xn − γ ∇f (xn ), (4.3)
for some γ > 0. The parameter γ is called the step length or the learn-
ing rate.
To understand the basics of gradient descent (GD) method thoroughly, we

start with the algorithm for solving
• unconstrained minimization problems

• defined in the one-dimensional (1D) space.
4.1.1. The gradient descent method in 1D
Problem 4.5. Consider the minimization problem in 1D:
min f (x), x ∈ S, (4.4)

x
where S is a closed interval in R. Then its gradient descent method reads
xn+1 = xn − γ f 0 (xn ). (4.5)
Picking the step length γ : Assume that the step length was chosen to
be independent of n, although one can play with other choices as well. The
question is how to select γ in order to make the best gain of the method. To
turn the right-hand side of (4.5) into a more manageable form, we invoke
Taylor’s Theorem:2
ˆ x+t
0
f (x + t) = f (x) + t f (x) + (x + t − s) f 00 (s) ds. (4.6)
x
00
Assuming that |f (s)| ≤ L, we have
t2
f (x + t) ≤ f (x) + t f 0 (x) + L.
2
Now, letting x = xn and t = −γ f 0 (xn ) reads
f (xn+1 ) = f (xn − γ f 0 (xn ))
1
≤ f (xn ) − γ f 0 (xn ) f 0 (xn ) + L [γ f 0 (xn )]2 (4.7)
2
L
= f (xn ) − [f 0 (xn )]2 γ − γ 2 .
2
The gain (learning) from the method occurs when
L 2
γ − γ2 > 0 ⇒ 0 < γ < , (4.8)
2 L
and it will be best when γ − L2 γ 2 is maximal. This happens at the point
1
γ= . (4.9)
L
2
Taylor’s Theorem with integral remainder: Suppose f ∈ C n+1 [a, b] and x0 ∈ [a, b]. Then, for every
n ˆ
X f (k) (x0 ) 1 x
x ∈ [a, b], f (x) = (x − x0 )k + Rn (x), Rn (x) = (x − s)n f (n+1) (s) ds.
k! n! x0
k=0
Thus an effective gradient descent method (4.5) can be written as

1 0 1
xn+1 = xn − γ f 0 (xn ) = xn − f (xn ) = xn − 00
f 0 (xn ). (4.10)
L max |f (x)|
Furthermore, it follows from (4.7) and (4.9) that
1 0
f (xn+1 ) ≤ f (xn ) − [f (xn )]2 . (4.11)
2L
Remark 4.6. (Convergence of gradient descent method).

Thus it is obvious that the method defines a sequence of points {xn } along
which {f (xn )} decreases.
• If f is bounded from below and the level sets of f are bounded,
{f (xn )} converges; so does {xn }. That is, there is a point x
b such
that
lim xn = x
b. (4.12)
n→∞
• Now, we can rewrite (4.11) as
[f 0 (xn )]2 ≤ 2L [f (xn ) − f (xn+1 )]. (4.13)
Since f (xn ) − f (xn+1 ) → 0, also f 0 (xn ) → 0.

• When f 0 is continuous, using (4.12) reads
f 0 (b
x) = lim f 0 (xn ) = 0, (4.14)
n→∞
which implies that the limit x

b is a critical point.
• The method thus generally finds a critical point but that could still
be a local minimum or a saddle point. Which it is cannot be decided
at this level of analysis.
4.1.2. The full gradient descent algorithm

We can implement the full gradient descent algorithm as follows. The algo-
rithm has only one free parameter: γ.
Algorithm 4.7. (The Gradient Descent Algorithm).
input: initial guess x0 , step size γ > 0;

for n = 0, 1, 2, · · · do
xn+1 = xn − γ ∇f (xn ); (4.15)
end for
return xn+1 ;
Remark 4.8. In theory, the step length γ can be found as in (4.9):

1
γ= , where L = max k∇2 f (x)k. (4.16)
L x
Here || · || denotes an induced matrix norm and ∇2 f (x) is the Hessian

of f defined by
 2 
∂ f ∂ 2f ∂ 2f
 ∂x 2 ∂x ∂x · · · ∂x ∂x 
 1 1 2 1 d
 2 2 2
 ∂ f ∂ f ∂ f 

···
∇2 f =  ∂x2 ∂x1 ∂x2 2 ∂x2 ∂xd  ∈ Rd×d .
 
 . (4.17)
 .. .. . .. .
.. 

 . 
 2 
 ∂ f ∂ 2f ∂ 2f 
···
∂xd ∂x1 ∂xd ∂x2 ∂xd 2
• However, in practice, the computation of the Hessian (and L) can be
expensive.
Remark 4.9. Gradient Descent vs. Newton’s Method

The gradient descent method can be viewed as a simplification of the
Newton’s method (Section 4.2 below), replacing the inverse of Hessian,
(∇2 f )−1 , with a constant γ.
Convergence of Gradient Descent: Constant γ

Here we examine convergence of gradient descent on three examples: a
well-conditioned quadratic, an poorly-conditioned quadratic, and a non-
convex function, as shown by Dr. Fabian Pedregosa, UC Berkeley.
γ = 0.2
Figure 4.3: On a well-conditioned quadratic function, the gradient descent converges in a
few iterations to the optimum
γ = 0.02
Figure 4.4: On a poorly-conditioned quadratic function, the gradient descent converges and
takes many more iterations to converge than on the above well-conditioned problem. This
is partially because gradient descent requires a much smaller step size on this problem
to converge.
γ = 0.02
Figure 4.5: Gradient descent also converges on a poorly-conditioned non-convex problem.
Convergence is slow in this case.
The Choice of Step Size: Backtracking Line Search

Note: The convergence of the gradient descent method can be extremely
sensitive to the choice of step size. It often requires to choose the
step size adaptively: the step size would better be chosen small in re-
gions of large variability of the gradient, while in regions with small
variability we would like to take it large.
Strategy 4.10. Backtracking line search procedures allow to select

a step size depending on the current iterate and the gradient. In this
procedure, we select an initial (optimistic) step size γn and evaluate the
following inequality (known as sufficient decrease condition):
γn
f (xn − γn ∇f (xn )) ≤ f (xn ) − k∇f (xn )k2 . (4.18)
2
If this inequality is verified, the current step size is kept. If not, the step
size is divided by 2 (or any number larger than 1) repeatedly until (4.18)
is verified. To get a better understanding, refer to (4.11) on p. 60, with
(4.9).
The gradient descent algorithm with backtracking line search then becomes
Algorithm 4.11. (The Gradient Descent Algorithm, with Back-
tracking Line Search).
input: initial guess x0 , step size γ0 > 0;
for n = 0, 1, 2, · · · do

initial step size estimate γn ;
 while (TRUE) do
γn


 if f (xn − γn ∇f (xn )) ≤ f (xn ) − 2 k∇f (xn )k2

 break; (4.19)
 else γn = γn /2;
end while
xn+1 = xn − γn ∇f (xn );
end for
return xn+1 ;
Convergence of Gradient Descent: Backtracking line search

The following examples show the convergence of gradient descent with the
aforementioned backtracking line search strategy for the step size.
Figure 4.6: On a well-conditioned quadratic function, the gradient descent converges in a

few iterations to the optimum. Adding the backtracking line search strategy for the step
size does not change much in this case.
Figure 4.7: In this example we can clearly see the effect of the backtracking line search
strategy: once the algorithm in a region of low curvature, it can take larger step sizes. The
final result is a much improved convergence compared with the fixed step-size equivalent.
Figure 4.8: The backtracking line search also improves convergence on non-convex prob-
lems.
See Exercise 1 on p. 86.

4.1.3. Surrogate minimization: A unifying principle

Now, we aim to solve an optimization problem as in (4.1):
min f (x). (4.20)

x∈Ω
Key Idea 4.12. Start at an initial estimate x0 and successively mini-

mize an approximating function Qn (x) [43]:
xn+1 = arg min Qn (x). (4.21)

x∈Ω
We will call Qn a surrogate function. It is also known as a merit

function. A good surrogate function should be:
• Easy to optimize.
• Flexible enough to approximate a wide range of functions.
Gradient descent method: Approximates the objective function near xn

with a quadratic surrogate of the form
1
Qn (x) = cn + Gn · (x − xn ) + (x − xn )T (x − xn ), (4.22)
2γ
which coincides with f in its value and first derivative, i.e.,
Qn (xn ) = f (xn ) ⇒ cn = f (xn ),
(4.23)
∇Qn (xn ) = ∇f (xn ) ⇒ Gn = ∇f (xn ).
The gradient descent method thus updates its iterates minimizing the fol-
lowing surrogate function:
1
Qn (x) = f (xn ) + ∇f (xn ) · (x − xn ) + kx − xn k2 . (4.24)
2γ
Differentiating the function and equating to zero reads
xn+1 = arg min Qn (x) = xn − γ ∇f (xn ). (4.25)

x
Multiple Local Minima Problem
Remark 4.13. Optimizing Optimization Algorithms

Although you can choose the step size smartly, there is no guarantee for
your algorithm to converge to the desired solution (the global minimum),
particularly when the objective is not convex.
Here, we consider the so-called Gaussian homotopy continuation
method [53], which may overcome the local minima problem for cer-
tain classes of optimization problems.
• The method begins by trying to find a convex approximation of an op-
timization problem, using a technique called Gaussian smoothing.
• Gaussian smoothing converts the cost function into a related function,
each of whose values is a weighted average of all the surrounding
values.
• This has the effect of smoothing out any abrupt dips or ascents in the
cost function’s graph, as shown in Figure 4.9.
• The weights assigned the surrounding values are determined by a
Gaussian function, or normal distribution.
Figure 4.9: Smooth sailing, through a Gaussian smoothing.
However, there will be many ways to incorporate Gaussian smoothing; a

realization of the method will be challenging, particularly for ML optimiza-
tion. See P.3 (p. 379).
4.2. Newton’s Method 67
4.2. Newton’s Method

4.2.1. Derivation
Scheme 4.14. The Newton’s method is an iterative method to solve
the unconstrained optimization problem in (4.1), p. 56, when f is twice
differentiable. In Newton’s method, we approximate the objective with a
quadratic surrogate of the form
1
Qn (x) = cn + Gn · (x − xn ) + (x − xn )T H n (x − xn ). (4.26)
2γ
Compared with gradient descent, the quadratic term is not fixed to be
the identity but instead incorporates an invertible matrix H n .
• A reasonable condition to impose on this surrogate function is that at

xn it coincides with f at least in its value and first derivatives, as
in (4.23).
• An extra condition the method imposes is that
H n = ∇2 f (xn ), (4.27)
where ∇2 f is the Hessian of f defined as in (4.17).

• Thus the Newton’s method updates its iterates minimizing the fol-
lowing surrogate function:
1
Qn (x) = f (xn ) + ∇f (xn ) · (x − xn ) + (x − xn )T ∇2 f (xn ) (x − xn ).
2γ
(4.28)
• We can find the optimum of the function differentiating and equat-
ing to zero. This way we find (assuming the Hessian is invertible)
−1
xn+1 = arg min Qn (x) = xn − γ ∇2 f (xn )

∇f (xn ). (4.29)
x
Note: When γ = 1, Qn (x) in (4.28) is the second-order approximation of

the objective function near xn .
Remark 4.15. Where applicable, Newton’s method converges much

faster towards a local maximum or minimum than the gradient descent.
• In fact, every local minimum has a neighborhood such that, if we start
within this neighborhood, Newton’s method with step size γ = 1 con-
verges quadratically assuming the Hessian is invertible and Lips-
chitz continuous.
Remark 4.16. The Newton’s method can be seen as to find the critical
b such that ∇f (b
points of f , i.e., x x) = 0. Let
xn+1 = xn + ∆x. (4.30)
Then
∇f (xn+1 ) = ∇f (xn + ∆x) = ∇f (xn ) + ∇2 f (xn ) ∆x + O(|∆x|2 ).
Truncating high-order terms of ∆x and equating the result to zero reads

−1
∆x = − ∇2 f (xn ) ∇f (xn ). (4.31)
Implementation of Newton’s Method

Only the difference from the gradient descent algorithm is to compute the
Hessian matrix ∇2 f (xn ) to be applied to the gradient.
Algorithm 4.17. (Newton’s method).
input: initial guess x0 , step size γ > 0;

for n = 0, 1, 2, · · · do
−1
xn+1 = xn − γ ∇2 f (xn ) ∇f (xn ); (4.32)
end for
return xn+1 ;
For the three example functions in Section 4.1.2, the Newton’s method per-
forms better as shown in the following.
γ=1
Figure 4.10: In this case the approximation is exact and it converges in a single iteration.
γ=1
Figure 4.11: Although badly-conditioned, the cost function is quadratic; it converges in a
single iteration.
γ=1
Figure 4.12: When the Hessian is close to singular, there might be some numerical insta-
bilities. However, it is better than the result of the gradient descent method in Figure 4.5.
4.2.2. Hessian and principal curvatures
Claim 4.18. The Hessian (or Hessian matrix) describes the local
curvature of a function. The eigenvalues and eigenvectors of the Hes-
sian have geometric meaning:
• The first principal eigenvector (corresponding to the largest eigen-
value in modulus) is the direction of greatest curvature.
• The last principal eigenvector (corresponding to the smallest eigen-
value in modulus) is the direction of least curvature.
• The corresponding eigenvalues are the respective amounts of these
curvatures.
The eigenvectors of the Hessian are called principal directions, which
are always orthogonal to each other. The eigenvalues of the Hessian
are called principal curvatures and are invariant under rotation and
always real-valued.
Observation 4.19. Let a Hessian matrix H ∈ Rd×d be positive definite

and its eigenvalue-eigenvector pairs be given as {(λj , uj )}, j = 1, 2, · · · , d.
• Then, given a vector v ∈ Rd , it can be expressed as
d
X
v= ξj uj ,
j=1
and therefore
d
−1
X 1
H v = ξj uj , (4.33)
j=1
λj
where components of v in leading principal directions of H have
been diminished with larger factors.
• Thus the angle measured from H −1 v to the least principal direc-
tion of H becomes smaller than the angle measured from v.
• It is also true when v is the gradient vector (in fact, the negation of
the gradient vector).
Note: The above observation can be rephrased mathematically as fol-

lows. Let ud be the least principal direction of H. Then
angle(ud , H −1 v) < angle(ud , v), ∀ v, (4.34)
where a·b
angle(a, b) = arccos .
kak kbk
This implies that by setting v = −∇f (xn ), the adjusted vector H −1 v is
a rotation (and scaling) of the steepest descent vector towards the least
curvature direction.
Figure 4.13: The effect of the Hessian inverse H −1 .
Claim 4.20. The net effect of H −1 is to rotate and scale the gradi-
ent vector to face towards the minimizer by a certain degree, which may
make the Newton’s method converge much faster than the gradient de-
scent method.
Example 4.21. One can easily check that at each point (x, y) on the ellip-
soid
(x − h)2 (y − k)2
z = f (x, y) = + , (4.35)
a2 b2
−1
the vector − ∇2 f (x, y) ∇f (x, y) is always facing towards the minimizer
(h, k). See Exercise 2.
4.3. Quasi-Newton Methods

Note: The central issue with Newton’s method is that we need to be able
to compute the inverse Hessian matrix.
• For ML applications, the dimensionality of the problem can be of
the order of thousands or millions; computing the Hessian or its
inverse is often impractical.
• Because of these reasons, Newton’s method is rarely used in prac-
tice to optimize functions corresponding to large problems.
• Luckily, Newton’s method can still work even if the Hessian is re-
placed by a good approximation.
The BFGS Algorithm (1970)

Note: One of the most popular quasi-Newton methods is the BFGS
algorithm, which is named after Charles George Broyden [9], Roger
Fletcher [21], Donald Goldfarb [24], and David Shanno [72].
Key Idea 4.22. As a byproduct of the optimization, we observe many

gradients. Can we use these gradients to iteratively construct an ap-
proximation of the Hessian?
Derivation of BFGS algorithm

• At each iteration of the method, we consider the surrogate function:
1
Qn (x) = cn + Gn · (x − xn ) + (x − xn )T H n (x − xn ), (4.36)
2
where in this case H n is an approximation to the Hessian matrix,
which is updated iteratively at each stage.
• A reasonable thing to ask to this surrogate is that its gradient coin-
cides with ∇f at the last two iterates xn+1 and xn :
∇Qn+1 (xn+1 ) = ∇f (xn+1 ),
(4.37)
∇Qn+1 (xn ) = ∇f (xn ).
4.3. Quasi-Newton Methods 73
• From the definition of Qn+1 :

1
Qn+1 (x) = cn+1 + Gn+1 · (x − xn+1 ) + (x − xn+1 )T H n+1 (x − xn+1 ),
2
we have
∇Qn+1 (xn+1 ) − ∇Qn+1 (xn ) = Gn+1 − ∇Qn+1 (xn ) = −H n+1 (xn − xn+1 ).
Thus we reach at the following condition on H n+1 :
H n+1 (xn+1 − xn ) = ∇f (xn+1 ) − ∇f (xn ), (4.38)
which is the secant equation.
• Let
sn = xn+1 − xn and yn = ∇f (xn+1 ) − ∇f (xn ).
Then H n+1 sn = yn , which requires to satisfy the curvature condition
yn · sn > 0, (4.39)
with which H n+1 becomes positive definite. (Pre-multiply sTn to the secant
equation to prove it.)
• In order to maintain the symmetry and positive definiteness of H n+1 ,
the update formula can be chosen as3
H n+1 = H n + αuuT + βvvT . (4.40)
• Imposing the secant condition H n+1 sn = yn and with (4.40), we get the
update equation of H n+1 :
yn ynT (H n sn )(H n sn )T
H n+1 = Hn + − . (4.41)
y n · sn sn · H n sn
• Let B n = H −1
n , the inverse of H n . Then, applying the Sherman-Morrison
formula, we can update B n+1 = H −1 n+1 as follows.
sn ynT yn sTn sn sTn
B n+1 = I− Bn I − + . (4.42)
yn · sn yn · sn yn · sn
See Exercise 4.4.
3
Rank-one matrices: Let A be an m × n matrix. Then rank(A) = 1 if and only if there exist column
vectors v ∈ Rm and w ∈ Rn such that A = vwT .
Now, we are ready to summarize the BFGS algorithm.

Algorithm 4.23. (The BFGS algorithm). The n-th step:
1. Obtain the search direction: pn = B n (−∇f (xn )).
2. Perform line-search to find an acceptable stepsize γn .
3. Set sn = γn pn and update xn+1 = xn + sn .
4. Get yn = ∇f (xn+1 ) − ∇f (xn ).
5. Update B = H −1 :
B n+1 = I− Bn I − + .
Remark 4.24. The BFGS Algorithm

• The algorithm begins with B0 , an estimation of H0−1 . It is often better
when B0 = H0−1 .
• The resulting algorithm is a method which combines the low-cost of
gradient descent with the favorable convergence properties of
Newton’s method.
4.3. Quasi-Newton Methods 75
Examples, with the BFGS algorithm
Figure 4.14: BFGS, on the well-conditioned quadratic objective function.
Figure 4.15: On the poorly-conditioned quadratic problem, the BFGS algorithm

quickly builds a good estimator of the Hessian and is able to converge very fast towards
the optimum. Note that this, just like the Newton method (and unlike gradient descent),
BFGS does not seem to be affected (much) by a bad conditioning of the problem.
Figure 4.16: Even on the ill-conditioned nonconvex problem, the BFGS algorithm also
converges extremely fast, with a convergence that is more similar to Newton’s method than
to gradient descent.
4.4. The Stochastic Gradient Method

The stochastic gradient method (SGM), introduced by Robbins-Monro
in 1951 [63], is
• one of the most widely-used methods for large-scale optimization, and
• one of the main methods behind the current AI revolution.
Note: The SGM was considered earlier in Section 3.3.2, as a variant

of the gradient descent method for Adaline classification. Here we will
discuss it in details for more general optimization problems.
• The stochastic gradient method (a.k.a. stochastic gradient descent

or SGD) can be used to solve optimization problems in which the ob-
jective function is of the form
f (x) = E[fi (x)],
where the expectation is taken with respect to i.

• The most common case is when i can take a finite number of values,
in which the problem becomes
m
1 X
min f (x), f (x) = fi (x). (4.43)
x∈Rp m i=1
• The SGM can be motivated as an approximation to gradient descent

in which at each iteration we approximate the gradient as
∇f (xn ) ≈ ∇fi (xn ). (4.44)

4.4. The Stochastic Gradient Method 77
We can write the full stochastic gradient algorithm as follows. The algo-
rithm has only one free parameter: γ.
Algorithm 4.25. (Stochastic Gradient Descent).
input: initial guess x0 , step size sequence γn > 0;

for n = 0, 1, 2, · · · do
Choose i ∈ {1, 2, · · · , m} uniformly at random;
(4.45)
xn+1 = xn − γn ∇fi (xn );
end for
return xn+1 ;
The SGD can be much more efficient than gradient descent in the
case in which the objective consists of a large sum, because at each
iteration we only need to evaluate a partial gradient and not the full gradi-
ent.
Example 4.26. A least-squares problem can be written in the form accept-
able by SGD since
m
1 2 1 X
kAx − bk = (Ai x − bi )2 , (4.46)
m m i=1
where Ai is the i-th row of A.

Step Size for the SGD

• The choice of step size is one of the most delicate aspects of the SGD.
For the SGD, the backtracking line search is not an option since
it would involve to evaluate the objective function at each iteration,
which destroys the computational advantage of this method.
• Two popular step size strategies exist for the SGD: constant step
size and decreasing step size.
(a) Constant step size: In the constant step size strategy,
γn = γ
for some pre-determined constant γ.

The method converges very fast to neighborhood of a local mini-
mum and then bounces around. The radius of this neighborhood
will depend on the step size γ [44, 51].
(b) Decreasing step size: One can guarantee convergence to a local
minimizer choosing a step size sequence that satisfies
∞
X ∞
X
γn = ∞ and γn2 < ∞. (4.47)
n=1 n=1
The most popular sequence to verify this is

C
γn = , (4.48)
n
for some constant C. This is often referred to as a decreasing
step-size sequence, although in fact the sequence does not need
to be monotonically decreasing.
4.4. The Stochastic Gradient Method 79
Examples, with SGD
γ = 0.2
Figure 4.17: For the well-conditioned convex problem, stochastic gradient with constant
step size converges quickly to a neighborhood of the optimum, but then bounces around.
Figure 4.18: Stochastic Gradient with decreasing step sizes is quite robust to the choice
of step size. On one hand there is really no good way to set the step size (e.g., no equivalent
of line search for Gradient Descent) but on the other hand it converges for a wide range of
step sizes.
Convergence of the SGD

Quesiton. Why does the SGD converge, despite its update being a very
rough estimate of the gradient?
To answer this question, we must first understand the unbiasedness prop-

erty of its update.
Proposition 4.27. (Unbiasedness of the SGD update). Let En de-

note the expectation with respect to the choice of random sample (i) at
iteration n. Then since the index i is chosen uniformly at random, we
have m
X
En [∇fin (xn )] = ∇fi (xn )P (in = i)
i=1
m (4.49)
1 X
= ∇fi (xn ) = ∇f (xn )
m i=1
This is the crucial property that makes SGD work. For a full proof, see
e.g. [7].
4.5. The Levenberg–Marquardt Algorithm, for Nonlinear Least-Squares Problems 81
4.5. The Levenberg–Marquardt Algorithm, for

Nonlinear Least-Squares Problems
The Levenberg-Marquardt algorithm (LMA), a.k.a. the damped
least-squares (DLS) method, is used for the solution of nonlinear
least-squares problems which arise especially in curve fitting.
• In fitting a function yb(x; p) of an independent variable x and a pa-
rameter vector p ∈ Rn to a set of m data points (xi , yi ), it is customary
and convenient to minimize the sum of the weighted squares of
the errors (or weighted residuals) between the measured data yi
and the curve-fit function yb(xi ; p).
m h
X yi − yb(xi ; p) i2
f (p) =
i=1
ηi (4.50)
= (y − y b(p))T W (y − y b(p))
where ηi is the measurement error for yi and the weighting matrix

W is defined as
W = diag{1/ηi2 } ∈ Rm×m .
• However, more formally, W can be set to the inverse of the mea-

surement error covariance matrix; more generally, the weights can
be set to pursue other curve-fitting goals.
Definition 4.28. The measurement error (also called the observa-

tional error) is the difference between a measured quantity and its true
value. It includes random error and systematic error (caused by a
mis-calibrated instrument that affects all measurements).
Note: The goodness-of-fit measure in (4.50) is called the chi-squared

error criterion because the sum of squares of normally-distributed
variables is distributed as the χ-squared distribution.
If the function yb(x; p) is nonlinear in the model parameters p, then the

minimization of the χ-squared function f with respect to the parame-
ters must be carried out iteratively:
p := p + ∆p. (4.51)
The goal of each iteration is to find the parameter update ∆p that re-
duces f . We will begin with the gradient descent method and the Gauss-
Newton method.
4.5.1. The gradient descent method
Recall: The gradient descent method is a general minimization method

which updates parameter values in the “steepest downhill” direction:
the direction opposite to the gradient of the objective function.
• The gradient descent method converges well for problems with sim-
ple objective functions.
• For problems with thousands of parameters, gradient descent meth-
ods are sometimes the only workable choice.
The gradient of the objective function with respect to the parameters is

∂ ∂
b(p))T W
f = 2(y − y (y − y
b(p))
∂p ∂p
h ∂b
y(p) i
T (4.52)
= −2(y − yb(p)) W
∂p
T
= −2(y − yb(p)) W J ,
where J = ∂by∂p(p) ∈ Rm×n is the Jacobian matrix. The parameter update

∆p that moves the parameters in the direction of steepest descent is given
by
∆pgd = γ J T W (y − y
b(p)), (4.53)
where γ > 0 is the step length.

4.5.2. The Gauss-Newton method

The Gauss-Newton method is a method for minimizing a sum-of-
squares objective function.
• It assumes that the objective function is approximately quadratic
near the minimizer [6], and utilizes an approximate Hessian.
• For moderately-sized problems, the Gauss-Newton method typically
converges much faster than gradient-descent methods [52].
Algorithm Derivation
• The function evaluated with perturbed model parameters may be locally

approximated through a first-order Taylor series expansion.
h ∂b
y(p) i
b(p + ∆p) ≈ y
y b(p) + ∆p = y
b(p) + J ∆p. (4.54)
∂p
• Substituting the approximation into (4.50), p. 81, we have

f (p + ∆p) ≈ yT W y − 2yT W y b(p)T W y
b(p) + y b(p)
(4.55)
b(p))T W J ∆p + (J ∆p)T W J ∆p.
−2(y − y
Note: The above approximation for f (p + ∆p) is quadratic in the pa-

rameter perturbation ∆p.
• The parameter update ∆p can be found from ∂f /∂∆p = 0:
∂
b(p))T W J + 2(J ∆p)T W J = 0,
f (p + ∆p) ≈ −2(y − y (4.56)
∂∆p
and therefore the resulting normal equation for the Gauss-Newton up-
date reads
[J T W J ] ∆pgn = J T W (y − y
b(p)). (4.57)
Note: The matrix J T W J ∈ Rn×n is an approximate Hessian of the

objective function. Here, we require m ≥ n; otherwise, the approximate
Hessian must be singular.
4.5.3. The Levenberg-Marquardt algorithm

The Levenberg-Marquardt algorithm adaptively varies the parameter
updates between the gradient descent and the Gauss-Newton methods:
[J T W J + λI] ∆plm = J T W (y − y
b(p)), (4.58)
where λ ≥ 0 is the damping parameter. Small values of λ result in a

Gauss-Newton update and large values of it result in a gradient descent
update.
Remark 4.29. Implementation of the Levenberg-Marquardt Al-

gorithm.
• The damping parameter λ is often initialized to be large so that
first updates are small steps in the steepest-descent direction.
• As the solution improves, λ is decreased; the Levenberg-
Marquardt method approaches the Gauss-Newton method, and the
solution typically accelerates to the local minimum [49, 52].
• If any iteration happens to result in a bad approximation, e.g.,
f (p + ∆plm ) > f (p),
then λ is increased.
Acceptance of the Step

There have been many variations of the Levenberg-Marquardt method, par-
ticularly for acceptance criteria.
Example 4.30. Acceptance Criterion. Recall (4.50) and (4.54):

T
f (p) = y−y b(p) W y − y b(p) , (4.50)
b(p + ∆plm ) ≈ y
y b(p) + J ∆plm . (4.54)
Then the Sum of Squared Error (SSE), f (p + ∆plm ), can be approximated

by
T
f (p + ∆plm ) = y − y
b(p + ∆plm ) W y − yb(p + ∆plm )
T (4.59)
≈ y − [b
y(p) + J ∆plm ] W y − [b
y(p) + J ∆plm ] .
• At the k-th step, we first compute

f (p) − f (p + ∆plm )
ρk (∆plm ) = T
f (p) − y − y
b − J ∆plm W y − y b − J ∆plm
(4.60)
f (p) − f (p + ∆plm )
= . [⇐ (4.58)]
∆pTlm (λk ∆plm + J T W (y − yb(p))
• Then the step is accepted when ρk (∆plm ) > ε0 , for a threshold ε0 > 0.
An example implementation reads


Initialize p0 , λ0 , and ε0 ; (e.g. λ0 = 0.01 & ε0 = 0.1)
 Compute ∆plm from (4.58);


 Evaluate ρk from (4.60);

 If ρ > ε : (4.61)
 k 0
pk+1 = pk + ∆plm ; λk+1 = λk · max[1/3, 1 − (2ρk )3 )]; νk = 2;


otherwise: λk+1 = λk νk ; νk+1 = 2νk ;
4.1. (Gradient descent method). Implement the gradient descent algorithm (4.15) and
the gradient descent algorithm with backtracking line search (4.19).
(a) Compare their performances with the Rosenbrock function in 2D (4.2).

(b) Find an effective strategy for initial step size estimate for (4.19).
4.2. (Net effect of the inverse Hessian matrix). Verify the claim in Example 4.21.
4.3. (Newton’s method). Implement a line search version of the Newton’s method (4.32)
with the Rosenbrock function in 2D.
(a) Recall the results in Exercise 1. With the backtracking line search, is the New-
ton’s method better than the gradient descent method?
(b) Now, we will approximate the Hessian matrix by its diagonal. That is,
 2  2
∂ f ∂ f ∂ 2f
 
 ∂x2 0  2
Dn =   ∂x
def 
(xn ) ≈ ∇2 f (xn ) == ∂x∂y (xn ). (4.62)
∂ 2f   ∂ 2f ∂ 2f 
0
∂y 2 ∂y∂x ∂y 2
How does the Newton’s method perform when the Hessian matrix is replaced by
Dn ?
4.4. (BFGS update). Consider H n+1 and B n+1 in (4.41) and (4.42), respectively:
yn ynT (H n sn )(H n sn )T
H n+1 = H n + − ,
yn · sn sn · H n sn
B n+1 = I− Bn I − + .
(a) Verify the the secant condition H n+1 sn = yn .

(b) Verify H n+1 B n+1 = I, assuming that H n B n = I.
Continued on the next page =⇒

4.5. (Curve fitting; Optional for undergraduates). Consider a set of data consisting
of four points
1 2 3 4
xi 0.0 1.0 2.0 3.0
yi 1.1 2.6 7.2 21.1
Fit the data with a fitting function of the form
yb(x, p) = a ebx , where p = [a, b], (4.63)
by minimizing the sum of the square-errors:
(a) Implement the three algorithms introduced in Section 4.5: the gradient descent
method, the Gauss-Newton method, and the Levenberg-Marquardt method.
(b) Ignore the weight vector W , i.e., set W = I.
(c) For each method, set p0 = [a0 , b0 ] = [1.0, 0.8].
(d) Discuss how to choose γ for the gradient descent and λ for the Levenberg-Marquardt.
Hint : The Jacobian for this example must be in R4×2 ; more precisely,
 
1 0
∂  eb a eb 
J= y
b(x, p) = 
 e2b 2a e2b  ,

∂p
e3b 3a e3b
b(x, p) = [a, a eb , a e2b , a e3b ]T from (4.63) and {xi }.

because we have y
5
C HAPTER
Popular Machine Learning Classifiers
In this chapter, we will study a selection of popular and powerful machine

learning algorithms, which are commonly used in academia as well as in
the industry. While learning about the differences between several super-
vised learning algorithms for classification, we will also develop an intuitive
appreciation of their individual strengths and weaknesses.
The topics that we will learn about throughout this chapter are as follows:
• Introduction to the concepts of popular classification algorithms such

as logistic regression, support vector machine (SVM), decision
trees, and k-nearest neighbors.
• Questions to ask when selecting a machine learning algorithm
• Discussions about the strengths and weaknesses of classifiers with lin-
ear and nonlinear decision boundaries
5.1. Logistic Sigmoid Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
5.2. Classification via Logistic Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
5.3. Support Vector Machine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
5.4. Decision Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
5.5. k-Nearest Neighbors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
89
90 Chapter 5. Popular Machine Learning Classifiers
Choosing a classification algorithm

Choosing an appropriate classification algorithm for a particular prob-
lem task requires practice:
• Each algorithm has its own quirks/characteristics and is based
on certain assumptions.
• No Free Lunch theorem: No single classifier works best across all
possible scenarios.
• In practice, it is recommended that you compare the perfor-
mance of at least a handful of different learning algorithms
to select the best model for the particular problem.
Eventually, the performance of a classifier, computational power as well
as predictive power, depends heavily on the underlying data that are
available for learning. The five main steps that are involved in training
a machine learning algorithm can be summarized as follows:
1. Selection of features.
2. Choosing a performance metric.
3. Choosing a classifier and optimization algorithm.
4. Evaluating the performance of the model.
5. Tuning the algorithm.
5.1. Logistic Sigmoid Function 91
5.1. Logistic Sigmoid Function

A logistic sigmoid function (or logistic curve) is a common “S" shape
curve with equation:
L
f (x) = −k(x−x
, (5.1)
1+e 0)
where L denotes the curve’s maximum value, x0 is the sigmoid’s midpoint,

and k is the logistic growth rate or steepness of the curve.
In statistics, the logistic model is a widely used statistical model that
uses a logistic function to model a binary dependent variable; many more
complex extensions exist.
5.1.1. The standard logistic sigmoid function

Setting L = 1, k = 1, and x0 = 0 gives the standard logistic sigmoid
function:
1
s(x) = . (5.2)
1 + e−x
Figure 5.1: Standard logistic sigmoid function s(x) = 1/(1 + e−x ).

Remark 5.1. (The standard logistic sigmoid function):

1 ex
s(x) = =
1 + e−x 1 + ex
• The standard logistic function is the solution of the simple first-
order non-linear ordinary differential equation
d 1
y = y(1 − y), y(0) = . (5.3)
dx 2
It can be verified easily as
0 ex (1 + ex ) − ex · ex ex
s (x) = = = s(x)(1 − s(x)). (5.4)
(1 + ex )2 (1 + ex )2
• s0 is even: s0 (−x) = s0 (x).

• Rotational symmetry about (0, 1/2):
1 1 2 + ex + e−x
s(x) + s(−x) = + = ≡ 1. (5.5)
1 + e−x 1 + ex 2 + ex + e−x
ˆ ˆ
ex
• s(x) dx = x
dx = ln(1 + ex ), which is known as the softplus
1+e
function in artificial neural networks. It is a smooth approxi-
mation of the the rectifier (an activation function) defined as
f (x) = x+ = max(x, 0). (5.6)
Figure 5.2: The rectifier and it smooth approximation, softplus function ln(1 + ex ).
5.1. Logistic Sigmoid Function 93
5.1.2. The logit function

Logistic regression uses the sigmoid function for activation. We first wish to
explain the idea behind logistic regression as a probabilistic model.
• Let p be the probability of a particular event (having class label y = 1).
• Then the odds ratio of the particular event is defined as
p
.
1−p
• We can then define the logit function, which is simply the logarithm
of the odds ratio (log-odds):
p
logit(p) = ln . (5.7)
1−p
• The logit function takes input values in (0, 1) and transforms them to
values over the entire real line,
which we can use to express a linear relationship between fea-
ture values and the log-odds:
logit(p(y = 1|x)) = w0 x0 + w1 x1 + · · · + wm xm = wT x, (5.8)
where p(y = 1|x) is the conditional probability that a particular sample

(given its features x) belongs to class 1.
Remark 5.2. What we are actually interested in is
predicting the probability
that a certain sample belongs to a particular class, which is the inverse

form of the logit function:
p(y = 1|x) = logit−1 (wT x). (5.9)
Quesiton. What is the inverse of the logit function?

Example 5.3. Find the inverse of the logit function

p
logit(p) = ln .
1−p
Solution.
1
Ans: logit−1 (z) = , the standard logistic sigmoid function.
1 + e−z
Figure 5.3: The standard logistic sigmoid function, again.
Note: The Sigmoid Function as an Activation Function

• When the standard logistic sigmoid function is adopted as an activa-
tion function, the prediction may be considered as the probability
that a certain sample belongs to a particular class.
• This explains why the logistic sigmoid function is one of most popular
activation functions.
5.2. Classification via Logistic Regression 95
5.2. Classification via Logistic Regression

Logistic regression is a probabilistic model.
• Logistic regression maximizes the likelihood of the parameter
w; in realization, it is similar to Adaline.
• Only the difference is the activation function (the sigmoid function),
as illustrated in the figure:
Figure 5.4: Adaline vs. Logistic regression.
• The prediction (the output of the sigmoid function) is interpreted as

the probability of a particular sample belonging to class 1,
φ(z) = p(y = 1|x; w), (5.10)
given its features x parameterized by the weights w, z = wT x.
Remark 5.4. Logistic Regression can be applied not only for classi-
fication (class labels) but also for class-membership probability.
• For example, logistic regression is used in weather forecasting (to
predict the chance of rain).
• Similarly, it can be used to predict the probability that a patient has
a particular disease given certain symptoms.
– This is why logistic regression enjoys great popularity in the field
of medicine.
5.2.1. The logistic cost function

Logistic regression incorporates a cost function in which the likeli-
hood is maximized.1
Definition 5.5. The binomial distribution with parameters n and p ∈
[0, 1] is the discrete probability distribution of the number of successes
in a sequence of n independent experiments, each asking a success–
failure question, with probability of success being p.
• The probability of getting exactly k successes in n trials is given by
the probability mass function
f (k, n, p) = P (k; n, p) = n Ck pk (1 − p)n−k . (5.11)
Definition 5.6. (Likelihood). Let X1 , X2 , · · · , Xn have a joint den-

sity function f (X1 , X2 , · · · , Xn |θ). Given X1 = x1 , X2 = x2 , · · · , Xn = xn
observed, the function of θ defined by
L(θ) = L(θ|x1 , x2 , · · · , xn ) = f (x1 , x2 , · · · , xn |θ) (5.12)
is the likelihood function, or simply the likelihood.
Note: The likelihood describes the joint probability of the observed

data, as a function of the parameters of the chosen statistical model.
• The likelihood function indicates which parameter values are more
likely than others, in the sense that they would make the observed
data more probable.
• The maximum likelihood estimator selects the parameter values
(θ = w) that give the observed data the largest possible probability.
1
Note that the Adaline minimizes the sum-squared-error (SSE) cost function defined as J (w) =
1 X 2
φ(z (i) ) − y (i) , where z (i) = wT x(i) , using the gradient descent method; see Section 3.3.1.
2 i
Derivation of the Logistic Cost Function

• Assume that the individual samples in our dataset are independent
of one another. Then we can define the likelihood L as
L(w) = P (y|x; w) = Πni=1 P (y (i) |x(i) ; w)
y(i) 1−y(i) (5.13)
n (i) (i)
= Πi=1 φ(z ) 1 − φ(z ) ,
where z (i) = wT x(i) .

• In practice, it is easier to maximize the (natural) log of this equation,
which is called the log-likelihood function:
n h
X i
(i) (i) (i) (i)
`(w) = ln(L(w)) = y ln φ(z ) + (1 − y ) ln 1 − φ(z ) .
i=1
(5.14)
Remark 5.7. Log-Likelihood

• Firstly, applying the log function reduces the potential for numerical
underflow, which can occur if the likelihoods are very small.
• Secondly, we can convert the product of factors into a summation of
factors, which makes it easier to obtain the derivative of this function
via the addition trick, as you may remember from calculus.
• We can adopt the negation of the log-likelihood as a cost func-
tion J that can be minimized using gradient descent.
Now, we define the logistic cost function to be minimized:

Xn h i
(i) (i) (i) (i)
J (w) = −y ln φ(z ) −(1 − y ) ln 1 − φ(z ) , (5.15)
i=1
where z (i) = wT x(i) .

Note: Looking at the equation, we can see that the first term becomes
zero if y (i) = 0, and the second term becomes zero if y (i) = 1.
Figure 5.5: Plot of J (w), when n = 1 (one single-sample):

− ln(φ(z)), if y = 1,
J (w) =
− ln(1 − φ(z)), if y = 0.
Observation 5.8. We can see that

• (Solid curve, in blue). If we correctly predict that a sample belongs
to class 1, the cost approaches 0.
• (Dashed curve, in orange). If we correctly predict y = 0, the cost
also approaches 0.
• However, if the prediction is wrong, the cost goes towards infinity.

• Here, the main point is that we penalize wrong predictions with an
increasingly larger cost, which will enforce the model to fit the sample.
• For general n ≥ 1, it would try to fit all the samples in the training
dataset.
5.2.2. Gradient descent learning for logistic regression

Let’s start by calculating the partial derivative of the logistic cost function
(5.15) with respect to the j–th weight, wj :
n
∂φ(z (i) )

∂J (w) X 1 1
= −y (i) (i) )
+ (1 − y (i)
) (i) )
, (5.16)
∂wj i=1
φ(z 1 − φ(z ∂w j
where, using z (i) = wT x(i) and (5.4),

∂φ(z (i) ) 0 (i) ∂z
(i)
(i)

(i)

(i)
= φ (z ) = φ(z ) 1 − φ(z ) xj .
∂wj ∂wj
Thus, if follows from the above and (5.16) that
n h
∂J (w) X
(i)

(i)

(i) (i)
i
(i)
= −y 1 − φ(z ) + (1 − y )φ(z ) xj
∂wj i=1
n h i
(i)
X
(i) (i)
= − y − φ(z ) xj
i=1
and therefore n h
X i
∇J (w) = − y − φ(z ) x(i) .
(i) (i)
(5.17)
i=1
Algorithm 5.9. Gradient descent learning for logistic regression is for-

mulated as
w := w + ∆w, (5.18)
where η > 0 is the step length (learning rate) and
n h
X i
(i)
∆w = −η∇J (w) = η y − φ(z ) x(i) .
(i)
(5.19)
i=1
Note: The above gradient descent rule for logistic regression has the
same shape as that of Adaline; see (3.15) on p. 48.
Only the difference is the activation function φ(z).
5.2.3. Regularization: bias-variance tradeoff
• Overfitting is a common problem in ML.

– If a model performs well on the training data but does not gener-
alize well to unseen (test) data, then it is most likely the sign of
overfitting.
– Due to a high variance, from randomness (noise) in the training
data.
– Variance measures the consistency (or variability) of the model
prediction for a particular sample instance.
• Similarly, our model can also suffer from underfitting.
– Our model is not complex enough to capture the pattern in
the training data well, and therefore also suffers from low per-
formance on unseen data.
– Due to a high bias.
– Bias is the measure of the systematic error that is not due to
randomness.
Figure 5.6
Regularization
• One way of finding a good bias-variance tradeoff.
• It is useful to prevent overfitting, also handling
– collinearity (high correlation among features)
– filter-out noise from data
– multiple local minima problem
• The concept behind regularization is to introduce additional in-
formation (bias) to penalize extreme parameter (weight) values.
• The most common form of regularization is so-called L2 regulariza-
tion (sometimes also called L2 shrinkage or weight decay):
m
λ λX 2
kwk2 = w , (5.20)
2 2 j=1 j
where λ is the regularization parameter.
The cost function for logistic regression can be regularized by adding a sim-
ple regularization term, which will shrink the weights during model train-
ing: for z (i) = wT x(i) ,
n h
X i λ
(i)
J (w) = −y (i) (i) (i)
ln φ(z ) − (1 − y ) ln 1 − φ(z ) + kwk2 . (5.21)
i=1
2
Note: Regularization
• Regularization is another reason why feature scaling such as stan-
dardization is important.
• For regularization to work properly, we need to ensure that all our
features are on comparable scales.
• Then, via the regularization parameter λ, we can control how well we
fit the training data while keeping the weights small. By increasing
the value of λ, we increase the regularization strength.
• See § 6.3 for details on feature scaling.
5.3. Support Vector Machine

• Support vector machine (SVM), developed in 1995 by Cortes-
Vapnik [12], can be considered as an extension of the Percep-
tron/Adaline, which maximizes the margin.
• The rationale behind having decision boundaries with large margins
is that they tend to have a lower generalization error, whereas
models with small margins are more prone to overfitting.
5.3.1. Linear SVM
Figure 5.7: Linear support vector machine.
To find an optimal hyperplane that maximizes the margin, let’s begin with
considering the positive and negative hyperplanes that are parallel to the
decision boundary:
w0 + wT x+ = 1,
(5.22)
w0 + wT x− = −1.
where w = [w1 , w2 , · · · , wd ]T . If we subtract those two linear equations from
each other, then we have
w · (x+ − x− ) = 2
and therefore
w 2
· (x+ − x− ) = . (5.23)
kwk kwk
5.3. Support Vector Machine 103
Note: w = [w1 , w2 , · · · , wd ]T is a normal vector a to the decision bound-

ary (a hyperplane) so that the left side of (5.23) is the distance between
the positive and negative hyperplanes.
a
See Exercise 5.1.
Maximizing the distance (margin) is equivalent to minimizing its reciprocal

1 1 2
2 kwk, or minimizing 2 kwk .
Problem 5.10. The linear SVM is formulated as

1
min kwk2 , subject to
w,w0 2
(5.24)
w0 + wT x(i) ≥ 1 if y (i) = 1,

w0 + wT x(i) ≤ −1 if y (i) = −1.
Remark 5.11. The constraints in Problem 5.10 can be written as
y (i) (w0 + wT x(i) ) − 1 ≥ 0, ∀ i. (5.25)
• The beauty of linear SVM is that if the data is linearly separable,

there is a unique global minimum value.
• An ideal SVM analysis should produce a hyperplane that completely
separates the vectors (cases) into two non-overlapping classes.
• However, perfect separation may not be possible, or it may result
in a model with so many cases that the model does not classify cor-
rectly.
Note: Constrained optimization problems such as (5.24) are typically

solved using the method of Lagrange multipliers.
5.3.2. The method of Lagrange multipliers

In this subsection, we briefly consider Lagrange’s method to solve the
problem of the form
min / max f (x) subj.to g(x) = c. (5.26)
x
Figure 5.8: The method of Lagrange multipliers in R2 : ∇f // ∇g, at optimum.
Strategy 5.12. (Method of Lagrange multipliers). For the maxi-

mum and minimum values of f (x) subject to g(x) = c,
(a) Find x and λ such that
∇f (x) = λ∇g(x) and g(x) = c.
(b) Evaluate f at all these points, to find the maximum and minimum.
Self-study 5.13. Use the method of Lagrange multipliers to find the ex-
treme values of f (x, y) = x2 + 2y 2 on the circle x2
+ y 2 = 1.
 2x = 2x λ 1
2x 2x
Hint : ∇f = λ∇g =⇒ =λ . Therefore, 4y = 2y λ 2
4y 2y  2 2
x +y =1 3
From 1 , x = 0 or λ = 1.
Ans: min: f (±1, 0) = 1; max: f (0, ±1) = 2

Lagrange multipliers – Dual variables
For simplicity, consider

minx x2 subj.to x ≥ 1. (5.27)
Rewriting the constraint

x − 1 ≥ 0,
introduce Lagrangian (objective):

L(x, α) = x2 − α (x − 1). (5.28)
Now, consider
minx maxα L(x, α) subj.to α ≥ 0.
Figure 5.9: minx x2 subj.to x ≥ 1.
(5.29)
Claim 5.14. The minimization problem (5.27) is equivalent to the min-

max problem (5.29).
Proof. 1 Let x > 1. ⇒ maxα≥0 {−α(x − 1)} = 0 and α∗ = 0. Thus,
L(x, α) = x2 . (original objective)
2 Let x = 1. ⇒ maxα≥0 {−α(x − 1)} = 0 and α is arbitrary. Thus, again,
L(x, α) = x2 . (original objective)
3 Let x < 1. ⇒ maxα≥0 {−α(x − 1)} = ∞. However, minx won’t make this
happen! (minx is fighting maxα ) That is, when x < 1, the objective L(x, α)
becomes huge as α grows; then, minx will push x % 1 or increase it to become
x ≥ 1. In other words, minx forces maxα to behave, so constraints will
be satisfied.
Now, the goal is to solve (5.29). In the following, we will define the dual
problem of (5.29), which is equivalent to the primal problem.
Recall: The min-max problem in (5.29), which is equivalent to the (orig-

inal) primal problem:
min max L(x, α) subj.to α ≥ 0, (Primal) (5.30)

x α
where
L(x, α) = x2 − α (x − 1).
Definition 5.15. The dual problem of (5.30) is formulated by swap-

ping minx and maxα as follows:
max min L(x, α) subj.to α ≥ 0, (Dual) (5.31)

α x
The term minx L(x, α) is called the Lagrange dual function and the
Lagrange multiplier α is also called the dual variable.
How to solve it. For the Lagrange dual function minx L(x, α), the minimum
occurs where the gradient is equal to zero.
d α
L(x, α) = 2x − α = 0 ⇒ x = . (5.32)
dx 2
Plugging this to L(x, α), we have
α 2 α α2
L(x, α) = −α −1 =α− .
2 2 4
We can rewrite the dual problem (5.31) as

h α2 i
max α − . (Dual) (5.33)
α≥0 4
⇒ the maximum is 1 when α∗ = 2 (for the dual problem).

Plugging α = α∗ into (5.32) to get x∗ = 1. Or, using the Lagrangian objective,
we have
L(x, α) = x2 − 2(x − 1) = (x − 1)2 + 1. (5.34)
⇒ the minimum is 1 when x∗ = 1 (for the primal problem).
5.3.3. Karush-Kuhn-Tucker conditions and Complemen-

tary slackness
Allowing inequality constraints, the KKT approach generalizes the method
of Lagrange multipliers which allows only equality constraints.
Recall: The linear SVM formulated in Problem 5.10:

1
min kwk2 , subj.to
w,w0 2 (Primal) (5.35)
y (i) (w0 + wT x(i) ) − 1 ≥ 0, ∀ i = 1, 2, · · · , N.
To solve the problem, let’s begin with its Lagrangian:

N
1 2
X
L([w, w0 ], α) = kwk − αi [y (i) (w0 + wT x(i) ) − 1], (5.36)
2 i=1
where α = [α1 , α2 , · · · , αN ]T , the dual variable (Lagrange multipliers).
• The primal problem of the SVM is formulated equivalently as
min max L([w, w0 ], α) subj.to α ≥ 0, (Primal) (5.37)

w,w0 α
while its dual problem reads
max min L([w, w0 ], α) subj.to α ≥ 0. (Dual) (5.38)

α w,w0
• Solve the “min” problem of (5.38) first, using calculus techniques.
Definition 5.16. Karush-Kuhn-Tucker (KKT) conditions

In optimization, the KKT conditions [36, 42] are first derivative tests
for a solution in nonlinear programming to be optimized. It is also called
the first-order necessary conditions.
Writing the KKT conditions, starting with Lagrangian stationarity, where

we need to find the first-order derivatives w.r.t. w and w0 :
N
X N
X
(i) (i)
∇w L([w, w0 ], α) = w− αi y x =0 ⇒ w= αi y (i) x(i) ,
i=1 i=1
N N
∂ X X
L([w, w0 ], α) = − αi y (i) = 0 ⇒ αi y (i) = 0,
∂w0 i=1 i=1
αi ≥ 0, (dual feasibility)
αi [y (i) (w0 + wT x(i) ) − 1] = 0, (complementary slackness)
y (i) (w0 + wT x(i) ) − 1 ≥ 0. (primal feasibility)
(5.39)
Complementary slackness will be discussed in detail on page 111.
Using the KKT conditions (5.39), we can simplify the Lagrangian:

N N N
1 2
X
(i)
X
(i) T (i)
X
L([w, w0 ], α) = kwk − αi y w 0 − αi y w x + αi
2 i=1 i=1 i=1
N
1 X
= kwk2 − 0 − wT w + αi (5.40)
2 i=1
N
1 X
= − kwk2 + αi .
2 i=1
Again using the first KKT condition, we can rewrite the first term.
N N
1 2 1 X (i) (i)
X
(j) (j)

− kwk = − αi y x · αj y x
2 2 i=1 j=1
N N (5.41)
1 XX
= − αi αj y (i) y (j) x(i) · x(j) .
2 i=1 j=1
Plugging (5.41) into the (simplified) Lagrangian (5.40), we see that the La-
grangian now depends on α only.
Problem 5.17. The dual problem of (5.35) is formulated as

N N N
hX 1 XX (i) (j) (i) (j)
i
max αi − αi αj y y x · x , subj.to
α 2
i=1 i=1 j=1 (5.42)
αi ≥ 0, ∀ i,
PN (i)
i=1 αi y = 0.
Remark 5.18. (Solving the dual problem).

• We can solve the dual problem (5.42), by using either a generic
quadratic programming solver or the Sequential Minimal Opti-
mization (SMO), which we will discuss in § 5.3.6, p. 120.
• For now, assume that we solved it to have α∗ = [α1∗ , · · · , αn∗ ]T .
• Then we can plug it into the first KKT condition to get
N
X
∗
w = αi∗ y (i) x(i) . (5.43)
i=1
• We still need to get w0∗ .
Remark 5.19. The objective function L(α) in (5.42) is a linear com-

bination of the dot products of data samples {x(i) · x(j) }, which will be
used when we generalize the SVM for nonlinear decision boundaries;
see § 5.3.5.
Support vectors
Assume momentarily that we have w0∗ . Consider the complementary slack-
ness KKT condition along with the primal and dual feasibility conditions:
αi∗ [y (i) (w0∗ + w∗T x(i) ) − 1] = 0



 αi∗ > 0 ⇒ y (i) (w0∗ + w∗T x(i) ) = 1
 α∗ < 0 (can’t happen)

(5.44)
i
⇒

 y (w0∗ + wT x(i) ) − 1 > 0 ⇒ αi∗ = 0
(i)
 (i) ∗

y (w0 + wT x(i) ) − 1 < 0 (can’t happen).
We define the optimal (scaled) scoring function:
f ∗ (x(i) ) = w0∗ + w∗T x(i) . (5.45)
Then
αi∗ > 0 ⇒ y (i) f ∗ (x(i) ) = scaled margin = 1,

(5.46)
y (i) f ∗ (x(i) ) > 1 ⇒ αi∗ = 0.
Definition 5.20. The examples in the first category, for which the
scaled margin is 1 and the constraints are active, are called support
vectors. They are the closest to the decision boundary.
Finding the optimal value of w0

To get w0∗ , use the primal feasibility condition:
y (i) (w0∗ + w∗T x(i) ) ≥ 1 and min y (i) (w0∗ + w∗T x(i) ) = 1.
i
If you take a positive support vector (y (i) = 1), then
w0∗ = 1 − min w∗T x(i) . (5.47)

i:y (i) =1
Here, you’d better refer to Summary of SVM in Algorithm 5.27, p. 115.

Complementary Slackness
Definition 5.21. Types of Constraints
• A binding constraint is one where some optimal solution is on the
hyperplane for the constraint.
• A non-binding constraint is one where no optimal solution is on
the line for the constraint.
• A redundant constraint is one whose removal would not change
the feasible region.
Theorem 5.22. Complementary Slackness

Assume the primal problem (P) has a solution w∗ and the dual problem
(D) has a solution α∗ .
(a) If wj∗ > 0, then the j-th constraint in (D) is binding.
(b) If αi∗ > 0, then the i-th constraint in (P) is binding.
The term complementary slackness refers to a relationship between the

slackness in a primal constraint and the slackness (positivity) of the associ-
ated dual variable.
• Notice that the number of variables in the dual is the same as the num-
ber of constraints in the primal, and the number of constraints in the
dual is equal to the number of variables in the primal.
• This correspondence suggests that variables in one problem are com-
plementary to constraints in the other.
• We say that a constraint has slack if it is not binding.
Example 5.23. The contrapositive statement of Theorem 5.22 (b):
If the i-th constraint in (P) is not binding, then α∗i = 0.

or, equivalently,
If the i-th constraint in (P) has slack, then α∗i = 0.
See (5.46).
5.3.4. The inseparable case: Soft-margin classification

When the dataset is inseparable, there would be no separating hyperplane;
there is no feasible solution to the linear SVM.
Figure 5.10: Slack variable: ξi .
Let’s fix our SVM so it can accommodate the inseparable case.

• The new formulation involves the slack variable; it allows some
instances to fall off the margin, but penalize them.
• So we are allowed to make mistakes now, but we pay a price.
Remark 5.24. The motivation for introducing the slack variable ξ is:
1. The linear constraints need to be relaxed for inseparable data.
2. Allow the optimization to converge
• under appropriate cost penalization,
• in the presence of misclassifications.
Such strategy of the SVM is called the soft-margin classification.
Recall: The linear SVM formulated in Problem 5.10:

1
min kwk2 , subj.to
w,w0 2 (Primal) (5.48)
y (i) (w0 + wT x(i) ) − 1 ≥ 0, ∀ i.
Let’s change it to this new primal problem:
Problem 5.25. (Soft-margin classification). The SVM with the slack

variable is formulated as
N
1 2
X
min kwk + C ξi subj.to
w,w0 ,ξ 2
(i) i=1 (Primal) (5.49)
T (i)
y (w0 + w x ) ≥ 1 − ξi ,
ξi ≥ 0.
Via the variable C, we can then con- variance trade-off, as illustrated

trol the penalty for misclassification. in the following figure:
Large values of C correspond to
large error penalties, whereas we
are less strict about misclassification
errors if we choose smaller values
for C. We can then use the C pa-
rameter to control the width of the
margin and therefore tune the bias-
Figure 5.11: Bias-variance trade-off, via C.
The constraints allow some slack of size ξi , but we pay a price for it in the
objective. That is,
if y (i) f (x(i) ) ≥ 1, then ξi = 0 and penalty is 0. Otherwise, y (i) f (x(i) ) = 1 − ξi
and we pay price ξi > 0
.
The Dual for soft-margin classification

Form the Lagrangian of (5.49):
N N
1 X X
L([w, w0 ], ξ, α, r) = kwk2 + C ξi − ri ξi
2 i=1 i=1
N (5.50)
X
− αi [y (i) (w0 + wT x(i) ) − 1 + ξi ],
i=1
where αi ’s and ri ’s are Lagrange multipliers (constrained to be ≥ 0).

After some work, the dual turns out to be
Problem 5.26. The dual problem of (5.48) is formulated as
N N N
hX 1 XX i
max αi − αi αj y (i) y (j) x(i) · x(j) , subj.to
α 2 i=1 j=1
i=1 (5.51)
0 ≤ αi ≤ C, ∀ i,
PN (i)
i=1 αi y = 0.
So the only difference from the original problem’s dual, (5.42), is that
αi ≥ 0 is changed to 0 ≤ αi ≤ C . Neat!
See § 5.3.6, p. 120, for the solution of (5.51), using the SMO algorithm.
Algebraic expression for the dual problem :

Let    
y (1) x(1) 1
 y (2) x(2)  1
Z=  ∈ RN ×m , 1 =  ..  ∈ RN .
   
..
 .  .
y (N ) x(N ) 1
Then dual problem (5.51) can be written as

1
max α · 1 − αT ZZ T α subj.to α · y = 0.

(5.52)
0≤α≤C 2
Note:
• G = ZZ T ∈ RN ×N is called the Gram matrix. That is,
Gij = y (i) y (j) x(i) · x(j) . (5.53)
• The optimization problem (5.52) is a typical quadratic program-

ming (QP) problem.
• It admits a unique solution.
Algorithm 5.27. (Summary of SVM)

• Training
– Compute Gram matrix: Gij = y (i) y (j) x(i) · x(j)
– Solve QP to get α∗ (Chapter 11, or § 5.3.6)
– Compute the weights: w∗ = N ∗ (i) (i)
P
i=1 αi y x (5.43)
– Compute the intercept: w0∗ = 1 − mini:y(i) =1 w∗T x(i) (5.47)
• Classification (for a new sample x)

– Compute ki = x · x(i) for support vectors x(i)
– Compute f (x) = w0∗ + i αi∗ y (i) ki (:= w0∗ + w∗T x)
P
(5.24)
– Test sign(f (x)).
5.3.5. Nonlinear SVM and kernel trick

Note: A reason why the SVM is popular is that
• It can be easily kernelized to solve nonlinear classification prob-
lems incorporating linearly inseparable data.
The basic idea behind kernel methods is
• To transform the data to a higher-dimensional space where the
data becomes linearly separable.
For example, for the inseparable data set in Figure 5.12, we define
φ(x1 , x2 ) = (x1 , x2 , x21 + x22 ).
Figure 5.12: Inseparable dataset, feature expansion, and kernel SVM.
To solve a nonlinear problem using an SVM, we would

(a) 1 Transform the training data to a higher-dimensional
space, via a mapping φ, and 2 train a linear SVM model.
(b) Then, for new unseen data, classify using 1 the same map-
ping φ to transform and 2 the same linear SVM model.
Kernel Trick
Recall: the dual problem to the soft-margin SVM given in (5.51):
N N N
hX 1 XX (i) (j) (i) (j)
i
α 2
i=1 i=1 j=1 (5.54)
0 ≤ αi ≤ C, ∀ i,
PN (i)
i=1 αi y = 0.
Observation 5.28. The objective is a linear combination of dot prod-

ucts {x(i) · x(j) }. Thus,
• If the kernel SVM transforms the data samples through φ,
the dot product x(i) · x(j) must be replaced by φ(x(i) ) · φ(x(j) ).
• The dot product φ(x(i) ) · φ(x(j) ) is performed in a higher-dimension,
which may be costly.
Definition 5.29. In order to save the expensive step of explicit

computation of this dot product (in a higher-dimension), we define a
so-called kernel function:
K(x(i) , x(j) ) ≈ φ(x(i) ) · φ(x(j) ). (5.55)
One of the most widely used kernels is the Radial Basis Function
(RBF) kernel or simply called the Gaussian kernel:
(i) (j) kx(i) − x(j) k2

(i) (j) 2

K(x , x ) = exp − = exp −γkx − x k , (5.56)
2σ 2
where γ = 1/(2σ 2 ). Occasionally, the parameter γ plays an important
role in controlling overfitting.
Note: Roughly speaking, the term kernel can be interpreted as a

similarity function between a pair of samples.
This is the big picture behind the kernel trick.

Kernel SVM: It can also be summarized as in Algorithm 5.27, p. 115;

only the difference is that dot products x(i) · x(j) and x · x(i) are replaced
by K(x(i) , x(j) ) and K(x, x(i) ), respectively.
Common Kernels
• Polynomial of degree exactly k (e.g. k = 2):

K(x(i) , x(j) ) = (x(i) · x(j) )k (5.57)
• Polynomial of degree up to k: for some c > 0,

K(x(i) , x(j) ) = (c + x(i) · x(j) )k (5.58)
• Sigmoid:
K(x(i) , x(j) ) = tanh(a x(i) · x(j) + b) (5.59)
• Gaussian RBF:
(i) (j) kx(i) − x(j) k2

K(x , x ) = exp − (5.60)
2σ 2
• And many others: Fisher kernel, graph kernel, string kernel, ...
very active area of research!
Example 5.30. (Quadratic kernels). Let K(x, z) = (c + x · z)2 :

m
X m
X
2
(c + x · z) = c + xj zj c + x` z`
j=1 `=1
Xm m
XX m
= c2 + 2c xj zj + xj zj x` z` (5.61)
j=1 j=1 `=1
m m
X √
2
√ X
= c + ( 2cxj )( 2czj ) + (xj x` )(zj z` ).
j=1 j,`=1
Define a feature expansion as

√ √
φ([x1 , · · · , xm ]) = [x21 , x1 x2 , · · · , xm xm−1 , x2m , 2cx1 , · · · , 2cxm , c], (5.62)
2
which is in Rm +m+1
. Then φ(x) · φ(z) = K(x, z) = (c + x · z)2 .
Note: Kernel Functions

• Kernels may not be expressed as φ(x) · φ(z).
• The mapping φ may transform x to infinite dimensions.
Summary 5.31. Linear Classifiers

• They are a simple and popular way to learn a classifier
• They suffer from inefficient use of data, overfitting, or lack of expres-
siveness
• SVM
– It can fix these problems using 1 maximum margins and
2 feature expansion (mapping to a higher-dimension).
– In order to make feature expansion computationally feasible,
we need the 3 kernel trick, which avoids writing out high-
dimensional feature vectors explicitly.
Remark 5.32. Kernel Trick

• There is no explicit feature expansion.
• The kernel K(x, z) must be formulated meaningfully.
• The kernel function K must be considered as

a nonlinear measure for the data to become separable.
5.3.6. Solving the dual problem with SMO

SMO (Sequential Minimal Optimization) is
• a type of coordinate ascent algorithm,

• but adapted to the SVM so that the solution always stays within the
feasible region.
Recall: The dual problem of the soft-margin SVM, formulated in

(5.51):
N N N
hX 1 XX (i) (j) (i) (j)
i
α 2
i=1 i=1 j=1 (5.63)
0 ≤ αi ≤ C, ∀ i,
PN (i)
i=1 αi y = 0.
Quesiton. Start with (5.63). Let’s say you want to hold α2 , · · · , αN fixed
and take a coordinate step in the first direction. That is, change α1 to
maximize the objective in (5.63). Can we make any progress? Can we
get a better feasible solution by doing this?
PN (i)
Turns out, no. Let’s see why. Look at the constraint in (5.63), i=1 αi y = 0.
This means
N
X N
X
(1) (i) (1)
α1 y =− αi y ⇒ α1 = −y αi y (i) .
i=2 i=2
So, since α2 , · · · , αN are fixed, α1 is also fixed.

Thus, if we want to update any of the αi ’s, we need to update at least
2 of them simultaneously to keep the solution feasible (i.e., to keep the
constraints satisfied).
• Start with a feasible vector α.

• Let’s update α1 and α2 , holding α3 , · · · , αN fixed.
Question: What values of α1 and α2 are we allowed to choose?
N
X
(1) (2)
• The constraint is: α1 y + α2 y = − αi y (i) =: ξ.
i=3
Figure 5.13
We are only allowed to choose α1 and α2 on the line.

• When we pick α2 , we can get α1 from
1
α1 = (1)
(ξ − α2 y (2) ) = y (1) (ξ − α2 y (2) ). (5.64)
y
• Optimization for α2 : The other constraints in (5.63) says 0 ≤

α1 , α2 ≤ C. Thus, α2 needs to be within [L, H] on the figure (∵ α1 ∈
[0, C]). To do the coordinate ascent step, we will optimize the objective
over α2 , keeping it within [L, H]. Using (5.64), (5.63) becomes
N N N
h
(1) (2)
X 1 XX (i) (j) (i) (j)
i
max y (ξ−α2 y )+α2 + αi − αi αj y y x ·x , (5.65)
α2 ∈[L,H]
i=3
2 i=1 j=1
of which the objective is quadratic in α2 . This means we can just set

its derivative to 0 to optimize it =⇒ get α2 .
• After updating α1 using (5.64), move to the next iteration of SMO.
Note: There are heuristics to choose the order of αi ’s chosen to update.

5.4. Decision Trees

Decision tree classifiers are attractive models if we care about
interpretability. As the name decision tree suggests, we can think of this
model as breaking down our data by making decision based on asking a
series of questions. Decision tree was invented by a British researcher,
William Belson, in 1959 [1].
Note: Decision trees are commonly used in operations research,

specifically in decision analysis, to help identify a strategy most likely
to reach a goal, but are also a popular tool in ML.
Figure 5.14: A decision tree to decide upon an activity on a particular day.
Key Idea 5.33. (Decision tree).

• Start at the tree root
• Split the data so as to result in the largest Information Gain (IG)
• Repeat the splitting at each child node until the leaves are pure
(This means the samples at each node all belong to the same class)
• In practice , this can result in a very deep tree with many nodes,
which can easily lead to overfitting
(We typically set a limit for the maximal depth of the tree)
5.4. Decision Trees 123
5.4.1. Decision tree objective

• Decision tree also needs to incorporate an objective function, to be
optimized via the tree learning algorithm.
• Here, the objective function is to maximize the information gain at
each split, which we define as follows:
m
X Nj
IG(DP , f ) = I(DP ) − I(Dj ), (5.66)
j=1
N P
where
f : the feature to perform the split
DP : the parent dataset
Dj : the dataset of the j-th child node
I : the impurity measure
NP : the total number of samples at the parent note
Nj : the number of samples in the j-th child node
• The information gain is simply the difference between the impurity

of the parent node and the average of the child node impurities
– The lower the impurity of the child nodes, the larger the informa-
tion gain.
• However, for simplicity and to reduce the combinatorial search space,
most libraries implement binary decision trees, where each parent
node is split into two child nodes, DL and DR :
NL NR
IG(DP , f ) = I(DP ) − I(DL ) − I(DR ). (5.67)
NP NP
Impurity measure?
Commonly used in binary decision trees:
• Entropy c
X
IH (t) = − p(i|t) log2 p(i|t) (5.68)
i=1
• Gini impurity
c
X c
X
p(i|t)2

IG (t) = p(i|t) 1 − p(i|t) = 1 − (5.69)
i=1 i=1
• Classification error
IE (t) = 1 − max{p(i|t)} (5.70)
i
where p(i|t) denotes the proportion of the samples that belong to class
i for a particular node t.
Mind simulation: When c = 2

• Entropy: It is maximal, if we have a uniform class distribution;
it is 0, if all samples at the node t belong to the same class.
IH (t) = 0, if p(i = 1|t) = 1 or p(i = 2|t) = 0
IH (t) = 1, if p(i = 1|t) = p(i = 2|t) = 0.5
⇒ We can say that the entropy criterion attempts to maximize the mu-
tual information in the tree.
• Gini impurity: Intuitively, it can be understood as a criterion to min-
imize the probability of misclassification. The Gini impurity is maxi-
mal, if the classes are perfectly mixed.
IG (t) = 1 − ci=1 0.52 = 0.5
P
⇒ In practice, both Gini impurity and entropy yield very similar results.
• Classification error: It is less sensitive to changes in the class prob-
abilities of the nodes.
⇒ The classification error is a useful criterion for pruning, but not rec-
ommended for growing a decision tree.
Figure 5.15: A decision tree result with Gini impurity measure, for three classes with
two features (petal length, petal width). Page 99, Python Machine Learning, 3rd Ed..
Quesiton. How can the decision tree find questions such as

petal width <= 0.75 petal length <= 4.75 ... ?
Algorithm 5.34. (Decision tree split rule).

(i)
1. For each and every feature in DP , fj :
◦ make a question to split Dp into DL and DR
(k) (i)
(e.g. fj ≤ fj , for which k’s?)
◦ compute the impurities: I(DL ) and I(DR )
◦ compute the information gain:
(i) NL NR
IG(DP , fj ) = I(DP ) − I(DL ) − I(DR ).
NP NP
2. Let
(i)
fq(p) = arg max IG(DP , fj ). (5.71)
i,j
3. Then, the best split question (at the current node) is
fq(k) ≤ fq(p) , for which k’s ? (5.72)
The maximum in (5.71) often happens when one of the child impurities
is zero or very small.
5.4.2. Random forests: Multiple decision trees

Random forests (or random decision forests) are an ensemble learn-
ing method for classification, regression, and other tasks that operates by
constructing multiple decision trees at training time and outputting the
class that is the mode of the predicted classes (classification) or mean
prediction (regression) of the individual trees [32].
• Random forests have gained huge popularity in applications of ML

during the last decade due to their good classification performance,
scalability, and ease of use.
• The idea behind a random forest is to average multiple (deep) de-
cision trees that individually suffer from high variance, to build a
more robust model that has a better generalization performance and
is less susceptible to overfitting.
Algorithm 5.35. Random Forest.

The algorithm can be summarized in four simple steps:
1. Draw a random bootstrap sample of size n
(Randomly choose n samples from the training set with replace-
ment).
2. Grow a decision tree from the bootstrap sample.
3. Repeat Steps 1-2 k times.
4. Aggregate the prediction by each tree to assign the class label by
majority vote.
Note: In Step 2, when we are training the individual decision tree:

• instead of evaluating all features to determine the best split at each
node,
• we can consider a random (without replacement) subset of those (of
size d ).
Remark 5.36. A big advantage of random forests is that

we don’t have to worry so much about
choosing good hyperparameter values.
• A smaller n increases randomness of the random forest; the bigger n

is, the larger the degree of overfitting becomes.
Default n = size(the original training set), in most implementations
√
• Default d = M , where M is the number of features in the training
set
• The only parameter that we really need to care about in practice is
the number of trees k (Step 3).
Typically, the larger the number of trees, the better the performance
of the random forest classifier at the expense of an increased compu-
tational cost.
5.5. k-Nearest Neighbors 129
5.5. k-Nearest Neighbors

The k-nearest neighbor (k-NN) classifier is a typical example of a
lazy learner.
• It is called lazy not because of its apparent simplicity, but because it
doesn’t learn a discriminative function from the training data,
but memorizes the training dataset instead.
• Analysis of the training data is delayed until a query is made to
the system.
Algorithm 5.37. (k-NN algorithm). The algorithm itself is fairly

straightforward and can be summarized by the following steps:
1. Choose the number k and a distance metric.
2. For the new sample, find the k-nearest neighbors.
3. Assign the class label by majority vote.
Figure 5.16: Illustration for how a new data point (?) is assigned the triangle class label,
based on majority voting, when k = 5.
k-NN: pros and cons

• Since it is memory-based, the classifier immediately adapts as we
collect new training data.
• The computational complexity for classifying new samples grows
linearly with the number of samples in the training dataset in the
worst-case scenario.a
• Furthermore, we can’t discard training samples since no training
step is involved. Thus, storage space can become a challenge if we
are working with large datasets.
a
J. H. Friedman, J. L. Bentley, and R.A. Finkel (1977). An Algorithm for Finding Best Matches in
Logarithmic Expected Time, ACM transactions on Mathematical Software (TOMS), 3, no. 3, pp. 209–
226. The algorithm in the article is called the KD-tree.
k-NN: what to choose k and a distance metric?

• The right choice of k is crucial to find a good balance between
overfitting and underfitting.
(For sklearn.neighbors.KNeighborsClassifier, default n_neighbors = 5.)
• We also choose a distance metric that is appropriate for the features
in the dataset. (e.g., the simple Euclidean distance, along with data
standardization)
• Alternatively, we can choose the Minkowski distance:
m
X 1/p
def p
d(x, z) = kx − zkp == |xi − zi | . (5.73)
i=1
(For sklearn.neighbors.KNeighborsClassifier, default p = 2.)
Remark 5.38. The k-NN algorithm is very susceptible (wide open) to

overfitting due to the curse of dimensionality.a
Since regularization is not applicable for k-NN, we can use feature se-
lection and dimensionality reduction techniques to help us avoid the
curse of dimensionality and avoid overfitting. This will be discussed in
more details later.
a
The curse of dimensionality describes the phenomenon where the feature space becomes increas-
ingly sparse for an increasing number of dimensions of a fixed-size training dataset. Intuitively, we
can think of even the closest neighbors being too far away in a high-dimensional space to give a good
estimate.
5.5. k-Nearest Neighbors 131
5.1. The equation c1 x1 + c2 x2 + · · · + cn xn = d determines a hyperplane in Rn . Prove that

the vector [c1 , c2 , · · · , cn ] is a normal vector of the hyperplane.
5.2. For this problem, you would modify the code used for Problem 3.2 in Chapter 3. For
the standardized data (XSD ),
(a) Apply the logistic regression gradient descent (Algorithm 5.9).

(b) Compare the results with that of Adaline descent gradient.
5.3. (Continuation of Problem 5.2). Perturb the standardized data (XSD ) by a random
Gaussian noise Gσ of an observable σ (so as for Gσ (XSD ) not to be linearly separable).
(a) Apply the logistic regression gradient descent (Algorithm 5.9) for the noisy data
Gσ (XSD ).
(b) Modify the code for the logistic regression with regularization (5.21) and apply
the resulting algorithm for Gσ (XSD ).
(c) Compare their performances
5.4. (Optional for Undergraduate Students) Verify the formulation in (5.51), which is
dual to the minimization of (5.50).
5.5. Experiment examples on pp. 84–91, Python Machine Learning, 3rd Ed., in order to
optimize the performance of kernel SVM by finding a best kernel and optimal hyper-
parameters (gamma and C).
Choose one of Exercises 6 and 7 below to implement and experiment. The experiment
will guide you to understand how the LM software has been composed from scratch.
You may use the example codes thankfully shared by Dr. Jason Brownlee, who is
the founder of machinelearningmastery.com.
5.6. Implement a decision tree algorithm that incorporates the Gini impurity measure,
from scratch, to run for the data used on page 96, Python Machine Learning, 3rd Ed..
Compare your results with the figure on page 97 of the book. You may refer to
https://machinelearningmastery.com/implement-decision-tree-algorithm-scratch-python/
5.7. Implement a k-NN algorithm, from scratch, to run for the data used on page 106,
Python Machine Learning, 3rd Ed.. Compare your results with the figure on page
103 of the book. You may refer to
https://machinelearningmastery.com/tutorial-to-implement-k-nearest-neighbors-in-python-from-
scratch/
6
C HAPTER
Data Preprocessing in Machine

Learning
Data preprocessing (or, data preparation) is a data mining technique,

which is the most time consuming (often, the most important) step in
machine learning.
6.1. General Remarks on Data Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . 134
6.2. Dealing with Missing Data & Categorical Data . . . . . . . . . . . . . . . . . . . . . . . 136
6.3. Feature Scaling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
6.4. Feature Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
6.5. Feature Importance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
133
134 Chapter 6. Data Preprocessing in Machine Learning
6.1. General Remarks on Data Preprocessing

Data preprocessing is a data mining technique.
• It involves transforming raw data into a understandable and
more tractable format.
• Real-world data is often incomplete, redundant, inconsistent,
and/or lacking in certain behaviors or trends, and is likely to
contain many errors.
• Data preprocessing is a proven method of resolving such issues.
• Often, data preprocessing is the most important phase of a ma-
chine learning project, especially in computational biology.
Summary 6.1. Different steps involved for data preprocessing can be

summarized as follows.
1. Data Cleaning: In this first step, the primary focus is on handling
missing data, noisy data, detection and removal of outliers, and min-
imizing duplication and computed biases within the data.
2. Data Integration: This process is used when data is gathered from
various data sources and data are combined to form consistent data.
3. Data Transformation: This step is used to convert the raw data
into a specified format according to the need of the model.
(a) Normalization – Numerical data is converted into the specified
range (e.g., feature scaling → ∼ N (0, 1)).
(b) Aggregation – This method combines some features into one.
4. Data Reduction: Redundancy within the data can be removed and
efficiently organize the data.
The more disciplined you are in your handling of data, the more
consistent and better results you are likely to achieve.
6.1. General Remarks on Data Preprocessing 135
Remark 6.2. Data preparation is difficult because the process is

not objective, and it is important because ML algorithms learn from
data. Consider the following.
• Preparing data for analysis is one of the most important steps in
any data-mining project – and traditionally, one of the most time
consuming.
• Often, it takes up to 80% of the time.
• Data preparation is not a once-off process; that is, it is iterative
as you understand the problem deeper on each successive pass.
• It is critical that you feed the algorithms with the right data for
the problem you want to solve. Even if you have a good dataset, you
need to make sure that it is in a useful scale and format and that
meaningful features are included.
Questions in ML, in practice

• What would reduce the generalization error?
• What is the best form of the data to describe the problem?
(It is difficult to answer, because it is not objective.)
• Can we design effective methods and/or smart algorithms for
automated data preparation?
6.2. Dealing with Missing Data & Categorical

Data
6.2.1. Handling missing data
Software: [pandas.DataFrame].isnull().sum() > 1
For missing values, three different steps can be executed.

• Removal of samples (rows) or features (columns):
– It is the simplest and efficient method for handling the missing
data.
– However, we may end up removing too many samples or features.
Software: pandas.dropna
• Filling the missing values manually:
– This is one of the best-chosen methods.
– But there is one limitation that when there are large data set, and
missing values are significant.
• Imputing missing values using computed values:
– The missing values can also be occupied by computing mean, me-
dian, or mode of the observed given values.
– Another method could be the predictive values that are computed
by using any ML or Deep Learning algorithms.
– But one drawback of this approach is that it can generate bias
within the data as the calculated values are not accurate concern-
ing the observed values.
Software: from sklearn.preprocessing import Imputer
6.2. Dealing with Missing Data & Categorical Data 137
6.2.2. Handling categorical data

It is common that real-world datasets contain one or more categorical fea-
ture columns. These categorical features must be effectively handled to fit
in numerical computing libraries.
When we are talking about categorical data, we should further distin-

guish between ordinal features and nominal features.
• Mapping ordinal features: e.g.,
   
M 0
size : L
  ←→ 1 .
 (6.1)
XL 2
– This is called an ordinal encoding or an integer encoding.

– The integer values have a natural ordered relationship between
each other; machine learning algorithms may understand and
harness this relationship.
• Encoding nominal features: one-hot encoding, e.g.,
     
blue 0 1 0 0
color :  green  ←→ 1 ←→ 0 1 0 . (6.2)
red 2 0 0 1
Software: from sklearn.preprocessing import OneHotEncoder
Remark 6.3. For categorical variables where no ordinal relationship

exists, the integer encoding is not enough.
• In fact, assuming a natural ordering between categories and using
the integer encoding may result in poor performance or unexpected
results.
• The one-hot encoding can be used, although ordinal relationship
exists.
6.3. Feature Scaling

Note: Feature scaling is a method used to standardize the range of
independent variables or features of the data.
• It is one of data normalization methodsa in a broad sense.
• It is generally performed during the data preprocessing step.
• There are some scale-invariant algorithms such as decision trees

and random forests.
• Most of other algorithms (we have learned) perform better with fea-
ture scaling.
a
In a broad sense, data normalization is a process of reorganizing data, by cleaning and adjusting
data values measured on different scales to a notionally common scale; its intention is to bring the
entire probability distributions of adjusted values into alignment.
There are two common approaches to bring different features onto the same
scale:
• min-max scaling (normalization):
(i)
(i) xj − xj,min
xj,norm = ∈ [0, 1], (6.3)
xj,max − xj,min
where xj,min and xj,max are the minimum and maximum of the j-th fea-
ture column (in the training dataset), respectively.
• standardization:
(i)
(i) xj − µj
xj,std = , (6.4)
σj
where µj is the sample mean of the j-th feature column and σj is the
corresponding standard deviation.
– The standardized data has the standard normal distribution.

6.3. Feature Scaling 139
Remark 6.4. Standardization is more practical than the min-max

scaling for many ML methods, especially for optimization algorithms
such as the gradient descent method.
• Reason 1 : For many linear models such as the logistic regression and
the SVM, we can easily initialize the weights to 0 or small random
values close to 0.
⇐ Standardization possibly results in w∗ small.
• Reason 2 : It makes regularization perform more effectively; see
Sections 5.2.3 and 6.4.3 for regularization.
⇐ The minimizer of the penalty term is 0.
6.4. Feature Selection

6.4.1. Selecting meaningful variables
Remark 6.5. Overfitting. If we observe that a model performs much

better on a training dataset than on the test dataset, it is a strong
indicator of overfitting.
• Overfitting means the model fits the parameters too closely
with regard to the particular observations in the training dataset,
but does not generalize well to new data. (The model has a high
variance.)
• The reason for the overfitting is that our model is too complex for
the given training data.
Common solutions to reduce the generalization error (via bias-variance

tradeoff) are listed as follows:
• Collect more training data (often, not applicable)
• Introduce regularization (penalty for complexity)
• Choose a simpler model (fewer parameters)
• Reduce the dimensionality (feature selection)
Feature Selection (a.k.a. Variable Selection)

Its objective is four-fold:
• enhancing generalization by reducing overfitting/variance,
• providing faster and more cost-effective predictors,
• reducing training time, and
• providing a better understanding of the underlying process that gen-
erated the data.
Recall: Curse of Dimensionality. It describes the phenomenon where

the feature space becomes increasingly sparse for an increasing number
of dimensions of a fixed-size training dataset.
6.4. Feature Selection 141
Methods for “automatic” feature selection

• Filter methods: Filter methods suppress the least interesting
features, after assigning a scoring to each feature and ranking the
features. The methods consider the feature independently, or with re-
gard to the dependent variable.
Examples: Chi-squared test & correlation coefficient scores.
• Wrapper methods: Wrapper methods evaluate subsets of features
which allows, unlike filter approaches, to detect the possible inter-
actions between features. They prepare various combinations
of features, to evaluate and compare with other combinations. The
two main disadvantages of these methods are:
– Increasing overfitting risk, when the data size is not enough.
– Significant computation time, for a large number of variables.
Example: The recursive feature elimination algorithm
• Embedded methods: Embedded methods have been recently pro-
posed that try to combine the advantages of both previous meth-
ods. They learn which features contribute the best to the accuracy of
the model while the model is being created. The most common types of
embedded feature selection methods are regularization methods.
Examples: ridge regressiona , LASSOb , & elastic net regularizationc
a
The ridge regression (a.k.a. Tikhonov regularization) is the most commonly used method of reg-
ularization of ill-posed problems. In machine learning, ridge regression is basically a regularized linear
regression model: minw Q(X, y; w) + λ2 kwk22 , in which the regularization parameter λ should be learned
as well, using a method called cross validation. It is related to the Levenberg-Marquardt algorithm for
non-linear least-squares problems.
b
LASSO (least absolute shrinkage and selection operator) is a regression analysis method that per-
forms both variable selection and regularization in order to enhance the prediction accuracy and inter-
pretability of the statistical model it produces. It includes an L1 penalty term: minw Q(X, y; w) + λkwk1 .
It was originally developed in Geophysics [68, (Santosa-Symes, 1986)], and later independently redis-
covered and popularized in 1996 by Robert Tibshirani [75], who coined the term and provided further
insights into the observed performance.
c
The elastic net regularization is a regularized regression method that linearly combines the L1
and L2 penalties of the LASSO and ridge methods, particularly in the fitting of linear or logistic regres-
sion models.
We will see how L1 -regularization can reduce overfitting (serving as a fea-

ture selection method).
6.4.2. Sequential backward selection (SBS)

The idea behind the sequential backward selection (SBS) algorithm is
quite simple:
• The SBS sequentially removes features one-by-one until the new
feature subspace contains the desired number of features.
• In order to determine which feature is to be removed at each stage,
we need to define the criterion function C, e.g., performance of the
classifier after the removal of a particular feature.
• Then, the feature to be removed at each stage can simply be defined
as the feature that maximizes this criterion; or in more intuitive
terms, at each stage we eliminate the feature that causes the least
performance loss after removal.
Algorithm 6.6. Sequential Backward Selection

We can outline the algorithm in four simple steps:
1. Initialize the algorithm with k = d, where d is the dimensionality of
the full feature space Fd .
2. Determine the feature fb such that
fb = arg max C(Fk − f ).
f ∈Fk
3. Remove the feature fb from the feature set:

Fk−1 = Fk − fb; k = k − 1;
4. Terminate if k equals the number of desired features; otherwise, go

to step 2.
6.4.3. Ridge regression vs. LASSO
Observation 6.7. When an Lp -penalty term is involved (p = 1, 2), the

minimization problem can be written as follows:
w∗ = arg min Q(X, y; w) + λRp (w), (6.5)

w
where
1
Rp (w) := kwkpp , p = 1, 2. (6.6)
p
• Regularization can be considered as adding a penalty term to the
cost function to encourage smaller weights; or in other words, we
penalize large weights.
• Thus, by increasing the regularization strength (λ ↑),
– we can shrink the weights towards zero, and
– decrease the dependence of our model on the training data.
• The minimizer w∗ must be the point where the Lp -ball intersects
with the minimum-valued contour of the unpenalized cost function.
– The variable λ in (6.5) is a kind of Lagrange multiplier.
Pm
Figure 6.1: L2 -regularization (kwk22 = i=1 wi2 ) and L1 -regularization (kwk1 =
P m
i=1 |wi |).
LASSO (L1 -regularization). In the right figure, the L1 -ball touches the
minimum-valued contour of the cost function at w1 = 0; the optimum
is more likely located on the axes, which encourages sparsity (zero
entries in w∗ ).
Remark 6.8. LASSO (L1 -regularization)

• We can enforce sparsity (more zero entries) by increasing the reg-
ularization strength λ.
• A sparse model is a model where many of the weights are 0 or close
to 0. Therefore L1 -regularization is more suitable to create desired
0-weights, particularly for sparse models.
Remark 6.9. Regularization

In general, regularization can be understood as adding bias and pre-
ferring a simpler model to reduce the variance (overfitting),
in the absence of sufficient training data, in particular.
• L1 -regularization encourages sparsity.
• We can enforce sparsity (more zero entries) by increasing the reg-
ularization strength λ.
• Thus it can reduce overfitting, serving as a feature selection
method.
• L1 -regularization may introduce oscillation, particularly when the

regularization strength λ is large.
• A post-processing operation may be needed to take into account
oscillatory behavior of L1 -regularization.
Example 6.10. Consider a model consisting of the weights w =

(w1 , · · · , wm )T and
m m
X 1X 2
R1 (w) = |wi |, R2 (w) = wi . (6.7)
i=1
2 i=1
Let us minimize Rp (w), p = 1, 2, using gradient descent.
Solution. The gradients read
∇w R1 (w) = sign(w), ∇w R2 (w) = w, (6.8)
where 
1, if wi > 0

sign(wi ) = −1, if wi < 0
0, if wi = 0.

Thus the gradient descent becomes

R1 : wk+1 = wk − λ sign(wk ),
(6.9)
R2 : wk+1 = wk − λ wk = (1 − λ) wk = (1 − λ)k+1 w0 .
• The L2 -gradient is linearly decreasing towards 0 as the weight goes

towards 0. Thus L2 -regularization will move any weight towards 0,
but it will take smaller and smaller steps as a weight approaches 0.
(The model never reaches a weight of 0.)
• In contrast, L1 -regularization will move any weight towards 0 with
the same step size λ, regardless the weight’s value.
– The iterates for minimizing R1 may oscillate endlessly near 0.
(e.g., w0 = 0.2 and λ = 0.5
⇒ w1 = −0.3 ⇒ w2 = 0.2 ⇒ w3 = −0.3 ⇒ w4 = 0.2 ⇒ · · · )
– The oscillatory phenomenon may not be severe for real-world prob-
lems where R1 is used as a penalty term for a cost function.
– However, we may need a post-processing to take account of os-
cillation, when λ is set large.
6.5. Feature Importance

The concept of feature importance is straightforward: it is the increase
in the model’s prediction error after we permuted the feature’s values,
which breaks the relationship between the feature and the true outcome.
• A feature is “important" if shuffling its values increases the
model error, because in this case the model relied on the feature
for the prediction.
• A feature is “unimportant" if shuffling its values leaves the model
error unchanged, because in this case the model ignored the feature
for the prediction.
• The permutation feature importance measurement was introduced

by Breiman (2001) [8] for random forests.
• Based on this idea, Fisher, Rudin, and Dominici (2018) [19] proposed a
model-agnostic version of the feature importance and called it model
reliance.
Algorithm 6.11. Permutation feature importance (FI)
input: Trained model f , feature matrix X, target vector y,

error measure L(f, X, y);
1. Estimate the original model error εorig = L(f, X, y);
2. For each feature j = 1, 2, · · · , d; do:

Permute feature j in the data X to get X (j) ;
 Estimate error ε(j) = L(f, X (j) , y);

Calculate permutation FI: F I (j) = ε(j) /εorig (or, ε(j) − εorig );

3. Sort features by descending F I;
6.5. Feature Importance 147
Should we compute FI on training or test data?
To answer the question, you need to decide whether

• you want to know how much the model relies on each feature
for making predictions (→ training data) or
• how much the feature contributes to the performance of the
model on unseen data (→ test data).
There is no research addressing the question of training vs. test data; more
research and more experience are needed to gain a better understanding.
First, read pp. 135-143, Python Machine Learning, 3rd Ed..

6.1. On pp. 135-143, the sequential backward selection (SBS) is implemented as a
feature selection method and experimented with a k-NN classifier (n_neighbors=5),
using the wine dataset.
(a) Perform the same experiment with the k-NN classifier replaced by the support
vector machine (soft-margin SVM classification).
(b) In particular, analyze accuracy of the soft-margin SVM and plot the result
as in the figure on p. 139.
6.2. On pp. 141-143, the permutation feature importance is assessed from the ran-
dom forest classifier, using the wine dataset.
(a) Discuss whether or not you can derive feature importance for a k-NN classifier.
(b) Assess feature importance with the logistic regression classifier, using the
same dataset.
(c) Based on the computed feature importance, analyze and plot accuracy of the
logistic regression classifier for k_features = 1, 2, · · · , 13.
7
C HAPTER
Feature Extraction: Data Compression

There are two main categories of dimensionality reduction methods:
• Feature selection: Select a subset of the original features.
• Feature extraction: Construct a new feature subspace.
Feature Extraction
• It can be understood as an approach to dimensionality reduction
and data compression.
– with the goal of maintaining most of the relevant information
• In practice, feature extraction is used
– to improve storage space or the computational efficiency
– to improve the predictive performance by reducing the curse
of dimensionality
In this chapter, we will study three fundamental techniques for dimen-

sionality reduction:
• Principal component analysis (PCA)
• Linear discriminant analysis (LDA), maximizing class separability
• Kernel principal component analysis, for nonlinear PCA
7.1. Principal Component Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
7.2. Singular Value Decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
7.3. Linear Discriminant Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
7.4. Kernel Principal Component Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
149
150 Chapter 7. Feature Extraction: Data Compression
7.1. Principal Component Analysis

• Principal component analysis (PCA) (a.k.a. orthogonal linear
transformation) was invented in 1901 by K. Pearson [58], as an ana-
logue of the principal axis theorem in mechanics; it was later indepen-
dently developed and named by H. Hotelling in the 1930s [33, 34].
• The PCA is a statistical procedure that uses an orthogonal trans-
formation to convert a set of observations (of possibly correlated vari-
ables) to a set of linearly uncorrelated variables called the prin-
cipal components.
• The orthogonal axes of the new subspace can be interpreted as the
directions of maximum variance given the constraint that the new
feature axes are orthogonal to each other:
Figure 7.1: Principal components.
• As an unsuperviseda linear transformation technique, the PCA is

widely used across various fields – in ML, most prominently for fea-
ture extraction and dimensionality reduction.
• The PCA identifies patterns in data based on the correlation be-
tween features.
• The PCA directions are highly sensitive to data scaling, and we
need to standardize the features prior to PCA.
a
The PCA is a unsupervised technique, because it does not use any class label information.
7.1. Principal Component Analysis 151
7.1.1. Computation of principal components
• Consider a data matrix X ∈ RN ×d :

– each of the N rows represents a different data point,
– each of the d columns gives a particular kind of feature, and
– each column has zero empirical mean (e.g., after standardization).
• The goal is to find an orthogonal weight matrix W ∈ Rd×d such
that
Z = XW (7.1)
maximizes the variance (⇒ minimizes the reconstruction error).
• Here Z ∈ RN ×d is called the score matrix, of which columns represent
principal components of X.
First weight vector w1 : the first column of W :

In order to maximize variance of z1 , the first weight vector w1 should satisfy
w1 = arg max kz1 k2 = arg max kXwk2
kwk=1 kwk=1
wT X T Xw (7.2)
T T
= arg max w X Xw = arg max ,
kwk=1 w6=0 wT w
where the quantity to be maximized can be recognized as a Rayleigh quo-
tient.
Theorem 7.1. For a positive semidefinite matrix (such as X T X), the

maximum of the Rayleigh quotient is the same as the largest eigenvalue
of the matrix, which occurs when w is the corresponding eigenvector, i.e.,
wT X T Xw v1
w1 = arg max = , (X T X)v1 = λ1 v1 , (7.3)
w6=0 wT w kv1 k
where λ1 is the largest eigenvalue of X T X ∈ Rd×d .
Example 7.2. With w1 found, the first principal component of a data

(i)
vector x(i) can then be given as a score z1 = x(i) · w1 .
Further weight vectors wk :

The k-th weight vector can be found by 1 subtracting the first (k − 1) prin-
cipal components from X:
k−1
X
Xk := X −
b Xwi wiT , (7.4)
i=1
and then 2 finding the weight vector which extracts the maximum vari-
ance from this new data matrix Xbk :
wk = arg max kX bk wk2 . (7.5)
kwk=1
Claim 7.3. The above turns out to give the (normalized) eigenvectors of
X T X. That is, the transformation matrix W is the stack of eigenvec-
tors of X T X:
W = [w1 |w2 | · · · |wd ], (X T X) wj = λj wj , wiT wj = δij , (7.6)
where λ1 ≥ λ2 ≥ · · · ≥ λd ≥ 0.
With W found, a data vector x is transformed to a d-dimensional row vec-

tor of principal components
z = xW, (7.7)
of which components zj , j = 1, 2, · · · , d, are decorrelated.
Remark 7.4. From Singular Value Decomposition:

While the weight matrix W ∈ Rd×d is the collection of eigenvectors of
X T X, the score matrix Z ∈ RN ×d is the stack of eigenvectors of XX T ,
scaled by the square-root of eigenvalues:
p p p
Z = [ λ1 u1 | λ2 u2 | · · · | λd ud ], (XX T ) uj = λj uj , uTi uj = δij . (7.8)
See (7.14) and § 7.2.

7.1.2. Dimensionality reduction

The transformation Z = XW maps data points in Rd to a new d-dimensional
space of principal components. Keeping only the first k principal com-
ponents (k < d) gives a truncated transformation:
Zk = X Wk : x(i) ∈ Rd 7→ z(i) ∈ Rk , (7.9)
where Zk ∈ RN ×k and Wk ∈ Rd×k . Define the truncated data as
Xk := Zk WkT = X Wk WkT . (7.10)
Quesitons. How can we choose k ?

Is the difference kX − Xk k small ?
Remark 7.5. The principal components transformation can also be as-

sociated with the singular value decomposition (SVD) of X:
X = U ΣV T , (7.11)
where
U : n × d orthogonal (the left singular vectors of X.)
Σ : d × d diagonal (the singular values of X.)
V : d × d orthogonal (the right singular vectors of X.)
• The matrix Σ explicitly reads

Σ = diag(σ1 , σ2 , · · · , σd ), (7.12)
where σ1 ≥ σ2 ≥ · · · ≥ σd ≥ 0.
• In terms of this factorization, the matrix X T X reads
X T X = (U ΣV T )T U ΣV T = V ΣU T U ΣV T = V Σ2 V T . (7.13)
• Comparing with the eigenvector factorization of X T X, we have

– the right singular vectors V ∼ = the eigenvectors of X T X ⇒ V ∼
=W
– the square of singular values of X are equal to the eigenvalues of X T X
⇒ σj2 = λj , j = 1, 2, · · · , d.
Further considerations for the SVD

• Using the SVD, the score matrix Z reads
Z = XW = U ΣV T W = U Σ, (7.14)
and therefore each column of Z is given by one of the left singular

vectors of X multiplied by the corresponding singular value. This form
is also the polar decomposition of Z. See (7.8) on p. 152.
• As with the eigen-decomposition, the SVD, the truncated score ma-
trix Zk ∈ RN ×k can be obtained by considering only the first k largest
singular values and their singular vectors:
Zk = XWk = U ΣV T Wk = U Σk , (7.15)
where
Σk := diag(σ1 , · · · , σk , 0, · · · , 0). (7.16)
• Now, using (7.15), the truncated data matrix reads
Xk = Zk WkT = U Σk WkT = U Σk W T = U Σk V T . (7.17)
Claim 7.6. It follows from (7.11) and (7.17) that
kX − Xk k2 = kU ΣV T − U Σk V T k2
= kU (Σ − Σk )V T k2 (7.18)
= kΣ − Σk k2 = σk+1 ,
where k · k2 is the induced matrix L2 -norm.
Remark 7.7. Efficient algorithms exist to calculate the SVD of X with-

out having to form the matrix X T X. Computing the SVD is now the
standard way to carry out the PCA. See [27, 79].
7.1.3. Explained variance
Note: Since we want to reduce the dimensionality of our dataset by com-

pressing it onto a new feature subspace, we only select the subset of the
eigenvectors (principal components) that contains most of the infor-
mation (variance). The eigenvalues define the magnitude of the
eigenvectors, so we have to sort the eigenvalues by decreasing magni-
tude; we are interested in the top k eigenvectors based on the values of
their corresponding eigenvalues.
Definition 7.8. Let λj (= σj2 ) be eigenvalues of X T X: (X T X)vj = λj vj .

Define the explained variance ratio of each eigenvalue as
λi
evr(λi ) = Pd , i = 1, 2, · · · , d, (7.19)
λ
j=1 j
and cumulative explained variance as

k
X k
X d
.X
cev(λk ) = evr(λi ) = λi λj , k = 1, 2, · · · , d. (7.20)
i=1 i=1 j=1
Then, we may choose k satisfying

cev(λk−1 ) < ε and cev(λk ) ≥ ε, (7.21)
for a tolerance ε. ( The smallest k such that cev(λk ) ≥ ε. )
Figure 7.2: evr and cev for the wine dataset.

7.2. Singular Value Decomposition

Here we will deal with the SVD in detail.
Theorem 7.9. (SVD Theorem). Let A ∈ Rm×n with m ≥ n. Then we

can write
A = U ΣV T, (7.22)
where U ∈ Rm×n and satisfies U T U = I, V ∈ Rn×n and satisfies V T V = I,
and Σ = diag(σ1 , σ2 , · · · , σn ), where
σ1 ≥ σ2 ≥ · · · ≥ σn ≥ 0.
Remark 7.10. The matrices are illustrated pictorially as

   
  
   
   
 A  =  U  Σ  V T , (7.23)
   
   
where
U : m × n orthogonal (the left singular vectors of A.)
Σ : n × n diagonal (the singular values of A.)
V : n × n orthogonal (the right singular vectors of A.)
• For some r ≤ n, the singular values may satisfy
σ ≥ σ2 ≥ · · · ≥ σr > σr+1 = · · · = σn = 0. (7.24)

|1 {z }
nonzero singular values
In this case, rank(A) = r.

• If m < n, the SVD is defined by considering AT .
7.2. Singular Value Decomposition 157
Proof. (of Theorem 7.9) Use induction on m and n: we assume that

the SV D exists for (m − 1) × (n − 1) matrices, and prove it for m × n. We
assume A 6= 0; otherwise we can take Σ = 0 and let U and V be arbitrary
orthogonal matrices.
• The basic step occurs when n = 1 (m ≥ n). We let A = U ΣV T with
U = A/||A||2 , Σ = ||A||2 , V = 1.
• For the induction step, choose v so that
||v||2 = 1 and ||A||2 = ||Av||2 > 0.

Av
• Let u = ||Av||2 , which is a unit vector. Choose Ũ , Ṽ such that
U = [u Ũ ] ∈ Rm×n and V = [v Ṽ ] ∈ Rn×n
are orthogonal.
• Now, we write
" #
uT uT Av uT AṼ

T
U AV = · A · [v Ṽ ] =
Ũ T Ũ T Av Ũ T AṼ
Since
T (Av)T (Av) ||Av||22

u Av = = = ||Av||2 = ||A||2 ≡ σ,
||Av||2 ||Av||2
Ũ T Av = Ũ T u||Av||2 = 0,
we have
T
σ 0 1 0 σ 0 1 0
U T AV = = ,
0 U1 Σ1 V1T 0 U1 0 Σ1 0 V1
or equivalently
T
1 0 σ 0 1 0
A= U V . (7.25)
0 U1 0 Σ1 0 V1
Equation (7.25) is our desired decomposition.

7.2.1. Interpretation of the SVD

Algebraic interpretation of the SVD
Let rank(A) = r. let the SVD of A be A = U Σ V T , with
U = [u1 u2 · · · un ],
Σ = diag(σ1 , σ2 , · · · , σn ),
V = [v1 v2 · · · vn ],
and σr be the smallest positive singular value. Since
A = U Σ V T ⇐⇒ AV = U ΣV T V = U Σ,
we have
AV = A[v1 v2 vn ] = [Av1 Av2 · · · Avn ]
···
 
σ1
 ... 
 
= [u1 · · · ur · · · un ] 

σr

 (7.26)

 . ..


0
= [σ1 u1 · · · σr ur 0 · · · 0].
Therefore,

Avj = σj uj , j = 1, 2, · · · , r
A = U ΣV T ⇔ (7.27)
Avj = 0, j = r + 1, · · · , n
Similarly, starting from AT = V Σ U T ,

T
T T A uj = σj vj , j = 1, 2, · · · , r
A = V ΣU ⇔ (7.28)
AT uj = 0, j = r + 1, · · · , n
Summary 7.11. It follows from (7.27) and (7.28) that
• (vj , σj2 ), j = 1, 2, · · · , r, are eigenvector-eigenvalue pairs of AT A.

AT A vj = AT (σj uj ) = σj2 vj , j = 1, 2, · · · , r. (7.29)
So, the singular values play the role of eigenvalues.

• Similarly, we have
AAT uj = A(σj vj ) = σj2 uj , j = 1, 2, · · · , r. (7.30)
• Equation (7.29) gives how to find the singular values {σj } and the
right singular vectors V , while (7.27) shows a way to compute the
left singular vectors U .
• (Dyadic decomposition) The matrix A ∈ Rm×n can be expressed as
n
X
A= σj uj vjT . (7.31)
j=1
When rank(A) = r ≤ n,
r
X
A= σj uj vjT . (7.32)
j=1
This property has been utilized for various approximations and ap-
plications, e.g., by dropping singular vectors corresponding to small
singular values.
Geometric interpretation of the SVD

The matrix A maps an orthonormal basis
B1 = {v1 , v2 , · · · , vn }
of Rn onto a new “scaled” orthogonal basis
B2 = {σ1 u1 , σ2 u2 , · · · , σr ur }
for a subspace of Rm :
A
B1 = {v1 , v2 , · · · , vn } −
→ B2 = {σ1 u1 , σ2 u2 , · · · , σr ur } (7.33)
Consider a unit sphere S n−1 in Rn :

n Xn o
n−1 2
S = x xj = 1 .

j=1
Then, ∀x ∈ S n−1 ,
x = x1 v1 + x2 v2 + · · · + xn vn
Ax = σ1 x1 u1 + σ2 x2 u2 + · · · + σr xr ur (7.34)
= y1 u1 + y2 u2 + · · · + yr ur , (yj = σj xj )
So, we have
yj
yj = σj xj ⇐⇒ xj =
σj
n r
X yj2 (7.35)
x2j
P
= 1 (sphere) ⇐⇒ = α ≤ 1 (ellipsoid)
j=1 j=1
σj2
Example 7.12. We build the set A(S n−1 ) by multiplying one factor of A =
U ΣV T at a time. Assume for simplicity that A ∈ R2×2 and nonsingular. Let

3 −2
A = = U ΣV T
−1 2

−0.8649 0.5019 4.1306 0 −0.7497 0.6618
=
0.5019 0.8649 0 0.9684 0.6618 0.7497
Then, for x ∈ S 1 ,
Ax = U ΣV T x = U Σ(V T x)

In general,
• V T : S n−1 → S n−1 (rotation in Rn )

• Σ : ej 7→ σj ej (scaling from S n−1 to Rn )
• U : Rn → Rm (rotation)
7.2.2. Properties of the SVD
Theorem 7.13. Let A ∈ Rm×n with m ≥ n. Let A = U ΣV T be the SVD

of A, with
σ1 ≥ · · · ≥ σr > σr+1 = · · · = σn = 0.
Then, 

 rank(A) = r

 Null(A)


 = Span{vr+1 , · · · , vn }
Range(A) = Span{u1 , · · · , ur } (7.36)

 X r

A = σi ui viT



i=1
and 

 ||A||2 = σ1 (See Exercise 2.)
||A||2F = σ12 + · · · + σr2 (See Exercise 3.)





||Ax||2


min = σn (m ≥ n) (7.37)
x6=0 ||x||2
σ1


κ2 (A) = ||A||2 · ||A−1 ||2 =





 σn
( when m = n, & ∃A−1 )

Theorem 7.14. Let A ∈ Rm×n , m ≥ n, rank(A) = n, with singular

values
σ1 ≥ σ2 ≥ · · · σn > 0.
Then
||(AT A)−1 ||2 = σn−2 ,
||(AT A)−1 AT ||2 = σn−1 ,
(7.38)
||A(AT A)−1 ||2 = σn−1 ,
||A(AT A)−1 AT ||2 = 1.
Definition 7.15. (AT A)−1 AT is called the pseudoinverse of A, while

A(AT A)−1 is called the pseudoinverse of AT . Let A = U ΣV T be the SVD
of A. Then def
(AT A)−1 AT = V Σ−1 U T == A+ . (7.39)
Theorem 7.16. Let A ∈ Rm×n with rank(A) = r > 0. Let A = U ΣV T be

the SVD of A, with singular values
σ1 ≥ · · · ≥ σr > 0.
Define, for k = 1, · · · , r − 1,
k
X
Ak = σj uj vjT (sum of rank-1 matrices).
j=1
Then, rank(Ak ) = k and

||A − Ak ||2 = min{||A − B||2 rank(B) ≤ k}
= σk+1 ,
(7.40)
min{||A − B||2F rank(B) ≤ k}

||A − Ak ||2F =
2
= σk+1 + · · · + σr2 .
That is, of all matrices of rank ≤ k, Ak is closest to A.
Note: The matrix Ak can be written as
A k = U Σk V T , (7.41)
where Σk = diag(σ1 , · · · , σk , 0, · · · , 0). The pseudoinverse of Ak reads
A+ + T
k = V Σk U , (7.42)
where
Σ+
k = diag(1/σ1 , 1/σ2 , · · · , 1/σk , 0, · · · , 0). (7.43)
Corollary 7.17. Suppose A ∈ Rm×n has full rank; rank(A) = n. Let

σ1 ≥ · · · ≥ σn be the singular values of A. Let B ∈ Rm×n satisfy
||A − B||2 < σn .
Then B also has full rank.

Full SVD
• For A ∈ Rm×n ,
A = U ΣV T ⇐⇒ U T AV = Σ,
where U ∈ Rm×n and Σ, V ∈ Rn×n .
• Expand
[U U#2 ] ∈ Rm×m ,
U → Ũ = " (orthogonal)
Σ
Σ → Σ̃ = ∈ Rm×n ,
O
where O is an (m − n) × n zero matrix.
• Then, " #
Σ
Ũ Σ̃V T = [U U2 ] V T = U ΣV T = A (7.44)
O
7.2.3. Computation of the SVD
For A ∈ Rm×n , the procedure is as follows.

1. Form AT A (AT A – covariance matrix of A).
2. Find the eigen-decomposition of AT A by orthogonalization process,
i.e., Λ = diag(λ1 , · · · , λn ),
AT A = V ΛV T ,
where V = [v1 ··· vn ] is orthogonal, i.e., V T V = I.

3. Sort the eigenvalues according to their magnitude and let
p
σj = λj , j = 1, 2, · · · , n.
4. Form the U matrix as follows,

1
uj = Avj , j = 1, 2, · · · , r.
σj
If necessary, pick up the remaining columns of U so it is orthogonal.
(These additional columns must be in Null(AAT ).)
 T 
v1
5. A = U ΣV T = [u1 · · · ur · · · un ] diag(σ1 , · · · , σr , 0, · · · , 0)  ... 
 
vnT
Lemma 7.18. Let A ∈ Rn×n be symmetric. Then (a) all the eigenvalues
of A are real and (b) eigenvectors corresponding to distinct eigenvalues
are orthogonal.
Proof. See Exercise 4.

 
1 2
Example 7.19. Find the SV D for A =  −2 1 .
3 2
Solution.

T 14 6
1. A A = .
6 9
2. Solving det(AT A − λI) = 0 gives the eigenvalues of AT A
λ1 = 18 and λ2 = 5,
of which corresponding eigenvectors are

√3 − √213
" # " # " #
3 −2 13
ṽ1 = , ṽ2 = . =⇒ V =
2 3 √2 √3
13 13
√ √ √ √ √
3. σ1 = λ1 = 18 = 3 2, σ2 = λ2 = 5. So
√
18 √0
Σ=
0 5
√7
   
" # 7
√3 234
1 1 13 1 √1    − √4 
4. u1 = σ1 Av1 = 18 A
√ = 18 13  −4  = 
√
√2 234 
13 √13
13 234
   4 
"
−2
# 4 √
1 1
√
13 1 √1    √765 
u2 = σ2 Av2 = 5 A
√ = 5 13  7  =  65  .
√
√3
13
0 0
 
√7 √4
234 65 √ " √3 √2 #

√4 √7 
 18 √0 13 13
5. A = U ΣV T =  − 234 65  2 3
 0 5 − √13 √13
√13 0
234
Example 7.20. Find the pseudoinverse of A,
A+ = (AT A)−1 AT = V Σ−1 U T ,

 
1 2
when A =  −2 1 .
3 2
Solution. From Example 7.19, we have
7 4
 
234
√ √
65
√ " √3 √2
#
4 √7
18 √0
A = U ΣV T =  − 234
√ 13 13
 
65 0 5 − √213 √3

√13 0 13
234
Thus,
" #" #" #
√3 − √213 √1 0 √7 4
− √234 √13
A+ = V Σ−1 U T = 13
√2 √3
18
√1
234
√4 √7
234
13 13
0 5 65 65
0
1 4 1

− 30 − 15 6
= 11 13 1
45 45 9
Computer implementation [25]
Figure 7.3: A two-phase procedure for the SVD: A = U ΣV T .
Algorithm 7.21. (Golub and Reinsch, 1970) [26]. Let A ∈ Rm×n .

• Phase 1: It constructs two finite sequences of Householder
transformations to find an upper bidiagonal matrix:
Pn · · · P1 A Q1 · · · Qn−2 = B (7.45)
• Phase 2: It is to iteratively diagonalize B using the QR method.
Golub-Reinsch SVD algorithm

• It is extremely stable.
• Computational complexity:
– Computation of U , V , and Σ: 4m2 n + 8mn2 + 9n3 .
– Computation of V and Σ: 4mn2 + 8n3 .
• Phases 1 & 2 take O(mn2 ) and O(n2 ) flops, respectively.
(when Phase 2 is done with O(n) iterations)
• Python: U,S,V = numpy.linalg.svd(A)

• Matlab/Maple: [U,S,V] = svd(A)
• Mathematica: {U,S,V} = SingularValueDecomposition[A]
Numerical rank
In the absence of round-off errors and uncertainties in the data, the SVD
reveals the rank of the matrix. Unfortunately the presence of errors makes
rank determination problematic. For example, consider
 
1/3 1/3 2/3
2/3 2/3 4/3
 
A=  1/3 2/3 3/3 
 (7.46)
2/5 2/5 4/5
3/5 1/5 4/5
• Obviously A is of rank 2, as its third column is the sum of the first two.
• Matlab “svd" (with IEEE double precision) produces
σ1 = 2.5987, σ2 = 0.3682, and σ3 = 8.6614 × 10−17 .
• What is the rank of A, 2 or 3? What if σ3 is in O(10−13 )?

• For this reason we must introduce a threshold T . Then we say that A
has numerical rank r if A has r singular values larger than T , that
is,
σ1 ≥ σ2 ≥ · · · ≥ σr > T ≥ σr+1 ≥ · · · (7.47)
In Matlab
• Matlab has a “rank" command, which computes the numerical rank
of the matrix with a default threshold
T = 2 max{m, n} ||A||2 (7.48)
where is the unit round-off error.

• In Matlab, the unit round-off error can be found from the parameter
“eps"
eps = 2−52 = 2.2204 × 10−16 .
• For the matrix A in (7.46),
T = 2 · 5 · eps · 2.5987 = 5.7702 × 10−15
and therefore rank(A)=2.

See Exercise 5.
7.2.4. Application of the SVD to image compression
• A ∈ Rm×n is a sum of rank-1 matrices (dyadic decomposition):

V = [v1 , · · · , vn ], U = [u1 , · · · , un ],
n
T
X (7.49)
A = U ΣV = σi ui viT , ui ∈ Rm , vi ∈ Rn .
i=1
• The approximation
k
X
T
A k = U Σk V = σi ui viT (7.50)
i=1
is closest to A among matrices of rank≤ k, and

||A − Ak ||2 = σk+1 . (7.51)
• It only takes (m + n) · k words to store u1 through uk , and σ1 v1 through

σk vk , from which we can reconstruct Ak .
Image compression using k singular values

peppers_SVD.m
1 img = imread('Peppers.png'); [m,n,d]=size(img);
2 [U,S,V] = svd(reshape(im2double(img),m,[]));
3 %%---- select k <= p=min(m,n)
4 k = 20;
5 img_k = U(:,1:k)*S(1:k,1:k)*V(:,1:k)';
6 img_k = reshape(img_k,m,n,d);
7 figure, imshow(img_k)
The “Peppers" image is in [270, 270, 3] ∈ R270×810 .

Original (k = 270) k=1 k = 10
k = 20 k = 50 k = 100
Peppers: Singular values
Peppers: Storage: It requires (m + n) · k words.For example, when k = 50,

(m + n) · k = (270 + 810) · 50 = 54,000 , (7.52)
which is approximately a quarter the full storage space
270 × 270 × 3 = 218,700 .

7.3. Linear Discriminant Analysis

Linear discriminant analysis is a method to find a linear combination
of features that characterizes or separates two or more classes of ob-
jects or events.
• The LDA is sometimes also called Fisher’s LDA. Fisher initially for-
mulated the LDA for two-class classification problems in 1936 [20], and
later generalized for multi-class problems by C. Radhakrishna Rao under
the assumption of equal class covariances and normally distributed
classes in 1948 [61].
• The LDA may be used as a linear classifier, or, more commonly, for
dimensionality reduction (§ 7.3.4) for a later classification.
• The general concept behind the LDA is very similar to PCA.1
LDA objective
• The LDA objective is to perform dimensionality reduction.
– So what? PCA does that, too!
• However, we want to preserve as much of the class discriminatory
information as possible.
– OK, this is new!
LDA
• Consider a pattern classification problem, where we have c classes.
• Suppose each class has Nk samples in Rd , where k = 1, 2, · · · , c.
• Let Xk = {x(1) , x(2) , · · · , x(Nk ) } be the set of d-dimensional samples for
class k.
• Let X ∈ Rd×N be the data matrix, stacking all the samples from
all
P classes, such that each column represents a sample, where N =
k Nk .
• The LDA seeks to obtain a transformation of X to Z through projecting
the samples in X onto a hyperplane with dimension c − 1.
1
In PCA, the main idea is to re-express the available dataset to extract the relevant information by re-
ducing the redundancy and to minimize the noise. While (unsupervised) PCA attempts to find the
orthogonal component axes of maximum variance in a dataset, the goal in the (supervised) LDA is to find
the feature subspace that optimizes class separability.
7.3. Linear Discriminant Analysis 173
7.3.1. Fisher’s LDA (classifier): two classes

Let us define a transformation of samples x onto a line [(c − 1)-space, for
c = 2]:
z = wT x = w · x, (7.53)
where w ∈ Rd is a projection vector.
Of all the possible lines, we would like to select the one that maximizes the
separability of the scalars {z}.
• In order to find a good projection vector, we need to define a measure
of separation between the projections.
• The mean vector of each class in x and z feature space is
1 X 1 X 1 X T
µk = x, µek = z= w x = w T µk , (7.54)
Nk Nk Nk
x∈Xk x∈Xk x∈Xk
i.e., projecting x to z will lead to projecting the mean of x to the mean

of z.
• We could then choose the distance between the projected means
as our objective function:
Jb(w) = |e e2 | = |wT (µ1 − µ2 )|.

µ1 − µ (7.55)
the projected means is not a

very good measure, since it
does not take into account the
sample distribution within the
classes.
• The maximizer w∗ of (7.55) must
be parallel to (µ1 − µ2 ):
Figure 7.4: The x1 -axis has a larger dis- w∗ // (µ1 − µ2 );
tance between means, while the x2 -axis
yields a better class separability. the projection to a parallel line of
(µ1 − µ2 ) is not an optimal trans-
• However, the distance between formation.
Fisher’s LDA: The Key Idea

The solution proposed by Fisher is to maximize a function that rep-
resents the difference between the means, normalized by a measure
of the within-class variability (called the scatter).
• For each class k, we define the scatter (an equivalent of the variance)
as 2
X
sek = ek )2 , z = wT x.
(z − µ (7.56)
x∈Xk
• The quantity se2k measures the variability within class Xk after project-
ing it on the z-axis.
• Thus, se21 + se22 measures the variability within the two classes at hand
after projection; it is called the within-class scatter of the projected
samples.
• Fisher’s linear discriminant is defined as the linear function wT x
that maximizes the objective function:
∗ µ1 − µ
(e e2 )2
w = arg max J (w), where J (w) = . (7.57)
w se21 + se22
• Therefore, Fisher’s LDA searches for a projection where samples from

the same class are projected very close to each other; at the same time,
the projected means are as farther apart as possible.
Figure 7.5: Fisher’s LDA.

7.3.2. Fisher’s LDA: the optimum projection

Rewrite the Fisher’s objective function:
µ1 − µ
(e e2 )2
J (w) = , (7.58)
se21 + se22
where 1 X X
µk = x, ek = wT µk ,
µ se2k = ek )2 .
(z − µ
Nk
x∈Xk x∈Xk
• In order to express J (w) as an explicit function of w, we first define a

measure of the scatter in the feature space x:
X
Sw = S1 + S2 , for Sk = (x − µk )(x − µk )T , (7.59)
x∈Xk
where Sw ∈ Rd×d is called the within-class scatter matrix of samples x,

while Sk is the covariance matrix of class Xk .
Then, the scatter of the projection z can then be expressed as
X X
se2k = ek )2 =
(z − µ (wT x − wT µk )2
x∈X
Xk x∈Xk
= w (x − µk )(x − µk )T w
T
(7.60)
x∈Xk
X
T
= w (x − µk )(x − µk ) w = wT Sk w.
T
x∈Xk
Thus, the denominator of the objective function gives
se21 + se22 = wT S1 w + wT S2 w = wT Sw w =: Sew , (7.61)
where Sew is the within-class scatter of projected samples z.

• Similarly, the difference between the projected means (in z-space) can be
expressed in terms of the means in the original feature space (x-space).
e2 )2 = (wT µ1 − wT µ2 )2 = wT (µ1 − µ2 )(µ1 − µ2 )T w

µ1 − µ
(e
(7.62)
| {z }
=:Sb
T
= w Sb w =: Seb ,
where the rank-one matrix Sb ∈ Rd×d is called the between-class scatter

matrix of the original samples x, while Seb is the between-class scatter
of the projected samples z.
• Since Sb is the outer product of two vectors, rank(Sb ) ≤ 1.
We can finally express the Fisher criterion in terms of Sw and Sb as
µ1 − µ
(e e2 )2 wT Sb w
J (w) = = T . (7.63)
se21 + se22 w Sw w
Hence, J (w) is a measure of the difference between class means (en-
coded in the between-class scatter matrix), normalized by a measure
of the within-class scatter matrix.
• To find the maximum of J (w), we differentiate it with respect to w and

equate to zero. Applying some algebra leads (Exercise 6)
Sw−1 Sb w = J (w) w. (7.64)
Note that Sw−1 Sb is a rank-one matrix.

Equation (7.64) is a generalized eigenvalue problem:
Sw−1 Sb w = λ w ⇐⇒ Sb w = λ Sw w; (7.65)
the maximizer w∗ of J (w) is the eigenvector associated with the nonzero

eigenvalue λ∗ = J (w).
Summary 7.22. Finding the eigenvector of Sw−1 Sb associated with the

largest eigenvalue yields
wT Sb w
w∗ = arg max J (w) = arg max T . (7.66)
w w w Sw w
This is known as Fisher’s Linear Discriminant, although it is not a discrim-

inant but a specific choice of direction for the projection of the data down to
one dimension.
Fisher’s LDA: an example

We will compute the Linear Discriminant projection for the following two-
dimensional dataset of two classes (c = 2).
lda_Fisher.m
1 m=2; n=5;
2
3 X1=[2,3; 4,3; 2,1; 3,4; 5,4];

4 X2=[7,4; 6,8; 7,6; 8,9; 10,9];
5
6 Mu1 = mean(X1)'; % Mu1 = [3.2,3.0]

7 Mu2 = mean(X2)'; % Mu2 = [7.6,7.2]
8
9 S1 = cov(X1,0)*n;
10 S2 = cov(X2,0)*n;
11 Sw = S1+S2; % Sw = [20,13; 13,31]
12
13 Sb = (Mu1-Mu2)*(Mu1-Mu2)'; % Sb = [19.36,18.48; 18.48,17.64]

14
15 invSw_Sb = inv(Sw)*Sb;
16 [V,L] = eig(invSw_Sb); % V1 = [ 0.9503,0.3113]; L1 = 1.0476
17 % V2 = [-0.6905,0.7234]; L2 = 0.0000
Figure 7.6: A synthetic dataset and Fisher’s LDA projection.

7.3.3. LDA for multiple classes

• Now, we have c-classes instead of just two.
• We are now seeking (c−1) projections [z1 , z2 , · · · , zc−1 ] by means of (c−1)
projection vectors wk ∈ Rd .
• Let W = [w1 |w2 | · · · |wc−1 ], a collection of column vectors, such that
zk = wkT x =⇒ z = W T x ∈ Rc−1 . (7.67)
• If we have N sample (column) vectors, we can stack them into one ma-
trix as follows.
Z = W T X, (7.68)
where X ∈ Rd×N , W ∈ Rd×(c−1) , and Z ∈ R(c−1)×N .
Recall: For the two classes case, the within-class scatter matrix was
computed as
Sw = S1 + S2 .
This can be generalized in the c-classes case as:

c
X X
Sw = Sk , Sk = (x − µk )(x − µk )T , (7.69)
k=1 x∈Xk
where µk = N1k x∈Xk x, where Nk is the number of data samples in class

P
Xk , and Sw ∈ Rd×d .
Recall: For the two classes case, the between-class scatter matrix
was computed as
Sb = (µ1 − µ2 )(µ1 − µ2 )T .
For c-classes case, we will measure the between-class scatter matrix

with respect to the mean of all classes as follows:
c
X 1 X
Sb = Nk (µk − µ)(µk − µ)T , µ= x, (7.70)
N
k=1 ∀x
where rank(Sb ) = c − 1.
Definition 7.23. As an analogue to (7.66), we may define the LDA

optimization, for c classes case, as follows.
∗ W T Sb W
W = arg max J (W ) = arg max T . (7.71)
W W W Sw W
Recall: For two-classes case, when we set ∂J∂w

(w)
= 0, the optimization
problem is reduced to the eigenvalue problem
Sw−1 Sb w∗ = λ∗ w∗ , where λ∗ = J (w∗ ).
For c-classes case, we have (c − 1) projection vectors. Hence the eigenvalue

problem can be generalized to the c-classes case:
Sw−1 Sb wk∗ = λ∗k wk∗ , λ∗k = J (wk∗ ), k = 1, 2, · · · , c − 1. (7.72)
Thus, it can be shown that the optimal projection matrix
W ∗ = [w1∗ |w1∗ | · · · |wc−1

∗
] ∈ Rd×(c−1) (7.73)
is the one whose columns are the eigenvectors corresponding to the

eigenvalues of the following generalized eigenvalue problem:
Sw−1 Sb W ∗ = λ∗ · W ∗ , λ∗ = [λ∗1 , · · · , λ∗c−1 ], (7.74)
where Sw−1 Sb ∈ Rd×d and (·) denotes the pointwise product.

Illustration – 3 classes
• Let us generate a dataset for each class to illustrate the LDA transfor-
mation.
• For each class:
– Use the random number generator to generate a uniform stream of
500 samples that follows U(0, 1).
– Using the Box-Muller approach, convert the generated uniform stream
to N (0, 1).
– Then use the method of eigenvalues and eigenvectors to manip-
ulate the standard normal to have the required mean vector and
covariance matrix .
– Estimate the mean and covariance matrix of the resulted dataset.
Figure 7.7: Generated and manipulated dataset, for 3 classes.

lda_c3.m
1 close all;
2 try, pkg load statistics; end % for octave
3
4 %% uniform stream
5 U = rand(2,1000); u1 = U(:,1:2:end); u2 = U(:,2:2:end);
6
7 %% Box-Muller method to convert to N(0,1)

8 X = sqrt((-2).*log(u1)).*(cos(2*pi.*u2)); % 2 x 500
9 clear u1 u2 U;
10
11 %% manipulate for required Mean and Cov

12 Mu = [5;5];
13
14 Mu1= Mu +[-3;7]; Cov1 =[5 -1; -3 3];

15 X1 = denormalize(X,Mu1,Cov1);
16 Mu2= Mu +[-3;-4]; Cov2 =[4 0; 0 4];
18 Mu3= Mu +[7; 5]; Cov3 =[4 1; 3 3];
20
21 %%Begin the comptation of the LDA Projection Vectors

22 % estimate mean and covariance
23 N1 = size(X1,2); N2 = size(X2,2); N3 = size(X3,2);
24 Mu1 = mean(X1')'; Mu2 = mean(X2')'; Mu3 = mean(X3')';
25 Mu = (Mu1+Mu2+Mu3)/3.;
26
27 % within-class scatter matrix

28 S1 = cov(X1'); S2 = cov(X2'); S3 = cov(X3');
29 Sw = S1+S2+S3;
30
31 % between-class scatter matrix

32 Sb1 = N1 * (Mu1-Mu)*(Mu1-Mu)'; Sb2 = N2 * (Mu2-Mu)*(Mu2-Mu)';
33 Sb3 = N3 * (Mu3-Mu)*(Mu3-Mu)';
34 Sb = Sb1+Sb2+Sb3;
35
36 % computing the LDA projection

37 invSw_Sb = inv(Sw)*Sb; [V,D] = eig(invSw_Sb);
38 w1 = V(:,1); w2 = V(:,2);
39 if D(1,1)<D(2,2), w1 = V(:,2); w2 = V(:,1); end
40
41 lda_c3_visualize;
Figure 7.8: lda_c3.m

denormalize.m
1 function Xnew = denormalize(X,Mu,Cov)
2 % it manipulates data samples in N(0,1) to something else.
3
4 [V,D] = eig(Cov); VsD = V*sqrt(D);

5
6 Xnew = zeros(size(X));
7 for j=1:size(X,2)
8 Xnew(:,j)= VsD * X(:,j);
9 end
10
11 %Now, add "replicated and tiled Mu"

12 Xnew = Xnew +repmat(Mu,1,size(Xnew,2));
lda_c3_visualize.m
1 figure, hold on; axis([-10 20 -5 20]);
2 xlabel('x_1 - the first feature','fontsize',12);
3 ylabel('x_2 - the second feature','fontsize',12);
4 plot(X1(1,:)',X1(2,:)','ro','markersize',4,"linewidth",2)
5 plot(X2(1,:)',X2(2,:)','g+','markersize',4,"linewidth",2)
6 plot(X3(1,:)',X3(2,:)','bd','markersize',4,"linewidth",2)
7 hold off
8 print -dpng 'LDA_c3_Data.png'
9
10 figure, hold on; axis([-10 20 -5 20]);

11 xlabel('x_1 - the first feature','fontsize',12);
12 ylabel('x_2 - the second feature','fontsize',12);
13 plot(X1(1,:)',X1(2,:)','ro','markersize',4,"linewidth",2)
14 plot(X2(1,:)',X2(2,:)','g+','markersize',4,"linewidth",2)
15 plot(X3(1,:)',X3(2,:)','bd','markersize',4,"linewidth",2)
16
17 plot(Mu1(1),Mu1(2),'c.','markersize',20)
18 plot(Mu2(1),Mu2(2),'m.','markersize',20)
19 plot(Mu3(1),Mu3(2),'r.','markersize',20)
20 plot(Mu(1),Mu(2),'k*','markersize',15,"linewidth",3)
21 text(Mu(1)+0.5,Mu(2)-0.5,'\mu','fontsize',18)
22
23 t = -5:20; line1_x = t*w1(1); line1_y = t*w1(2);

24 plot(line1_x,line1_y,'k-',"linewidth",3);
25 t = -5:10; line2_x = t*w2(1); line2_y = t*w2(2);
26 plot(line2_x,line2_y,'m--',"linewidth",3);
27 hold off
28 print -dpng 'LDA_c3_Data_projection.png'
29
30 %Project the samples through w1

31 wk = w1;
32 z1_wk = wk'*X1; z2_wk = wk'*X2; z3_wk = wk'*X3;
33
34 z1_wk_Mu = mean(z1_wk); z1_wk_sigma = std(z1_wk);

35 z1_wk_pdf = mvnpdf(z1_wk',z1_wk_Mu,z1_wk_sigma);
36

39

42
43 figure, plot(z1_wk,z1_wk_pdf,'ro',z2_wk,z2_wk_pdf,'g+',...
44 z3_wk,z3_wk_pdf,'bd')
45 xlabel('z','fontsize',12); ylabel('p(z|w1)','fontsize',12);
46 print -dpng 'LDA_c3_Xw1_pdf.png'
47
48 %Project the samples through w2

49 wk = w2;
50 z1_wk = wk'*X1; z2_wk = wk'*X2; z3_wk = wk'*X3;
51

54

57

60
61 figure, plot(z1_wk,z1_wk_pdf,'ro',z2_wk,z2_wk_pdf,'g+',...
62 z3_wk,z3_wk_pdf,'bd')
63 xlabel('z','fontsize',12); ylabel('p(z|w2)','fontsize',12);
64 print -dpng 'LDA_c3_Xw2_pdf.png'
Figure 7.9: w1∗ (solid line in black) and w2∗ (dashed line in magenta).
• w1∗ = [0.85395, 0.52036]T , w2∗ = [−0.62899, 0.77742]T .

• Corresponding eigenvalues read
λ1 = 3991.2, λ2 = 1727.7.
Figure 7.10: Classes PDF, along the first projection vector w1∗ ; λ1 = 3991.2.
Figure 7.11: Classes PDF, along the second projection vector w2∗ ; λ2 = 1727.7.
Apparently, the projection vector that has the highest eigenvalue pro-
vides higher discrimination power between classes.
7.3.4. The LDA: dimensionality reduction
Let X ∈ RN ×d be the data matrix, in which each row represents a sam-

ple.
We summarize the main steps that are required to perform the LDA for
dimensionality reduction.
1. Standardize the d-dimensional dataset (d is the number of features).
2. For each class j, compute the d-dimensional mean vector µj .
3. Construct the within-class scatter matrix Sw (7.69) and the
between-class scatter matrix Sb (7.70).
4. Compute the eigenvectors and corresponding eigenvalues of the ma-
trix Sw−1 Sb (7.72).
5. Sort the eigenvalues by decreasing order to rank the corresponding
eigenvectors.
6. Choose the k eigenvectors that correspond to the k largest eigenvalues
to construct a transformation matrix
W = [w1 |w2 | · · · |wk ] ∈ Rd×k ; (7.75)
the eigenvectors are the columns of this matrix.

7. Project the samples onto a new feature subspace: X → Z := XW .
Remark 7.24.
• rank(Sw−1 Sb ) ≤ c − 1; we must have k ≤ c − 1.
• The projected feature Zij is x(i) · wj in the projected coordinates and
(x(i) · wj ) wj in the original coordinates.
Limitations of the LDA (classifier)

• The LDA produces at most (c − 1) feature projections.
– If the classification error estimates establish that more features
are needed, some other method must be employed to provide those
additional features.
• The LDA is a parametric method, since it assumes unimodal Gaus-
sian likelihoods.
– If the distributions are significantly non-Gaussian, the LDA pro-
jections will not be able to preserve any complex structure of the
data, which may be needed for classification.
• The LDA will fail when the discriminatory information is not in the
mean but rather in the variance of the data.
LDA vs. PCA
Figure 7.12: PCA vs. LDA.
The (supervised) LDA classifier must work better than the (unsuper-
vised) PCA, for datasets in Figures 7.9 and 7.12.
Recall: Fisher’s LDA was generalized under the assumption of equal class
covariances and normally distributed classes.
However, even if one or more of those assumptions are (slightly) violated,
the LDA for dimensionality reduction can still work reasonably well.
7.4. Kernel Principal Component Analysis 189
7.4. Kernel Principal Component Analysis

The kernel principal component analysis (kernel PCA) [70] is an ex-
tension of the PCA using kernel techniques and performing the origi-
nally linear operations of the PCA in a kernel Hilbert space.
Recall: (PCA). Consider a data matrix X ∈ RN ×d :

◦ each of the N rows represents a different data point,
◦ each of the d columns gives a particular kind of feature, and
◦ each column has zero empirical mean (e.g., after standardization).
• The goal of the standard PCA is to find an orthogonal weight matrix

Wk ∈ Rd×k such that
Zk = X Wk , k ≤ d, (7.76)
where Zk ∈ RN ×k is call the truncated score matrix and Zd = Z.
Columns of Z represent the principal components of X.
• (Claim 7.3, p. 152). The transformation matrix Wk turns out to be the
collection of normalized eigenvectors of X T X:
Wk = [w1 |w2 | · · · |wk ], (X T X) wj = λj wj , wiT wj = δij , (7.77)
where λ1 ≥ λ2 ≥ · · · ≥ λk ≥ 0.
• (Remark 7.4, p. 152). The matrix Zk ∈ RN ×k is scaled eigenvectors of
XX T :
p p p
Zk = [ λ1 u1 | λ2 u2 | · · · | λk uk ], (XX T ) uj = λj uj , uTi uj = δij .
(7.78)
• A data (row) vector x (new or old) is transformed to a k-
dimensional row vector of principal components
z = xWk ∈ R1×k . (7.79)
• (Remark 7.5, p. 153). Let X = U ΣV T be the SVD of X, where

Σ = diag(σ1 , σ2 , · · · , σd ), σ1 ≥ σ2 ≥ · · · ≥ σd ≥ 0.
Then,
V ∼
= W ; σj2 = λj , j = 1, 2, · · · , d,
(7.80)
Zk = [σ1 u1 |σ2 u2 | · · · |σk uk ].
7.4.1. Principal components of the kernel PCA

1
Note: Let C = N X T X, the covariance matrix of X. Then,
N N
1 X (i) (i) T 1 X (i) (i)
C= x x ∈ Rd×d , Cjk = x x . (7.81)
N i=1 N i=1 j k
Here, we consider x(i) as a column vector (when standing alone), while it

lies in X as a row.
• The kernel PCA is a generalization of the PCA, where the dataset X

is transformed into a higher dimensional space (by creating non-
linear combinations of the original features):
φ : X ∈ RN ×d → φ(X) ∈ RN ×p , d < p, (7.82)
and the covariance matrix is computed via outer products between

such expanded samples:
N
1 X 1
C= φ(x(i) )φ(x(i) )T = φ(X)T φ(X) ∈ Rp×p . (7.83)
N i=1 N
• To obtain the eigenvectors – the principal components – from the

covariance matrix, we should solve the eigenvalue problem:
Cv = λv. (7.84)
• Assume (7.84) is solved.

– Let, for λ1 ≥ λ2 ≥ · · · ≥ λk ≥ · · · ≥ λp ≥ 0,
Vk = [v1 |v2 | · · · |vk ] ∈ Rp×k , Cvj = λj vj , viT vj = δij . (7.85)
– Then, the score matrix Zk (principal components) for the ker-

nel PCA reads
Zk = φ(X)Vk ∈ RN ×k , (7.86)
which is an analogue to (7.76).
However, it is computationally expensive or impossible to solve the

eigenvalue problem (7.84), when p is large or infinity.
An Alternative to the Computation of the Score Matrix
Claim 7.25. Let v be an eigenvector of C as in (7.84). Then it can be

expressed as linear combination of data points:
N
X
v= αi φ(x(i) ). (7.87)
i=1
Proof. Since Cv = λv, we get

N
1 X
φ(x(i) )φ(x(i) )T v = λv
N i=1
and therefore
N N
1 X (i) (i) T 1 X
v= φ(x )φ(x ) v = [φ(x(i) ) · v] φ(x(i) ), (7.88)
λN i=1 λN i=1
where φ(x(i) ) · v is a scalar and αi := (φ(x(i) ) · v)/(λN ).
Note:
• The above claim means that all eigenvectors v with λ 6= 0 lie in the
span of φ(x(1) ), · · · , φ(x(N ) ).
• Thus, finding the eigenvectors in (7.84) is equivalent to finding the
coefficients α = (α1 , α2 , · · · , αN )T .
How to find α
• Let Cvj = λj vj with λj 6= 0. Then, (7.87) can be written as

N
X
vj = α`j φ(x(`) ) = φ(X)T αj . (7.89)
`=1
• By substituting this back into the equation and using (7.83), we get
1
Cvj = λj vj ⇒ φ(X)T φ(X)φ(X)T αj = λj φ(X)T αj . (7.90)
N
and therefore
1
φ(X)φ(X)T φ(X)φ(X)T αj = λj φ(X)φ(X)T αj . (7.91)
N
• Let K be the similarity (kernel) matrix:
def
K == φ(X)φ(X)T ∈ RN ×N . (7.92)
• Then, (7.91) can be rewritten as

K 2 αj = (N λj ) Kαj . (7.93)
• We can remove a factor of K from both sides of the above equation:a

Kαj = µj αj , µj = N λj . (7.94)
which implies that αj are eigenvectors of K .

• It should be noticed that αj are analogues of uj , where X =
U ΣV T .
a
This will only affects the eigenvectors with zero eigenvalues, which will not be a principle component
anyway.
Note: There is a normalization condition for the αj vectors:

√
kvj k = 1 ⇐⇒ kαj k = 1/ µj .
"
1 = vjT vj = (φ(X)T αj )T φ(X)T αj = αTj φ(X)φ(X)T αj ⇐ (7.89)
∵
= αTj Kαj = αTj (µj αj ) = µj kαj k2 ⇐ (7.94)
(7.95)
7.4.2. Computation of the kernel PCA
Remark 7.26. Let the eigenvalue-eigenvector pairs of the kernel ma-

trix K be given as
Kαj = µj αj , j = 1, 2, · · · , N ; αTi αj = δij . (7.96)
• Then, referring (7.78) derived for the standard PCA, we may conclude
that the k principal components for the kernel PCA are
√ √ √
Ak = [ µ1 α1 | µ2 α2 | · · · | µk αk ] ∈ RN ×k . (7.97)
• It follows from (7.86), (7.89), and (7.95)-(7.96) that for a new point x,
its projection onto the principal components is:
N N
T 1 T
X
(`) 1 X
zj = φ(x) vj = √ φ(x) α`j φ(x ) = √ α`j φ(x)T φ(x(`) )
µj µj
`=1 `=1
N
1 X 1
= √ α`j K(x, x(`) ) = √ K(x, X)T αj .
µj µj
`=1
(7.98)
That is, due to (7.95) and (7.96), when vj is expressed in terms of αj ,
√
it must be scaled by 1/ µj .
Construction of the kernel matrix K
• The kernel trick is to avoid calculating the pairwise dot products of

the transformed samples φ(x) explicitly by using a kernel function.
• For a selected kernel function K,
 
K(x(1) , x(1) ) K(x(1) , x(2) ) · · · K(x(1) , x(N ) )
 K(x(2) , x(1) ) K(x(2) , x(2) ) · · · K(x(2) , x(N ) ) 
K=  ∈ RN ×N . (7.99)
 
.. .. ... ..
 . . . 
K(x , x ) K(x , x(2) )
(N ) (1) (N )
· · · K(x(N ) , x(N ) )
where K is called the kernel function.a

a
As for nonlinear SVM, the most commonly used kernels are the polynomial kernel, the hyperbolic
tangent (sigmoid) kernel, and the Gaussian Radial Basis Function (RBF) kernel. See (5.57)-(5.60), p. 118.
Normalizing the feature space
• In general, φ(x(i) ) may not be zero mean.

• Thus K = φ(X)φ(X)T would better be normalized before start finding
its eigenvectors and eigenvalues.
• Centered features:
N
(i) 1 X
(i)
φ(x
e ) = φ(x ) − φ(x(k) ), ∀ i. (7.100)
N
k=1
• The corresponding kernel is

e (i) , x(j) ) = φ(x
K(x e (i) )T φ(x
e (j) )
N N

(i) 1 X (k)
T
(j) 1 X (k)

= φ(x ) − φ(x ) φ(x ) − φ(x )
N N
k=1 k=1
N N
1 X 1 X (7.101)
= K(x(i) , x(j) ) − K(x(i) , x(k) ) − K(x(k) , x(j) )
N N
k=1 k=1
N
1 X
+ 2 K(x(k) , x(`) ).
N
k,`=1
• In a matrix form
e = K − K1 − 1 K + 1 K1 ,
K (7.102)
1/N 1/N 1/N 1/N
where 11/N is an N × N matrix where all entries are equal to 1/N .

Summary 7.27. (Summary of the Kernel PCA).

• Pick a kernel function K.
• For data X ∈ RN ×d , construct the kernel matrix
K = [K(x(i) , x(j) )] ∈ RN ×N . (7.103)
• Normalize the kernel matrix K:

e = K − K1 − 1 K + 1 K1 .
K (7.104)
1/N 1/N 1/N 1/N
• Solve an eigenvalue problem:
Kα
e j = µj αj , αTi αj = δij . (7.105)
• Then, the k principal components for the kernel PCA are

Ak = [µ1 α1 |µ2 α2 | · · · |µk αk ] ∈ RN ×k , k ≤ N. (7.106)
• For a data point x (new or old), we can represent it as

N
X N
X
T T (`)
zj = φ(x) vj = φ(x) α`j φ(x ) = α`j K(x, x(`) ), j = 1, 2, · · · , k.
`=1 `=1
(7.107)
Note: Formulas in (7.106)-(7.107) are alternatives of (7.97)-(7.98).
Properties of the KPCA
• With an appropriate choice of kernel function, the kernel PCA can give
a good re-encoding of the data that lies along a nonlinear manifold.
• The kernel matrix is in (N × N )-dimensions, so the kernel PCA will
have difficulties when we have lots of data points.
7.1. Read pp. 145–158, Python Machine Learning, 3rd Ed., about the PCA.
(a) Find the optimal number of components k ∗ which produces the best classifica-
tion accuracy (for logistic regression), by experimenting the example code with
n_components = 1, 2, · · · , 13.
(b) What is the corresponding cumulative explained variance?
7.2. Let A ∈ Rm×n . Prove that ||A||2 = σ1 , the largest singular value of A. Hint: Use the
following
||Av1 ||2 σ1 ||u1 ||2
= = σ1 =⇒ ||A||2 ≥ σ1
||v1 ||2 ||v1 ||2
and arguments around Equations (7.34) and (7.35) for the opposite directional in-
equality.
7.3. Recall that the Frobenius matrix norm is defined by
m X
X n 1/2
||A||F = |aij |2 , A ∈ Rm×n .
i=1 j=1
Show that ||A||F = (σ12 + · · · + σk2 )1/2 , where σj are nonzero singular values of A. Hint:
You may use the norm-preserving property of orthogonal matrices. That is, if U is
orthogonal, then ||U B||2 = ||B||2 and ||U B||F = ||B||F .
7.4. Prove Lemma 7.18. Hint: For (b), let Avi = λi vi , i = 1, 2, and λ1 6= λ2 . Then
(λ1 v1 ) · v2 = (Av1 ) · v2 = v1 · (Av2 ) = v1 · (λ2 v2 ).

| {z }
∵ A is symmetric
For (a), you may use a similar argument, but with the dot product being defined for
complex values, i.e.,
u · v = uT v,
where v is the complex conjugate of v.
7.5. Use Matlab to generate a random matrix A ∈ R8×6 with rank 4. For example,
A = randn(8,4);
A(:,5:6) = A(:,1:2)+A(:,3:4);
[Q,R] = qr(randn(6));
A = A*Q;
(a) Print out A on your computer screen. Can you tell by looking if it has (numerical)
rank 4?
(b) Use Matlab’s “svd" command to obtain the singular values of A. How many are
“large?" How many are “tiny?" (You may use the command “format short e" to get
a more accurate view of the singular values.)
(c) Use Matlab’s “rank" command to confirm that the numerical rank is 4.
(d) Use the “rank" command with a small enough threshold that it returns the value
6. (Type “help rank" for information about how to do this.)
7.6. Verify (7.64). Hint : Use the quotient rule for ∂J∂w
(w)
and equate the numerator to zero.
7.7. Try to understand the kernel PCA more deeply by experimenting pp. 175–188, Python
Machine Learning, 3rd Ed.. Its implementation is slightly different from (but equiv-
alent to) Summary 7.27.
(a) Modify the code, following Summary 7.27, and test if it works as expected as in
Python Machine Learning, 3rd Ed..
(b) The datasets considered are transformed via the Gaussian radial basis function
(RBF) kernel only. What happens if you use the following kernels?
K1 (x(i) , x(j) ) = (a1 + b1 x(i) · x(j) )2 (polynomial of degree up to 2)
K2 (x(i) , x(j) ) = tanh(a2 + b2 x(i) · x(j) ) (sigmoid)
Can you find ai and bi , i = 1, 2, appropriately?

8
C HAPTER
Cluster Analysis
Cluster analysis or clustering is the task of finding groups of objects

such that the objects in a group will be similar (or related) to one another
and different from (or unrelated to) the objects in other groups. It is a main
task of exploratory data mining, and a common technique for statisti-
cal data analysis, used in many fields, including machine learning, pat-
tern recognition, image analysis, information retrieval, bioinformatics, data
compression, and computer graphics.
History . Cluster analysis was originated in anthropology by Driver

and Kroeber in 1932 [16], introduced to psychology by Zubin in 1938
[84] and Robert Tryon in 1939 [76], and famously used by Cattell begin-
ning in 1943 [11] for trait theory classification in personality psychol-
ogy.
8.1. Basics for Cluster Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
8.2. K-Means and K-Medoids Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
8.3. Hierarchical Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224
8.4. DBSCAN: Density-based Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
8.5. Cluster Validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
8.6. Self-Organizing Maps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247
199
200 Chapter 8. Cluster Analysis
8.1. Basics for Cluster Analysis
Figure 8.1: Intra-cluster distance vs. inter-cluster distance.
Applications of Cluster Analysis

• Understanding
– group related documents or browsings
– group genes/proteins that have similar functionality, or
– group stocks with similar price fluctuations
• Summarization
– reduce the size of large data sets
Not Cluster Analysis

• Supervised classification – Uses class label information
• Simple segmentation – Dividing students into different registration
groups alphabetically, by last name
• Results of a query – Groupings are a result of an external specification
Clustering uses only the data (unsupervised learning):

to discover hidden structures in data
8.1. Basics for Cluster Analysis 201
8.1.1. Quality of clustering

• A good clustering method will produce high quality clusters with
– high intra-class similarity
– low inter-class similarity
• The quality of a clustering result depends on both the similarity
measure and its implementation
• The quality of a clustering method is also measured by its ability to
discover some or all of the hidden patterns
Measuring the Quality of Clustering

• Dissimilarity/Similarity/Proximity metric: Similarity is ex-
pressed in terms of a distance function d(i, j)
• The definitions of distance functions are usually very different for
interval-scaled, boolean, categorical, ordinal ratio, and vector vari-
ables.
• There is a separate “quality” function that measures the “goodness” of
a cluster.
• Weighted measures: Weights should be associated with different vari-
ables based on applications and data semantics.
• It is hard to define “similar enough” or “good enough”
– the answer is typically highly subjective
Notion of a Cluster can be Ambiguous
Figure 8.2: How many clusters?
The answer could be:
Figure 8.3: Two clusters.
Figure 8.4: Four clusters.
Figure 8.5: Six clusters.

Similarity and Dissimilarity Between Objects
• Distances are normally used to measure the similarity or dissimilar-

ity between two data objects
• Some popular ones include: Minkowski distance
d(i, j) = kxi − xj kp = (|xi1 − xj1 |p + |xi2 − xj2 |p · · · |xid − xjd |p )1/p , (8.1)
where xi , xj ∈ Rd , two d-dimensional data objects.

– When p = 1, it is Manhattan distance
– When p = 2, it is Euclidean distance
• Other Distances: Also, one can use weighted distance, parametric
Pearson product moment correlation, or other dissimilarity measures
• Various similarity measures have been studied for

– Binary variables
– Nominal variables & ordinal variables
– Ratio-scaled variables
– Variables of mixed types
– Vector objects
8.1.2. Types of clusters

• Center-based clusters
• Contiguity/connectivity-based clusters
• Density-based clusters
• Conceptual clusters
Note: (Well-separated clusters). A cluster is a set of objects such that

an object in a cluster is closer (more similar) to every/some of points
in the cluster, than any points not in the cluster.
Center-based Clusters
• The center of a cluster is often

– a centroid, the average of all the points in the cluster, or
– a medoid, the most representative point of a cluster.
• A cluster is a set of objects such that an object in a cluster is closer (more
similar) to the “center” of a cluster, than to the center of any other
clusters.
Figure 8.6: Well-separated, 4 center-based clusters.

Contiguity-based Clusters
• Contiguous cluster (nearest neighbor or transitive)

• A cluster is a set of points such that a point in a cluster is closer (or
more similar) to one or more other points in the cluster, than to any
points not in the cluster.
Figure 8.7: 8 contiguous clusters.
Density-based Clusters
• A cluster is a dense region of points, which is separated by low-density

regions, from other regions of high density.
• Used when the clusters are irregular or intertwined, and when noise
and outliers are present.
Figure 8.8: 6 density-based clusters.

Conceptual Clusters
• Points in a cluster share some general property.

– Conceptual clusters are hard to detect, because they are often none
of the center-based, contiguity-based, or density-based.
– Points in the intersection of the circles belong to both.
Figure 8.9: Conceptual clusters
Clusters Defined by an Objective Function
• Find clusters that minimize or maximize an objective function.

• Enumerate all possible ways of dividing the points into clusters and
evaluate the “goodness” of each potential set of clusters by using the
given objective function. (NP-Hard)
• Can have global or local objectives. Typically,
– Partitional clustering algorithms have global objectives
– Hierarchical clustering algorithms have local objectives
Computational Complexity Theory

Problem Types
• P (Polynomial Time): Problems which are solvable in polyno-
mial time (when running on a deterministic Turing machinea ).
• NP (Non-deterministic Polynomial Time): Decision problems
which can be verified in polynomial time.
• NP-Hard: These are at least as hard as the hardest problems in
NP, in both solution and verification.
• NP-Complete: These are the problems which are both NP and NP-
Hard.
a
A Turing machine is a theoretical machine that manipulates symbols on a strip of tape according to
a table of rules. A deterministic Turing machine is a theoretical machine, used in thought experiments
to examine the abilities and limitations of algorithms. In a deterministic Turing machine, the set of
rules impose at most one action to be performed for any given situation. In a nondeterministic Turing
machine, it may have a set of rules that prescribes more than one action for a given situation [13].
Problem Type Verifiable in P-time Solvable in P-time

P Yes Yes
NP Yes Yes or No
NP-Complete Yes Unknown
NP-Hard Yes or No Unknown
Quesiton. P = NP? (P versus NP problem)

• This one is the most famous problem in computer science, and one
of the most important outstanding questions in the mathematical
sciences.
• In fact, the Clay Institute is offering one million dollars for a solu-
tion to the problem.
– It’s clear that P is a subset of NP.
– The open question is whether or not NP problems have deter-
ministic polynomial time solutions.
8.1.3. Types of clustering and Objective functions

• Partitional clustering
• Hierarchical clustering (agglomerative; divisive)
• Density-based clustering (DBSCAN)
Partitional Clustering
Divide data objects into non-overlapping subsets (clusters) such that
each data object is in exactly one subset
Figure 8.10: Original points & A partitional clustering
Examples are
• K-Means, Bisecting K-Means

• K-Medoids (PAM: partitioning around medoids)
• CLARA, CLARANS (Sampling-based PAMs)
Hierarchical Clustering
A set of nested clusters, organized as a hierarchical tree
Figure 8.11: Dendrogram and Nested cluster diagram.
Other Distinctions Between Sets of Clusters

• Exclusive (hard) vs. Non-exclusive (soft)
– In non-exclusive clusterings, points may belong to multiple clus-
ters.
• Fuzzy vs. Non-fuzzy
– In fuzzy clustering, a point belongs to every cluster with some
membership weight between 0 and 1
– Membership weights must sum to 1
– Probabilistic clustering has similar characteristics
• Partial vs. Complete
– In some cases, we only want to cluster some of the data
• Homogeneous vs. Heterogeneous
– Cluster of widely different sizes, shapes, and densities
Objective Functions
Global objective function
• Typically used in partitional clustering

– K-Means minimizes the Sum of Squared Errors (SSE):
K X
X
SSE = kx − µi k22 , (8.2)
i=1 x∈Ci
where x is a data point in cluster Ci and µi is the center for cluster

Ci as the mean of all points in the cluster.
• Mixture models: assume that

the dataset is a “mixture” of a
number of parametric statistical
distributions
(e.g., Gaussian mixture models).
Figure 8.12: A two-component Gaussian

mixture model: data points, and equi-
probability surfaces of the model.
Local objective function
• Hierarchical clustering algorithms typically have local objectives

• Density-based clustering is based on local density estimates
• Graph based approaches: Graph partitioning and shared nearest
neighbors
We will consider the objective functions when we talk about individual

clustering algorithms.
8.2. K-Means and K-Medoids Clustering 211
8.2. K-Means and K-Medoids Clustering

• Given
– X, a dataset of N objects
– K, the number of clusters to form
• Organize the objects into K partitions (K ≤ N ), where each partition
represents a cluster
• The clusters are formed to optimize an objective partitioning cri-
terion:
– Objects within a cluster are similar
– Objects of different clusters are dissimilar
8.2.1. The (basic) K-Means clustering
• Partitional clustering approach

• Each cluster is associated with a centroid (mean)
• Each point is assigned to the cluster with the closest centroid
• Number of clusters, K, must be specified
Algorithm 8.1. Lloyd’s algorithm (a.k.a. Voronoi iteration):

(Lloyd, 1957) [48]
1. Select K points as the initial centroids;
2. repeat
3. Form K clusters by assigning all points to the closest centroid;
4. Recompute the centroid of each cluster;
5. until (the centroids don’t change)
iteration=1 iteration=2
iteration=3 iteration=15
Figure 8.13: Lloyd’s algorithm: The Voronoi diagrams, the given centroids (•), and the
+
updated centroids ( ), for iteration = 1, 2, 3, and 15.
The K-Means Clustering – Details

• Initial centroids are often chosen randomly.
– Clusters produced vary from one run to another.
• The centroid is (typically) the mean of the points in the cluster.
• “Closeness” is measured by Euclidean distance, cosine similarity,
correlation, etc..
• The K-Means will converge typically in the first few iterations.
– Often the stopping condition is changed to “until (relatively few
points change clusters)” or some measure of clustering doesn’t
change.
• Complexity is O(N ∗ d ∗ K ∗ I), where
N : the number of points
d: the number of attributes
K: the number of clusters
I: the number of iterations
Evaluating the K-Means Clusters

• Most common measure is Sum of Squared Error (SSE):
XK X
SSE = kx − µi k22 , (8.3)
i=1 x∈Ci
where x is a data point in cluster Ci and µi is the center for cluster Ci .

• Multiple runs: Given sets of clusters, we can choose the one with the
smallest error.
• One easy way to reduce SSE is to increase K, the number of clusters.
– A good clustering with smaller K can have a lower SSE than a
poor clustering with higher K.
The K-Means is heuristic to minimize the SSE.

Figure 8.14: K-Means clustering example.

Problems with Selecting Initial Points

The chance of selecting one centroid from each cluster is small.
• Chance is relatively small when K is large
• If clusters are the same size, n, then
# of ways to select a centroid from each cluster
P =
# of ways to select K centroids
K
K!n K!
= = .
(Kn)K KK
• For example, if K = 5 or 10, then probability is:
5!/55 = 0.0384, 10!/1010 = 0.00036.
• Sometimes the initial centroids will readjust themselves in “right”

way, and sometimes they don’t.
Figure 8.15: Importance of choosing initial centroids.

Solutions to Initial Centroids Problem

• Multiple runs
– Helps, but probability is not on your side
• Sample and use hierarchical clustering to determine initial cen-
troids
• Select more than K initial centroids and then, among these initial
centroids
– Select most widely separated
• Post-processing
• Bisecting K-Means
– Not as susceptible to initialization issues
Pre-processing and Post-processing
• Pre-processing
– Normalize the data
– Eliminate outliers
• Post-processing
– Eliminate small clusters that may represent outliers
– Split “loose” clusters, i.e., clusters with relatively high SSE
– Merge clusters that are “close” and that have relatively low SSE
* Can use these steps during the clustering process – ISODATA

8.2.2. Bisecting K-Means algorithm

A variant of the K-Means that can produce a partitional or a hierarchical
clustering
1. Initialize (a list of clusters), containing all points.

2. Repeat
(a) Select a cluster from the list of clusters
(b) for i = 1 to iter_runs do
Bisect the selected cluster using the basic K-Means
end for
(c) Add the two clusters from the bisection with the lowest
SSE to the list of clusters.
until (the list of clusters contains K clusters)
Figure 8.16: Bisecting K-Means algorithm, with K = 4.
Note: The bisecting K-Means algorithm is not as susceptible to initial-

ization issues as the basic K-Means clustering.
Limitations of K-Means Algorithms

• The K-Means have problems when clusters are of differing
– sizes, densities, and non-globular shapes
• The K-Means have problems when the data contains outliers
Figure 8.17: The K-Means with 3 clusters of different sizes.
Figure 8.18: The K-Means with 3 clusters of different densities.
Figure 8.19: The K-Means with 2 non-globular clusters.

Overcoming K-Means Limitations
• Use a larger number of clusters

• Several clusters represent a true cluster
Figure 8.20: Unequal-sizes.
Figure 8.21: Unequal-densities.

Figure 8.22: Non-spherical shapes.
Overcoming the K-Means Outlier Problem
• The K-Means algorithms are sensitive to outliers.

– Since an object with an extremely large value may substantially
distort the distribution of the data.
• Solutions:
(a) Instead of taking the mean value of the object in a cluster as a ref-
erence point, medoids can be used, which is the most centrally
located object in a cluster.
(b) Develop an effective outlier removal algorithm. We will do
it as a project which combines clustering and supervised learning
for classification.
8.2.3. The K-Medoids algorithm

The K-Medoids algorithm (or PAM algorithm) is a clustering al-
gorithm similar to the K-Means algorithm. Both the K-Means and
K-Medoids algorithms are partitional (breaking the dataset up into
groups) and both attempt to minimize the distance between points la-
beled to be in a cluster and a point designated as the center of that clus-
ter. The K-Medoids chooses data points as centers (medoids) and can
be used with arbitrary distances, while the K-Means only minimizes
the squared Euclidean distances from cluster means. The PAM method
was proposed by (Kaufman & Rousseeuw, 1987) [37] for the work with
L1 -norm and other distances.
• Find representative objects, called medoids, in clusters.

• The PAM (partitioning around medoids) starts from an initial set of
medoids and iteratively replaces one of the medoids by one of the
non-medoids if it improves the total distance of the resulting clus-
tering.
• The PAM works effectively for small datasets, but does not scale
well for large data sets.
• CLARA (Clustering LARge Applications): sampling-based method
(Kaufmann & Rousseeuw, 1990) [38]
• CLARANS: CLARA with randomized search (Ng & Han, 1994) [54]
PAM (Partitioning Around Medoids) : Use real objects to represent the

clusters (called medoids).
Initialization: select K representative objects;

Associate each data point to the closest medoid;
while (the cost of the configuration decreases) :

For each medoid m and each non-medoid data point o :

 swap m and o;

 associate each data point to the closest medoid;
 recompute the cost (sum of distances of points to their medoid);
If the total cost of the configuration increased, undo the swap;
Pros and cons of the PAM

• The PAM is more robust than the K-Means in the presence of noise
and outliers, because a medoid is less influenced by outliers or other
extreme values than a mean.
• The PAM works efficiently for small datasets but does not scale well
for large data sets.
• The run-time complexity of the PAM is O(K(N − K)2 ) for each itera-
tion, where N is the number of data points and K is the number of
clusters.
⇒ CLARA (Clustering LARge Applications): sampling-based method

(Kaufmann & Rousseeuw, 1990) [38]
The PAM finds the best K-medoids among a given data, and the CLARA
finds the best K-medoids among the selected samples.
8.2.4. CLARA and CLARANS

CLARA (Clustering LARge Applications)
• Sampling-based PAM (Kaufmann & Rousseeuw, 1990) [38]
• It draws multiple samples of the dataset, applies the PAM on each
sample, and gives the best clustering as the output.
⊕ Strength: deals with larger data sets than the PAM.

Weakness:
– Efficiency depends on the sample size.
– A good clustering based on samples will not necessarily represent
a good clustering of the whole data set, if the sample is biased.
• Medoids are chosen from the sample:
The algorithm cannot find the best solution if one of the best K-
Medoids is not among the selected samples.
CLARANS (“Randomized” CLARA)

• CLARANS (CLARA with Randomized Search) (Ng & Han; 1994,2002)
[54, 55]
• The CLARANS draws sample of neighbors dynamically.
– CLARANS draws a sample of neighbors in each step of a search,
while CLARA draws a sample of nodes at the beginning of a
search.
• The clustering process can be presented as searching a graph where
every node is a potential solution, that is, a set of K medoids.
• If a local optimum is found, the CLARANS starts with new ran-
domly selected node in search for a new local optimum.
• Finds several local optimums and output the clustering with the best
local optimum.
⊕ It is more efficient and scalable than both the PAM and the CLARA;
handles outliers.
⊕ Focusing techniques and spatial access structures may further im-
prove its performance; see (Ng & Han, 2002) [55] and (Schubert &
Rousseeuw, 2018) [71].
Yet, the computational complexity of the CLARANS is O(N 2 ), where

N is the number of objects.
The clustering quality depends on the sampling method.
8.3. Hierarchical Clustering

8.3.1. Basics of AGNES and DIANA
Hierarchical clustering can be divided into two main types: agglomerative
and divisive.
Figure 8.23: AGNES and DIANA
Agglomerative hierarchical clustering

(a.k.a. AGNES: Agglomerative Nesting). It works in a bottom-up manner.
• Each object is initially considered as its own singleton cluster (leaf).

• At each iteration, the two closest clusters are merged into a new
bigger cluster (nodes).
• This procedure is iterated until all points are merged into a single
cluster (root).
• The result is a tree which can be plotted as a dendrogram.
Divisive hierarchical clustering

(a.k.a. DIANA: Divisive Analysis). It works in a top-down manner; the
algorithm is an inverse order of the AGNES.
• It begins with the root, where all objects are included in a single cluster.
• Repeat: the most heterogeneous cluster is divided into two.
• Until: all objects are in their own cluster.
8.3. Hierarchical Clustering 225
Note: Agglomerative clustering is good at identifying small clusters,

while divisive hierarchical clustering is good for large clusters.
Complexity
• The optimum cost is O(N 2 ), because it uses the proximity matrix. (N

is the number of points)
• In practice, O(N 3 ) in many cases.
– There are O(N ) steps and at each step the proximity matrix of size
O(N 2 ) must be updated and searched.
– Complexity can be reduced to O(N 2 log(N )) for some approaches.
Limitations
• Greedy: Once a decision is made to combine two clusters, it cannot be

undone.
• No global objective function is directly minimized.
• Most algorithms have problems with one or more of:
– Sensitivity to noise and outliers
– Difficulty handling different sized clusters and non-convex shapes
– Chaining, breaking large clusters
Hierarchical Clustering vs. K-Means
• Recall that K-Means or K-Medoids requires

– The number of clusters K
– An initial assignment of data to clusters
– A distance measure between data d(xi , xj )
• Hierarchical clustering requires only a similarity measure between
groups/clusters of data points.
8.3.2. AGNES: Agglomerative clustering
Quesiton. How do we measure the similarity (or dissimilarity) between

two groups of observations?
A number of different cluster agglomeration methods (i.e, linkage meth-

ods) have been developed to answer to the question. The most popular
choices are:
• Single linkage
• Complete linkage
• Group linkage
• Centroid linkage
• Ward’s minimum variance
1. Single linkage: the similarity of the closest pair
dSL (G, H) = min d(i, j). (8.4)

i∈G,j∈H
• Single linkage can produce “chaining”, where a sequence of

close observations in different groups cause early merges of those
groups
• It tends to produce long “loose” clusters.
2. Complete linkage: the similarity of the furthest pair
dCL (G, H) = max d(i, j). (8.5)

i∈G,j∈H
• Complete linkage has the opposite problem; it might not merge

close groups because of outlier members that are far apart.
• It tends to produce more compact clusters.
3. Group average: the average similarity between groups

1 XX
dGA (G, H) = d(i, j). (8.6)
NG NH
i∈G j∈H
– Group average represents a natural compromise, but depends

on the scale of the similarities. Applying a monotone transforma-
tion to the similarities can change the results.
4. Centroid linkage: It computes the dissimilarity between the cen-
troid for group G (a mean vector of length d variables) and the centroid
for group H.
5. Ward’s minimum variance: It minimizes the total within-cluster
variance. More precisely, at each step, the method finds the pair of
clusters that leads to minimum increase in total within-cluster
variance after merging. It uses the squared error (as an objective
function).
Interpretable Visualization of AGNES
• Each level of the resulting tree is a segmentation of data

• The algorithm results in a sequence of groupings
• It is up to the user to choose a natural clustering from this se-
quence
Dendrogram
• Agglomerative clustering is monotonic
– The similarity between merged clusters is monotone decreas-
ing with the level of the merge.
• Dendrogram: Plot each merge at the dissimilarity between the two
merged groups
– Provides an interpretable visualization of the algorithm and
data
– Useful summarization tool, part of why hierarchical clustering
is popular
Figure 8.24: Six observations and a dendrogram showing their hierarchical clustering.
Remark 8.2.
• The height of the dendrogram indicates the order in which the clus-
ters were joined; it reflects the distance between the clusters.
• The greater the difference in height, the more dissimilarity.
• Observations are allocated to clusters by drawing a horizontal line
through the dendrogram. Observations that are joined together below
the line are in the same clusters.
Single Link
Figure 8.25: Single link clustering of six points.
• Pros: Non-spherical, non-convex clusters

• Cons: Chaining
Complete Link
Figure 8.26: Complete link clustering of six points.
• Pros: more robust against noise (no chaining)

• Cons: Tends to break large clusters; biased towards globular clusters
Average Link
Figure 8.27: Average link clustering of six points.
• Compromise between single and complete links
Ward’s Minimum Variance Method
• Similarity of two clusters is based on the increase in squared error

when two clusters are merged
• Less susceptible to noise and outliers
• Biased towards globular clusters.
• Hierarchical analogue of the K-Means; it can be used to initialize
the K-Means. (Note that the K-Means works with a global objective
function.)
8.4. DBSCAN: Density-based Clustering 231
8.4. DBSCAN: Density-based Clustering

In density-based clustering:
• Clusters are defined as areas of higher density than the remainder

of the data set. (core points)
• Objects in sparse areas are usually considered to be noise and border
points.
• The most popular density-based clustering method is
– DBSCANa (Ester, Kriegel, Sander, & Xu, 1996) [18].
a
Density-Based Spatial Clustering of Applications with Noise (DBSCAN).
DBSCAN
• Given a set of points, it groups points that are closely packed together
(points with many nearby neighbors),
– marking as outliers points that lie alone in low-density regions.
• It is one of the most common clustering algorithms and also most
cited in scientific literature. (Citation #: 28,008, as of Apr. 15, 2023)
– In 2014, the algorithm was awarded the test of time awarda at
the leading data mining conference,
KDD 2014: https://www.kdd.org/kdd2014/.
a
The test of time award is an award given to algorithms which have received substantial attention in
theory and practice.
Preliminary for DBSCAN
• Consider a dataset to be clustered.

• Let ε be a parameter specifying the radius of a neighborhood with
respect to some point.
• In DBSCAN clustering, the points are classified as core points,
reachable points, and outliers, as follows:
– A point p is a core point if at least m (=minPts) points are within
distance ε of it (including p itself).
– A point q is directly reachable from p if point q is within dis-
tance ε from the core point p.
(Points are only said to be directly reachable from core points.)
– A point q is reachable from p if there is a path p1 , · · · , pn with
p1 = p and pn = q, where each pi+1 is directly reachable from pi .
(Note that this implies that all points on the path must be core
points, with the possible exception of q.)
– All points not reachable from any other points are outliers or
noise points.
• Now, a core point forms a cluster together with all points (core or
non-core) that are reachable from it.
– Each cluster contains at least one core point;
non-core points can be part of a cluster, but they form its “edge”,
since they cannot be used to reach more points.
– A non-core reachable point is also called a border point.
User parameters:
• ε: the radius of a neighborhood
• minPts: the minimum number of points in the ε-neighborhood
Illustration of the DBSCAN
Figure 8.28: Illustration of the DBSCAN, with m (= minPts) = 4.
• Point A and 5 other red points are core points. They are all reachable
from one another, so they form a single cluster.
• Points B and C are not core points, but are reachable from A (via
other core points) and thus belong to the cluster as well.
• Point N is a noise point that is neither a core point nor directly-
reachable.
Note: Reachability is not a symmetric relation since, by definition,
no point may be reachable from a non-core point, regardless of distance.
(A non-core point may be reachable, but nothing can be reached from it.)
Definition 8.3. Two points p and q are density-connected if there

is a point c such that both p and q are reachable from c. Density-
connectedness is symmetric.
A DBSCAN cluster satisfies two properties:

1. All points within the cluster are mutually density-connected.
2. If a point is density-reachable from any point of the cluster, then
it is part of the cluster as well.
DBSCAN: Pseudocode
DBSCAN
1 DBSCAN(D, eps, MinPts)
2 C=0 # Cluster counter
3 for each unvisited point P in dataset D
4 mark P as visited
5 NP = regionQuery(P, eps) # Find neighbors of P
6 if size(NP) < MinPts
7 mark P as NOISE
8 else
9 C = C + 1
10 expandCluster(P, NP, C, eps, MinPts)
11
12 expandCluster(P, NP, C, eps, MinPts)

13 add P to cluster C
14 for each point Q in NP
15 if Q is not visited
16 mark Q as visited
17 NQ = regionQuery(Q, eps)
18 if size(NQ) >= MinPts
19 NP = NP joined with NQ
20 if Q is not yet member of any cluster
21 add Q to cluster C
Figure 8.29: Original points (left) and point types of the DBSCAN clustering with eps=10
and MinPts=4 (right): core (green), border (blue), and noise (red).
Note: In Pseudocode: Line 7 may classify a border point as noise,

which would be corrected by Lines 20-21.
Properties of DBSCAN Clustering
• Resistant to Noise
• Can handle clusters of different shapes and sizes
• Eps and MinPts depend on each other and can be hard to specify
When the DBSCAN does NOT work well
• Varying densities
• High-dimensional data
Figure 8.30: Original points.
Figure 8.31: The DBSCAN clustering. For both cases, it results in 3 clusters.
Overall, DBSCAN is a great density-based clustering algorithm.

8.5. Cluster Validation

8.5.1. Basics of cluster validation
• For supervised classification (= using class label), we have a variety of
measures to evaluate how good our model is.
◦ Accuracy, precision, recall
• For cluster analysis (= unsupervised), the analogous question is:
How to evaluate the “goodness” of the resulting clusters?
• But “clusters are in the eye of the beholder”!

• Yet, we want to evaluate them. Why?
◦ To avoid finding patterns in noise
◦ To compare clustering algorithms
◦ To compare two sets of clusters
◦ To compare two clusters
Aspects of Cluster Validation

1. Understanding the clustering tendency of a set of data,
(i.e., distinguishing non-random structures from all the retrieved).
2. Validation Methods?
• External validation: Compare the results of a cluster analysis
to externally known class labels (ground truth).
• Internal validation: Evaluating how well the results of a cluster
analysis fit the data without reference to external information –
use only the data.
3. Compare clusterings to determine which is better.
4. Determining the “correct” number of clusters.
For 2 and 3, we can further distinguish whether we want to evaluate the

entire clustering or just individual clusters.
8.5. Cluster Validation 237
Definition 8.4. Precision and Recall

Precision is the fraction of relevant instances among all the retrieved,
while recall (a.k.a. sensitivity) is the fraction of relevant instances that
were retrieved.
Figure 8.32: Illustration of precision and recall, Wikipedia.
Note: Therefore, both precision and recall are about relevance of

the retrieval, measured respectively from all the retrieved instances
and all the relevant instances in the database.
Measures of Cluster Validity
Numerical measures for judging various aspects of cluster validity are

classified into the following three types.
• External Measures: Used to measure the extent to which cluster
labels match externally supplied class labels.
– Entropy, Purity, Rand index
– Precision, Recall
• Internal Measures: Used to measure the goodness of a clustering
structure without respect to external information.
– Correlation, Similarity matrix
– Sum of Squared Error (SSE), Silhouette coefficient
• Relative Measures:
– Used to compare 2 different clusterings or clusters.
– Often an external or internal measure is used for this function,
e.g., SSE or entropy
Definition 8.5. The correlation coefficient ρX,Y between two random

variables X and Y with expected values µX and µY and standard devia-
tions σX and σY is defined as
cov(X, Y ) E[(X − µX )(Y − µY )]
ρX,Y = corr(X, Y ) = = ∈ [−1, 1]. (8.7)
σ X σY σX σY
If X and Y are independent, ρX,Y = 0. (The reverse may not be true.)
Note: The correlation between two vectors u and v is the cosine of

the angle between the two vectors.
u·v
corr(u, v) = . (8.8)
kuk kvk
The correlation between two matrices can be defined similarly, by con-
sidering the matrices as vectors.
Measuring Cluster Validity Via Correlation
• Two matrices
– Proximity matrixa (P ∈ RN ×N )
– Incidence matrix (I ∈ RN ×N )
* One row and one column for each data point
* An entry is 1 if the associated pair of points belong to the same cluster
* An entry is 0 if the associated pair of points belongs to different clusters
• Compute the correlation between the two matrices

– Since the matrices are symmetric, only the correlation between
N (N − 1)/2 entries needs to be calculated.
• High correlation indicates that points that belong to the same clus-
ter are close to each other.
Example: For K-Means clusterings of two data sets, the correlation
coefficient are:
Figure 8.33: ρP,I = −0.924. Figure 8.34: ρP,I = −0.581.
• Not a good measure for some density- or contiguity-based clus-

ters (e.g., single link HC).
A proximity matrix is a square matrix in which the entry in cell (i, j) is some
a
measure of the similarity (or distance) between the items to which row i and column j
correspond.
Using Similarity Matrix for Cluster Validation
Order the similarity matrix with respect to cluster labels and inspect
visually.
Figure 8.35: Clusters are so crisp!
Figure 8.36: Clusters in random data are not so crisp.

Figure 8.37: Similarity matrix for cluster validation, for DBSCAN.
8.5.2. Internal and external measures of cluster validity
Internal Measures
• (Average) SSE is good for comparing two clusterings or two clusters.
XK X
SSE = ||x − µi k2 . (8.9)
i=1 x∈Ci
• It can also be used to estimate the number of clusters
10 clusters
Figure 8.38: Estimation for the number of clusters.

Cohesion and Separation
• Cluster cohesion: Measure how closely related are objects in a cluster

– Example: Within-cluster sum of squares (WSS=SSE)
K X
X
W SS = ||x − µi k2 . (8.10)
i=1 x∈Ci
• Cluster separation: Measures how distinct or well-separated a clus-

ter is from other clusters
– Example: Between-cluster sum of squares (BSS)
K
X
BSS = |Ci | ||µ − µi k2 . (8.11)
i=1
• Total sum of squares: T SS = W SS + BSS

– TSS is a constant for a given data set (independently of the number
of clusters)
– Example: a cluster {1, 2, 4, 5} can be separated into two clusters
{1, 2} ∪ {4, 5}. It is easy to check the following.
* 1 cluster: T SS = W SS + BSS = 10 + 0 = 10.
* 2 clusters: T SS = W SS + BSS = 1 + 9 = 10.
Silhouette Coefficient
• Silhouette coefficient combines ideas of both cohesion and sepa-

ration, but for individual points. For an individual point i:
– Calculate a(i) = average distance of i to all other points in its cluster
– Calculate b(i) = min {average distance of i to points in another cluster}
– The silhouette coefficient for the point i is then given by
s(i) = 1 − a(i)/b(i). (8.12)
– Typically, s(i) ∈ [0, 1].
– The closer to 1, the better.
• We can calculate the average silhouette width for a cluster or a
clustering
Selecting K with Silhouette Analysis on K-Means Clustering
Figure 8.39: n_clusters = 2; Figure 8.42: n_clusters = 5;

average silhouette score = 0.705. average silhouette score = 0.564.
Figure 8.40: n_clusters = 3; Figure 8.43: n_clusters = 6;

average silhouette score = 0.588. average silhouette score = 0.450.
• The silhouette plot shows that

(n_clusters = 3, 5, and 6) are
bad picks for the data, due to
– the presence of clusters with below
average silhouette scores
– wide fluctuations in the size of the
silhouette plots
Figure 8.41: n_clusters = 4;
• Silhouette analysis is ambiva-
average silhouette score = 0.651.
lent in deciding between 2 and
4.
• When n_clusters = 4, all the
silhouette subplots are more or
less of similar thickness.
https://scikit-learn.org
Another way of Picking K with Silhouette Analysis
10 clusters
Figure 8.44: Average silhouette coefficient vs. number of clusters.

External Measures of Cluster Validity
• Entropya
– For cluster j, let pij be the probability that a member of cluster j
belongs to class i, defined as
pij = nij /Nj , (8.13)
where Nj is the number of points in cluster j and nij is the number

of points of class i in cluster j.
– The entropy of each cluster j is defined as
XL
ej = − pij log2 pij , (8.14)
i=1
where L is the number of classes and

– The total entropy is calculated as the sum of entropies
PK of each clus-
ter weighted by the size of each cluster: for N = j=1 Nj ,
K
1 X
e= Nj e j . (8.15)
N j=1
• Purity
– The purity of cluster j is given by
purityj = max pij . (8.16)

i
– The overall purity of a clustering is

K
1 X
purity = Nj purityj . (8.17)
N j=1
a
The concept of entropy was introduced earlier in § 5.4.1. Decision tree objectives, when we defined
impurity measures. See (5.68) on p. 124.
Final Comment on Cluster Validity

The following is a claim in an old book by (Jain & Dubes, 1988) [35].
However, today, the claim is yet true.
“The validation of clustering structures is the most difficult and frus-

trating part of cluster analysis. Without a strong effort in this direction,
cluster analysis will remain a black art accessible only to those true be-
lievers who have experience and great courage.”
8.6. Self-Organizing Maps 247
8.6. Self-Organizing Maps

8.6.1. Basics of the SOM
• Self-Organizing Map (SOM) refers to a process in which the inter-
nal organization increases automatically without being guided or man-
aged by an outside source.
– This process is due to local interaction with simple rules.
– Local interaction gives rise to a global structure.
• Why SOM?
A high-dimensional dataset is represented as an one/two-dimensional
discretized pattern using self-organizing maps or Kohonen maps.
• Advantage of SOM?
The primary benefit of employing an SOM is that the data is sim-
ple to read and comprehend. Grid clustering and the decrease of
dimensionality make it simple to spot patterns in the data.
We can interpret emerging global structures as learned structures,

which in turn appear as clusters of similar objects.
Note: The SOM acts as a unsupervised clustering algorithm and a

powerful visualization tool as well.
• It considers a neighborhood structure among the clusters.
⊕ The SOM is widely used in many application domains, such as econ-
omy, industry, management, sociology, geography, text mining, etc..
⊕ Many variants have been suggested to adapt the SOM to the pro-
cessing of complex data, such as time series, categorical data, nominal
data, dissimilarity or kernel data.
However, the SOM has suffered from a lack of rigorous results on
its convergence and stability.
Game of Life : – Most famous example of self-organization.
Simple local rules (en.wikipedia.org/wiki/Conway’s_Game_of_Life):

Suppose that every cell interacts with its eight neighbors.
• Any live cell with fewer than two live neighbors dies, as if
caused by under-population.
• Any live cell with two or three live neighbors lives on to the
next generation.
• Any live cell with more than three live neighbors dies, as if by
overcrowding.
• Any dead cell with exactly three live neighbors becomes a live
cell, as if by reproduction.
Figure 8.45: Still life, oscillator, and spaceship.

SOM Architecture
• A feed-forward neural network architecture based on competi-

tive learninga , invented by Teuvo Kohonen in 1982 [39].
• Neurobiological studies indicate that different sensory inputs (motorb ,
visual, auditory, etc.) are mapped onto corresponding areas of the cere-
bral cortex in an orderly fashion.
– Our interest is in building artificial topographic maps that
learn through self-organization in a neurobiologically inspired
manner.
a
One particularly interesting class of unsupervised system is based on competitive
learning, in which the output neurons compete amongst themselves to be activated,
with the result that only one is activated at any one time. This activated neuron is
called a winner-takes-all neuron or simply the winning neuron. Such competi-
tion can be induced/implemented by having lateral inhibition connections (nega-
tive feedback paths) between the neurons. The result is that the neurons are forced to
organize themselves.
b
Motor output is a response to the stimuli received by the nervous system.
• The principal goal of an SOM is to transform an incoming sig-

nal pattern of arbitrary dimension into a one/two-dimensional dis-
crete map, and to perform this transformation adaptively in a topo-
logically ordered fashion.
– We therefore set up our SOM by placing neurons at the nodes of a
one/two-dimensional lattice.
– Higher dimensional maps are also possible, but not so common.
• The neurons become selectively tuned, and the locations of the
neurons so tuned (i.e. the winning neurons) become ordered, and a
meaningful coordinate system for the input features is created on
the lattice.
– The SOM thus forms the required topographic map of the input
patterns.
• We can view this as a non-linear generalization of principal com-
ponent analysis (PCA).
Versions of the SOM
• Basic version: a stochastic process

• Deterministic version:
– For industrial applications, it can be more convenient to use a
deterministic version of the SOM, in order to get the same results
at each run of the algorithm when the initial conditions and the
data remain unchanged (repeatable!).
– To address this issue, T. Kohonen has introduced the batch SOM
in 1995 [40].
Remark 8.6. The following are quite deeply related to each other.
(a) Repeatability
(b) Optimality
(c) Convergence
(d) Interpretability
Indeterministic Issue & Deterministic Clustering

Clustering algorithms are to partition objects into groups based on their
similarity.
• Many clustering algorithms face indeterministic issue.
– For example, the standard K-means algorithm randomly selects
its initial centroids, which causes to produce different results in
each run.
• There have been several studies on how to achieve deterministic
clustering.
(a) Multiple runs
(b) Initialization, using hierarchical clustering approaches and PCA
(c) Elimination of randomness (Zhang et al., 2018) [83]
8.6.2. Kohonen SOM networks

We will see some details of the Kohonen SOM network or Kohonen
network.
• The Kohonen SOM network has a feed-forward structure with a
single computational layer arranged in rows and columns.
• Each neuron is fully connected to all the source nodes in the input
layer
Figure 8.46: Kohonen network.
Data Types for the Kohonen SOM
Originally, the SOM algorithm was defined for data described by numer-
ical vectors which belong to a subset X of Rd .
• Continuous setting: the input space X ⊂ Rd is modeled by a proba-
bility distribution with a density function f ,
• Discrete setting: the input space X comprises N data points
x1 , x2 , · · · , xN ∈ R d .
Here the discrete setting means a finite subset of the input space.
Components of Self-Organization
Preliminary 8.7. The self-organization process involves four major
components:
1. Initialization: All the connection weights are initialized with
small random values.
2. Competition: For each input pattern, the neurons compute their re-
spective values of a discriminant function which provides the basis
for competition.
• The particular neuron with the smallest value of the discrim-
inant function is declared the winner.
3. Cooperation: The winning neuron determines the spatial lo-
cation of a topological neighborhood of excited neurons, thereby
providing the basis for cooperation among neighboring neurons.
(smoothing the neighborhood of the winning neuron)
4. Adaptation: The excited neurons decrease their individual values
of the discriminant function in relation to the input pattern through
suitable adjustment of the associated connection weights, such that
the response of the winning neuron to the subsequent application of a
similar input pattern is enhanced.
(making the winning neuron look more like the observation)
Remark 8.8. The SOM is an unsupervised system based on competi-

tive learning.
• The output neurons compete amongst themselves to be activated,
with the result that only one is activated at any one time.
• This activated neuron is called a winner-takes-all neuron or sim-
ply the winning neuron.
• Such competition can be implemented by having lateral inhibition
connections (negative feedback paths) between the neurons.
• The result: The neurons are forced to organize themselves.
The Competitive Process

• The input space is d-dimensional (i.e. there are d input units).
• The connection weights between the d input units and the kth
output neuron (in the computational layer) can be written
 
w1k
w 
 2k 
wk =  .. , k = 1, · · · , K, (8.18)
 . 
wdk
where K is the total number of neurons in the computational layer.
Definition 8.9. We can define the discriminant function to be the

squared Euclidean distance between the input vector x and the weight
vector wk for each neuron k:
d
X
2
dk (x) = ||x − wk || = (xi − wik )2 . (8.19)
i=1
Thus, the neuron whose weight vector comes closest to the input vector (i.e.
is most similar to it) is declared the winner; see Preliminary 8.7, item 2.
The Cooperative Process

In neurobiological studies, it is found that there is lateral interaction
between a set of excited neurons.
• When one neuron fires, its closest neighbors tend to get excited more
than those further away.
• There is a topological neighbourhood that decays with distance.
Definition 8.10. Neighbourhood Function
• Let us take K units on a regular lattice

(string-like for 1D, or grid-like for 2D).
• If K = {1, 2, · · · , K} and t is the time, a neighborhood function h(t)
is defined on K × K. It has to satisfy the following properties:
(a) h is symmetric with hkk = 1,
(b) hk` depends only on the distance dist(k, `) between units k and ` on
the lattice, and
(c) h decreases with increasing distance.
• Several choices are possible for h.
– The most classical is the step function; equal to 1 if the distance
between k and ` is less than a specific radius (this radius can de-
crease with time), and 0 otherwise.
– Another very classical choice is a Gaussian-shaped function
dist2 (k, `)
hk` (t) = exp − , (8.20)
2σ 2 (t)
where σ 2 (t) can decrease over time to reduce the intensity and
the scope of the neighborhood relations. A popular time depen-
dence is an exponential decay:
σ(t) = σ0 exp(−t/τσ ). (8.21)
The Adaptive Process

The SOM must involve an adaptive (learning) process by which
• the outputs become self-organized and
• the feature map between inputs and outputs is formed.
Learning Rule in the Adaptive Process

• The point of the topographic neighborhood is twofold:
– The winning neuron gets its weights updated.
– Its neighbors will have their weights updated as well, although
by not as much as the winner itself.
• An appropriate weight update rule is formulated as
∆wk = η(t) · hI(x),k (t) · (x − wk ), ∀ k ∈ K, (8.22)
where I(x) is the index of the winning neuron and η(t) is a learning
rate (0 < η(t) < 1, constant or decreasing).
• The effect of each weight update is to move the weight vectors
wk of the winning neuron and its neighbors towards the input vector
x.
– Repeated presentations of the training data thus leads to topo-
logical ordering.
Remark 8.11. The learning rule (8.22) has several properties:

• Maximal at the winning neuron.
• Symmetric about that neuron.
• Decreases monotonically to zero as the distance goes to infinity.
• Translation invariant (i.e., independent of the location of the win-
ning neuron).
8.6.3. The SOM algorithm and its interpretation

The stages of the SOM can be summarized as follows.
Algorithm 8.12. The Stochastic SOM
• Initialization: A connection weight wk ∈ Rd is attached to each unit

k, whose initial values are chosen at random and denoted by
W (0) = [w1 (0), w2 (0), · · · , wK (0)].
• For t = 0, 1, 2, · · ·
(a) Sampling: A data point x is randomly drawn (according to the
density function f or from the finite set X)
(b) Matching: The best matching unit is defined by
I(x) = arg min ||x − wk (t)||2 (8.23)
k∈K
(c) Updating: All the weights are updated via

wk (t + 1) = wk (t) + ∆wk , ∀ k ∈ K, (8.24)
where, as defined in (8.22),
∆wk = η(t) · hI(x),k (t) · (x − wk ).
(d) Continuation: Keep returning to the sampling step until the

feature map stops changing.
Results of the SOM

• After learning, cluster Ck can be defined as the set of inputs closer to
wk than to any other one.
• The Kohonen map is the representation of
– the weights or
– the cluster contents,
displayed according to the neighborhood structure.
Example: Data approximation
Figure 8.47: Data approximation: One-dimensional SOM vs. PCA.

SOM is a red broken line with squares, 20 nodes. The first principal component is pre-
sented by a blue line. Data points are the small gray circles. The fraction of variance
unexplained in this example is 6.86% for SOM and 23.23% for PCA.
(“Self-organizing map”, Wikipedia)
Properties of the Kohonen Maps
• The quantization property: the weights represent the data space

as accurately as possible, as do other quantization algorithms.
– To get a better quantization, the learning rate η(t) decreases with
time as well as the scope of the neighborhood function h.
• The self-organization property, that means that the weights pre-
serve the topology of the data:
– close inputs belong to the same cluster (as do any clustering
algorithms) or to neighboring clusters.
Theoretical Issues
• The algorithm is easy to define and to use, and a lot of practical studies
confirm that it works.
– However, the theoretical study of its convergence when t tends to
∞ remains without complete proof and provides open problems.
– The main question is to know if the solution obtained from a finite
sample converges to the true solution that might be obtained from
the true data distribution.
• When t tends to ∞, the Rd -valued stochastic processes [wk (t)]k=1,2,··· ,K
can present oscillations, explosion to infinity, convergence in distribu-
tion to an equilibrium process, convergence in distribution or almost
sure to a finite set of points in Rd , etc.. Some of the open questions are:
– Is the algorithm convergent in distribution or almost surely, when
t tends to ∞?
– What happens when η(t) is constant? (when it decreases?)
– If a limit state exists, is it stable?
– How to characterize the organization?
In Practice: Ordering and Convergence

Note: The SOM algorithm may start from an initial state of complete
disorder, and it will gradually lead to an organized representation of ac-
tivation patterns drawn from the input space.
There are two identifiable phases of the adaptive process:

1. Ordering or Self-organizing phase – during which the topologi-
cal ordering of the weight vectors takes place.
• Typically this will take as many as 1000 iterations of the SOM
algorithm.
• Careful consideration needs to be given to the choice of neigh-
bourhood and learning rate parameters.
2. Convergence phase – during which the feature map is fine tuned
and comes to provide an accurate statistical quantification of the
input space.
• Typically the number of iterations in this phase will be at least
500 times the number of neurons in the network.
• Again, the parameters must be chosen carefully.
8.1. We will experiment the K-Means algorithm following the first section of Chapter 11,
Python Machine Learning, 3rd Ed., in a little bit different fashion.
(a) Make a dataset of 4 clusters (modifying the code on pp. 354–355).
(b) For K = 1, 2, · · · , 10, run the K-Means clustering algorithm with the initialization
init=’k-means++’.
(c) For each K, compute the within-cluster SSE (distortion) for an elbow analysis
to select an appropriate K. Note: Rather than using inertia_ attribute, imple-
ment a function for the computation of distortion.
(d) Produce silhouette plots for K = 3, 4, 5, 6.
8.2. Now, let’s experiment DBSCAN, following Python Machine Learning, 3rd Ed., pp. 376–
381.
(a) Produce a dataset having three half-moon-shaped structures each of which con-
sists of 100 samples.
(b) Compare performances of K-Means, AGNES, and DBSCAN.
(Set n_clusters=3 for K-Means and AGNES.)
(c) For K-Means and AGNES, what if you choose n_clusters much larger than 3 (for
example, 9, 12, 15)?
(d) Again, for K-Means and AGNES, perform an elbow analysis to select an appro-
priate K.
9
C HAPTER
Neural Networks and Deep Learning
Deep learning is a family of machine learning methods based on learning

data representations (features), as opposed to task-specific algorithms.
Learning can be supervised, semi-supervised, or unsupervised [3, 5, 69].
9.1. Basics for Deep Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262
9.2. Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266
9.3. Back-Propagation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278
9.4. Deep Learning: Convolutional Neural Networks . . . . . . . . . . . . . . . . . . . . . . 285
261
262 Chapter 9. Neural Networks and Deep Learning
9.1. Basics for Deep Learning

Conventional Machine Learning
• Limited in their ability to process data in their raw form
• Feature!!
– Coming up with features:
Difficult, time-consuming, requiring expert knowledge.
– Tuning the features: We spend a lot of time, before and during
learning.
Feature Learning
−→ −→
Representation Algorithm
Examples of features: Histogram of oriented gradients (HOG),

the scale-invariant feature transform (SIFT) (Lowe, 1999) [50], etc.
Representation Learning
• Discover representations, automatically

⇒ The machine is fed with raw data
• Deep Learning methods are representation-learning methods with
multiple levels of representation/abstraction
– Simple non-linear modules ⇒ higher and abstract representation
– With the composition of enough such transformations, very com-
plex functions can be learned.
• Key Aspects
– Layers of features are not designed by human engineers.
– Learn features from data using a general-purpose learning pro-
cedure.
9.1. Basics for Deep Learning 263
Advances in Deep Learning

• Image recognition [23, 29, 41]
• Speech recognition [29, 66]
• Natural language understanding [23, 73, 81]
– Machine translation
– Image 2 text
– Sentiment analysis
– Question-answering (QA) machine:
IBM’s Watson, 2011, defeated legendary Jeopardy champions
Brad Rutter and Ken Jennings, winning the first place prize of
$1 million
• Many other domains
– Predicting the activity of potential drug molecules
– Analyzing particle accelerator data
– Reconstructing brain circuits
– Predicting the effects of mutations in non-coding DNA on gene ex-
pression and disease
• Image-based Classifications: Deep learning has provided break-
through results in speech recognition and image classification.
Why/What about Deep Learning?
• Why is it generally better than other methods on image, speech,

and certain other types of data? Short answers:
– Deep learning means using a neural network with several lay-
ers of nodes between input and output
– The series of layers between input & output do feature identifi-
cation and processing in a series of stages, just as our brains
seem to do.
• Multi-layer neural networks have been more than 30 years (Rina
Dechter, 1986) [14]. What is actually new?
– We have always had good algorithms for learning the weights in
networks with 1 hidden layer.
But these algorithms are not good at learning the weights for net-
works with more hidden layers
– The New are: methods for training many-layer networks
Terms: AI vs. ML vs. Deep Learning

• Artificial intelligence (AI): Intelligence exhibited by machines
• Machine learning (ML): An approach to achieve AI
• Deep learning (DL): A technique for implementing ML
– Feature/Representation-learning
– Multi-layer neural networks (NN)
– Back-propagation
(In the 1980s and 1990s, researchers did not have much luck, except
for a few special architectures.)
– New ideas enable learning in deep NNs, since 2006
9.1. Basics for Deep Learning 265
Back-propagation to Train Multi-layer Architectures

• Nothing more than a practical application of the chain rule,
for derivatives
• Forsaken because poor local minima
• Revived around 2006 by unsupervised learning procedures with unla-
beled data
– CIFAR (Canadian Institute for Advanced Research): [4, 30, 31, 46]
– Recognizing handwritten digits or detecting pedestrians
– Speech recognition by GPUs, with 10 or 20 times faster
(Raina et al., 2009) [60] and (Bengio, 2013) [2].
– Local minima become rarely a problem.
• Convolutional neural network (CNN)
– Widely adopted by computer-vision community
(LeCun et al., 1989) [45]
• Activations
– Non-linear functions: max(z, 0) (ReLU), tanh(z), 1/(1 + e−z )
Machine Learning Challenges We’ve Yet to Overcome
• Interpretability: Although ML has come very far, researchers still

don’t know exactly how deep training nets work.
– If we don’t know how training nets actually work,
⇒ how do we make any real progress?
• One-Shot Learning: We still haven’t been able to achieve one-shot

learning. Traditional networks need a huge amount of data, and
are often in the form of extensive iterative training.
– Instead, we should find a way to enable neural networks to learn,
using just a few examples.
– Current neural networks are gradient-and-iteration-based;
⇒ can we modify/replace it?
9.2. Neural Networks

Recall: In 1957, Frank Rosenblatt invented the perceptron algorithm:
• For input values: x = (x1 , x2 , · · · , xd )T ,
• Learn weight vector: w = (w1 , w2 , · · · , wd )T
• Get the net input z = w1 x1 + w2 x2 + · · · + wd xd = w · x
• Classify, using the activation function

1 if z ≥ θ,
φ(z) = z = w · x, (9.1)
−1 otherwise,
or, equivalently,

1 if z ≥ 0,
φ(z) = z = b + w · x, (9.2)
−1 otherwise,
where b = −θ is the bias. (See (3.2) and (3.3), p. 40.)
Figure 9.1: Perceptron: The simplest artificial neuron.
Perceptron is the simplest artificial neuron:

• It makes decisions by weighting up evidence.
• However, it is not a complete model for decision-making!
9.2. Neural Networks 267
Complex Network of Perceptrons
• Perceptron as a building block:

– What the example illustrates is how a perceptron can weigh
up different kinds of evidence in order to make decisions.
Figure 9.2: A complex network of perceptrons.
• It should seem plausible that a complex network of perceptrons

could make quite subtle decisions.
An Issue on Perceptron Networks
• Thresholding. A small change in the weights or bias of any single

perceptron in the network can sometimes cause the output of that per-
ceptron to completely flip, say from −1 to 1.
– That flip may then cause the behavior of the rest of the net-
work to completely change in some very complicated way.
• We can overcome this problem by introducing a new type of artificial
neuron, e.g., a sigmoid neuron.
9.2.1. Sigmoid neural networks
Recall: The logistic sigmoid function is defined as

1
σ(z) = . (9.3)
1 + e−z
Figure 9.3: The standard logistic sigmoid function σ(z) = 1/(1 + e−z ).
Sigmoid Neural Networks

• They are built with sigmoid neurons.
• The output of a sigmoid neuron with inputs x, weights w, and bias b is
1
σ(z) = , (9.4)
1 + exp(−b − w · x)
which we considered as the logistic regression model in Section 5.2.
• Advantages of the sigmoid activation:
– It allows calculus to design learning rules. (σ 0 = σ(1 − σ))
– Small changes in weights and bias produce a corresponding
small change in the output.
X ∂ output ∂ output
∆output ≈ ∆wj + ∆b. (9.5)
j
∂wj ∂b
Figure 9.4: ∆output is a linear combination of ∆wj and ∆b.

The Architecture of (Smooth) Neural Networks

• The leftmost layer is called the input layer, and the neurons within
the layer are called input neurons.
• The rightmost layer is the output layer.
• The middle layers are called hidden layers.
• The design of the input and output layers in a network is of-

ten straightforward. For example, for the classification of hand-
written digits:
– If the images are in 28 × 28 grayscale pixels, then we’d have 784(=
28 × 28) input neurons.
– It is heuristic to set 10 neurons in the output layer. (rather than
4, where 24 = 16 ≥ 10)
• There can be quite an art to the design of the hidden layers.
– In particular, it is not possible to sum up the design process
for the hidden layers with a few simple rules of thumb.
– Instead, neural networks researchers have developed many de-
sign heuristics for the hidden layers, which help people get the
behavior they want out of their nets.
– For example, such heuristics can be used to help determine how
to trade off the number of hidden layers against the accuracy
and the time required to train the network.
9.2.2. A simple network to classify handwritten digits

• The problem of recognizing handwritten digits has two components:
segmentation and classification.
=⇒
Figure 9.5: Segmentation.
• We’ll focus on algorithmic components for the classification of individ-

ual digits.
MNIST Dataset :
A modified subset of two datasets collected by NIST (US National Insti-
tute of Standards and Technology):
• The first part contains 60,000 images (for training)
• The second part is 10,000 images (for test)
Each image is in 28 × 28 grayscale pixels.
A Simple Feed-forward Network
Figure 9.6: A sigmoid network having a hidden layer.

What the Neural Network Will Do

• Let’s concentrate on the first
output neuron, the one that is
trying to decide whether or not
the input digit is a 0.
• It does this by weighing up ev-
idence from the hidden layer of
neurons.
• What are those hidden neurons doing?

– Let’s suppose for the sake of argument that the first neuron
in the hidden layer may detect whether or not an image like the
following is present
It can do this by heavily weighting input pixels which overlap

with the image, and only lightly weighting the other inputs.
– Similarly, let’s suppose that the second, third, and fourth
neurons in the hidden layer detect whether or not the following
images are present
– As you may have guessed, these four images together make up

the 0 image that we saw in the line of digits shown in Figure 9.5:
– So if all four of these hidden neurons are firing, then we can

conclude that the digit is a 0.
Learning with Gradient Descent
• Dataset: {(x(i) , y(i) )}, i = 1, 2, · · · , N

– y(i) ? For example, if an image x(i) depicts a 2, then
y(i) = (0, 0, 1, 0, 0, 0, 0, 0, 0, 0)T .
• Cost function
1 X (i)
C(W , B) = ||y − a(x(i) )||2 , (9.6)
2N i
where W denotes the collection of all weights in the network, B all the
biases, and a(x(i) ) is the vector of outputs from the network when x(i)
is input.
• Gradient descent method

W W ∆W
← + , (9.7)
B B ∆B
where
∆W ∇W C
= −η .
∆B ∇B C
Note: To compute the gradient ∇C, we need to compute the gradients

∇Cx(i) separately for each training input, x(i) , and then average them:
1 X
∇C = ∇Cx(i) . (9.8)
N i
• Unfortunately, when the number of training inputs is very large, it

can take a long time, and learning thus occurs slowly.
• An idea called stochastic gradient descent can be used to speed
up learning.
Stochastic Gradient Descent

The idea is to estimate the gradient ∇C by computing ∇Cx(i) for a small
sample of randomly chosen training inputs. By averaging over this
small sample, it turns out that we can quickly get a good estimate of
the true gradient ∇C; this helps speed up gradient descent, and thus
learning.
• Pick out a small number of randomly chosen training inputs (m N ):
e(1) , x
x e(2) , · · · , x
e(m) ,
which we refer to as a mini-batch.

• Average ∇Cxe(k) to approximate the gradient ∇C. That is,
m
1 X def 1
X
∇Cxe(k) ≈ ∇C == ∇Cx(i) . (9.9)
m N i
k=1
• For classification of handwritten digits for the MNIST dataset, you

may choose: batch_size = 10.
Note: In practice, you can implement the stochastic gradient descent as

follows. For an epoch,
• Shuffle the dataset
• For each m samples (selected from the beginning), update (W , B)
using the approximate gradient (9.9).
Implementing a Network to Classify Digits [56]

network.py
1 """
2 network.py (by Michael Nielsen)
3 ~~~~~~~~~~
4 A module to implement the stochastic gradient descent learning
5 algorithm for a feedforward neural network. Gradients are calculated
6 using backpropagation. """
7 #### Libraries
8 # Standard library
9 import random
10 # Third-party libraries
12
13 class Network(object):
14 def __init__(self, sizes):
15 """The list ``sizes`` contains the number of neurons in the
16 respective layers of the network. For example, if the list
17 was [2, 3, 1] then it would be a three-layer network, with the
18 first layer containing 2 neurons, the second layer 3 neurons,
19 and the third layer 1 neuron. """
20
21 self.num_layers = len(sizes)
22 self.sizes = sizes
23 self.biases = [np.random.randn(y, 1) for y in sizes[1:]]
24 self.weights = [np.random.randn(y, x)
25 for x, y in zip(sizes[:-1], sizes[1:])]
26
27 def feedforward(self, a):

28 """Return the output of the network if ``a`` is input."""
29 for b, w in zip(self.biases, self.weights):
30 a = sigmoid(np.dot(w, a)+b)
31 return a
32
33 def SGD(self, training_data, epochs, mini_batch_size, eta,

34 test_data=None):
35 """Train the neural network using mini-batch stochastic
36 gradient descent. The ``training_data`` is a list of tuples
37 ``(x, y)`` representing the training inputs and the desired
38 outputs. """
39
40 if test_data: n_test = len(test_data)

41 n = len(training_data)
42 for j in xrange(epochs):
43 random.shuffle(training_data)
44 mini_batches = [
45 training_data[k:k+mini_batch_size]
46 for k in xrange(0, n, mini_batch_size)]

47 for mini_batch in mini_batches:
48 self.update_mini_batch(mini_batch, eta)
49 if test_data:
50 print "Epoch {0}: {1} / {2}".format(
51 j, self.evaluate(test_data), n_test)
52 else:
53 print "Epoch {0} complete".format(j)
54
55 def update_mini_batch(self, mini_batch, eta):

56 """Update the network's weights and biases by applying
57 gradient descent using backpropagation to a single mini batch.
58 The ``mini_batch`` is a list of tuples ``(x, y)``, and ``eta``
59 is the learning rate."""
60 nabla_b = [np.zeros(b.shape) for b in self.biases]
61 nabla_w = [np.zeros(w.shape) for w in self.weights]
62 for x, y in mini_batch:
63 delta_nabla_b, delta_nabla_w = self.backprop(x, y)
64 nabla_b = [nb+dnb for nb, dnb in zip(nabla_b, delta_nabla_b)]
65 nabla_w = [nw+dnw for nw, dnw in zip(nabla_w, delta_nabla_w)]
66 self.weights = [w-(eta/len(mini_batch))*nw
67 for w, nw in zip(self.weights, nabla_w)]
68 self.biases = [b-(eta/len(mini_batch))*nb
69 for b, nb in zip(self.biases, nabla_b)]
70
71 def backprop(self, x, y):

72 """Return a tuple ``(nabla_b, nabla_w)`` representing the
73 gradient for the cost function C_x. ``nabla_b`` and
74 ``nabla_w`` are layer-by-layer lists of numpy arrays, similar
75 to ``self.biases`` and ``self.weights``."""
76 nabla_b = [np.zeros(b.shape) for b in self.biases]
77 nabla_w = [np.zeros(w.shape) for w in self.weights]
78 # feedforward
79 activation = x
80 activations = [x] #list to store all the activations, layer by layer
81 zs = [] # list to store all the z vectors, layer by layer
82 for b, w in zip(self.biases, self.weights):
83 z = np.dot(w, activation)+b
84 zs.append(z)
85 activation = sigmoid(z)
86 activations.append(activation)
87 # backward pass
88 delta = self.cost_derivative(activations[-1], y) * \
89 sigmoid_prime(zs[-1])
90 nabla_b[-1] = delta
91 nabla_w[-1] = np.dot(delta, activations[-2].transpose())
92
93 for l in xrange(2, self.num_layers):

94 z = zs[-l]
95 sp = sigmoid_prime(z)
96 delta = np.dot(self.weights[-l+1].transpose(), delta) * sp
97 nabla_b[-l] = delta
98 nabla_w[-l] = np.dot(delta, activations[-l-1].transpose())
99 return (nabla_b, nabla_w)
100
101 def evaluate(self, test_data):

102 test_results = [(np.argmax(self.feedforward(x)), y)
103 for (x, y) in test_data]
104 return sum(int(x == y) for (x, y) in test_results)
105
106 def cost_derivative(self, output_activations, y):

107 """Return the vector of partial derivatives \partial C_x /
108 \partial a for the output activations."""
109 return (output_activations-y)
110
111 #### Miscellaneous functions

112 def sigmoid(z):
113 return 1.0/(1.0+np.exp(-z))
114
115 def sigmoid_prime(z):

116 return sigmoid(z)*(1-sigmoid(z))
The code is executed using

Run_network.py
1 import mnist_loader
2 training_data, validation_data, test_data = mnist_loader.load_data_wrapper()
3
4 import network
5 n_neurons = 20
6 net = network.Network([784 , n_neurons, 10])
7
8 n_epochs, batch_size, eta = 30, 10, 3.0

9 net.SGD(training_data , n_epochs, batch_size, eta, test_data = test_data)
len(training_data)=50000, len(validation_data)=10000, len(test_data)=10000

Validation Accuracy
Validation Accuracy
1 Epoch 0: 9006 / 10000
2 Epoch 1: 9128 / 10000
3 Epoch 2: 9202 / 10000
4 Epoch 3: 9188 / 10000
5 Epoch 4: 9249 / 10000
6 ...
7 Epoch 25: 9356 / 10000
8 Epoch 26: 9388 / 10000
9 Epoch 27: 9407 / 10000
10 Epoch 28: 9410 / 10000
11 Epoch 29: 9428 / 10000
Accuracy Comparisons
• scikit-learn’s SVM classifier using the default settings: 9435/10000
• A well-tuned SVM: ≈98.5%
• Well-designed (convolutional) NN: 9979/10000 (only 21 missed!)
Note: For well-designed neural networks, the performance is close

to human-equivalent, and is arguably better, since quite a few of
the MNIST images are difficult even for humans to recognize with confi-
dence, e.g.,
Figure 9.7: MNIST images difficult even for humans to recognize.
Moral of the Neural Networks

• Let all the complexity be learned, automatically, from data
• Simple algorithms can perform well for some problems:
(sophisticated algorithm) ≤ (simple learning algorithm + good training data)
9.3. Back-Propagation
• In the previous section, we saw an example of neural networks that

could learn their weights and biases using the stochastic gradient de-
scent algorithm.
• In this section, we will see how to compute the gradient, more
precisely, the derivatives of the cost function with respect to
weights and biases in all layers.
• The back-propagation is a practical application of the chain
rule for the computation of derivatives.
• The back-propagation algorithm was originally introduced in the
1970s, but its importance was not fully appreciated until a famous
1986 paper by Rumelhart-Hinton-Williams [65], in Nature.
9.3.1. Notations
• Let’s begin with notations which let us refer to weights, biases, and
activations in the network in an unambiguous way.
`
wjk : the weight for the connection from the k-th neuron in the
(` − 1)-th layer to the j-th neuron in the `-th layer
`
bj : the bias of the j-th neuron in the `-th layer
`
aj : the activation of the j-th neuron in the `-th layer
`
Figure 9.8: The weight wjk .
9.3. Back-Propagation 279
Figure 9.9: The bias b`j and activation a`j .
• With these notations, the activation a`j reads

X
` ` `−1 `
aj = σ wjk ak + bj , (9.10)
k
where the sum is over all neurons k in the (` − 1)-th layer. Denote the
weighted input by
X
zj` := ` `−1
wjk ak + b`j . (9.11)
k
• Now, define
W ` = [wjk `
] : the weight matrix for layer `
` `
b = [bj ] : the bias vector for layer `
(9.12)
z` = [zj` ] : the weighted input vector for layer `
a` = [a`j ] : the activation vector for layer `
• Then, (9.10) can be rewritten (in a vector form) as
a` = σ(z` ) = σ(W ` a`−1 + b` ). (9.13)

9.3.2. The cost function

The Cost Function: With the notations, the quadratic cost function
(9.6) has the form
1 X
C= ||y(x) − aL (x)||2 , (9.14)
2N x
where N is the total number of training examples, y(x) is the correspond-

ing desired output for the training example x, and L denotes the number
of layers in the network.
Two Assumptions for the Cost Function
1. The cost function can be written as an average

1 X
C= Cx , (9.15)
N x
over cost functions Cx for individual training examples x.

2. The cost function can be written as a function of the outputs from the
neural network (aL ).
Remark 9.1. Thus the cost function in (9.14) satisfies the assumptions,
with
1 1X
Cx = ||y(x) − aL (x)||2 = (yj (x) − aLj (x))2 . (9.16)
2 2 j
• The reason we need the first assumption is because what the back-
propagation actually lets us do is compute the partial derivatives
`
∂Cx /∂wjk and ∂Cx /∂b`j for a single training example.
`
– We then can recover ∂C/∂wjk and ∂C/∂b`j by averaging over
training examples.
• With the assumptions in mind, we may focus on computing the
partial derivatives for a single example.
9.3.3. The four fundamental equations behind the back-

propagation
The back-propagation is about understanding how changing the weights
and biases in a network changes the cost function, which means com-
`
puting the partial derivatives ∂C/∂wjk and ∂C/∂b`j .
Definition 9.2. Define the learning error (or, error) of neuron j in

layer ` by
def ∂C
δj` == . (9.17)
∂zj`
The back-propagation will give us a way of computing δ ` = [δj` ] for every

layer `, and then relating those errors to the quantities of real interest,
`
∂C/∂wjk and ∂C/∂b`j .
Theorem 9.3. Suppose that the cost function C satisfies the two as-
sumptions in Section 9.3.2 so that it represents the cost for a single train-
ing example. Assume the network contains L layers, of which the feed-
forward model is given as in (9.10):
X
` ` ` ` `−1
aj = σ(zj ), zj = wjk ak + b`j ; ` = 2, 3, · · · , L.
k
Then,
∂C 0 L
(a) δjL = σ (zj ),
∂aLj
X
` `+1 `+1 0 `
(b) δj = wkj δk σ (zj ), ` = L − 1, · · · , 2,
k
∂C (9.18)
(c) `
= δj` , ` = 2, · · · , L,
∂bj
∂C
(d) `
= a`−1 `
k δj , ` = 2, · · · , L.
∂wjk
Proof. Here, we will prove (b) only; see Exercise 1 for the others. Using the
definition (9.17) and the chain rule, we have
` ∂C X ∂C ∂zk`+1 X ∂zk`+1 `+1
δj = ` = `+1 ∂z `
= δ
` k
(9.19)
∂zj k
∂z k j k
∂z j
Note X X
zk`+1 = `+1 `
wki ai + b`+1
k = `+1
wki σ(zi` ) + b`+1
k . (9.20)
i i
Differentiating it, we obtain
∂zk`+1 X `+1 ∂σ(zi` ) `+1 0 `
`
= wki `
= wkj σ (zj ). (9.21)
∂zj i
∂z j
Substituting back into (9.19), we complete the proof.
The Hadamard product / Schur product

Definition 9.4. A frequently used algebraic operation is the element-
wise product of two vectors/matrices, which is called the Hadamard
product or the Schur product, and defined as
(c d)j = cj dj . (9.22)
• For example,
1 3 1·3 3
= = . (9.23)
2 4 2·4 8
• In Numpy, A*B denotes the Hadamard product of A and B, while A.dot(B)

or A@B produces the regular matrix-matrix multiplication.
Remark 9.5. The four fundamental equations (9.18) can be written in

a vector form as
(a) δ L = ∇aL C σ 0 (zL ),
(b) δ ` = ((W `+1 )T δ `+1 ) σ 0 (z` ), ` = L − 1, · · · , 2,
(9.24)
(c) ∇b` C = δ ` , ` = 2, · · · , L,
(d) ∇W ` C = δ ` (a`−1 )T , ` = 2, · · · , L.
The Back-Propagation Algorithm
Algorithm 9.6. Let’s summarize the back-propagation algorithm:

1. Input x: Set the corresponding activation a1 for the input layer.
2. Feed-forward:
z` = W ` a`−1 + b` ; a` = σ(z` ); ` = 2, 3, · · · , L
3. Output error δ L :
δ L = ∇aL C σ 0 (zL );
4. Back-propagate the error:
δ ` = ((W `+1 )T δ `+1 ) σ 0 (z` ); ` = L − 1, · · · , 2
5. The gradient of the cost function:
∇b` C = δ ` ; ∇W ` C = δ ` (a`−1 )T ; ` = 2, · · · , L
An SDG Learning Step, Based on a Mini-batch
1. Input a set of training examples of size m;

2. Initialize: ∆W = 0 and ∆B = 0;
3. For each training example x:
(a) Apply the back-propagation algorithm to find
∇b` Cx = δ x,` ; ∇W ` Cx = δ x,` (ax,`−1 )T ; ` = 2, · · · , L
(b) Update the gradient:

∆B = ∆B + [∇b2 Cx | · · · |∇bL Cx ];
∆W = ∆W + [∇W 2 Cx | · · · |∇W L Cx ];
4. Gradient descent: Update the biases and weights

η η
B=B− ∆B; W =W− ∆W ;
m m
See the method “update_mini_batch”, Lines 55–69 in network.py, p. 274.

Remarks 9.7. Saturated Neurons

• The four fundamental equations satisfy independently of choices of
the cost function C and the activation σ.
• A consequence of (9.24.d) is that if a`−1 is small (in modulus), gradient
term ∂C/∂W ` will also tend to be small. In this case, we’ll say the
weight learns slowly, meaning that it’s not changing much during
gradient descent.
– In other words, a consequence of (9.24.d) is that weights output
from low-activation neurons learn slowly.
– The sigmoid function σ becomes very flat when σ(zjL ) is approxi-
mately 0 or 1. When this occurs we will have σ 0 (zjL ) ≈ 0. So, a weight
in the final layer will learn slowly if the output neuron is either low
activation (≈ 0) or high activation (≈ 1). In this case, we usually say
the output neuron has saturated and, as a result, the weight is
learning slowly (or stopped).
• Similar remarks hold also in other layers and for the biases as well.
• Summing up, weights and biases will learn slowly if

– either the in-neurons (upwind) are in low-activation
– or the out-neurons (downwind) have saturated.
Designing Activation Functions

The four fundamental equations can be used to design activation func-
tions which have particular desired learning properties.
• For example, suppose we were to choose a (non-sigmoid) acti-
vation function σ so that σ 0 is always positive, and never gets
close to zero.
– That would prevent the slow-down of learning that occurs
when ordinary sigmoid neurons saturate.
• Learning accuracy and efficiency can be improved by finding
more effective cost and activation functions.
9.4. Deep Learning: Convolutional Neural Networks 285
9.4. Deep Learning: Convolutional Neural Net-

works
In this section, we will consider deep neural networks; the focus is on un-
derstanding core fundamental principles behind them, and applying those
principles for the easy-to-understand context of the MNIST problem.
Example 9.8. Consider neural networks for the classification of hand-

written digits, as shown in the following images:
Figure 9.10: A few images in the MNIST dataset.
A neural network can be built, with three hidden layers, as follows:
Figure 9.11
• Let each of hidden layers have 30 neurons:

– n_weights = 282 · 30 + 30 · 30 + 30 · 30 + 30 · 10 = 25, 620
– n_biases = 30 + 30 + 30 + 10 = 100
• Optimization is difficult
– The number of parameters to teach is huge (low efficiency)
– Multiple local minima problem (low solvability)
– Adding hidden layers is not necessarily improving accuracy
In fully-connected networks, deep neural networks have been

hardly practical, except for some special applications.
Remarks 9.9. The neural network exemplified in Figure 9.11 can pro-
duce a classification accuracy better than 98%, for the MNIST hand-
written digit dataset.
• But upon reflection, it’s strange to use networks of fully-
connected layers to classify images.
– The network architecture does not take into account the spatial
structure of the images.
– For instance, it treats input pixels which are far apart and close
together, on exactly the same footing.
What if, instead of starting with a network architecture which is tabula

rasa (blank mind), we use an architecture which tries to take advan-
tage of the spatial structure? Could it be better than 99%?
9.4.1. Introducing convolutional networks
Here, we will introduce convolutional neural networks (CNN),

which use a special architecture which is particularly well-adapted to
classify images.
• The architecture makes the convolutional networks fast to train.
• This, in turn, helps train deep, many-layer networks.
• Today, deep CNNs or some close variants are used in most neural net-
works for image recognition.
• CNNs use three basic ideas:

(a) local receptive fields,
(b) shared weights and biases, &
(c) pooling.
(a) Local Receptive Fields
Figure 9.12: An illustration for local receptive fields.
• In CNNs, the geometry of neurons (units) in the input layer is

exactly the same as that of images (e.g., 28 × 28).
(rather than a vertical line of neurons as in fully-connected networks)
• As per usual, we’ll connect the input neurons (pixels) to a layer of
hidden neurons.
– But we will not connect fully from every input pixel to every hid-
den neuron.
– Instead, we only make connections in small, localized regions of
the input image.
– For example: Each neuron in the first hidden layer will be con-
nected to a small region of the input neurons, say, a 5 × 5 region
(Figure 9.12).
• That region in the input image is called the local receptive field for
the hidden neuron.
• We slide the local receptive field across the entire input image.
– For each local receptive field, there is a different hidden neuron in
the first hidden layer.
Figure 9.13: Two of local receptive fields, starting from the top-left corner.
(Geometry of neurons in the first hidden layer is 24 × 24.)
Note: We have seen that the local receptive field is moved by one pixel
at a time (stride_length=1).
• In fact, sometimes a different stride length is used.
– For instance, we might move the local receptive field 2 pixels to
the right (or down).
– Most software gives a hyperparameter for the user to set the stride
length.
(b) Shared Weights and Biases

Recall that each hidden neuron has a bias and 5 × 5 weights connected
to its corresponding local receptive field.
• In CNNs, we use the same weights and bias for each of the 24 × 24
hidden neurons. In other words, for the (j, k)-th hidden neuron, the
output is:
X4 X4
σ b+ wp,q aj+p,k+q , (9.25)
p=0 q=0
where σ is the neural activation function (e.g., the sigmoid function),

b is the shared value for the bias, and wp,q is a 5 × 5 array of shared
weights.
– The weighting in (9.25) is just a form of convolution; we may
rewrite it as
a1 = σ(b + w ∗ a0 ). (9.26)
– So the network is called a convolutional network.
• We sometimes call the map, from the input layer to the hidden layer,
a feature map.
– Suppose the weights and bias are such that the hidden neuron
can pick out a feature (e.g., a vertical edge) in a particular local
receptive field.
– That ability is also likely to be useful at other places in the image.
– And therefore it is useful to apply the same feature detector
everywhere in the image.
• We call the weights and bias defining the feature map the shared
weights and the shared bias, respectively.
• A set of the shared weights and bias defines clearly a kernel or filter.
To put it in slightly more abstract terms, CNNs are well adapted to the
translation invariance of images.a
a
Move a picture of a cat a little ways, and it’s still an image of a cat.
Remark 9.10. Multiple feature maps.

• The network structure we have considered so far can detect just a
single localized feature.
• To do more effective image recognition, we’ll need more than one
feature map.
• Thus, a complete convolutional layer consists of several different
feature maps:
Figure 9.14: A convolutional network, consisting of 3 feature maps.
Modern CNNs are often built with 10 to 50 feature maps, each associated
to a r × r local receptive field: r = 3 ∼ 9.
Figure 9.15: The 20 images corresponding to 20 different feature maps, which are ac-
tually learned when classifying the MNIST dataset (r = 5).
The number of parameters to learn

A big advantage of sharing weights and biases is that it greatly reduces
the number of parameters involved in a convolutional network.
• Convolutional networks: (5 × 5 + 1) ∗ 20 = 520
• Fully-connected networks: (28 × 28 + 1) ∗ 20 = 15, 700
(c) Pooling
• CNNs also contain pooling layers, in addition to the convolutional
layers just mentioned.
• Pooling layers are usually used right after convolutional layers.
• What they do is to simplify the information in the output from the
convolutional layer.
Figure 9.16: Pooling: summarizing a region of 2 × 2 neurons in the convolutional layer.
From Figure 9.14: Since we have 24×24 neurons output from the convo-
lutional layer, after pooling we will have 12 × 12 neurons for each feature
map:
Figure 9.17: A convolutional network, consisting of 3 feature maps and pooling.
Types of pooling
1. max-pooling: simply outputs the maximum activation in the 2 × 2

input neurons.
2. L2 -pooling: outputs the L2 -average of the 2 × 2 input neurons.
...
9.4.2. CNNs, in practice
Figure 9.18: A simple CNN of three feature maps, to classify MNIST digits.
• To form a complete CNN by putting all these ideas, we need to add

some extra layers, below the convolution-pooling layers.
– Figure 9.18 shows a CNN that involves an extra layer of 10 output
neurons, for the 10 possible values for MNIST digits.
• The final layer of connections is a fully-connected layer. e.g.,:
– Let filter_shape =(20, 1,5,5), poolsize =(2,2)
(20 feature maps; 1 × 5 × 5 kernel; 2 × 2 pooling)
– Then, the number of parameters to teach:
(52 + 1) · 20 + (20 · 122 + 1) · 10 = 29, 330.
– Classification accuracy for the MNIST dataset . 99%
• Add a second convolution-pooling layer:
– Its input is the output of the first convolution-pooling layer.
– Let filter_shape =(40, 20,5,5), poolsize =(2,2)
– The output of the second convolution-pooling layer: 40 × 42
– Then, the number of parameters to teach:
(52 + 1) · 20 + (52 + 1) · 40 + (40 · 42 + 1) · 10 = 7, 970
– Classification accuracy for the MNIST dataset & 99%
• Add a fully-connected layer (up the output layer):
– Let choose 40 neurons: =⇒ 27, 610 parameters
– Classification accuracy ≈ 99.5%
• Use an ensemble of networks
– Using 5 CNNs, classification accuracy = 99.67% (33 missed!)
Figure 9.19: The images missed by an ensemble of 5 CNNs. The label in the top right is
the correct classification, while in the bottom right is the label classified output.
Remarks 9.11. Intuitively speaking:

• (Better representation). The use of translation invariance by the
convolutional layer will reduce the number of parameters it needs to
get the same performance as the fully-connected model.
• (Convolution kernels). The filters try to detect localized features,
producing feature maps.
• (Efficiency). Pooling simplifies the information in the output from
the convolutional layer.
– That, in turn, will result in faster training for the convolutional
model, and, ultimately, will help us build deep networks using
convolutional layers.
• Fully-connected hidden layers try to collect information for more

widely formed features.
Deep Learning Packages
• Keras
– A high-level neural network application programming interface
(API)
– It can run on top of TensorFlow, Theano, and Microsoft Cognitive
Toolkit (CNTK).
– Keras focuses on being modular, user-friendly, and extensible.
– Written in Python
• Tensorflow
– Low and High API levels: An end-to-end open-source deep
learning framework developed by Google in 2015
– It is a symbolic math library used for neural networks ⇒ fast!
– Tensorflow is difficult to use and debug.
– Written in Python, C++, CUDA
• Pytorch
– Low and High API levels: It is a deep learning framework based
on Torch, developed by Facebook in 2017, and taken over by the
PyTorch Foundation (part of Linux Foundation) in late 2022.
– It has outstanding community support and development.
– Pytorch is much easier to use and debug than Tensorflow.
– Written in Lua
Remark 9.12. You do not need to learn C++ or Lua.

• Currently, Keras is most popular due to its simplicity and long history.
• Pytorch is most rapidly growing.
– It can be viewed as a trade-off between Keras and Tensorflow.
– It is particularly good for natural language processing appli-
cations.
– Mathematicians and experienced researchers will find
Pytorch better than Keras, for many applications.
9.1. Complete proof of Theorem 9.3. Hint : The four fundamental equations in (9.18) can
be obtained by simple applications of the chain rule.
9.2. The core equations of back-propagation in a network with fully-connected layers are
given in (9.18). Suppose we have a network containing a convolutional layer, a max-
pooling layer, and a fully-connected output layer, as in the network shown in Fig-
ure 9.18. How are the core equations of back-propagation modified?
9.3. (Designing a deep network). First, download a CNN code (including the MNIST
dataset) by accessing to
https://github.com/mnielsen/neural-networks-and-deep-learning.git
or ‘git clone’ it. In the ‘src’ directory, there are 8 python source files:
conv.py mnist_average_darkness.py mnist_svm.py network2.py
expand_mnist.py mnist_loader.py network.py network3.py
On lines 16–22 in Figure 9.20 below, I put a design of a CNN model, which involved 2
hidden layers, one for a convolution-pooling layer and the other for a fully-connected
layer. Its test accuracy becomes approximately 98.8% in 30 epochs.
(a) Set ‘GPU = False’ in network3.py, if you are NOT using a GPU.
(Default: ‘GPU = True’, set on line 50-some of network3.py)
(b) Modify Run_network3.py appropriately to design a CNN model as accurate as
possible. Can your network achieve an accuracy better than 99.5%? Hint : You
may keep using the SoftmaxLayer for the final layer. ReLU (Rectified Linear
Units) seems comparable with the sigmoid function (default) for activation. The
default p_dropout=0.0. You should add some more layers, meaningful, and tune
well all of hyperparameters: the number of feature maps for convolutional layers,
the number of fully-connected neurons, η, p_dropout, etc..
(c) Design an ensemble of 5 such networks to further improve the accuracy. Hint :
Explore the function ‘ensemble’ defined in conv.py.
Run_network3.py
1 """ Run_network3.py:
2 -------------------
3 A CNN model, for the MNIST dataset,
4 which uses network3.py written by Michael Nielsen.
5 The source code can be downloaded from
6 https://github.com/mnielsen/neural-networks-and-deep-learning.git
7 or 'git clone' it
8 """
9 import network3
10 from network3 import Network, ReLU
11 from network3 import ConvPoolLayer, FullyConnectedLayer, SoftmaxLayer
12
13 training_data, validation_data, test_data = network3.load_data_shared()

14
15 mini_batch_size = 10
16 net = Network([
17 ConvPoolLayer(image_shape=(mini_batch_size, 1, 28, 28),
18 filter_shape=(20, 1, 5, 5),
19 poolsize=(2, 2), activation_fn=ReLU),
20 FullyConnectedLayer(
21 n_in=20*12*12, n_out=100, activation_fn=ReLU, p_dropout=0.0),
22 SoftmaxLayer(n_in=100, n_out=10, p_dropout=0.5)], mini_batch_size)
23
24 n_epochs, eta = 30, 0.1

25 net.SGD(training_data, n_epochs, mini_batch_size, eta, \
26 validation_data, test_data)
Figure 9.20: Run_network3.py

10
C HAPTER
Data Mining
10.1.Introduction to Data Mining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298
10.2.Vectors and Matrices in Data Mining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302
10.3.Text Mining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 310
10.4.Eigenvalue Methods in Data Mining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 320
297
298 Chapter 10. Data Mining
10.1. Introduction to Data Mining

Why Mine Data?
Commercial Viewpoint
• Lots of data is being collected and warehoused.
– Web data, e-commerce
– Purchases at department/grocery stores
– Bank/Credit Card transactions
• Computers have become cheaper and more powerful.
• Competitive pressure is strong.
– Provide better, customized services for an edge (e.g. in Customer
Relationship Management)
Scientific Viewpoint
• Data collected and stored at enormous speeds (GB/hour)
– Remote sensors on a satellite
– Telescopes scanning the skies
– Microarrays generating gene expression data
– Scientific simulations generating terabytes of data
• Traditional techniques infeasible for raw data
• Data mining may help scientists
– in classifying and segmenting data
– in Hypothesis Formation
10.1. Introduction to Data Mining 299
Mining Large Data Sets - Motivation

• There is often information “hidden” in the data that is not readily
evident.
• Human analysts may take weeks to discover useful information.
• Much of the data is never analyzed at all.
– Data gap becomes larger and larger.
What is Data Mining?

• Data mining is a process to turn raw data into useful informa-
tion/patterns.
– Non-trivial extraction of implicit, previously unknown and poten-
tially useful information from data.
– Exploration & analysis, by automatic or semi-automatic means, of
large quantities of data in order to discover meaningful patterns.
– Patterns must be: valid, novel, understandable, and poten-
tially useful.
Note: Data mining is also called Knowledge Discovery in Data

(KDD).
Origins of Data Mining

• Draws ideas from machine learning/AI, pattern recognition, statis-
tics, and database systems.
• Traditional Techniques may be unsuitable, due to
– Enormity of data
– High dimensionality of data
– Variety: Heterogeneous, distributed nature of data
Data Mining Methods

• Prediction Methods
– Use some variables to predict unknown or future values of other
variables.
• Description Methods
– Find human-interpretable patterns that describe the data.
Data Mining Tasks

• Classification [Predictive]
– Given a collection of records (training set), find a model for class
attribute as a function of the values of other attributes.
• Regression [Predictive]
– Predict a value of a given continuous valued variable based on the
values of other variables, assuming a linear or nonlinear model of
dependency.
• Clustering [Descriptive]
– Given a set of data points and a similarity measure among them,
find clusters such that data points in one cluster are more similar
to one another than points in other clusters.
• Association Rule Discovery [Descriptive]
– Given a set of records each of which contain some number of items
from a given collection, produce dependency rules which will pre-
dict occurrence of an item based on occurrences of other items.
• Sequential Pattern Discovery [Descriptive]
– Given is a set of objects, with each object associated with its own
timeline of events, find rules that predict strong sequential depen-
dencies among different events.
• Deviation/Anomaly Detection [Predictive]
– Detect significant deviations from normal behavior.
10.1. Introduction to Data Mining 301
Challenges in Data Mining

• Scalability
• Dimensionality
• Complex and Heterogeneous Data
– Spatial and temporal data
– Point and interval data
– Categorical data
– Graph data
– semi/un-structured Data
• Data Quality
• Data Ownership and Distribution
• Privacy Preservation
• Streaming Data
Related Fields
Figure 10.1: Related fields.

10.2. Vectors and Matrices in Data Mining

Note: Often the data are numerical, and the data points can be thought
of as belonging to a high-dimensional vector space. Ensembles of data
points can then be organized as matrices. In such cases it is natural to
use concepts and techniques from linear algebra. Here, we present Nu-
merical Linear Algebra in Data Mining, following and modifying (Eldén,
2006) [17].
10.2.1. Examples
Example 10.1. Term-document matrices are used in information re-
trieval. Consider the following set of five documents. Key words, referred
to as terms, are marked in boldface.
Document 1: The Google matrix P is a model of the Internet.
Document 2: Pij is nonzero if there is a link from web page j to i.
Document 3: The Google matrix is used to rank all web pages.
Document 4: The ranking is done by solving a matrix eigenvalue
problem.
Document 5: England dropped out of the top 10 in the FIFA ranking.
• Counting the frequency of terms in each document we get the following

result.
Term Doc 1 Doc 2 Doc 3 Doc 4 Doc 5
eigenvalue 0 0 0 1 0
England 0 0 0 0 1
FIFA 0 0 0 0 1
Google 1 0 1 0 0
Internet 1 0 0 0 0
link 0 1 0 0 0
matrix 1 0 1 1 0
page 0 1 1 0 0
rank 0 0 1 1 1
web 0 1 1 0 0
The set of terms is called the dictionary.

10.2. Vectors and Matrices in Data Mining 303
• Each document is represented by a vector in R10 , and we can organize

the data as a term-document matrix,
 
0 0 0 1 0
0 0 0 0 1
 
0 0 0 0 1
 
1 0 1 0 0
 
1 0 0 0 0
 
A=  ∈ R10×5 . (10.1)
0 1 0 0 0
1 0 1 1 0
 
 
0 1 1 0 0
 
0 0 1 1 1
0 1 1 0 0
• Assume that we want to find all documents that are relevant with respect
to the query “ranking of web pages”. This is represented by a query vec-
tor, constructed in an analogous way as the term-document matrix, using
the same dictionary.  
0
0
 
0
 
0
 
0
 
q =   ∈ R10 . (10.2)
0
0
 
 
1
 
1
1
Thus the query itself is considered as a document.
• Now, the information retrieval task can be formulated as a mathematical
problem: find the columns of A that are close to the vector q.
– To solve this problem we use some distance measure in R10 .

Remark 10.2. Information Retrieval, with term-document ma-

trix A ∈ Rm×n .
• It is common that m is large, of the order 106 , say.
• The matrix A is sparse, because most of the documents only con-
tain a small fraction of the terms in the dictionary.
• In some methods for information retrieval, linear algebra tech-
niques, such as singular value decomposition (SVD), are used
for data compression and retrieval enhancement.
Note: The very idea of data mining is to extract useful informa-

tion from large and often unstructured datasets. Therefore
• the methods must be efficient and often specially designed for
large problems.
Example 10.3. Google Pagerank algorithm. The task of extracting

information from all the web pages available on the Internet, is performed
by search engines.
• The core of the Google search engine is a matrix computation, prob-

ably the largest that is performed routinely.
• The Google matrix P is assumed to be of dimension of the order
billions (2005), and it is used as a model of (all) the web pages on the
Internet.
• In the Google Pagerank algorithm, the problem of assigning ranks
to all the web pages is formulated as a matrix eigenvalue problem.
The Google Pagerank algorithm

• Let all web pages be ordered from 1 to n, and let i be a particular web
page.
• Then Oi will denote the set of pages that i is linked to, the outlinks. The
number of outlinks is denoted Ni = |Oi |.
• The set of inlinks, denoted Ii , are the pages that have an outlink to i.
• Now define Q to be a square matrix of dimension n, and let

1/Nj , if there is a link from j to i,
Qij = (10.3)
0, otherwise.
– This definition means that row i has nonzero elements in those posi-
tions that correspond to inlinks of i.
– Similarly, column j has nonzero elements equal to 1/Nj in those posi-
tions that correspond to the outlinks of j.
– Thus, the sum of each column is either 0 or 1.
• The following link graph illustrates a set of web pages with outlinks and
inlinks.
Figure 10.2: A link graph, for six web pages.
The corresponding link matrix becomes

 
0 1/3 0 0 0 0
1/3 0 0 0 0 0
 
 0 1/3 0 0 1/3 1/2
Q= 1/3
 (10.4)
 0 0 0 1/3 0
1/3 1/3 0 0 0 1/2
0 0 1 0 1/3 0
• Define a pagerank vector r, which holds the ranks of all pages.

• Then, the vector r can be found as the eigenvector corresponding to the
eigenvalue λ = 1 of Q:
Qr = λr. (10.5)
We discuss numerical aspects of the Pagerank computation in Section 10.4.
10.2.2. Data compression: Low rank approximation

Note: Rank Reduction.
• One way of measuring the information contents in a data matrix is to
compute its rank.
• Obviously, linearly dependent column or row vectors are redundant.
• Therefore, one natural procedure for extracting information from a
data matrix is to systematically determine a sequence of lin-
early independent vectors, and deflate the matrix by subtracting
rank one matrices, one at a time.
• It turns out that this rank reduction procedure is closely related to
matrix factorization, data compression, dimensionality reduc-
tion, and feature selection/extraction.
• The key link between the concepts is the Wedderburn rank reduc-
tion theorem.
Theorem 10.4. (Wedderburn, 1934) [78]. Suppose A ∈ Rm×n , f ∈ Rn ,

and g ∈ Rm . Then
Af g T A
rank A − = rank(A) − 1 ⇐⇒ ω = g T Af 6= 0. (10.6)
ω
Algorithm 10.5. Wedderburn rank-reduction process.

Based on Wedderburn rank reduction theorem, a stepwise rank reduc-
tion procedure can be defined.
• Let A(0) = A.
• Define a sequence of matrices {A(i) }:
T
(i+1) (i) A(i) f (i) g (i) A(i)
A =A − , (10.7)
ωi
where f (i) ∈ Rn and g (i) ∈ Rm such that
T
ωi = g (i) A(i) f (i) 6= 0. (10.8)
• The sequence defined in (10.7) terminates when r = rank(A(i+1) ),

since each time the rank of the matrix decreases by one. The matri-
ces A(i) are called Wedderburn matrices.
Remark 10.6. The Wedderburn rank-reduction process gives a matrix

decomposition called the rank-reduction decomposition.
A = FbΩ−1 G,
b (10.9)
where
Fb = (f1 , · · · , fr ) ∈ Rm×r , fi = A(i) f (i) ,
Ω = diag(ω1 , · · · , ωr ) ∈ Rr×r , (10.10)
Gb = (g1 , · · · , gr ) ∈ Rn×r , gi = A(i) T g (i) .
Theorem 10.4 can be generalized to the case where the reduction of rank is
larger than one, as shown in the next theorem.
Theorem 10.7. (Guttman, 1957) [28]. Suppose A ∈ Rm×n , F ∈ Rn×k ,

and G ∈ Rm×k . Then
rank(A − AF R−1 GT A) = rank(A) − rank(AF R−1 GT A)
(10.11)
⇐⇒ R = GT AF is nonsingular.
Note: There are many choices of F and G that satisfy the condition
(10.11).
• Therefore, various rank-reduction decompositions are possible.
• It is known that several standard matrix factorizations in numerical
linear algebra are instances of the Wedderburn formula:
– Gram-Schmidt orthogonalization,
– singular value decomposition,
– QR and Cholesky decomposition, and
– the Lanczos procedure.
Relation between the truncated SVD and the Wedderburn rank

reduction process
• Recall the truncated SVD (7.17), page 154.
• In the rank reduction formula (10.11), define the error matrix E as
E = A − AF R−1 GT A = A − AF (GT AF )−1 GT A, (10.12)
where F ∈ Rn×k and G ∈ Rm×k .

• Assume that k ≤ rank(A) = r, and consider the problem
min ||E|| = min ||A − AF (GT AF )−1 GT A||, (10.13)
F ∈Rn×k ,G∈Rm×k
where the norm is orthogonally invariant such as the L2 -norm and

the Frobenius norm.
• According to Theorem 7.16, p.163, the minimum error is obtained
when
(AF )(GT AF )−1 (GT A) = U Σk V T = Uk Σk VkT , (10.14)
which is equivalent to choosing
F = Vk G = Uk . (10.15)
10.3. Text Mining

Definition 10.8. Text mining is methods that extract useful informa-
tion from large and often unstructured collections of texts.
• A related term is information retrieval.
• A typical application is search in data bases of abstract of scientific
papers.
– For instance, in medical applications one may want to find all the
abstracts in the data base that deal with a particular syndrome.
– So one puts together a search phrase, a query, with key words that
are relevant for the syndrome.
– Then the retrieval system is used to match the query to the docu-
ments in the data base, and present to the user all the documents
that are relevant, preferably ranked according to relevance.
Example 10.9. The following is a typical query (Eldén, 2006) [17].
9. the use of induced hypothermia in heart surgery, neurosurgery,

(10.16)
head injuries and infectious diseases.
We will refer to this query as Q9 in the sequel.
Note: Another well-known area of text mining is web search engines.

• There the search phrase is usually very short.
• Often there are so many relevant documents that it is out of the
question to present them all to the user.
• In that application the ranking of the search result is critical for
the efficiency of the search engine.
• We will come back to this problem in Section 10.4.
10.3. Text Mining 311
Public Domain Text Mining Software

A number of public domain software are available.
• R
– textmineR
• Python
– nltk (natural language toolkit)
– spaCy (written in Cython)
In this section
We will review one of the most common methods for text mining, namely
the vector space model (Salton et al., 1975) [67].
• In Example 10.1, we demonstrated the basic ideas of the construction
of a term-document matrix in the vector space model.
• Below we first give a very brief overview of the preprocessing that is
usually done before the actual term-document matrix is set up.
• Then we describe a variant of the vector space model: Latent Seman-
tic Indexing (LSI) (Deerwester et al., 1990) [15], which is based on
the SVD of the term-document matrix.
10.3.1. Vector space model: Preprocessing and query match-

ing
Note: In information retrieval, keywords that carry information about

the contents of a document are called terms.
• A basic task is to create a list of all the terms in alphabetic order, a
so called index.
• But before the index is made, two preprocessing steps should be
done:
(a) removal of stop words
(b) stemming
Removal of Stop Words

• Stop words are words that one can find in virtually any document.
• The occurrence of such a word in a document does not distinguish this
document from other documents.
• The following is the beginning of one stop list
a, able, about, above, according, accordingly, across, actually, after, after-
wards, again, against, ain’t, all, allow, allows, almost, alone, along, already,
also, although, always, am, among, amongst, an, and, ...
• Various sets of stop words are available on the Internet, e.g.

https://countwordsfree.com/stopwords.
Stemming
• Stemming is the process of reducing each word that is conjugated or
has a suffix to its stem.
• Clearly, from the point of view of information retrieval, no information
is lost in the following reduction.

computable 

computation 


computing =⇒ comput
computed




computational

• Public domain stemming algorithms are available on the Internet, e.g.

the Porter Stemming Algorithm
https://tartarus.org/martin/PorterStemmer/.
The Term-Document Matrix

• The term-document matrix A ∈ Rm×n , where
m = the number of terms in the dictionary
n = the number of documents
• It is common not only to count the occurrence of terms in docu-
ments but also to apply a term weighting scheme.
• Similarly, document weighting is usually done.
Example 10.10. For example, one can define the elements in A by
aij = fij log(n/ni ), (10.17)
where
• fij is term frequency,
the number of times term i appears in document j,
• ni is the number of documents that contain term i
(inverse document frequency).
If a term occurs frequently in only a few documents, then both factors
are large. In this case the term discriminates well between different
groups of documents, and it gets a large weight in the documents where
it appears.
Normally, the term-document matrix is sparse: most of the matrix elements

are equal to zero.
Example 10.11. For the stemmed Medline collection in Example 10.9,
p.310, the matrix is 4163 × 1063, with 48263 non-zero elements, i.e. approxi-
mately 1%. (It includes 30 query columns.) The first 500 rows and columns
of the matrix are illustrated in Figure 10.3.
Figure 10.3: The first 500 rows and columns of the Medline matrix. Each dot represents a
non-zero element.
Query Matching
• The query is parsed using the same dictionary as the documents, giv-
ing a vector q ∈ Rm .
• Query matching is the process of finding all documents that are con-
sidered relevant to a particular query q.
• This is often done using the cosine distance measure: All docu-
ments {aj } are returned for which
q · aj
≥ tol, (10.18)
||q|| ||aj ||
where tol is user-defined tolerance.
Example 10.12. Query matching is performed for query Q9 in the stemmed

Medline collection. With tol = 0.19 only a single document is considered rel-
evant. When the tolerance was lowered to 0.17, then three documents are
retrieved.
• Irrelevant documents may be returned.
– For a high value of the tolerance, the retrieved documents are likely
to be relevant.
– When the cosine tolerance is lowered, irrelevant documents may be
returned relatively more.
Definition 10.13. In performance modelling for information retrieval,

we define the following measures:
Tr Tr
P = (precision) R= (recall), (10.19)
Tt Br
where
Tr = the number of relevant documents retrieved
Tt = the total number of documents retrieved
Br = the total number of relevant documents in the database
See Definition 8.4 and Figure 8.32, p.237.
Note: With the cosine measure:

• We see that with a large value of tol, we have high precision, but
low recall.
• For a small value of tol, we have high recall, but low precision.
Example 10.14. Query matching is performed for query Q9 in the Medline

collection using the cosine measure, in order to obtain recall and precision
as illustrated in Figure 10.4.
• In the comparison of different methods, it is more illustrative to draw

the recall versus precision diagram.
• Ideally a method has high recall at the same time as the precision is
high. Thus, the closer the curve is to the upper right corner, the better
the method is.
Figure 10.4: Recall versus precision diagram for query matching for Q9, using the vector
space method.
10.3.2. Latent Semantic Indexing

Latent Semantic Indexing (LSI) is based on the assumption
• that there is some underlying latent semantic structure in the
data that is corrupted by the wide variety of words used
• and that this semantic structure can be enhanced by projecting the
data onto a lower-dimensional space using the singular value de-
composition.
Algorithm 10.15. Latent Semantic Indexing (LSI)
• Let A = U ΣV T be the SVD of the term-document matrix.

• Let Ak be its approximation of rank k:
Ak = Uk Σk VkT = Uk (Σk VkT ) =: Uk Dk , (10.20)
where Vk ∈ Rn×k so that Dk ∈ Rk×n .

– The columns of Uk live in the document space and are an orthogo-
nal basis that we use to approximate all the documents.
– Column j of Dk holds the coordinates of document j in terms of the
orthogonal basis.
• Note that
qT Ak = qT Uk Dk = (UkT q)T Dk ∈ R1×n . (10.21)
• Thus, in query matching, we compute the coordinates of the query in

terms of the new document basis and compute the cosines from
bk · (Dk ej )
q
cos θj = , bk = UkT q.
q (10.22)
||b
qk || ||Dk ej ||
• This means that the query-matching is performed in a k-dimensional

space.
Example 10.16. Query matching is carried out for Q9 in the Medline

collection, approximating the matrix using the truncated SVD with of rank
100 (k = 100). The recall-precision curve is given in Figure 10.5. It is seen
that for this query, the LSI improves the retrieval performance.
Figure 10.5: Recall versus precision diagram for query matching for Q9, using the full
vector space method (solid curve) and the rank 100 approximation (dashed).
Example 10.17. Recall Example 10.1. Consider the term-document ma-

trix A ∈ R10×5 and the query vector q ∈ R10 , of which the query is “ranking
of web pages”. See pages 302–303 for details.
Document 1: The Google matrix P is a model of the Internet.
Document 2: Pij is nonzero if there is a link from web page j to i.
Document 3: The Google matrix is used to rank all web pages.
Document 4: The ranking is done by solving a matrix eigenvalue problem.
Document 5: England dropped out of the top 10 in the FIFA ranking.
   
0 0 0 1 0 0
0 0 0 0 1 0
   
0 0 0 0 1 0
   
1 0 1 0 0 0
   
1 0 0 0 0 0
 ∈ R10×5 , 10
A= 0 ∈ R .
q= 
0 1 0 0 0
   
1 0 1 1 0 0
   
0 1 1 0 0 1
   
0 0 1 1 1 1
0 1 1 0 0 1
• (Eldén, 2006) [17]

“Obviously, Documents 1-4 are relevant with respect to the query,
while Document 5 is totally irrelevant. However, we obtain the fol-
lowing cosines for query and the original data
(0 0.6667 0.7746 0.3333 0.3333)
We then compute the SVD of the term-document matrix, and use a rank
2 approximation. After projection to the two-dimensional subspace the
cosines, computed according to (10.22), are
(0.7857 0.8332 0.9670 0.4873 0.1819)
It turns out that Document 1, which was deemed totally irrelevant for the
query in the original representation, is now highly relevant. In addition,
the scores for the relevant Documents 2-4 have been reinforced. At the
same time, the score for Document 5 has been significantly reduced.”
• However, I view it as a warning.
10.4. Eigenvalue Methods in Data Mining

Note: An Internet search performs two major operations, using a
search engine.
(a) Traditional text processing. The aim is to find all the web pages
containing the words of the query.
(b) Sorting out.
• Due to the massive size of the Web, the number of hits is likely
to be much too large to be handled by the user.
• Therefore, some measure of quality is needed to sort out the
pages that are likely to be most relevant to the particular query.
When one uses a web search engine, then typically the search phrase is
under-specified.
Example 10.18. A Google search conducted on October 21, 2022, using

the search phrase “university”:
• The result: links to universities, including Mississippi State University,

University of Arizona, University of Washington - Seattle, University of
Wisconsin–Madison, The University of Texas at Austin, and University
of Southern California - Los Angeles.
• The total number of web pages relevant to the search phrase was more
than 7 billions.
Remark 10.19. Google uses an algorithm (Pagerank) for ranking

all the web pages that agrees rather well with a common-sense quality
measure.
• Google assigns a high rank to a web page, if it has inlinks from
other pages that have a high rank.
• We will see that this “self-referencing” statement can be formu-
lated mathematically as an eigenvalue problem.
10.4. Eigenvalue Methods in Data Mining 321
10.4.1. Pagerank
Note: Google uses the concept of Pagerank as a quality measure of web

pages. It is based on the assumption that
the number of links to and from a page give information
about the importance of a page.
• Let all web pages be ordered from 1 to n, and let i be a particular web
page.
• Then Oi will denote the set of pages that i is linked to, the outlinks.
The number of outlinks is denoted Ni = |Oi |.
• The set of inlinks, denoted Ii , are the pages that have an outlink to i.
Note: In general, a page i can be considered as more important the

more inlinks it has.
• However, a ranking system based only on the number of inlinks is easy
to manipulate.
– When you design a web page i that you would like to be seen by
as many as possible, you could simply create a large number of
(information-less and unimportant) pages that have outlinks to i.
• In order to discourage this, one may define the rank of i in such a
way that if a highly ranked page j, has an outlink to i, this should
add to the importance of i.
• Here the manner is:
the rank of page i is a weighted sum of the ranks of the
pages that have outlinks to i.
Definition 10.20. The preliminary definition of Pagerank is

X rj
ri = , i = 1, 2, · · · , n. (10.23)
Nj
j∈Ii
That is, the weighting is such that the rank of a page j is divided evenly
among its outlinks.
Remark 10.21. Pagerank may not be solvable.

• As in (10.3)-(10.4), p. 305, let Q be a square matrix of dimension n,
and let

1/Nj , if there is a link from j to i,
Qij = (10.24)
0, otherwise,
where Q is sometimes called the normalized web matrix.

• Then, (10.23) can be written as
λr = Qr, λ = 1, (10.25)
i.e., r is an eigenvector of Q with eigenvalue λ = 1.

• However, it is not clear that Pagerank is well-defined, because we do
not know if there exists an eigenvalue equal to 1.
Reformulation of (10.23)
Modify the matrix Q to have an eigenvalue λ = 1.
• Assume that a surfer visiting a web page, always chooses the next page
among the outlinks with equal probability.
• Assume that the random surfer never get stuck.
– In other words, there should be no web pages without outlinks
(such a page corresponds to a zero column in Q).
• Therefore the model is modified so that zero columns are replaced by
a constant value in each position.
• Define the vectors

   
1 d1
1 d 
2 1, if Nj = 0,
e =  ..  ∈ Rn , d =  .. , dj = (10.26)
   
. . 0, otherwise.
1 dn
• Then the modified matrix is defined

1 T
P = Q+ ed . (10.27)
n
• Then P is a column-stochastic matrix, of which columns are prob-
ability vectors. That is, it has non-negative elements (P ≥ 0) and the
sum of each column is 1.
• Furthermore,
1 T T
eT P = eT Q + e ed = eT Q + dT = eT , (10.28)
n
which implies that λ = 1 is a left eigenvalue and therefore a right
eigenvalue. Note that
eT P = eT ⇐⇒ P T e = e,
(10.29)
det(A − λI) = det(AT − λI).
• Now, we define the Pagerank vector r as a unique eigenvector of P

with eigenvalue λ = 1,
P r = r. (10.30)
• However, uniqueness is still not guaranteed.

– To ensure this, the directed graph corresponding to the matrix
must be strongly connected
– Equivalently, in matrix terms, P must be irreducible.
– Equivalently, there must not exist any subgraph, which has no
outlinks.
The uniqueness of the eigenvalue is guaranteed by the Perron-Frobenius

theorem.
Theorem 10.22. (Perron-Frobenius) If A ∈ Rn×n is nonnegative,
then
• ρ(A) is an eigenvalue of A.
• There is a nonnegative eigenvector x such that Ax = ρ(A)x.
Theorem 10.23. (Perron-Frobenius) If A ∈ Rn×n is nonnegative and

irreducible, then
• ρ(A) is an eigenvalue of A.
• ρ(A) > 0.
• There is a positive eigenvector x such that Ax = ρ(A)x.
• ρ(A) is a simple eigenvalue.
Theorem 10.24. (Perron) If A ∈ Rn×n is positive, then

• Theorem 10.23 holds, and in addition,
• |λ| < ρ(A) for any eigenvalue λ with λ 6= ρ(A).
Corollary 10.25. Let A be an irreducible column-stochastic matrix.

Then
• The largest eigenvalue in magnitude is equal to 1.
• There is a unique corresponding eigenvector r satisfying r > 0 and
||r||1 = 1; this is the only eigenvector that is non-negative.
• If A > 0, then |λi | < 1, i = 2, 3, · · · , n.
Remark 10.26. Given the size of the Internet and reasonable assump-
tions about its structure,
• it is highly probable that the link graph is not strongly connected,
• which means that the Pagerank eigenvector of P may not be
well-defined.
10.4.2. The Google matrix

To ensure connectedness, i.e., to make it impossible for the random
walker to get trapped in a subgraph, one can add, artificially, a link
from every web page to all the other. In matrix terms, this can be
made by taking a convex combination of P and a rank one matrix.
One billion dollar idea, by Sergey Brin and Lawrence Page in 1996
• The Google matrix is the matrix
1
G = αP + (1 − α) eeT , (10.31)
n
for some α satisfying 0 < α < 1, called the damping factor.
• Obviously G is irreducible (since G > 0) and column-stochastic.a
• Furthermore,
1
eT G = αeT P + (1 − α)eT eeT = αeT + (1 − α)eT = eT . (10.32)
n
• The pagerank equation reads
h 1 Ti
Gr = αP + (1 − α) ee r = r. (10.33)
n
a
A n × n matrix is called a Markov matrix if all entries are nonnegative and the sum of each column
vector is equal to 1. A Markov matrix are also called a stochastic matrix.
Note: The random walk interpretation of the additional rank one term
is that each time step a page is visited, the surfer will jump to any page
in the whole web with probability 1 − α (sometimes referred to as tele-
portation).
1
• Recall (10.27): P = Q + edT , which can be interpreted as follows.
n
When a random surfer visits a web page of no outlinks, the
surfer will jump to any page with an equal probability 1/n.
1
• The convex combination in (10.31): G = αP + (1 − α) eeT .
n
Although there are outlinks, the surfer will jump to any page
with an equal probability (1 − α)/n.
Proposition 10.27. Let the eigenvalues of the column-stochastic ma-

trix P be {1, λ2 , λ3 , · · · , λn }. Then, the eigenvalues of G = αP + (1 −
α) n1 eeT are {1, αλ2 , αλ3 , · · · , αλn }.
• This means that even if P has a multiple eigenvalue equal to 1, the
second largest eigenvalue in magnitude of G is equal to α.
1
Remark 10.28. The vector e in (10.31) can be replaced by a non-
n
negative vector v with ||v||1 = 1, which can be chosen in order to make
the search biased towards certain kinds of web pages. Therefore, it is
referred to as a personalization vector.
10.4.3. Solving the Pagerank equation

Now, we should solve the Pagerank equation, an eigenvalue problem
Gr = r, (10.34)
where r ≥ 0 with ||r||1 = 1.
Observation 10.29. The Google matrix G ∈ Rn×n

• G is a full matrix, although it is not necessary to construct it explicitly.
• n represents the number of all web pages, which is order of billions.
• It is impossible to use sparse eigenvalue algorithms that require
the storage of more than very few vectors.
The only viable method so far for Pagerank computations on the whole
web seems to be the power method.
• The rate of convergence of the power method depends on the ratio of
the second largest and the largest eigenvalue in magnitude.
• Here, we have
|1 − λ(k) | = O(αk ), (10.35)
due to Proposition 10.27.
• In view of the huge dimension of the Google matrix, it is non-trivial
to compute the matrix-vector product. We will consider some details.
The power method: matrix-vector product

Recall: It follows from (10.24), (10.27), and (10.31) that the Google
matrix is formulated as
1
G = αP + (1 − α) eeT , (10.36)
n
where
1
P = Q + edT .
n
Here Q is the link matrix and e and d are defined as in (10.26).
Derivation 10.30. Let z = Gy .

• Normalization-free: Since G is column-stochastic (eT G = eT ),
||z||1 = eT z = eT Gy = eT y = ||y||1 . (10.37)
Thus, when the power method begins with y(0) with ||y(0) ||1 = 1, the
normalization step in the power method is unnecessary.
• Let us look at the multiplication in some detail:
h 1 Ti e
z = αP + (1 − α) ee y = αQy + β , (10.38)
n n
where
β = αdT y + (1 − α)eT y. (10.39)
• Apparently we need to know which pages lack outlinks (d), in order to

find β. However, in reality, we do not need to define d. It follows from
(10.37) and (10.38) that
β = 1 − αeT Qy = 1 − ||αQy||1 . (10.40)

Algorithm 10.31. The following Matlab code implements the matrix

vector multiplication: z = Gy.
zhat = alpha*Q*y;
beta = 1-norm(zhat,1);
z = zhat + beta*v;
residual = norm(y-z,1);
Here v is (1/n)e or a personalization vector; see Remark 10.28.
Note:
• From Proposition 10.27, we know that the second eigenvalue of the
Google matrix is αλ2 .
• A typical value of α = 0.85.
• Approximately k = 57 iterations are needed to reach 0.85k < 10−4 .
• This is reported to be close the number of iterations used by Google.
10.1. Consider Example 10.17, p.319. Compute vectors of cosines, for each subspace ap-
proximations, i.e., with Ak where k = 1, 2, · · · , 5.
10.2. Verify equations in Derivation 10.30, p.328, particularly (10.38), (10.39), and (10.40).
10.3. Consider the link matrix Q in (10.4) and its corresponding link graph in Figure 10.2.
Find the pagerank vector r by solving the Google pagerank equation.
• You may initialize the power method with any vector r(0) satisfying ||r(0) ||1 = 1.
• Set α = 0.85.
• Let the iteration stop, when residual < 10−4 .
10.4. Now, consider a modified link matrix Q,e by adding an outlink from page 4 to 5 in
Figure 10.2. Find the pagerank vector e
r, by setting parameters and initialization the
same way as for the previous problem.
• Compare r with e
r.
• Compare the number of iterations for convergence.
11
C HAPTER
Quadratic Programming
Quadratic programming (QP) is the process of solving a constrained
quadratic optimization problem. That is, the objective f is quadratic and
the constraints are linear in several variables x ∈ Rn . Quadratic program-
ming is a particular type of nonlinear programming. Its general form is
1 T
minn f (x) := x Ax − xT b, subj.to
x∈R 2
Cx = c, (11.1)
Dx ≤ d,
where A ∈ Rn×n is symmetric, C ∈ Rm×n , D ∈ Rp×n , b ∈ Rn , c ∈ Rm , and
d ∈ Rp .
In this chapter, we will study various meth-

ods for solving the QP problem (11.1), involv-
ing
• the method of Lagrange multipliers,
• direct solution methods, and
• iterative solution methods.
11.1.Equality Constrained Quadratic Programming . . . . . . . . . . . . . . . . . . . . . . . 332
11.2.Direct Solution for the KKT System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337
11.3.Linear Iterative Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 342
11.4.Iterative Solution of the KKT System . . . . . . . . . . . . . . . . . . . . . . . . . . . . 348
11.5.Active Set Strategies for Convex QP Problems . . . . . . . . . . . . . . . . . . . . . . . 350
11.6.Interior-point Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353
11.7.Logarithmic Barriers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 355
331
332 Chapter 11. Quadratic Programming
11.1. Equality Constrained Quadratic Program-

ming
If only equality constraints are imposed, the QP (11.1) reduces to
1
minn f (x) := xT Ax − xT b, subj.to
x∈R 2 (11.2)
Cx = c,
where A ∈ Rn×n , C ∈ Rm×n , m ≤ n. For the time being we assume that C

has full row rank m and m < n.
To solve the problem, let’s begin with its Lagrangian:

1
L(x, λ) = xT Ax − xT b − λ(c − Cx), (11.3)
2
where λ ∈ Rm is the the associated Lagrange multiplier. Then, we have
∇x L(x, λ) = Ax − b + C T λ. (11.4)
The KKT conditions (first-derivative tests) for the solution x ∈ Rn of

(11.2) give rise to the following linear system
A CT x

b
= , (11.5)
C 0 λ c
| {z }
:=K
where the second row is the primal feasibility.
Let Z ∈ Rn×(n−m) be a matrix whose columns span the null space of C, N (C),
i.e.,
CZ = 0 or Span(Z) = N (C). (11.6)
Definition 11.1. The matrix K in (11.5) is called the KKT matrix and
the matrix Z T AZ is referred to as the reduced Hessian.
Note: Now, a question is: “Is the KKT matrix nonsingular?"

11.1. Equality Constrained Quadratic Programming 333
Definition 11.2. Let A be a symmetric matrix. We say that A is (pos-

itive) semidefinite, and we write A 0, if all eigenvalues of A are
nonnegative. We say that A is (positive) definite, and write A 0, if
all eigenvalues of A are positive.
Lemma 11.3. Assume that A ∈ Rn×n is symmetric and (positive)

semidefinite (A 0) and C ∈ Rm×n has full row rank m ≤ n. Then
the following are equivalent.
(a) N (A) ∩ N (C) = {0}.
(b) Cx = 0, x 6= 0 ⇒ xT Ax > 0.
(c) Z T AZ is positive definite ( 0), where Z ∈ Rn×(n−m) is a matrix for
which Span(Z) = N (C).
(d) A + C T QC 0 for some Q 0.
Proposition 11.4. For a symmetric matrix A, the following are equiv-

alent.
1. A 0.
2. A = U T U for some U .
3. xT Ax ≥ 0 for all x ∈ Rn .
4. All principal minors of A are nonnegative.
5. There exist x1 , x2 , · · · , xk ∈ Rn such that
Xk
A= xi xTi . (11.7)
i=1
Definition 11.5. For A, B ∈ Rn×n , we define the dot product of ma-

trices as Xn Xn
A·B = Aij Bij = tr(AT B). (11.8)
i=1 j=1
Proposition 11.6.
• If A, B 0, then A · B ≥ 0, and A · B = 0 implies AB = 0.
• A symmetric matrix A is semidefinite if A · B ≥ 0 for every B 0.
Theorem 11.7. (Existence and uniqueness). Assume that C ∈ Rm×n

has full row rank m ≤ n and that the reduced Hessian Z T AZ is positive
definite. Then, the KKT matrix K is nonsingular and therefore the
KKT system (11.5) admits a unique solution (x∗ , λ∗ ).
Proof. Suppose that x ∈ N (A) ∩ N (C), x 6= 0. ⇒ Ax = Cx = 0 and therefore
A CT x

x
K = = 0, (11.9)
0 C 0 0
which implies the KKT matrix K is singular.
Now, we assume that the KKT matrix is singular. That is, there are
x ∈ Rn , z ∈ Rm , not both zero, such that
A CT x

x
K = = 0,
z C 0 z
which implies
Ax + C T z = 0 and Cx = 0. (11.10)
It follows from the above equations that
0 = xT Ax + xT C T z = xT Ax,
which contradicts (b) in Lemma 11.3, unless x = 0.

In the case (i.e., x = 0), we must have z 6= 0. But then C T z = 0 contracts the
assumption that C has full row rank.
Note: More generally, the nonsingularity of the KKT matrix is equiva-

lent to each of statements in Lemma 11.3.
11.1. Equality Constrained Quadratic Programming 335
Theorem 11.8. (Global minimizer of (11.2)). Let the assumptions

in Theorem 11.7 be satisfied and (x∗ , λ∗ ) the unique solution of the KKT
system (11.5). Then x∗ is the unique global solution of the QP (11.2).
Proof. When m = n, the theorem is trivial; we may assume m < n.

Let F = {x ∈ Rn | Cx = c}, the feasible set.
Clearly, x∗ is a solution of (11.2), i.e., x∗ ∈ F.
Let x ∈ F be another feasible point and p := x∗ − x 6= 0.
Then x = x∗ − p and
1 ∗
f (x) = (x − p)T A(x∗ − p) − (x∗ − p)T b
2 (11.11)
1 T
= p Ap − pT Ax∗ + pT b + f (x∗ ).
2
Now, (11.5) implies that Ax∗ = b − C T λ∗ and thus
pT Ax∗ = pT (b − C T λ∗ ) = pT b − pT C T λ∗ = pT b,
where we have used Cp = C(x∗ − x) = c − c = 0. Hence, (11.11) reduces to

1 T
f (x) = p Ap + f (x∗ ). (11.12)
2
Since p ∈ N (C), we can write p = Zy, for some nonzero y ∈ Rn−m , and
therefore
1
f (x) = yT Z T AZy + f (x∗ ). (11.13)
2
Since Z T AZ 0, we deduce f (x) > f (x∗ ); consequently, x∗ is the unique
global minimizer of (11.2).
Theorem 11.9. Let the assumptions in Theorem 11.7 be satisfied. Then

the KKT matrix K has exactly n positive and m negative eigenvalues.
Proof. From Lemma 11.3, A + C T C 0; also see (11.89), p. 358. Therefore there exists a
nonsingular matrix R ∈ Rn×n such that
RT (A + C T C)R = I. (11.14)
Let CR = U ΣV1T be the singular value decomposition of CR, where U ∈ Rm×m ,
Σ = diag(σ1 , σ2 , · · · , σm ), and V1 ∈ Rn×m . Let V2 ∈ Rn×(n−m) such that
V = [V1 V2 ]
is orthogonal, and define
S = [Σ 0] ∈ Rm×n .
Then, we have
CR = U SV T (11.15)
and therefore
V T RT (A + C T C)RV = V T RT ARV + (CRV )T CRV = I.
It follows from (11.15) that S = U T CRV and therefore
S T S = (U T CRV )T U T CRV = (CRV )T CRV.
Thus we have Λ := V T RT ARV = I − S T S is diagonal; we can write
Λ = V T RT ARV = diag(1 − σ12 , 1 − σ22 , · · · , 1 − σm
2
, 1, · · · , 1). (11.16)
Now, applying a congruence transformation to the KKT matrix gives
T T
V R 0 A C T RV 0 Λ ST
= (11.17)
0 UT C 0 0 U S 0
and the inertia of the KKT matrix is equal to the inertia of the matrix on the right.1
Applying a permutation to the matrix on the right of (11.17) gives a block diagonal matrix
with n diagonal blocks

λi σi
, i = 1, 2, · · · , m; λi , i = m + 1, , · · · , n, (11.18)
σi 0
where λi are as in (11.16). The eigenvalues of the 2 × 2-blocks are
p
λi ± λ2i + 4σi2
, i = 1, 2, · · · , m,
2
i.e., one eigenvalue is positive and one is negative. So we can conclude that there are
m + (n − m) = n positive eigenvalues and m negative eigenvalues.
1
Sylvester’s law of inertia is a theorem in matrix algebra about certain properties of the coefficient
matrix of a real quadratic form that remain invariant under a change of basis. Namely, if A is the symmetric
matrix that defines the quadratic form, and S is any invertible matrix such that D = SAS T is diagonal, then
the number of negative elements in the diagonal of D is always the same, for all such S; and the same goes
for the number of positive elements.
11.2. Direct Solution for the KKT System 337
11.2. Direct Solution for the KKT System

Recall: In (11.5), the KKT system for the solution x ∈ Rn of (11.2) reads
A CT x

b
= , (11.19)
C 0 λ c
| {z }
:=K
For direct solutions of the KKT system (11.19), this section considers sym-
metric factorization, the range-space approach, and the null-space approach.
11.2.1. Symmetric factorization

A method to solve the KKT system (11.19) is to provide a symmetric fac-
torization of the KKT matrix:
P KP T = LDLT , (11.20)
where P is a permutation matrix (appropriately chosen), L is lower triangu-

lar with diag(L) = I, and D is block diagonal. Based on (11.20), we rewrite
the KKT system (11.19) as
x x
b x
P = PK = P KP T P = LDLT P .
c λ λ λ
Thus it can be solved as follows.

b
solve Ly1 = P
c
solve Dy2 = y1
(11.21)
solve LT y3 = y2
∗
x T
set ∗ = P y3
λ
In python, scipy.linalg.ldl is available.

ldl_test.py
2 from scipy.linalg import ldl
3
4 A = np.array([[ 2, -1, 0],

5 [-1, 0, -1],
6 [ 0, -1, 4]])
7 L0,D,P = ldl(A, lower=1) # Use the default: the lower matrix
8
9 print('L0=\n',L0)
10 print('D=\n',D)
11 print('P=\n',P)
12 print('L0*D*L0^T=\n',L0.dot(D).dot(L0.T),'\n#-----------')
13
14 P_L0 = L0[P,:] # the row permutation of 'L0'

15 print('P*L0=\n',P_L0)
Result • As one can see from the result,

1 L0= P*L0 is a lower triangular matrix.
2 [[ 1. 0. 0. ]
3 [-0.5 -0.25 1. ] The output L0 of Python function
4 [ 0. 1. 0. ]] ldl is permuted as
5 D=
6 [[ 2. 0. 0. ] L0 = P T L. (11.22)
7 [ 0. 4. 0. ]
8 [ 0. 0. -0.75]]
9 P= • Reference: [10] J.R. Bunch and
10 [0 2 1]
L. Kaufman, Some stable methods
11 L0*D*L0^T=
12 [[ 2. -1. 0.] for calculating inertia and solving
13 [-1. 0. -1.] symmetric linear systems, Math.
14 [ 0. -1. 4.]]
15 #-----------
Comput. 31, 1977, pp.163-179.
16 P*L0=
17 [[ 1. 0. 0. ]
18 [ 0. 1. 0. ]
19 [-0.5 -0.25 1. ]]
11.2.2. Range-space approach
Recall: The KKT system for the solution x ∈ Rn of (11.2) is given by

A CT x

b
= , (11.23)
C 0 λ c
| {z }
:=K
The range-space approach applies, when A ∈ Rn×n is symmetric posi-

tive definite. Block Gauss elimination of the primal variable x leads to the
Schur complement system
CA−1 C T λ = CA−1 b − c, (11.24)
where S := CA−1 C T ∈ Rm×m is the Schur complement. See Exercise 2.
Note: Once the optimal Lagrange multipliers λ∗ is determined from

(11.24), the minimizer x∗ can be obtained by solving the first equation
of the KKT system
Ax = b − C T λ∗ . (11.25)
Theorem 11.10. Suppose that A ∈ Rn×n is symmetric positive definite,

C ∈ Rm×n , m ≤ n, such that rank(C) = m. Let S = CA−1 C T be the
Schur complement associated with the KKT-matrix. Then S is sym-
metric positive definite on Rm .
Remark 11.11. The range-space approach is particularly effective,

when
• The matrix A is well conditioned and efficiently invertible.
(e.g., diagonal or block-diagonal)
• Its inverse A−1 is known explicitly.
(e.g., by means of a quasi-Newton updating formula)
• The number of equality constraints (m) is small.
Note that C ∈ Rm×n and it can be considered as a map C : Rn → Rm .

11.2.3. Null-space approach

The null-space approach does not require regularity of A and thus
has a wider range of applicability than the range-space approach. The
method begins with some assumptions.
1. Assume that C ∈ Rm×n has full row rank m.

2. Assume that Z T AZ 0, where Z ∈ Rn×(n−m) is a matrix for which Span(Z) = N (C)
and CZ = 0; see (11.6).
3. Let Y ∈ Rn×m be a matrix such that [Y Z] ∈ Rn×n is nonsingular.

4. Partition the vector x ∈ Rn according to
x = Y wY + ZwZ , (11.26)
where wY ∈ Rm , wZ ∈ Rn−m .
5. Substitute (11.26) into the second equation of (11.19) to have

CZ wZ = c ⇒ CY wY = c,
Cx = CY wY + |{z} (11.27)
=0
i.e. Y wY is a particular solution of Cx = c.

6. Furthermore, wY is well determined by (11.27). ∵ Since C ∈ Rm×n has full row rank
m and [Y Z] ∈ Rn×n is nonsingular, the product C[Y Z] = [CY 0] ∈ Rm×n has full
row rank m and therefore CY ∈ Rm×m is nonsingular.
7. On the other hand, substituting (11.26) into the first equation of (11.19), we get
Ax + C T λ = AY wY + AZwZ + C T λ = b. (11.28)
Multiplying Z T and observing Z T C T = (CZ)T = 0 yield

Z T AZwZ = Z T b − Z T AY wY . (11.29)
The reduced KKT system (11.29) can be solved easily e.g. by a Cholesky factoriza-
tion of the reduced Hessian Z T AZ ∈ R(n−m)×(n−m) .
8. Once wY and wZ have been computed as solutions of (11.27) and (11.29), respectively,
x∗ is obtained from (11.26).
9. When Lagrange multipliers λ∗ is to be computed, we multiply (11.28) by Y T and
solve the resulting equation:
(CY )T λ∗ = Y T b − Y T Ax∗ . (11.30)
11.3. Linear Iterative Methods

Consider a linear algebraic system
Ax = b, (11.31)
for which we assume that A ∈ Rn×n is invertible.
Key Idea 11.12. Iterative methods for solving (11.31):

• Linear iterative methods begin with splitting the matrix A by
A = M − N, (11.32)
for some invertible matrix M .

• Then, the linear system equivalently reads
M x = N x + b. (11.33)
• Associated with the splitting is an iterative method
M xk = N xk−1 + b, (11.34)
or, equivalently,
xk = M −1 (N xk−1 + b) = xk−1 + M −1 (b − Axk−1 ), (11.35)
for an initial value x0 .
Note: Methods differ for different choices of M .

• M must be easy to invert (efficiency), and
• M −1 ≈ A−1 (convergence).
11.3. Linear Iterative Methods 343
11.3.1. Convergence theory

• Let
ek = x − x k .
• It follows from (11.33) and (11.34) that the error equation reads
M ek = N ek−1 (11.36)
or, equivalently,
ek = M −1 N ek−1 . (11.37)
• Since
kek k ≤ kM −1 N k · kek−1 k ≤ kM −1 N k2 · kek−2 k
(11.38)
≤ · · · ≤ kM −1 N kk · ke0 k,
a sufficient condition for the convergence is
kM −1 N k < 1. (11.39)
Definition 11.13. Let σ(B) be the spectrum, the set of eigenvalues of

the matrix B, and ρ(B) denote the spectral radius defined by
ρ(B) = max |λi |.

λi ∈σ(B)
Theorem 11.14. The iteration converges if and only if
ρ(M −1 N ) < 1. (11.40)
11.3.2. Graph theory: Estimation of the spectral radius

Definition 11.15. A permutation matrix is a square matrix in which
each row and each column has one entry of unity, all others zero.
Definition 11.16. For n ≥ 2, an n × n complex-valued matrix A is

reducible if there is a permutation matrix P such that

T A11 A12
P AP = ,
0 A22
where A11 and A22 are respectively r × r and (n − r) × (n − r) submatrices,

0 < r < n. If no such permutation matrix exists, then A is irreducible.
The geometrical interpretation of the concept of the irreducibility by means

of graph theory is useful.
Geometrical interpretation of irreducibility
in the plane, which we will call

nodes or nodal points.
• For any nonzero entry aij of A, we
−→
connect Pi to Pj by a path Pi Pj , di-
Figure 11.1: The directed paths for nonzero
aii and aij .
rected from the node Pi to the node
Pj ; a nonzero aii is joined to itself
by a directed loop, as shown in Fig-
ure 11.1.
• In this way, every n × n matrix
A can be associated a directed
graph G(A). For example, the ma-
trix
Figure 11.2: The directed graph G(A) for A  
in (11.41). 2 −1 0
A =  −1 2 −1  (11.41)
• Given A = [aij ] ∈ Cn×n , consider n 0 −1 2
distinct points
has a directed graph shown in Fig-
P1 , P2 , · · · , Pn ure 11.2.
Definition 11.17. A directed graph is strongly connected if, for any

ordered pair of nodes (Pi , Pj ), there is a directed path of a finite length
−→ −→ −→
Pi Pk1 , Pk1 Pk2 , · · · , Pkr−1 Pkr =j ,
connecting from Pi to Pj .
The theorems to be presented in this subsection can be found in [77] along

with their proofs.
Theorem 11.18. An n × n complex-valued matrix A is irreducible if and
only if its directed graph G(A) is strongly connected.
11.3.3. Eigenvalue locus theorem

For A = [aij ] ∈ Cn×n , let n
X
Λi := |aij | (11.42)
j=1
j 6= i
Theorem 11.19. (Eigenvalue locus theorem) Let A = [aij ] be an

irreducible n × n complex matrix. Then,
1. (Gerschgorin [22]) All eigenvalues of A lie in the union of the disks
in the complex plane
|z − aii | ≤ Λi , 1 ≤ i ≤ n. (11.43)
2. (Taussky [74]) In addition, assume that λ, an eigenvalue of A, is a

boundary point of the union of the disks |z − aii | ≤ Λi . Then, all the n
circles |z − aii | = Λi must pass through the point λ, i.e., |λ − aii | = Λi
for all 1 ≤ i ≤ n.
Example 11.20. For example, for

 
2 −1 0
A =  −1 2 −1 
0 −1 2
Λ1 = 1, Λ2 = 2, and Λ3 = 1. Since aii = 2, for i = 1, 2, 3,
|λ − 2| < 2
for all eigenvalues λ of A.
Positiveness
Definition 11.21. An n × n complex-valued matrix A = [aij ] is diago-
nally dominant if
n
X
|aii | ≥ Λi := |aij |, (11.44)
j=1
j 6= i
for all 1 ≤ i ≤ n. An n × n matrix A is irreducibly diagonally dominant if

A is irreducible and diagonally dominant, with strict inequality holding
in (11.44) for at least one i.
Theorem 11.22. Let A be an n × n strictly or irreducibly diagonally

dominant complex-valued matrix. Then, A is nonsingular. If all the di-
agonal entries of A are in addition positive real, then the real parts of all
eigenvalues of A are positive.
Corollary 11.23. A Hermitian matrix satisfying the conditions in The-

orem 11.22 is positive definite.
11.3.4. Regular splitting and M-matrices

Definition 11.24. For n × n real matrices, A, M , and N , A = M − N is
a regular splitting of A if M is nonsingular with M −1 ≥ 0, and N ≥ 0.
Theorem 11.25. If A = M − N is a regular splitting of A and A−1 ≥ 0,

then
−1 ρ(A−1 N )
ρ(M N ) = < 1. (11.45)
1 + ρ(A−1 N )
Thus, the matrix M −1 N is convergent and the iterative method of (11.34)
converges for any initial value x0 .
Definition 11.26. An n × n real matrix A = [aij ] with aij ≤ 0 for all i 6= j

is an M-matrix if A is nonsingular and A−1 ≥ 0.
Theorem 11.27. Let A = (aij ) be an n × n M -matrix. If M is any n × n

matrix obtained by setting certain off-diagonal entries of A to zero, then
A = M − N is a regular splitting of A and ρ(M −1 N ) < 1.
Theorem 11.28. Let A be an n × n real matrix with A−1 > 0, and

A = M1 − N1 = M2 − N2 be two regular splittings of A. If N2 ≥ N1 ≥ 0,
where neither N2 − N1 nor N1 is null, then
1 > ρ(M2−1 N2 ) > ρ(M1−1 N1 ) > 0. (11.46)

11.4. Iterative Solution of the KKT System

Recall: The KKT system for the solution x ∈ Rn of (11.2) is given by
A CT x

b
= , (11.47)
C 0 λ c
| {z }
:=K
When the direct solution of the KKT system (11.47) is computationally

too costly, the alternative is to use an iterative method. An iterative
solver can be applied
• either to the entire KKT system
• or to the special structure of the KKT matrix, as in the range-
space and null-space approach, and based on regular splittinga of
specifically transformed matrices of K.
a
For n × n real matrices, A, M , and N , A = M − N is a regular splitting of A if M is nonsingular
with M −1 ≥ 0, and N ≥ 0.
The transforming null-space iteration does not require regularity of A

and therefore has a wider range of applicability than the transforming
range-space iteration. Here we will deal with the transforming range-
space iteration only for simplicity.
11.4.1. Krylov subspace methods

The KKT matrix K ∈ R(n+m)×(n+m) is indefinite; if C has full row rank
m, then K has exactly n positive and m negative eigenvalues, as shown in
Theorem 11.9. Therefore, for iterative methods for the solution of entire
KKT system, appropriate candidates are Krylov subspace methods like
• GMRES (Generalized Minimum RESidual) and

• QMR (Quasi Minimum Residual).
11.4. Iterative Solution of the KKT System 349
11.4.2. The transforming range-space iteration
Assumption. The matrix A ∈ Rn×n is symmetric positive definite (SPD)

and A has an easily invertible SPD approximation A b−1 A ∼ I.
b such that A
1. We choose L ∈ R(n+m)×(n+m) as a lower triangular block matrix

I 0
L= b−1 I , (11.48)
−C A
which gives rise to the regular splitting of LK:
" # " #
T −1
A C
b A(I − A A) 0
b b
LK = − b−1 A) 0 =: M1 − M2 , (11.49)
0 Sb C(I − A
b−1 C T ∈ Rm×m . (Note M2 = M1 − LK ∼ 0.)

where Sb = −C A
2. Let
ψ := (x, λ)T , β := (b, c)T .
Then the KKT system (11.47) gives LKψ = (M1 − M2 ) ψ = Lβ so that
M1 ψ = M2 ψ + Lβ
(11.50)
= (M1 − LK) ψ + Lβ = M1 ψ + L(β − Kψ).
3. Given an initialization ψ 0 ∈ R(n+m)×(n+m) , we compute ψ k+1 by means of

the transforming range-space iteration
ψ k+1 = (I − M1−1 LK) ψ k + M1−1 Lβ
(11.51)
= ψ k + M1−1 L(β − Kψ k ), k ≥ 0.
Implementation of (11.51):
(1) (2)
T
compute rk " k , rk ) := β −#Kψ k ;
= (r
(1)
rk
compute Lrk = ;
−C A b−1 r(1) + r(2) (11.52)
k k
solve M1 ∆ψ k = Lrk ;
set ψ k+1 = ψ k + ∆ψ k ;
11.5. Active Set Strategies for Convex QP Prob-

lems
Recall: Quadratic programming formulated in a general form (11.1):
1
minn f (x) := xT Ax − xT b, subj.to
x∈R 2
Cx = c, (11.53)
Dx ≤ d,
where A ∈ Rn×n , C ∈ Rm×n , D ∈ Rp×n , and b ∈ Rn , c ∈ Rm , d ∈ Rp .

Here, we assume A is SPD.
Definition 11.29. The inequality constraint in (11.53) can be written

as
gi (x) ≤ 0, i = 1, 2, · · · , p.
Given a point x in the feasible region, a constraint gi (x) ≤ 0 is called
active at x if gi (x) = 0 and inactive if gi (x) 6= 0. The active set at
x is made up of those constraints that are active at the current point.
(Equality constraints are always active.)
Note: The active set is particularly important in optimization theory,

because it determines which constraints will influence the final result of
optimization. For example,
• For linear programming problem, the active set gives the hyper-
planes that intersect at the solution point.
• In quadratic programming, the solution is not necessarily on one
of the edges of the bounding polygon; an estimation of the active
set gives us a subset of inequalities to watch while searching the
solution, which reduces the complexity of the search [57].
11.5. Active Set Strategies for Convex QP Problems 351
11.5.1. Primal active set strategy

We rewrite the matrices
 C and D in the form 
C1 D1
 .. 
C =  .  , C i ∈ Rn ; D =  ...  , D i ∈ Rn . (11.54)
 
Cm Dp
Then the inequality constraints in (11.53) can be equivalently stated as

D Ti x ≤ di , i = 1, 2, · · · , p. (11.55)
(C i and D i are row vectors; we will deal with them like column vectors.)
The primal active set strategy is an iterative procedure:
• Given a feasible iterate xk , k ≥ 0, we determine its active set
Iac (xk ) ⊂ {1, 2, · · · , p} (11.56)
and consider the corresponding constraints as equality constraints,

whereas the remaining inequality constraints are disregarded.
• Setting
p = xk − x, rk = Axk − b, (11.57)
we find
1 T
f (x) = f (xk − p) = p Ap − rTk p + g, (11.58)
2
where g = 12 xTk Axk − bT xk .
• Then the equality constrained QP problem to be solved for the (k+1)-st
iteration step is:
1
T T
pk = arg minn p Ap − rk p , subj.to
p∈R 2
Cp = 0 (11.59)
D Ti p = 0, i ∈ Iac (xk ).
• The new iterate is then obtained according to
xk+1 = xk − αk pk , (11.60)
where αk is chosen such that xk+1 stays feasible.

Remark 11.30. (Determination of αk ). The parameter can be deter-

mined as follows.
• For each i ∈ Iac (xk ), we have
D Ti xk+1 = D Ti xk − αk D Ti pk = D Ti xk ≤ di . (11.61)
• If D Ti pk ≥ 0 for some i 6∈ Iac (xk ), it follows that

D Ti xk+1 = D Ti xk − αk D Ti pk ≤ D Ti xk ≤ di . (11.62)
• On the other hand, if D Ti pk < 0 for some i 6∈ Iac (xk ), we have

di − D Ti xk
D Ti xk+1 = D Ti xk − αk D Ti pk ≤ di ⇐⇒ αk ≤ . (11.63)
−D Ti pk
• Consequently, in order to guarantee feasibility, we choose

D Ti xk − di
bk ), where α
αk := min(1, α bk := min . (11.64)
i6∈Iac (xk ); D Ti pk <0 D Ti pk
Remark 11.31. (Update for Iac (xk+1 )). Let’s begin with defining the
set of blocking constraints:
def
n
T D Ti xk − di o
Ibl (pk ) == i 6∈ Iac (xk ) | D i pk < 0, ≤1 . (11.65)
D Ti pk
Then we specify Iac (xk+1 ) by adding the most restrictive blocking con-
straint to Iac (xk ):
def
n D Tj xk − dj o
Iac (xk+1 ) == Iac (xk ) ∪ j ∈ Ibl (pk ) | =α
bk . (11.66)
D Tj pk
For such a j (the index of the newly added constraint), we clearly have
D Tj xk+1 = D Tj xk − αk D Tj pk = D Tj xk − α
bk D Tj pk = dj , (11.67)
and therefore Iac (xk+1 ) contains active constraints only.

11.6. Interior-point Methods 353
11.6. Interior-point Methods

Interior-point methods are iterative schemes where the iterates approach
the optimal solution from the interior of the feasible set. For simplicity, we
consider inequality-constrained quadratic programming problems of
the form
1
minn Q(x) := xT Ax − xT b, subj.to
x∈R 2 (11.68)
Dx ≤ d,
where A ∈ Rn×n is SPD, D ∈ Rp×n , and b ∈ Rn and d ∈ Rp .
Note: Its Lagrangian reads

p
def
X
L(x, µ) == Q(x) + µi (D Ti x − di ), (11.69)
i=1
where µ = (µ1 , µ2 , · · · , µp )T and D = [D 1 , · · · , D p ]T , and therefore
∇x L(x, µ) = ∇x Q(x) + DT µ = Ax − b + DT µ. (11.70)
Thus the KKT conditions for (11.68) are stated as

Ax + DT µ − b = 0,
Dx − d ≤ 0,
(11.71)
µi (Dx − d)i = 0, i = 1, 2, · · · , p,
µi ≥ 0, i = 1, 2, · · · , p.
{µi (Dx − d)i = 0} is the complementary slackness.
By introducing a slack variable z = d − Dx, the above conditions can
be equivalently formulated as follows:
Ax + DT µ − b = 0,
Dx + z − d = 0,
(11.72)
µi zi = 0, i = 1, 2, · · · , p,
µi , zi ≥ 0, i = 1, 2, · · · , p.
The interior-point method begins with replacing µi zi = 0 by µi zi = θ >

0 for all i = 1, 2, · · · , p, and enforces θ & 0.
Equation (11.72) can be rewritten as a constrained system of nonlinear

equations. We define the nonlinear map
Ax + DT µ − b
 
def
F (x, µ, z) ==  Dx + z − d  , (11.73)
ZMe
where
Z = diag(z1 , · · · , zp ), M = diag(µ1 , · · · , µp ), e = (1, · · · , 1)T .
Definition 11.32. (Central path). The set of points (xτ , µτ , zτ ), τ > 0,

satisfying  
0
F (xτ , µτ , zτ ) =  0  , z, µ ≥ 0, (11.74)
τe
is called the central path.
Newton’s method:
• Given a feasible iterate (x, µ, z) = (xk , µk , zk ), we introduce a duality
measure θ: p
1X zT µ
θ := zi µi = . (11.75)
p i=1 p
• The idea is to apply Newton’s method to (11.73) to compute

(xσθ , µσθ , zσθ ) on the central path, where σ ∈ [0, 1] is an algorithm pa-
rameter.
• The Newton increments (∆x, ∆µ, ∆z) solve the linear system
   
∆x 0
∇F (x, µ, z) ∆µ = −F (x, µ, z) +  0  , (11.76)
∆z σθe
where A DT 0
 
def
∇F (x, µ, z) == D 0 I  .
0 Z M
• The new iterate (xk+1 , µk+1 , zk+1 ) is then determined by means of
(xk+1 , µk+1 , zk+1 ) = (xk , µk , zk ) + α(∆x, ∆µ, ∆z), (11.77)
with α chosen such that the new iterate stays feasible.

11.7. Logarithmic Barriers 355
11.7. Logarithmic Barriers

Recall: The inequality-constrained quadratic programming prob-
lem in (11.68):
1 T
minn Q(x) := x Ax − xT b, subj.to
x∈R 2 (11.78)
Dx ≤ d,
where A ∈ Rn×n is SPD, D ∈ Rp×n , and b ∈ Rn and d ∈ Rp .
Algorithms based on barrier functions are iterative methods where the it-
erates are forced to stay within the interior of the feasible set:
def
F int == {x ∈ Rn | D Ti x − di < 0, 1 ≤ i ≤ p}. (11.79)
Barrier functions commonly have the following properties:

• They are smooth within F int .
• They approach ∞ as x approaches the boundary of F int .
• They are infinite outside F int .
Definition 11.33. (Logarithmic barrier function). For the QP prob-

lem (11.78), the objective functional
p
def
X
Bβ (x) == Q(x) − β log(di − D Ti x), β > 0, (11.80)
i=1
is called the logarithmic barrier function.a The parameter β is re-

ferred to as the barrier parameter.
a
The function log stands for the natural logarithm (ln), as commonly accepted in the literature of
computational algorithms.
Theorem 11.34. (Properties of the logarithmic barrier function)

[80, (Wright,1992)]. Assume that the set S of solutions of (11.78) is
nonempty and bounded and that the interior F int of the feasible set is
nonempty. Let {βk } be a decreasing sequence of barrier parameters with
βk → 0, as k → ∞. (11.81)
Then there holds:
1. For any β > 0, the logarithmic barrier function Bβ (x) is convex in
F int and attains a minimizer xβ ∈ F int .
2. There is a unique minimizer; any local minimizer xβ is also a
global minimizer of Bβ (x).
3. If {xβk | k ∈ N} is a sequence of minimizers, then there exists N0 ⊂ N

such that
xβk → x∗ ∈ S, k ∈ N0 .
4. If Q∗ is the optimal value of the objective functional Q in (11.78),

then for any sequence {xβk } of minimizers,
Q(xβk ) → Q∗ , Bβk (xβk ) → Q∗ , as k → ∞. (11.82)
Objective: In the following:

We will have a closer look at the relation between a minimizer of Bβ (x)
and the solution of the KKT system, a point (x, µ) satisfying the KKT
conditions for (11.78).
Recall: The KKT conditions for (11.78) are given in (11.71), p. 353:
∇x Q(x) + DT µ = Ax − b + DT µ = 0,
Dx − d ≤ 0,
(11.83)
µi (Dx − d)i = 0, i = 1, 2, · · · , p,
µi ≥ 0, i = 1, 2, · · · , p.
If xβ is a minimizer of Bβ (x), we obviously have

p
X β
∇x Bβ (xβ ) = ∇x Q(xβ ) + T
D i = 0. (11.84)
i=1
di − D i x β
Definition 11.35. Perturbed (or, approximate) complementarity

is the vector zβ ∈ Rp having its components
β
(zβ )i = zβ,i := , 1 ≤ i ≤ p. (11.85)
di − D Ti xβ
In terms of the perturbed complementarity, (11.84) can be stated as

p
X
∇x Q(xβ ) + zβ,i D i = 0. (11.86)
i=1
Rewrite the first of the KKT conditions (11.83) as

p
X
∇x Q(x) + µi D i = 0. (11.87)
i=1
• Obviously, (11.87) looks the same as (11.86).

• Apparently, the 2nd and the 4th KKT conditions in (11.83) are sat-
isfied by (x, µ) = (xβ , zβ ).
• However, the 3rd KKT condition does not hold true, because it fol-
lows readily from (11.85) that
zβ,i (di − D Ti xβ ) = β > 0, 1 ≤ i ≤ p. (11.88)
As β → 0, the minimizer xβ and the associated zβ come closer and

closer to satisfying the 3rd KKT condition. This is why zβ is called
perturbed (approximate) complementarity.
11.1. Recall the KKT matrix

A CT
K= .
C 0
Here, we assume that A ∈ Rn×n is symmetric and (positive) semidefinite (A 0) and
C ∈ Rm×n has full row rank m ≤ n. Show the following are equivalent.
(a) N (A) ∩ N (C) = {0}.
(b) Cx = 0, x 6= 0 ⇒ xT Ax > 0.
(c) Z T AZ is positive definite ( 0), where Z ∈ Rn×(n−m) is a matrix for which Span(Z) =
N (C).
(d) A + C T QC 0 for some Q 0.
When (c) is considered, you may assume m < n.
Hint : (a)⇔(b): Use contradictions. For example, N (A) ∩ N (C) 6= {0}. ⇒ ∃x ∈
N (A) ∩ N (C), x 6= 0. ⇒ Ax = 0 and Cx = 0 and therefore xT Ax = 0, which is a
contradiction; this implies (a)⇐(b).
(b)⇐(c): Let Cx = 0, x 6= 0. ⇒ x must have the form x = Zy for some y 6= 0. (why?)
⇒ xT Ax = yT Z T P Zy > 0.
(b)⇔(d): If (b) holds, then
xT (A + C T C)x = xT Ax + kCxk2 > 0. (11.89)
⇒ (d) holds with Q = I. On the other hand, if (d) holds for some Q 0, then
xT (A + C T QC)x = xT Ax + xT C T QCx > 0, x 6= 0. (11.90)
When Cx = 0, xT Ax must be positive.
Now, you should fill out missing gaps: (a)⇒(b) and (b)⇒(c).
11.2. Derive the Schur complement system (11.24) from the KKT system (11.19).
11.3. Suppose that A ∈ Rn×n is symmetric positive definite, C ∈ Rm×n , m ≤ n, such that
rank(C) = m. Let S = CA−1 C T be the Schur complement associated with the KKT-
matrix, as in (11.24). Show that S is symmetric positive definite on Rm .
Hint : You may begin with claiming that C T x 6= 0 for every nonzero x ∈ Rm . (Figure
out why)
11.4. Verify the splitting (11.49).
11.5. Recall the dual problem of linear SVM in (5.52):
1
max α · 1 − αT Gα subj.to α · y = 0,

(11.91)
0≤α≤C 2
where G = ZZ T and Gij = y (i) y (j) x(i) · x(j) .
(a) Ignoring momentarily the inequality constraints on the dual problem, 0 ≤ α ≤ C,
prove that the problem has a unique solution. Hint : Formulate the KKT system
for the problem and check if it satisfies a statement in Lemma 11.3. You may use
the fact that G = ZZ T 0.
(b) Now, considering the inequality constraints, discuss why the problem is yet ad-
mitting a unique solution α∗ .
11.6. Consider the following equality-constrained QP problem
min 3x2 + 2y 2 + 2z 2 − 2yz − 8x − 3y − 3z, subj.to

(x,y,z)∈R3
x + z = 1, (11.92)
y + z = 3.
Begin with pencil-and-paper:

(a) Formulate the KKT system for (11.92) of the form (11.5) by identifying A ∈ R3×3 ,
C ∈ R2×3 , and vectors b ∈ R3 and c ∈ R2 .
(b) Solve the QP problem (11.92) by the null-space approach. In particular, spec-
ify the matrices Y and Z and compute wY and wZ .
Now, use your computer:
(c) Implement the null-space algorithm presented in Section 11.2.3 to find the min-
imizer of (11.92) numerically.
(d) Implement the range-space (Schur complement) method presented in Section 11.2.2
to find the minimizer of (11.92) numerically.
11.7. Now, consider the following inequality-constrained QP problem
min Q(x) := (x1 − 1)2 + 2(x2 − 2)2 + 2(x3 − 2)2 − 2x1 x2 , subj.to
x∈R3
x1 + x2 − 3x3 ≤ 0, (11.93)
4x1 − x2 + x3 ≤ 1.
Implement the interior-point method with the Newton’s iterative update, pre-
sented in Section 11.6, to solve the QP problem (11.93).
(a) As in Exercise 6, you should first formulate the KKT system for (11.93) by iden-
tifying A ∈ R3×3 , D ∈ R2×3 , and vectors b ∈ R3 and d ∈ R2 .
(b) Choose a feasible initial value (x0 , µ0 , z0 ).
(c) Select the algorithm parameter σ ∈ [0, 1] appropriately for Newton’s method.
(d) Discuss how to determine α in (11.77) in order for the new iterate (xk+1 , µk+1 , zk+1 )
to stay feasible.
P
A PPENDIX
Projects
Finally we add projects.
Contents of Projects
P.1. mCLESS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 362
P.2. Noise-Removal and Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 373
P.3. Gaussian Sailing to Overcome Local Minima Problems . . . . . . . . . . . . . . . . . . 379
P.4. Quasi-Newton Methods Using Partial Information of the Hessian . . . . . . . . . . . . 381
P.5. Effective Preprocessing Technique for Filling Missing Data . . . . . . . . . . . . . . . . 383
361
362 Appendix P. Projects
P.1. mCLESS
Note: Some machine learning algorithms are considered as black
boxes, because
• the models are sufficiently complex and
• they are not straightforwardly interpretable to humans.
Lack of interpretability in predictive models can undermine trust
in those models, especially in health care, in which so many decisions
are – literally – life and death issues [59].
Remark P.1. Why is Interpretability Important?

1. Understand the strengths and weaknesses of the model
2. Better feedback
3. Enhanced probability of adoption and success
4. Discover insights
Project Objectives
• Develop an interpretable machine learning algorithm.

– Formulated with the least-squares error.
– We call it the Multi-Class Least-Error-Square Sum (mCLESS).
• Compare it with traditional methods, for various datasets.
P.1. mCLESS 363
P.1.1. Review: Simple classifiers

The Perceptron [64] (or Adaline) is the simplest artificial neuron that
makes decisions for datasets of two classes by weighting up evidence.
• Inputs: feature values x = [x1 , x2 , · · · , xd ]
• Weight vector and bias: w = [w1 , w2 , · · · , wd ]T , w0
• Net input:
z = w0 + w1 x1 + w2 x2 + · · · + wd xd (P.1.1)
• Activation:
1, if z ≥ θ
φ(z) = (P.1.2)
0, otherwise,
where θ is a threshold. When the logistic sigmoid function is chosen
for the activation function, i.e., φ(z) = 1/(1 + e−z ), the resulting
classifier is called the Logistic Regression.
Remark P.2. Note that the net input in (P.1.1) represents a hyper-
plane in Rd .
• More complex neural networks can be built, stacking the simple ar-
tificial neurons as building blocks.
• Machine learning (ML) is to train weights from datasets of an arbi-
trary number of classes.
– The weights must be trained in such a way that data points in a
class are heavily weighted by the corresponding part of weights.
• The activation function is incorporated in order
(a) to keep the net input restricted to a certain limit as per
our requirement and, more importantly,
(b) to add nonlinearity to the network.
P.1.2. The mCLESS classifier

Here we present a new classifier which is based on a least-squares formu-
lation and able to classify datasets having arbitrary numbers of classes. Its
nonlinear expansion will also be suggested.
Two-layer Neural Networks
Figure P.1: A synthetic data of three classes.
• In order to describe the proposed algorithm effectively, we exemplify

a synthetic data of three classes, as shown in Figure P.1, in which
each class has 100 points.
• A point in the c-th class is expressed as
(c) (c)
x(c) = [x1 , x2 ] = [x1 , x2 , c] c = 0, 1, 2,
where the number in () in the superscript denotes the class that the
point belongs to.
• Let’s consider an artificial neural network of the identity activation
and no hidden layer, for simplicity.
P.1. mCLESS 365
A set of weights can be trained in a way that points in a class are

heavily weighted by the corresponding part of weights, i.e.,

(j) (j) (i) (j) (i) 1 if i = j
w0 + w1 x1 + w2 x2 = δij = (P.1.3)
0 if i 6= j
(j)
where δij is called the Kronecker delta and w0 is a bias for the class j.
• The weights can be determined by the least-squares method.
• We will call the algorithm the Multi-Class Least-Error-Square
Sum (mCLESS).
mCLESS: Algebraic Formulation

Training
• Dataset: We express the dataset {X, y} used for Figure P.1 by

   
x11 x12 c1
x
 21 x22   c2 
  
N ×2
X =  .. ..  ∈ R , y =  .. , (P.1.4)
 . .   . 
xN 1 xN 2 cN
where ci ∈ {0, 1, 2}, the class number.
• The algebraic system: It can be formulated using (P.1.3).
– Define the information matrix:
 
1 x11 x12
1 x
21 x22 

A=  ∈ RN ×3 . (P.1.5)

..
 . 
1 xN 1 xN 2
Note. The information matrix can be made using
A = np.column_stack((np.ones([N,]),X))
– The weight matrix to be learned is:
 (0) (1) (2) 
w 0 w 0 w0
W = [w(0) , w(1) , w(2) ] = w1(0) w1(1) w1(2) , (P.1.6)
 
(0) (1) (2)
w 2 w 2 w2
where the j-th column weights heavily points in the j-th class.
– Define the source matrix:
B = [δci ,j ] ∈ RN ×3 . (P.1.7)
For example, if the i-th point is in Class 0, then the i-th row of B is
[1, 0, 0].
• Then the multi-column least-squares (MC-LS) problem reads
c = arg min ||AW − B||2 ,

W (P.1.8)
W
which can be solved by the method of normal equations:
(AT A) W
c = AT B, AT A ∈ R3×3 . (P.1.9)
• The output of training: The weight matrix W

c.
Note: The normal matrix AT A is occasionally singular, particularly for

small datasets. In the case, the MC-LS problem can be solved using the
singular value decomposition (SVD).
Prediction
The prediction step in the mCLESS is quite simple:
(a) Let [x1 , x2 ] be a new point.
(b) Compute
[1, x1 , x2 ] W
c = [p0 , p1 , p2 ], c ∈ R3×3 .
W (P.1.10)
Note. Ideally, if the point [x1 , x2 ] is in class j, then pj is near 1, while
others would be near 0. Thus pj is the largest.
(c) Decide the class c:
c = np.argmax([p0 , p1 , p2 ], axis = 1). (P.1.11)

P.1. mCLESS 367
Experiment P.3. mCLESS, with a Synthetic Dataset

• As a data preprocessing, the dataset X is scaled column-wisely so that
the maximum value in each column is 1 in modulus.
• The training is carried out with randomly selected 70% the dataset.
• The output of training, W
c , represents three sets of parallel lines.
(j) (j) (j)
– Let [w0 , w1 , w2 ]T be the j-th column of W
c . Define Lj (x1 , x2 ) as
(j) (j) (j)
Lj (x1 , x2 ) = w0 + w1 x1 + w2 x2 , j = 0, 1, 2. (P.1.12)
– Figure P.2 depicts Lj (x1 , x2 ) = 0 and Lj (x1 , x2 ) = 1 superposed on

the training set.
• It follows from (P.1.11) that the mCLESS can be viewed as an one-
versus-rest (OVR) classifier; see Section 3.2.3.
Figure P.2: Lines represented by the weight vectors. mCLESS is interpretable!
The whole algorithm (training-prediction) is run 100 times, with ran-

domly splitting the dataset into 70:30 parts respectively for training and
prediction; which results in 97.87% and 0.00171 sec for the average ac-
curacy and e-time. The used is a laptop of an Intel Core i7-10750H CPU
at 2.60GHz.
P.1.3. Feature expansion
Remark P.4. Nonlinear mCLESS

• The mCLESS so far is a linear classifier.
• As for other classifiers, its nonlinear expansion begins with a data
transformation, more precisely, feature expansion.
• For example, the Support Vector Machine (SVM) replaces the dot
product of feature vectors (point) with the result of a kernel function
applied to the feature vectors, in the construction of the Gram matrix:
K(xi , xj ) ≈ σ(xi ) · σ(xj ),
where σ is a function for feature expansion.

– Thus, without an explicit expansion of feature vectors, the SVM
can incorporate the effect of data transformation effectively. Such
a technique is called the kernel trick. See Section 5.3.5.
• However, the mCLESS does not incorporate dot products between
points.
– As a result, we must perform feature expansion without a
kernel trick, which results in an augmented normal matrix, ex-
panded in both column and row directions.
P.1. mCLESS 369
Feature Expansion for mCLESS

• A feature expansion is expressed as

x = [x1 , x2 , · · · , xd ] e = [x1 , x2 , · · · , xd , σ(x)]
x
⇒ (P.1.13)
w = [w0 , w1 , · · · , wd ]T e = [w0 , w1 , · · · , wd , wd+1 ]T
w
where σ() is a feature function of x.

• Then, the expanded weights must be trained to satisfy
(j) (j) (i) (j) (i) (j)
e(i) ] w
[1, x e (j) = w0 + w1 x1 + · · · + wd xd + wd+1 σ(x(i) ) = δij , (P.1.14)
for all points in the dataset. Compare the equation with (P.1.3).
• The corresponding expanded information and weight matrices read
 
(0) (1) (C−1)
  w w0 · · · w0
1 x11 x12 · · · x1d σ(x1 )  0 
   w(0) w(1) · · · w(C−1) 
 1 1 1 
1 x
21 x 22 · · · x 2d σ(x )
2 
  .. ... ..
A
e=
.. . . , W
f= . ,
.
. .
 
 . . .  
 wd(0) wd(1) · · · wd(C−1) 
   
1 xN 1 xN 2 · · · xN d σ(xN ) 
(0) (1) (C−1)

wd+1 wd+1 · · · wd+1
(P.1.15)
N ×(d+2) f (d+2)×C
where A ∈ R
e ,W ∈R , and C is the number of classes.
• Feature expansion can be performed multiple times. When α features
are added, the optimal weight matrix W c ∈ R(d+1+α)×C is the least-
squares solution of
eT A)
(A e Wc=A eT B, (P.1.16)
eT A
where A e ∈ R(d+1+α)×(d+1+α) and B is the same as in (P.1.7).
Remark P.5. Various feature functions σ() can be considered. Here we

will focus on the feature function of the form
σ(x) = kx − pk, (P.1.17)
the Euclidean distance between x and a prescribed point p.

Now, the question is: “How can we find p?”
Generation of the Synthetic Data

synthetic_data.py
3 from GLOBAL_VARIABLES import *
4
5 def generate_data(n,scale,theta):
6 # Normally distributed around the origin
7 x = np.random.normal(0,1, n); y = np.random.normal(0,1, n)
8 P = np.vstack((x, y)).T
9 # Transform
10 sx,sy = scale
11 S = np.array([[sx,0],[0,sy]])
12 c,s = np.cos(theta), np.sin(theta)
13 R = np.array([[c,-s],[s,c]]).T #T, due to right multiplication
14 return P.dot(S).dot(R)
15
16 def synthetic_data():
17 N=0
18 plt.figure()
19 for i in range(N_CLASS):
20 scale = SCALE[i]; theta = THETA[i]; N+=N_D1
21 D1 = generate_data(N_D1,scale,theta) +TRANS[i]
22 D1 = np.column_stack((D1,i*np.ones([N_D1,1])))
23 if i==0: DATA = D1
24 else: DATA = np.row_stack((DATA,D1))
25 plt.scatter(D1[:,0],D1[:,1],s=15,c=COLOR[i],marker=MARKER[i])
26
27 np.savetxt(DAT_FILENAME,DATA,delimiter=',',fmt=FORMAT)
28 print(' saved: %s' %(DAT_FILENAME))
29
30 #xmin,xmax = np.min(DATA[:,0]), np.max(DATA[:,0])

31 ymin,ymax = np.min(DATA[:,1]), np.max(DATA[:,1])
32 plt.ylim([int(ymin)-1,int(ymax)+1])
33
34 plt.title('Synthetic Data: N = '+str(N))

35 myfigsave(FIG_FILENAME)
36 if __name__ == '__main__':
37 plt.show(block=False); plt.pause(5)
38
39 if __name__ == '__main__':
40 synthetic_data()
P.1. mCLESS 371
GLOBAL_VARIABLES.py
3
4 N_D1 = 100
5 FORMAT = '%.3f','%.3f','%d'
6
7 SCALE = [[1,1],[1,2],[1.5,1]]; TRANS = [[0,0],[6,0],[3,4]]

8 #SCALE = [[1,1],[1,1],[1,1]]; TRANS = [[0,0],[4,0],[8,0]]
9 THETA = [0,-0.25*np.pi, 0]
10 COLOR = ['r','b','c']
11 MARKER = ['.','s','+','*']
12 LINESTYLE = [['r--','r-'],['b--','b-'],['c--','c-']]
13
14 N_CLASS = len(SCALE)
15
16 DAT_FILENAME = 'synthetic.data'
17 FIG_FILENAME = 'synthetic-data.png'
18 FIG_INTERPRET = 'synthetic-data-interpret.png'
19
20 def myfigsave(figname):
21 plt.savefig(figname,bbox_inches='tight')
22 print(' saved: %s' %(figname))
What to do
1. Implement mCLESS:
• Training. You should implement modules for each of (P.1.5) and
(P.1.7). Then use Xtrain and ytrain to get A and B.
• Test. Use the same module (implemented for A) to get Atest from
Xtest. Then perform P = (Atest)*W c as in (P.1.10). Now, you can
get the prediction using
prediction = np.argmax(P,axis=1);
which may be compared with ytest to obtain accuracy.
2. Use following datasets:
• Synthetic datasets. Generate two different synthetic datasets:
1 Use Line 7 in GLOBAL_VARIABLES.py
2 Use Line 8 in GLOBAL_VARIABLES.py
• Real datasets. Use public datasets such as iris and wine.
To get the public datasets, you may use:
from sklearn import datasets
data_read1 = datasets.load_iris()
data_read2 = datasets.load_wine()
3. Compare the performance of mCLESS with
• LogisticRegression(max_iter = 1000)
• KNeighborsClassifier(5)
• SVC(gamma=2, C=1)
• RandomForestClassifier(max_depth=5, n_estimators=50, max_features=1)
See Section 1.3.
4. (Optional for Undergraduate Students) Add modules for feature
expansion, as described on page 369.
• For this, try to an interpretable strategy to find an effective point p such
that the feature expansion with (P.1.17) improves accuracy.
• Experiment Steps 1-3.
5. Report your experiments with the code and results.
You may start with the machine learning modelcode in Section 1.3; add
your own modules.
P.2. Noise-Removal and Classification 373
P.2. Noise-Removal and Classification

Machine Learning Tasks
• Many algorithms are sensitive to outliers or noise:
– An object with extreme values may substantially distort the dis-
tribution of the data.
• A good dataset is often better than a good algorithm.
Project Objectives
• Develop efficient noise-removal algorithms,

– using e.g., the k-NN and the Clustering-PCA.
• Merge the noise-removal algorithms to classification.
• Test and tune the resulting algorithms for public-domain datasets.
For each of selected datasets, you will design the best model for
noise-removal and classification.
Confidence Region
A confidence score indicates the likelihood that a machine learning
model assigns the respective intent correctly.
Definition P.6. A confidence region is the region where a new point

belongs to a specified class, given a confidence score/value.
Review: k-NN and PCA

k-Nearest Neighbors (k-NN)
Algorithm 5.37, p. 129. (k-NN algorithm). The algorithm itself is
fairly straightforward and can be summarized by the following steps:
1. Choose the number k and a distance metric.
2. For the new sample, find the k-nearest neighbors.
3. Assign the class label by majority vote.
Figure 5.16: Illustration for how a new data point (?) is assigned the triangle class label,
based on majority voting, when k = 5.
Example P.7. Along with the k-NN algorithm:

• Select k.
• Set a confidence value ξ ≤ k.
Then the confidence region for a class can be defined as the region
where the k-NN of a point includes at least ξ points from the same class.
For example, k = 5 and ξ = 4.
PCA
Recall: (PCA), p. 189. Consider a data matrix X ∈ RN ×d :
◦ each of the N rows represents a different data point,
◦ each of the d columns gives a particular kind of feature, and
◦ each column has zero empirical mean (e.g., after standardization).
• The goal of the standard PCA is to find an orthogonal weight matrix

Wk ∈ Rd×k such that
Zk = X Wk , k ≤ d, (P.2.1)
where Zk ∈ RN ×k is call the truncated score matrix and Zd = Z.
Columns of Z represent the principal components of X.
• (Claim 7.3, p. 152). The transformation matrix Wk turns out to be the
collection of normalized eigenvectors of X T X:
Wk = [w1 |w2 | · · · |wk ], (X T X) wj = λj wj , wiT wj = δij , (P.2.2)
where λ1 ≥ λ2 ≥ · · · ≥ λk ≥ 0.
• (Remark 7.4, p. 152). The matrix Zk ∈ RN ×k is scaled eigenvectors of
XX T :
p p p
Zk = [ λ1 u1 | λ2 u2 | · · · | λk uk ], (XX T ) uj = λj uj , uTi uj = δij .
(P.2.3)
• A data (row) vector x (new or old) is transformed to a k-
dimensional row vector of principal components
z = xWk ∈ R1×k . (P.2.4)
• (Remark 7.5, p. 153). Let X = U ΣV T be the SVD of X, where

Σ = diag(σ1 , σ2 , · · · , σd ), σ1 ≥ σ2 ≥ · · · ≥ σd ≥ 0.
Then,
V ∼
= W ; σj2 = λj , j = 1, 2, · · · , d,
(P.2.5)
Zk = [σ1 u1 |σ2 u2 | · · · |σk uk ].
Geometric Interpretation of PCA

Example P.8. Consider the following synthetic dataset.
Figure P.3: A synthetic dataset of three classes.
• For each class, one may perform PCA along with the SVD.
for c in range(nclass):
Xc = X[y==c]; CC = np.mean(Xc,axis=0)
U, s, VT = svd(Xc-CC,full_matrices=False)
(c) (c) (c)
• Let µ(c) = CC[c], V (c) = [v1 , · · · , vd ], and σj = s[j], the jth singular value
for Class c. Define an anisotropic distance as
d (c)
(c)
X (x − µ(c) ) · vj 2
γ (x) = (c)
. (P.2.6)
j=1 σj
– It is implemented in the function aniso_dist2, in util_PCA.py.

– For r > 0, γ (c) (x) = r2 assigns an ellipse.
Definition P.9. The minimum-volume enclosing ellipsoid (MVEE)

is the ellipsoid of smallest volume that fully contains all the objects.
(c)
Remark P.10. Let rmax be
2
(c)
rmax = max γ (c) (x). (P.2.7)
x∈X (c)
(c) 2
Then γ (c) (x) = rmax approximates the MVEE relatively well.
See Figure P.4 (a).
(a) (b)
Figure P.4: Approximate MVEEs for: (a) the dataset and (b) the confidence regions.
Example P.11. Along with PCA:

• Either set a threshold θ > 0
• or a portion 0 < p < 1.
The confidence region for a class can be defined as the region where
(a) either the points x satisfy γ (c) (x) ≤ θ
(b) or only the p-portion of the near-center points are picked from the
dataset ordered by the anisotropic distances.
Figure P.4(b) shows the confidence regions, for p = 0.9.
Note: You must first find confidence regions for the training dataset,
which can be viewed as denoising. You may then begin the training
step with the denoised dataset.
What to do
1. Download PCA-KNN-Denoising.PY.tar:
https://skim.math.msstate.edu/LectureNotes/data/PCA-KNN-
Denoising.PY.tar
2. Compose a denoising-and-classification code, using appropriate
functions from the downloaded package.
• You must implement both denoising algorithms: the k-NN-based
and the PCA-based.
3. Use similar datasets, utilized for Project 1. mCLESS, Section P.1:
• Select a synthetic dataset, using Line 7 or 8 in GLOBAL_VARIABLES.py.
• Real datasets. Use public datasets such as iris and wine.
To get the public datasets, you may use:
from sklearn import datasets
data_read1 = datasets.load_iris()
data_read2 = datasets.load_wine()
4. Compare performances of the classifiers with and without denois-
ing
• LogisticRegression(max_iter = 1000)
• KNeighborsClassifier(5)
• SVC(gamma=2, C=1)
• RandomForestClassifier(max_depth=5, n_estimators=50, max_features=1)
5. (Optional for Undergraduate Students)
Add modules for clustering-and-PCA denoising.
• For example, the MVEE does not make sense for a half-moon dataset.
• Add another dataset, such as
from sklearn.datasets import make_moons
X, y = make_moons(noise=0.2, n_samples=400, random_state=12)
• Preform k-Means cluster analysis for each class, with k = 4.
• For each cluster in each class, perform the PCA-based denoising.
• Carry out Steps 2–4.
6. Report your experiments with the code and results.
Note: You did already the portion: “without denoising”. Undergraduate

students may consider that a class is a cluster. For graduate students, Step
5 will be worth 40% your score.
P.3. Gaussian Sailing to Overcome Local Minima Problems 379
P.3. Gaussian Sailing to Overcome Local Min-

ima Problems
• A Gaussian smoothing for a 2D function f (x, y) can be achieved by
storing the function to a 2D-array A, and applying a built-in function
in scipy:
scipy.ndimage.filters.gaussian_filter
which requires a parameter σ (standard deviation for Gaussian ker-
nel).
• Alternatively, one can employ an averaging operator; for example,
apply a few iterations of the following convolution
 
1 cw 1
def 1 cw c2w cw  ,
S ∗ A, S == 2
(P.3.1)
(2 + cw )
1 cw 1
for cw ≥ 0. Since    
1 cw 1 1
cw c2w cw  = cw  [1 cw 1],
1 cw 1 1
the convolution smoothing can be implemented easily and conve-
niently as
  
1
1
S ∗A = cw  ∗ [1 cw 1] ∗ A  ,
(2 + cw )2
1
which is a horizontal filtering followed by a vertical filtering and fi-

nally a scaling with the factor 1/(2 + cw )2 .
• In this project, you will explore the gradient descent method with line
search (4.19) for the computation of the global minimizer of multiple
local minima problems.
Tasks to do
1. Go to http://www.sfu.ca/∼ssurjano/optimization.html (Virtual Li-
brary of Simulation Experiments) and select three functions of
your interests having multiple local minima. (e.g., two of them are
the Ackley function and Griewank function.)
2. Store each of the functions in a 2D-array A which has dimensions
large enough.
3. Compute
Aσ = gaussian_filter(A, σ), or At = S
| ∗ S ∗{z· · · ∗ S} ∗A,
t-times
which can be considered as a convex/smooth approximation of the

original function. You can use it for the estimation of f and its
derivatives at xn .
4. Design a set of σ/t-values T (including “0” as the last entry) so
that given an initial point x0 , the Gaussian homotopy continuation
method discussed in Remark 4.13 can locate the global minimum,
while the algorithm (4.19) can find only a local minimum, for each
of the functions.
Report. Submit hard copies of your experiences.

• Attach a “summary” or “conclusion” page at the beginning of report.
• Your work process.
• Your code.
• Figures; for each of the functions, include a figure that shows move-
ment of the minimizers for all σ’s or t’s in T .
• Discuss pros and cons of the Gaussian sailing strategy.
You may work in a group of two people; however, you must report individu-
ally.
P.4. Quasi-Newton Methods Using Partial Information of the Hessian 381
P.4. Quasi-Newton Methods Using Partial In-

formation of the Hessian
Consider a unconstrained optimization problem of the form
min f (x), (P.4.1)
x∈Rd
and Newton’s method −1
xn+1 = arg min Qn (x) = xn − γ ∇2 f (xn )

∇f (xn ), (P.4.2)
x
where 
∂2f ∂2f ∂2f

 ∂x1 2 ···
 ∂x1 ∂x2 ∂x1 ∂xd 

 ∂2f ∂2f ∂2f 
···
 
2 ∂x ∂x ∂x2 2 ∂x2 ∂xd  ∈ Rd×d .
 
∇ f =
 2 1 (P.4.3)
.. .. .. ..

. . . .
 
 
 
 ∂2f ∂2f ∂2f 
···
∂xd ∂x1 ∂xd ∂x2 ∂xd 2
Known . (Remark 4.15). Where applicable, Newton’s method con-

verges much faster towards a local extremum than gradient descent.
Every local minimum has a neighborhood such that, if we start within
this neighborhood, Newton’s method with step size γ = 1 converges
quadratically assuming the Hessian is invertible and Lipschitz contin-
uous.
Issues . The central issue with Newton’s method is that we need to be

able to compute the inverse Hessian matrix.
• For ML applications, the dimensionality of the problem can be of
the order of thousands or millions; computing the Hessian or its
inverse is often impractical.
• Because of these reasons, Newton’s method is rarely used in prac-
tice to optimize functions corresponding to large problems.
• Luckily, the above algorithm can still work even if the Hessian is
replaced by a good approximation.
• Various “quasi-Newton" methods have been developed so as to ap-
proximate and update the Hessian matrix (without evaluating the
second derivatives).
Objectives. You are required to perform tasks including the following.

• Algorithm Development:
– Note the Newton’s search direction is −H −1 ∇f (xn ), where H =
∇2 f (xn ).
– Select a set of k components (k d) from ∇f ∈ Rd which would
dominate the search direction. Then, for some permutation P ,
" #
∇f (x )
P ∇f (xn ) = f n .
c
(P.4.4)
∇f (xn )
– Construct H b n ) ∈ Rk×k (using finite differences) to solve

b = H(x
H
bqb = ∇f
c (xn ). (P.4.5)
– Find a scaling factor σ > 0 such that

T σq
−P
b
(P.4.6)
∇f
f (xn )
is the final search direction.

– Suggestions: For d ≥ 10, k = 2 ∼ 5 and
∼ ||∇f
c (xn )||
σ= . (P.4.7)
||b
q||
• Comparisons:
– Implement (or download codes for) the original Newton’s method
and one of quasi-Newton methods (e.g., BFGS).
– Let’s call our method the partial Hessian (PH)-Newton method.
Compare the PH-Newton with those known methods for: the num-
ber of iterations, the total elapsed time, convergence behavior, and
stability/robustness.
– Test with e.g. the Rosenbrock function defined on Rd , d ≥ 10, with
various initial points x0 .
P.5. Effective Preprocessing Technique for Filling Missing Data 383
P.5. Effective Preprocessing Technique for Fill-

ing Missing Data
(From Remark 6.2). Data preparation is difficult because the pro-
cess is not objective, and it is important because ML algorithms learn
from data. Consider the following.
• Preparing data for analysis is one of the most important steps in any
data-mining project – and traditionally, one of the most time consum-
ing.
• Often, it takes up to 80% of the time.
• Data preparation is not a once-off process; that is, it is iterative as
you understand the problem deeper on each successive pass.
Known . For missing values, three different steps can be executed.

• Removal of samples (rows) or features (columns):
It is the simplest and efficient method for handling the missing data.
However, we may end up removing too many samples or features.
• Filling the missing values manually:
This is one of the best-chosen methods.
But there is one limitation that when there are large data set, and
missing values are significant.
• Imputing missing values using computed values:
The missing values can also be occupied by computing mean, me-
dian, or mode of the observed given values. Another method could
be the predictive values that are computed by using any ML or Deep
Learning algorithm.
But one drawback of this approach is that it can generate bias
within the data as the calculated values are not accurate concerning
the observed values.
Objectives.
• Algorithm Development:
– Think a good strategy or two for filling the missing values, if it
is to be done manually.
– What information available from the dataset is useful and help us
build a good strategy?
– Suggestions: Use near-values to interpolate; try to employ the
concept of feature importance, if available.
• Comparisons:
– For the Wine dataset, for example, erase r% data values in ran-
dom; r = 5, 10, 20.
– Compare your new filling strategy with 1 the simple sample re-
moval method and 2 the imputation strategy using mean, me-
dian, or mode.
– Perform accuracy analysis for various classifiers, e.g., logistic
regression, support vector machine, and random forests.
Bibliography
[1] W. B ELSON, Matching and prediction on the principle of biological classification,
JRSS, Series C, Applied Statistics, 8 (1959), pp. 65–75.
[2] Y. B ENGIO, Deep learning of representations: Looking forward, in Statistical Lan-

guage and Speech Processing, A. H. Dediu, C. Martín-Vide, R. Mitkov, and B. Truthe,
eds., vol. 7978 of Lecture Notes in Computer Science, Springer, Berlin, Heidelberg,
2013, pp. 1–37.
[3] Y. B ENGIO, A. C OURVILLE , AND P. V INCENT, Representation learning: A review and

new perspectives, IEEE Transactions on Pattern Analysis and Machine Intelligence,
35 (2013), pp. 1798–1828.
[4] Y. B ENGIO, P. L AMBLIN, D. P OPOVICI , AND H. L AROCHELLE, Greedy layer-wise

training of deep networks, in NIPS’06 Proceedings of the 19th International Confer-
ence on Neural Information Processing Systems, NIPS, 2006, pp. 153–160.
[5] Y. B ENGIO, Y. L E C UN, AND G. H INTON, Deep learning, Nature, 521 (2015), pp. 436–
444.
[6] A. B JÖRCK, Numerical Methods for Least Squares Problems, SIAM, Philadelphia,
1996.
[7] L. B OTTOU, F. E. C URTIS, AND J. N OCEDAL, Optimization methods for large-scale

machine learning, SIAM Rev., 60 (2018), pp. 223–311.
[8] L. B REIMAN, Random forests, Machine Learning, 45 (2001), pp. 5–32.
[9] C. G. B ROYDEN, The convergence of a class of double-rank minimization algorithms

1. General considerations, IMA Journal of Applied Mathematics, 6 (1970), pp. 76–90.
[10] J. B UNCH AND L. K AUFMAN, Some stable methods for calculating inertia and solving
symmetric linear systems, Math. Comput., 31 (1977), pp. 163–179.
[11] R. B. C ATTELL, The description of personality: Basic traits resolved into clusters, Jour-
nal of Abnormal and Social Psychology., 38 (1943), pp. 476–506.
[12] C. C ORTES AND V. N. VAPNIK, Support-vector networks, Machine Learning, 20 (1995),

pp. 273–297.
[13] G. G. D ANIEL, Deterministic and Nondeterministic Turing Machine, Springer Nether-

lands, Dordrecht, 2013, pp. 624–624.
385
386 BIBLIOGRAPHY
[14] R. D ECHTER, Learning while searching in constraint-satisfaction-problems, in AAAI,

T. Kehler, ed., Morgan Kaufmann, 1986, pp. 178–185.
[15] S. D EERWESTER , S. D UMAIS, G. F URNAS, T. L ANDAUER , AND R. H ARSHMAN, In-

dexing by latent semantic analysis., Journal of the American Society for Information
Science, (1990), pp. 391–407.
[16] H. E. D RIVER AND A. L. K ROEBER, Quantitative expression of cultural relation-

ships, in University of California Publications in American Archaeology and Ethnol-
ogy, vol. Quantitative Expression of Cultural Relationships, 1932, pp. 211–256.
[17] L. E LDÉN, Numerical linear algebra in data mining, Acta Numerica, 15 (2006),
pp. 327 – 384.
[18] M. E STER , H.-P. K RIEGEL , J. S ANDER , AND X. X U, A density-based algorithm for

discovering clusters in large spatial databases with noise, in Proceedings of the Second
International Conference on Knowledge Discovery and Data Mining, KDD-96, AAAI
Press, 1996, pp. 226–231.
[19] A. F ISHER , C. R UDIN, AND F. D OMINICI, Model class reliance: Variable importance
measures for any machine learning model class, from the "rashomon" perspective,
(2018).
[20] R. A. F ISHER, The use of multiple measurements in taxonomic problems, Annals of

Eugenics, 7 (1936), pp. 179–188.
[21] R. F LETCHER, A new approach to variable metric algorithms, The Computer Journal,
13 (1970), pp. 317–322.
[22] S. G ERSCHGORIN, Über die abgrenzung der eigenwerte einer matrix, Izv. Akad. Nauk
SSSR Ser. Mat., 7 (1931), pp. 746–754.
[23] X. G LOROT, A. B ORDES, AND Y. B ENGIO, Deep sparse rectifier neural networks,
in Proceedings of the Fourteenth International Conference on Artificial Intelligence
and Statistics, G. Gordon, D. Dunson, and M. Dudík, eds., vol. 15 of Proceedings
of Machine Learning Research, Fort Lauderdale, FL, USA, 11–13 Apr 2011, PMLR,
pp. 315–323.
[24] D. G OLDFARB, A family of variable metric methods derived by variational means,

Mathematics of Computation, 24 (1970), pp. 23–26.
[25] F. G OLUB AND C. V. L OAN, Matrix Computations, 3rd Ed., The Johns Hopkins Uni-
versity Press, Baltimore, 1996.
[26] G. H. G OLUB AND C. R EINSCH, Singular value decomposition and least squares so-
lutions, Numer. Math., 14 (1970), pp. 403–420.
[27] B. G ROSSER AND B. L ANG, An o(n2 ) algorithm for the bidiagonal svd, Lin. Alg. Appl.,
358 (2003), pp. 45–70.
BIBLIOGRAPHY 387
[28] L. G UTTMAN, A necessary and sufficient formula for matric factoring, Psychometrika,
22 (1957), pp. 79–81.
[29] G. H INTON, , , G. D AHL , A.- R . M OHAMED, N. J AITLY, A. S ENIOR , V. VANHOUCKE ,

B. K INGSBURY, AND T. S AINATH, Deep neural networks for acoustic modeling in
speech recognition, IEEE Signal Processing Magazine, 29 (2012), pp. 82–97.
[30] G. E. H INTON, S. O SINDERO, AND Y.-W. T EH, A fast learning algorithm for deep
belief nets, Neural Comput., 18 (2006), pp. 1527–1554.
[31] G. E. H INTON AND R. S ALAKHUTDINOV, Reducing thedimensionality of data with

neural networks, Science, 313 (2006), pp. 504–507.
[32] T. K. H O, Random decision forests, in Proceedings of the 3rd International Conference

on Document Analysis and Recognition, Montreal, QC, 1995, pp. 278–282.
[33] H. H OTELLING, Analysis of a complex of statistical variables into principal compo-

nents, Journal of Educational Psychology, 24 (1933), pp. 417–441 and 498–520.
[34] , Relations between two sets of variates, Biometrika, 28 (1936), pp. 321–377.
[35] A. K. J AIN AND R. C. D UBES, Algorithms for Clustering Data, Prentice-Hall, Inc.,
Upper Saddle River, NJ, USA, 1988.
[36] W. K ARUSH, Minima of functions of several variables with inequalities as side con-
straints, M.Sc. Dissertation, Department of Mathematics, Univ. of Chicago, Chicago,
Illinois, 1939.
[37] L. K AUFMAN AND P. R OUSSEEUW, Clustering by means of medoids, in Statistical

Data Analysis Based on the L1 -Norm and Related Methods, Y. Dodge, ed., North-
Holland, 1987, pp. 405–416.
[38] L. K AUFMAN AND P. J. R OUSSEEUW, Finding Groups in Data: An Introduction to

Cluster Analysis, John Wiley & Sons, Inc., 1990.
[39] T. K OHONEN, Self-organized formation of topologically correct feature maps, Biologi-

cal Cybernetics, 43 (1982), pp. 59–69.
[40] T. K OHONEN, Self-Organizing Maps, Springer series in information sciences, 30,

Springer, Berlin, 3rd ed., Dec. 2001.
[41] A. K RIZHEVSKY, I. S UTSKEVER , AND G. E. H INTON, Imagenet classification with

deep convolutional neural networks, Commun. ACM, 60 (2017), pp. 84–90.
[42] H. W. K UHN AND A. W. T UCKER, Nonlinear programming, in Proceedings of 2nd

Berkeley Symposium, Berkeley, CA, USA, 1951, University of California Press.,
pp. 481–492.
[43] K. L ANGE , D. R. H UNTER , AND I. YANG, Optimization transfer using surrogate objec-
tive functions, Journal of Computational and Graphical Statistics, 9 (2000), pp. 1–20.
388 BIBLIOGRAPHY
[44] R. L EBLOND, F. P EDREGOSA , AND S. L ACOSTE -J ULIEN, Improved asynchronous par-

allel optimization analysis for stochastic incremental methods, CoRR, abs/1801.03749
(2018).
[45] Y. L E C UN, B. E. B OSER , J. S. D ENKER , D. H ENDERSON, R. E. H OWARD, W. E.

H UBBARD, AND L. D. J ACKEL, Handwritten digit recognition with a back-propagation
network, in Advances in Neural Information Processing Systems 2, D. S. Touretzky,
ed., Morgan-Kaufmann, 1990, pp. 396–404.
[46] Y. L E C UN, S. C HOPRA , R. H ADSELL , F. J. H UANG, AND ET AL ., A tutorial on energy-

based learning, in PREDICTING STRUCTURED DATA, MIT Press, 2006.
[47] C. L EMARECHAL, Cauchy and the gradient method, Documenta Mathematica, Extra
Volume, (2012), pp. 251–254.
[48] S. P. L LOYD, Least square quantization in pcm, Bell Telephone Laboratories Report,
1957. Published in journal: S. P. Lloyd (1982). Least squares quantization in PCM.
IEEE Transactions on Information Theory. 28 (2): pp. 129–137.
[49] M. L OURAKIS, A brief description of the Levenberg-Marquardt algorithm implemented

by levmar, Technical Report, Institute of Computer Science, Foundation for Research
and Technology – Hellas, 2005.
[50] D. G. L OWE, Object recognition from local scale-invariant features, in Proceedings

of the Seventh IEEE International Conference on Computer Vision, IEEE, 1999,
pp. 1150–1157.
[51] S. M ANDT, M. D. H OFFMAN, AND D. M. B LEI, Stochastic gradient descent as approx-

imate Bayesian inference, Journal of Machine Learning Research, 18 (2017), pp. 1–35.
[52] D. M ARQUARDT, An algorithm for least-squares estimation of nonlinear parameters,

Journal of the Society for Industrial and Applied Mathematics, 11 (1963), pp. 431–441.
[53] H. M OBAHI AND J. W. F ISHER III, On the link between gaussian homotopy continua-
tion and convex envelopes, in Energy Minimization Methods in Computer Vision and
Pattern Recognition. Lecture Notes in Computer Science, vol. 8932, X.-C. Tai, E. Bae,
T. F. Chan, and M. Lysaker, eds., Hong Kong, China, 2015, Springer, pp. 43–56.
[54] R. T. N G AND J. H AN, Efficient and effective clustering methods for spatial data
mining, in Proceedings of the 20th International Conference on Very Large Data
Bases, VLDB ’94, San Francisco, CA, USA, 1994, Morgan Kaufmann Publishers Inc.,
pp. 144–155.
[55] , CLARANS: A method for clustering objects for spatial data mining, IEEE Trans-
actions on Knowledge and Data Engineering, 14 (2002), pp. 1003–1016.
[56] M. N IELSEN, Neural networks and deep learning. (The online book can be found at
http://neuralnetworksanddeeplearning.com), 2013.
BIBLIOGRAPHY 389
[57] J. N OCEDAL AND S. J. W RIGHT, Numerical Optimization (2nd Ed.), Springer-Verlag,

Berlin, New York, 2006.
[58] K. P EARSON, On lines and planes of closest fit to systems of points in space, Philosoph-
ical Magazine, 2 (1901), pp. 559–572.
[59] J. P ETCH , S. D I , AND W. N ELSON, Opening the black box: The promise and limita-
tions of explainable machine learning in cardiology, Canadian Journal of Cardiology,
38 (2022), pp. 204–213.
[60] R. R AINA , A. M ADHAVAN, AND A. Y. N G, Large-scale deep unsupervised learning us-
ing graphics processors, in Proceedings of the 26th Annual International Conference
on Machine Learning, ICML ’09, New York, NY, USA, 2009, ACM, pp. 873–880.
[61] C. R. R AO, The utilization of multiple measurements in problems of biological classi-
fication, Journal of the Royal Statistical Society. Series B: Statistical Methodology, 10
(1948), pp. 159–203.
[62] S. R ASCHKA AND V. M IRJALILI, Python Machine Learning, 3rd Ed., Packt Publishing
Ltd., Birmingham, UK, 2019.
[63] H. R OBBINS AND S. M ONRO, A stochastic approximation method, Ann. Math. Statist.,
22 (1951), pp. 400–407.
[64] F. R OSENBLATT, The Perceptron, a Perceiving and Recognizing Automaton Project
Para, Report: Cornell Aeronautical Laboratory, Cornell Aeronautical Laboratory,
1957.
[65] D. E. R UMELHART, G. E. H INTON, AND R. J. W ILLIAMS, Learning representations
by back-propagating errors, Nature, 323 (1986), pp. 533–536.
[66] T. N. S AINATH , B. K INGSBURY, G. S AON, H. S OLTAU, M. A. R., G. D AHL , AND
B. R AMABHADRAN, Deep convolutional neural networks for large-scale speech tasks,
Neural Netw., 64 (2015), pp. 39–48.
[67] G. S ALTON, A. W ONG, AND C.-S. YANG, A vector space model for automatic indexing,
Communications of the ACM, 18 (1975), pp. 613–620.
[68] F. S ANTOSA AND W. W. S YMES, Linear inversion of band-limited reflection seismo-
grams, SIAM Journal on Scientific Computing, 7 (1986), pp. 1307–1330.
[69] J. S CHMIDHUBER, Deep learning in neural networks: An overview, Neural Networks,
61 (2015), pp. 85–117.
[70] B. S CHÖLKOPF , A. S MOLA , AND K.-R. M ÜLLER, Kernel principal component analy-
sis, in Artificial Neural Networks — ICANN’97, EDITORS, ed., vol. 1327 of Lecture
Notes in Computer Science, Springer, Berlin, Heidelberg, 1997, pp. 583–588.
[71] E. S CHUBERT AND P. J. R OUSSEEUW, Faster k-medoids clustering: Improving the
PAM, CLARA, and CLARANS algorithms, CoRR, abs/1810.05691 (2018).
390 BIBLIOGRAPHY
[72] D. F. S HANNO, Conditioning of quasi-Newton methods for function minimization,

Mathematics of Computation, 24 (1970), pp. 647–656.
[73] I. S UTSKEVER , O. V INYALS, AND Q. V. L E, Sequence to sequence learning with neural

networks, in Advances in Neural Information Processing Systems 27, Z. Ghahramani,
M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, eds., Curran Associates,
Inc., 2014, pp. 3104–3112.
[74] O. T AUSSKY, Bounds for characteristic roots of matrices, Duke Math. J., 15 (1948),
pp. 1043–1044.
[75] R. T IBSHIRANI, Regression shrinkage and selection via the LASSO, Journal of the
Royal Statistical Society. Series B (methodological), 58 (1996), pp. 267–288.
[76] R. C. T RYON, Cluster Analysis: Correlation Profile and Orthometric (factor) Analysis
for the Isolation of Unities in Mind and Personality, Edwards Brothers, 1939.
[77] R. VARGA, Matrix Iterative Analysis, 2nd Ed., Springer-Verlag, Berlin, Heidelberg,
2000.
[78] J. H. M. W EDDERBURN, Lectures on Matrices, Amer. Math. Soc., New York, 1934.
[79] P. R. W ILLEMS, B. L ANG, AND C. V ÖMEL, Computing the bidiagonal SVD using
multiple relatively robust representations, SIAM Journal on Matrix Analysis and Ap-
plications, 28 (2006), pp. 907–926.
[80] M. H. W RIGHT, Interior methods for constrained optimization, Acta Numerica, 1

(1992), pp. 341–407.
[81] K. X U, J. L. B A , R. K IROS, K. C HO, A. C OURVILLE , R. S ALAKHUTDINOV, R. S.

Z EMEL , AND Y. B ENGIO, Show, attend and tell: Neural image caption generation
with visual attention, in Proceedings of the 32Nd International Conference on Inter-
national Conference on Machine Learning - Volume 37, ICML’15, 2015, pp. 2048–
2057.
[82] S. X U, An Introduction to Scientific Computing with MATLAB and Python Tutorials,

CRC Press, Boca Raton, FL, 2022.
[83] W. Z HANG, J. WANG, D. J IN, L. O REOPOULOS, AND Z. Z HANG, A deterministic

self-organizing map approach and its application on satellite data based cloud type
classification, in 2018 IEEE International Conference on Big Data (Big Data), 2018,
pp. 2027–2034.
[84] J. Z UBIN, A technique for measuring like-mindedness, The Journal of Abnormal and
Social Psychology, 33 (1938), pp. 508–516.
Index
:, Python slicing, 21 bias-variance tradeoff, 101
_ _init_ _() constructor, 31 binary classifier, 38
binary decision tree, 123
acceptance criterion, 85 binding constraint, 111
activation function, 363 binomial distribution, 96
active set, 350 bisecting K-Means algorithm, 217
Adaline, 47, 95, 363 black boxes, 362
adaline, 37 bootstrap sample, 127
adaptive linear neurons, 37 border point, 232
adaptive step size, 63
agglomerative clustering, 224, 226 C++, 294
AGNES, 224, 228 call_get_cubes.py, 25
algorithmic parameter, 49 categorical data, 137
anisotropic distance, 376 Cauchy, Augustin-Louis, 56
API, 294 central path, 354
application programming interface, 294 centroid, 204
approximate complementarity, 357 centroid linkage, 227
approximate Hessian, 83 chain rule, 265
argmax, numpy, 366 chi-squared error criterion, 81
artificial neural networks, 92 child class, 34
artificial neurons, 38 CIFAR, 265
artificial topographic maps, 249 CLARA, 221, 222
association rule discovery, 300 CLARANS, 223
attributes, 31 class, 29, 30
averaged perceptron, 44 class attribute, 33
averaging operator, 379 class-membership probability, 95
Classes.py, 34
back-propagation, 278 classification, 2, 300
back-propagation algorithm, 283 classification error, 124
backtracking line search, 63 Clay Institute, 207
barrier functions, 355 cluster analysis, 199
barrier parameter, 355 cluster cohesion, 242
batch gradient descent, 51 cluster separation, 242
best split, 126 cluster validation, 236
between-class scatter, 176 clustering, 2, 11, 199, 300
between-class scatter matrix, 176, 178 CNN, 265, 286
BFGS algorithm, 72 code block, 20
bias, 40, 100, 266 column-stochastic matrix, 323
391
392 INDEX
column_stack, numpy, 365 definite, 333

competitive learning, 249, 252 dendrogram, 228
complementary slackness, 108, 111, 353 denoising, 377
complete linkage, 226 denormalize.m, 182
computational complexity theory, 207 density estimation, 3
confidence region, 373, 374, 377 density-based clustering, 231
confidence score, 373 density-connected, 233
constraint set, 56 description methods, 300
contrapositive, 111 desktop calculator, 19
convergence, 250 destructor, 33
convergence, iterative methods, 343 deterministic clustering, 250
convolution, 289 deviation/anomaly detection, 300
convolutional network, 289 diagonal dominance, 346
convolutional neural network, 265 DIANA, 224
convolutional neural networks, 286 dictionary, 302
core points, 232 dimensionality reduction, 149, 306
correlation, 238 directed graph, 344
cosine distance measure, 315 discriminant function, 253
cost function, 48 distance functions, 201
covariance matrix, 165, 175 divisive clustering, 224
criterion function, 142 DLS, 81
critical point, 60 document weighting, 313
critical points, 68 domain, 56
CUDA, 294 dot product of matrices, 333
cumulative explained variance, 155 dual feasibility, 108
curse of dimensionality, 130, 140 dual problem, 106, 109, 114, 117, 120
curvature condition, 73 dual variable, 106, 107
cython, 18 dyadic decomposition, 159
damped least-squares, 81 eigenvalue locus theorem, 345

damping factor, 325 elastic net regularization, 141
damping parameter, 84 elbow analysis, 260
data compression, 149, 306 element-wise product, 282
data gap, 299 ellipse, 376
data matrix, 151, 189, 375 embedded methods, 141
data mining, 299 empirical loss, 8
data normalization, 50, 138 empirical risk, 9
data preparation, 133, 135, 383 ensemble learning, 127
data preprocessing, 49, 133, 134 entropy, 124, 245
DBSCAN, 231 error, 281
decision tree, 122 Euclidean distance, 203
decreasing step-size sequence, 78 explained variance ratio, 155
deep learning, 261, 285
deep neural networks, 285 feasible solution, 55
deepcopy, 22 feature expansion, 119, 368
INDEX 393
feature extraction, 149 hierarchical clustering, 209, 224

feature function, 369 high bias, 100
feature importance, 146 high variance, 100
feature map, 255, 289, 290 hold-out method, 44
feature scaling, 50, 101, 134, 138 Horner’s method, 26, 33
feature selection, 140, 144, 149 horner, in Python, 27
feature selection, automatic, 141 horner.m, 28
feature selection/extraction, 306 Householder transformation, 168
feed-forward neural network, 249 hyperparameter, 49
filter methods, 141 hyperplane, 40, 363
first-order necessary conditions, 107 hypothesis formation, 298
Fisher’s LDA, 172 hypothesis space, 7
frequently_used_rules.py, 23
fully-connected networks, 285 identity function, 47
image compression, 170
Gauss-Newton method, 83 impurity measure, 123
Gaussian homotopy continuation, 66 incidence matrix, 239
Gaussian kernel, 117 indentation, 20
Gaussian sailing, 379 indeterministic issue, 250
Gaussian smoothing, 66, 379 index, 311
generalization error, 135, 140 induced matrix norm, 61
generalized eigenvalue problem, 176 inertia of matrix, 336
get_cubes.py, 25 inference, statistical, 2
Gini impurity, 124 information gain, 122, 123
GLOBAL_VARIABLES.py, 371 information matrix, 365
GMRES, 348 information retrieval, 304, 310
Golub-Reinsch SVD algorithm, 168 inheritance, 34
goodness-of-fit measure, 81 initial centroids problem, 216
Google, 320 inlinks, 321
Google matrix, 304, 325, 328 input layer, 269
Google Pagerank algorithm, 304 input neurons, 269
Google search, 320 inseparable, 44
Google search engine, 304 install Python packages, 15
gradient descent algorithm, 61, 63 instance, 30
gradient descent method, 48, 56, 60, 61, instantiation, 30
272 integer encoding, 137
Gram matrix, 115 intercept, 40
graph theory, 343 interior-point method, 353, 359
greatest curvature, 70 internal measures, 241
group average, 227 Internet search, 320
interpretability, 122, 250, 265, 362
Hadamard product, 282 inverse document frequency, 313
Hessian, 61, 67, 70 iris, 372, 378
Hessian matrix, 70, 381 Iris dataset, 53
hidden layers, 269 irreducible, 323
394 INDEX
irreducible matrix, 344 lda_c3.m, 181

lda_c3_visualize.m, 182
Jacobian matrix, 82 lda_Fisher.m, 177
K-Means clustering, 211 ldl_test.py, 338
K-Medoids algorithm, 221 learning error, 281
k-nearest neighbor, 129 learning rate, 41, 58, 255
k-nearest neighbors, 374 least curvature, 70
k-NN, 129, 373, 374 left eigenvalue, 323
Karush-Kuhn-Tucker conditions, 107 left singular vectors, 153, 156
KD-tree, 130 Levenberg-Marquardt algorithm, 81, 84
kernel function, 117, 193 likelihood, 96, 97, 373
kernel matrix, 192 likelihood function, 96
kernel method, 116 linear algebraic system, 342
kernel PCA, 189 linear classifier, 39, 368
kernel PCA, summary, 195 linear discriminant analysis, 149, 172
kernel principal component analysis, 149, linear iterative method, 342
189 linear separators, 45
kernel SVM, 116 linear SVM, 103, 107, 113
kernel trick, 117, 119, 193, 368 linearly separable, 42
keywords, 35 link graph, 305, 324
KKT approach, 107 link matrix, 305, 328
KKT conditions, 107, 332, 353 list, in Python, 21
KKT matrix, 332 Lloyd’s algorithm, 211
Knowledge Discovery in Data, 299 local minima problem, 66, 379
Kohonen map, 247, 256 local receptive field, 287
Kohonen network, 251 log-likelihood function, 97
Kohonen SOM network, 251 logarithmic barrier function, 355
Kohonen, Teuvo, 249 logistic cost function, 97
logistic curve, 91
L2 regularization, 101 logistic model, 91
L2 shrinkage, 101 Logistic Regression, 363
L2-pooling, 291 logistic regression, 95, 268
Lagrange dual function, 106 logistic sigmoid function, 91, 268
Lagrange multiplier, 143 logit, 93
Lagrange multipliers, 104, 107 loss function, 8
Lagrangian, 107, 332 LSI, 317
Lagrangian (objective), 105 Lua, 294
largest margin, 45
LASSO, 141 M-matrix, 347
latent semantic indexing, 311, 317 machine learning, 362, 373
lateral inhibition connections, 249 machine learning algorithm, 3
lateral interaction, 254 machine learning challenges, 265
law of parsimony, 10 machine learning modelcode, 12, 372
Lawrence Page, 325 Machine_Learning_Model.py, 12
lazy learner, 129 Manhattan distance, 203
INDEX 395
margin, 42 noise-removal algorithms, 373

Markov matrix, 325 nominal features, 137
matrix eigenvalue problem, 304 non-binding constraint, 111
matrix factorization, 306 nonlinear least-squares problems, 81
max-pooling, 291 nonlinear SVM, 116
maximum likelihood estimator, 96 normal vector, 103
mCLESS, 362, 365 normalization, 134, 138
mean square error, 8 normalization condition, 192
measurement error, 81 normalized web matrix, 322
medicine, 95 NP, 207
medoid, 204 NP-Complete, 207
membership weight, 209 NP-Hard, 206, 207
merit function, 65 null-space approach, 341, 359
method of Lagrange multipliers, 104, 107 numerical rank, 169
method of normal equations, 366 numerical underflow, 97
method, in Python class, 31 numpy, 18, 29
min-max scaling, 138
mini-batch, 273 object-oriented programming, 30
mini-batch learning, 52 objective function, 48, 56
minimum-volume enclosing ellipsoid, 376 observational error, 81
Minkowski distance, 130, 203 Occam’s razor principle, 10
missing data, 136 odds ratio, 93
mixture model, 210 one-hot encoding, 137
model, 300 one-shot learning, 265
model reliance, 146 one-versus-all, 46
motor output, 249 one-versus-rest, 367
multi-class least-error-square sum, 365 OOP, 30
multi-column least-squares, 366 optimality, 250
multi-line comments, 20 optimization, 55
multiple local minima problem, 66, 101 optimization problem, 55, 56
MVEE, 376 ordinal encoding, 137
ordinal features, 137
negative feedback paths, 252 orthogonal linear transformation, 150
neighborhood function, 254 orthogonally invariant, 309
neighbourhood function, 254 outlier problem, K-Means, 220
network.py, 274 outlier removal algorithm, 220
neural network, 264 outliers, 232, 373
neuron, 38 outlinks, 305, 321
Newton’s method, 26, 61, 67, 354, 381 output layer, 269
newton_horner, in Python, 27 OVA, 46
newton_horner.m, 28 overfit, 44
nltk, 311 overfitting, 100, 140
no free lunch theorem, 90 OVR, 46, 367
nodal point, 344
noise, 373 P, 207
396 INDEX
P versus NP problem, 207 python_startup.py, 19

Pagerank, 320–322
pagerank equation, 325 Q9, 310
Pagerank vector, 323 QMR, 348
pagerank vector, 306 QR method, 168
PAM algorithm, 221 quadratic programming, 115, 331
parent class, 34 quadratic programming, equality con-
partitional clustering, 208 strained, 332
PCA, 150 quadratic programming, general form,
Peppers image, 170 331, 350
peppers_SVD.m, 170 quadratic programming, inequality-
Perceptron, 363 constrained, 353, 355
perceptron, 37, 40, 266 quasi-Newton method, 381
permutation feature importance, 146, 148 quasi-Newton methods, 72
permutation matrix, 343 query matching, 315
Perron-Frobenius theorem, 324
personalization vector, 326, 329 radial basis function, 117
perturbed complementarity, 357 random decision forests, 127
polar decomposition, 154 random error, 81
Polynomial_01.py, 31 random forest, 127
Polynomial_02.py, 32 random forests, 127
pooling, 291 randomness, 100
pooling layers, 291 range, in Python, 22
Porter Stemming Algorithm, 312 range-space approach, 339
positive definite, 333 rank reduction procedure, 306
positive semidefinite, 333 rank-one matrix, 73, 176
positive semidefinite matrix, 151 rank-reduction decomposition, 307
power method, 327 Rayleigh quotient, 151
precision, 237, 315 reachable points, 232
prediction methods, 300 recall, 237, 315
preprocessing, 311 recall versus precision diagram, 316
primal active set strategy, 351 rectified linear units, 295
primal feasibility, 108 rectifier, 92
principal component analysis, 149, 150 reduced Hessian, 332
principal components, 150, 189, 375 reduced KKT system, 341
principal curvatures, 70 reducible matrix, 344
principal directions, 70 redundant constraint, 111
probabilistic clustering, 209 reference semantics, in Python, 22
probabilistic model, 95 regression, 300
projection vector, 173 regression analysis, 3
proximity matrix, 239 regression example, 7
pseudoinverse, 162, 163, 167 regular splitting, 347–349
purity, 245 regularization, 101, 143, 144
Python, 18 regularization methods, 141
Python essentials, 21 regularization strength, 101
INDEX 397
relevance, 237 sigmoid neurons, 268

ReLU, 295 silhouette coefficient, 242
remainder theorem, 26 silhouette plots, 260
repeatability, 250 similarity function, 117
representation learning, 262 similarity matrix, 192, 240
retrieving elements, in Python, 21 single linkage, 226
ridge regression, 141 singular value decomposition, 152, 153,
right eigenvalue, 323 156, 304, 317, 336, 366
right singular vectors, 153, 156 singular values, 153, 156
risk, 9 sklearn_classifiers.py, 13
Rosenbrock function, 57 slack variable, 112, 353
rotational symmetry, 92 slicing, in Python, 21
Run_network.py, 276 SMO, 109, 120
Run_network3.py, 296 soft-margin classification, 112
SoftmaxLayer, 295
scale-invariant feature transform, 262 softplus function, 92
scatter, 174 SOM, 247
Schur complement, 339, 358 source matrix, 366
Schur product, 282 spaCy, 311
scipy, 18 sparse, 314
score matrix, 151 sparse eigenvalue algorithms, 327
search direction, 48 sparse model, 144
search engine, 304 sparsity, 144
secant condition, 86 SPD, 349
secant equation, 73 spectral radius, 343
self, 31 spectrum, 343
self-organizing map, 247 Speed up Python Programs, 18
self-referencing, 320 SSE, 85, 210
semidefinite, 333 standard logistic sigmoid function, 91
sensitivity, 237 standard normal distribution, 138
sequential backward selection, 142 standardization, 50, 101, 138
Sequential Minimal Optimization, 109 statistical inference, 2
sequential minimal optimization, 120 steepest descent, 58
sequential pattern discovery, 300 steepest descent method, 56
Sergey Brin, 325 stemming, 312
set of blocking constraints, 352 step length, 48, 58, 99
SGD, 51, 76 stochastic gradient descent, 49, 51, 76, 77,
SGM, 76 273
shared bias, 289 stochastic gradient method, 76
shared weights, 289 stochastic matrix, 325
Sherman-Morrison formula, 73 stochastic SOM, 256
SIFT, 262 stop words, 312
sigmoid function, 91, 268 stride length, 288
sigmoid neural networks, 268 string, in Python, 21
sigmoid neuron, 267 strongly connected, 323, 345
398 INDEX
sufficient decrease condition, 63 transforming range-space iteration, 348,

sum of squared error, 85 349
sum of squared errors, 48, 210 translation invariance, 289, 293
sum-of-squares objective function, 83 truncated data, 153
sum-squared-error, 96 truncated score matrix, 154, 189, 375
summary of SVM, 115 truncated transformation, 153
supervised learning, 6 tuple, in Python, 21
support vector machine, 45, 102, 368 Turing machine, 207
support vectors, 110 two-class classification, 38
surrogate function, 65
SVD, 304, 366 unbiasedness property, 80
SVD theorem, 156 underfitting, 100
SVD, algebraic interpretation, 158 unit round-off error, 169
SVD, geometric interpretation, 160 unsupervised learning, 11, 200
SVM, 368 upper bidiagonal matrix, 168
SVM summary, 115 variable selection, 140
SVM, nonlinear, 116 variance, 100
Sylvester’s law of inertia, 336 vector space model, 311
symmetric factorization, 337 vectorization, 52
symmetric positive definite, 349 Voronoi iteration, 211
synthetic division, 26
synthetic_data.py, 370 Ward’s method, 230
systematic error, 81, 100 Ward’s minimum variance, 227
weather forecasting, 95
Taylor series expansion, 83 web search engines, 310
Taylor’s Theorem, with integral remain- Wedderburn matrices, 307
der, 59 Wedderburn rank reduction theorem, 306
teleportation, 326 Wedderburn rank-reduction process, 307
term, 302 weight decay, 101
term frequency, 313 weight matrix, 365
term weighting scheme, 313 William of Occam, 10
term-document matrix, 302, 303, 311, 313 wine, 372, 378
terms, 311 winner-takes-all neuron, 249, 252
text mining, 310 winning neuron, 249, 252
textmineR, 311 within-class scatter, 174, 175
the test of time award, 231 within-class scatter matrix, 175, 178
Tikhonov regularization, 141 within-class variability, 174
topographic neighborhood, 255 wrapper methods, 141
topological neighbourhood, 254
transformation matrix, 152 Zeros-Polynomials-Newton-Horner.py, 27
transforming null-space iteration, 348 zeros_of_poly_built_in.py, 27

Machine Learning Lecture-Notes

Uploaded by

Document Informationclick to expand document information

Copyright:

Available Formats

Machine Learning Lecture-Notes

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Machine Learning Lecture-Notes

Uploaded by

Copyright:

Available Formats

Mathematical Foundations

Department of Mathematics and Statistics

Updated: April 30, 2023

In organizing this lecture note, I am indebted by the following:

• S. R ASCHKA AND V. M IRJALILI, Python Machine Learning, 3rd Ed., 2019

3 Simple Machine Learning Algorithms for Classification 37

4 Gradient-based Methods for Optimization 55

5 Popular Machine Learning Classifiers 89

6 Data Preprocessing in Machine Learning 133

6.1. General Remarks on Data Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . 134

7 Feature Extraction: Data Compression 149

8 Cluster Analysis 199

8.3. Hierarchical Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224

9 Neural Networks and Deep Learning 261

10 Data Mining 297

11 Quadratic Programming 331

11.1.Equality Constrained Quadratic Programming . . . . . . . . . . . . . . . . . . . . . . . 332

What are we “learning” in Machine Learning (ML)?

• Algorithms, which solve some kinds of inference problems

1.1. Why and What in Machine Learning (ML)?

Loosely speaking, inference problems take in data, then output

Example 1.2. Inference algorithms can answer the following.

(a) Does this image have a tree in it?

(b) What words are in this picture?

Remark 1.3. What does a machine learning algorithm do?

Given a dataset, how can I succinctly describe it (in a quantita-

• Geometric models. The general problem is that we have example

Note: In both cases above, having a concise model can go a tremen-

1.1.3. Machine learning examples

1.2. Three Different Types of ML

• Supervised learning: classification, regression

Figure 1.1: Three different types of ML (by methods)

1.2.1. Supervised learning

Assumption. Given a data set {(xi , yi )}, ∃ a relation f : X → Y .

Figure 1.2: Supervised learning and prediction.

Figure 1.3: Classification and regression.

Example 1.5. Regression

• Wanted: Find a regression model

– We measure the error using a loss function L(y, yb).

L(yi , f (xi )) = ((yi − f (xi ))2 .

– The empirical loss of the function f applied to the training data is

Figure 1.4: Learning curve.

(xi , yi ) ∼ p(x, y).

• Then, given a loss function L:

f ∗ (DN ) = arg min R(f,

Remark 1.7. Key Issues in Machine Learning

Proposition 1.8. Occam’s Razor Principle (a.k.a. Law of parsimony):

“One should not increase, beyond what is necessary,

• William of Occam: A monk living in the 14-th century, England

1.2.2. Unsupervised learning

Figure 1.5: Clustering.

1.3. A Machine Learning Modelcode

28 rtrain = 0.7e0; run = 100

Exercises for Chapter 1

(a) Save the code written in two files, by copy-and-paste.

Installation: If you are on Ubuntu, you may begin with

2.1. Why Python?

Remark 2.1. Speed up Python Programs.

6 def init(self): #constructor

11 def del(self): #destructor