SEBASTIAN RASCHKA

Introduction to
Artificial Neural Networks
and Deep Learning
with Applications in Python
 Introduction to Artificial
Neural Networks
with Applications in Python

Sebastian Raschka

D RAFT
Last updated: May 31, 2018

This book will be available at http://leanpub.com/ann-and-deeplearning.

Please visit https://github.com/rasbt/deep-learning-book for more

information, supporting material, and code examples.

© 2016-2018 Sebastian Raschka

Contents

B Algebra Basics 4
B.1 Exponents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
B.1.1 Basic Principles . . . . . . . . . . . . . . . . . . . . . . 5
B.1.2 Laws of Exponents . . . . . . . . . . . . . . . . . . . . 5
B.1.3 Roots . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
B.2 Properties of Logarithms . . . . . . . . . . . . . . . . . . . . . 6
B.3 Order of Operations . . . . . . . . . . . . . . . . . . . . . . . 8
B.3.1 Commutative, Associative, and Distributive Laws . . 8
B.3.2 Removing Parentheses and Expanding . . . . . . . . 9
B.3.3 Polynomials . . . . . . . . . . . . . . . . . . . . . . . . 11
B.4 Summation and Products of Sequences . . . . . . . . . . . . 12
B.5 Algebra of Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
B.5.1 Set Notation . . . . . . . . . . . . . . . . . . . . . . . . 12
B.5.2 Set Theory . . . . . . . . . . . . . . . . . . . . . . . . . 13
B.5.3 Interval Notation . . . . . . . . . . . . . . . . . . . . . 14
B.6 Domain and Range of a Function . . . . . . . . . . . . . . . . 15
B.7 Inverse Functions . . . . . . . . . . . . . . . . . . . . . . . . . 16
B.8 Function Transformations . . . . . . . . . . . . . . . . . . . . 18

i
Website

Please visit the GitHub repository to download the code examples accom-
panying this book and other supplementary material.
If you like the content, please consider supporting the work by buy-
ing a copy of the book on Leanpub. Also, I would appreciate hearing
your opinion and feedback about the book, and if you have any ques-
tions about the contents, please don’t hesitate to get in touch with me via
mail@sebastianraschka.com. Happy learning!

Sebastian Raschka

1
About the Author

Sebastian Raschka received his doctorate from Michigan State University

developing novel computational methods in the field of computational bi-
ology. In summer 2018, he joined the University of Wisconsin–Madison
as Assistant Professor of Statistics. Among others, his research activities
include the development of new deep learning architectures to solve prob-
lems in the field of biometrics. Among his other works is his book "Python
Machine Learning," a bestselling title at Packt and on Amazon.com, which
received the ACM Best of Computing award in 2016 and was translated
into many different languages, including German, Korean, Italian, tradi-
tional Chinese, simplified Chinese, Russian, Polish, and Japanese.
Sebastian is also an avid open-source contributor and likes to contribute
to the scientific Python ecosystem in his free-time. If you like to find more
about what Sebastian is currently up to or like to get in touch, you can find
his personal website at https://sebastianraschka.com.

2
Acknowledgements

I would like to give my special thanks to the readers, who provided feed-
back, caught various typos and errors, and offered suggestions for clarify-
ing my writing.

• Appendix A: Artem Sobolev, Ryan Sun

• Appendix B: Brett Miller, Ryan Sun

• Appendix D: Marcel Blattner, Ignacio Campabadal, Ryan Sun, Denis

Parra Santander

• Appendix F: Guillermo Monecchi, Ged Ridgway, Ryan Sun, Patric

Hindenberger

• Appendix H: Brett Miller, Ryan Sun, Nicolas Palopoli, Kevin Zakka

DRAFT 3
Appendix B

Algebra Basics

In this appendix, we will refresh a subset of the fundamental Algebra con-

cepts that are relevant for following the mathematics behind the machine
learning algorithms presented in this book. While this appendix aims to
bring you up to speed regarding Algebra Basics, please bear in mind that it
is not a comprehensive algebra resource. Thus, I recommend you to consult
a more comprehensive resource, such as Beginning and Intermediate Alge-
bra1 by Tyler Wallace [Wallace, 2010], if you wish to study algebra in more
depth.
What is Algebra? Algebra is a subfield of mathematics that uses numbers
and letters (or symbols) in consent to describe relationships. Algebra is
often used to solve problems where at least one quantity is unknown or
may vary (we call these variables). In simple terms, we can think of algebra
as a language that combines letters, symbols, and numbers, to generalize
arithmetic operations and to solve equations with variables.

B.1 Exponents
Almost every equation or function in computer science and engineering in-
volves exponents. And based on the assumption that the vast majority of
readers have worked with exponents before, this section provides a sum-
mary of the most important concepts in tabular form, which is intended to
serve as a refresher as well as a look-up reference.
1
http://www.wallace.ccfaculty.org/book/book.html

DRAFT 4
APPENDIX B. ALGEBRA BASICS 5

B.1.1 Basic Principles

Rule Example
1 x0 = 1 50 = 1
2 x1 = x 51 = 5
3 xn = x · x · x · . . . · x 53 = 5 · 5 · 5 = 125
4 x−n = 1
xn 5−2 = 1 1
= 25
52
√ √
5 x1/2 = x 251/2 = 25 = 5
√ √
6 x1/n = n x 1251/3 = 3 125 = 5
√ √
3
7 xm/n = n xm 1252/3 = 1252 = 25

Table B.1: Basic Algebra Principles

B.1.2 Laws of Exponents

The following table lists the fundamental laws of exponents that we can
apply to manipulate algebraic equations:

Law Example
1 xm · xn = xm+n 53 · 52 = 55 = 3125
53
2 xm /xn = xm−n 52
= 51 = 5

n
3 xm = xm·n (52 )3 = 56 = 15, 625
4 (xy)n = xn · y n (2 · 5)2 = 22 · 52 = 100
x n xn 5 2 52 25




5 y = yn 2 = 22
= 4
x −n y
n 5 −2 2 2 22 4





6 y = x 2 = 5 = 52
= 25

Table B.2: Exponent Laws

While derivations of the exponent rules are beyond the scope of this al-
gebra refresher, notice that these rules are all closely related to each other,
and they are based on the basic principles that were presented in the previ-
ous section. For instance, while x0 = 1 may seem a bit unintuitive at first,

DRAFT
APPENDIX B. ALGEBRA BASICS 6

we can demonstrate why this relation is true by applying the second rule
from the Table B.2:

xn
1= = xn−n = x0 . (B.1)
xn

B.1.3 Roots
This section lists the basic concepts and rules regarding roots. The root of a
number x is a number that, if multiplied by itself a certain number of times,
√
equals x. Written in a different form, the equation m x = y is true if and only
if y m = x, where m is the root index and x is the radicand.

Equation Example
√ √ √ √
1 ( x)2 = x2 = x ( 4)2 = 42 = 4
for all positive numbers
√ √ √ √ √ √ √
2 x·y = x· y 36 = 9·4= 9· 4=3·2=6
√ q √
√x
q
x 9 √9 3
3 y = y 16 = 16
= 4
√ √ √ √
4 x + y 6= x+ y 16 + 9 6= 4 + 3
p
x2 + y 2 6= x + y

Table B.3: Properties of Logarithms

B.2 Properties of Logarithms

Logarithms are closely related to exponents; given the same base, the loga-
rithm is the inverse of the exponential function and vice versa. For instance,
if we are given an equation 2x = 9, we can obtain the value of the variable
x by applying the base 2 logarithm:

log2 (2x ) = x. (B.2)

In a more general form, we can express this relationship as follows:

logb (a) = c ⇐⇒ bc = a, (B.3)

where a is the power, b is the base, and c is the exponent.

DRAFT
APPENDIX B. ALGEBRA BASICS 7

The following table summarizes the most important properties of loga-

rithms that we use in practice:

Property Example
1 logb 1 = 0 log2 1 = 0
2 logb b = 1 log2 2 = 1
3 logb ba = a log2 23 = 3
4 blogb a = a 2log2 8 = 23 = 8
5 logb (m · n) = logb m + logb n log2 (2 · 4) = log2 2 + log2 4 = 1 + 2 = 3
m 4



6 logb n = logb m − logb n log2 2 = log2 4 − log2 2 = 2 − 1 = 1
7 logb (mp ) = p logb m log2 (23 ) = 3 log2 2 = 3
loga m log10 8
8 logb m = loga b log2 8 = log10 2 =3

Table B.4: Properties of Logarithms

Also here, we can derive the different properties from each other. For
example, the last property (row 8) in Table B.4 can be obtained as follows,
starting with the basic properties of logarithms:

logb m = c
(B.4)
bc = m.

Taking the desired logarithm on both sides, we have

loga bc = loga m. (B.5)

Next, we move the exponent outside, using rule 7 in Table B.4:

c loga b = loga m. (B.6)

Then, we divide by loga b and simplify:

loga b loga m
c =
loga b loga b
(B.7)
loga m
c= .
loga b

DRAFT
APPENDIX B. ALGEBRA BASICS 8

Finally, we resubstitute c from Equation B.4 to arrive at

loga m
logb m = . (B.8)
loga b

Infobox B.2.1 Natural Logarithms

Throughout this book, we will mostly work with natural logarithms –

logarithms to the base of the mathematical constant e. When we are
working with such natural logarithms, we will follow the common con-
vention and omit writing the base of the logarithm explicitly. For ex-
ample, instead of writing loge (ex ) = x, we will use the simpler notation
log(ex ) = x.

B.3 Order of Operations

This section provides a brief overview of the basic laws of algebra, which
are based on the following order of operations:

1. Perform all calculations inside parentheses.

2. Perform multiplication and division operations from left to right.

3. Perform addition and subtraction from left to right.

B.3.1 Commutative, Associative, and Distributive Laws

The commutative law says that we can disregard the order of two real num-
bers if we perform operations such as addition or multiplication:

x+y =y+x (B.9)

and

x · y = y · x. (B.10)
For example, 5 + 2 = 2 + 5 and 5 · 2 = 2 · 5.
Per associative law, we can disregard the grouping of numbers in opera-
tions such as addition and multiplication:

(x + y) + z = x + (y + z) (B.11)

DRAFT
APPENDIX B. ALGEBRA BASICS 9

and

(x · y) · z = x · (y · z). (B.12)
For example, (5 + 2) + 3 = 5 + (2 + 3) and (5 · 2) · 3 = 5 · (2 · 3).
The distributive law is extremely useful for multiplying and factoring
algebraic expressions. With respect to multiplication and addition, the dis-
tributive law says that

x · (y + z) = x · y + x · z, (B.13)
for example,

5 · (3 + 2) = 5 · 3 + 5 · 2
= 15 + 10 (B.14)
= 25.

Notice that this is similar to performing the addition operation inside

the parentheses first, for instance,

5 · (3 + 2) = 5 · 5
(B.15)
= 25.

Infobox B.3.1 Notation

In practice, we often omit the center "dot" symbol (·) as multiplication
operator whenever it is obvious from the context. For example, we
could write 2x instead of 2 · x or 5(3 + 2) instead of 5 · (3 + 2).

B.3.2 Removing Parentheses and Expanding

If an expression, enclosed by parentheses, is preceded by a plus sign, we
can simply remove the parentheses:

3 + (5x − 2y + 4) = 3 + 5x − 2y + 4
(B.16)
= 5x − 2y + 7.

DRAFT
APPENDIX B. ALGEBRA BASICS 10

However, if there is a minus sign in front of the opening parenthesis, we

have to be more careful and reverse addition and subtraction inside the
parentheses:

3 − (5x − 2y + 4) = 3 − (−1)(5x − 2y + 4)
= 3 − 5x + 2y − 4 (B.17)
= −5x + 2y − 1.

As we have seen in the previous subsection, we can get rid of paren-

theses by using the distributive law, which says that we have to multiply a
factor onto everything inside the parentheses:

5(3x + 7)
(B.18)
= 15x + 35.

Of course, the same is true for division, since division is simply multiplica-
tion by the reciprocal:

1
5/(5x + 10) = (5x + 10)
5 (B.19)
=x+2

Now, if we are multiplying two expressions containing variables, we can

simply use the distributive law and distribute the terms of the first expres-
sion over the second:

(a + b)(c + d) = a(c + d) + b(c + d)

(B.20)
= ac + ad + bc + bd

For example,

(2x − 3)(3x + 4) = 2x(3x + 4) − 3(3x + 4)

= 6x2 + 8x − 9x + 12 (B.21)
2
= 6x − x + 12.

DRAFT
APPENDIX B. ALGEBRA BASICS 11

B.3.3 Polynomials
At the end of the previous section, we multiplied two binomials – binomials
are polynomials with two terms. A polynomial is an expression that contains
multiple terms (poly means "many" in Greek).
Polynomials may consist of constants, variables, and exponents, but
they never divide a term by a variable. Thus, polynomials also have no
√
negative exponents. For example, x = x1/2 is not a polynomial, and
x− 2 = x12 is also not a polynomial. Finally, the following is also not a
polynomial because we cannot simplify it further:

15x2 + 10x + 1 1
= 3x + 2 + . (B.22)
5x 5x
In standard form, we write a polynomial as a combination of monomials
sorted by their degree. For instance, the following polynomial, written in
standard form, has three terms:

4x3 + 2x2 + y. (B.23)

The degree of a polynomial is defined by the degree of its highest term, and
the degree of the highest term is defined by the sum of its exponents. For
instance, the polynomial 4x3 + 2x2 + y has degree 3. However, note that the
following polynomial has degree 7, since 3 + 4 = 7:

4x3 y 4 + 2x2 + y. (B.24)

In general form, we can express a univariate nth-degree polynomial (a
polynomial containing only a single variable) as follows:

an xn + an−1 xn−1 + ... + a2 x2 + a1 x + a0 , (B.25)

where an , ...a0 are coefficients. In practice, however, we will mostly work
with polynomials with a degree between 0 and 3:

Degree Common Name Example

0 constant 2x0 = 2
1 linear 2x + 1
2 quadratic 2x2 + 3x + 1
3 cubic 2x3 + 5x2 + x − 5

Table B.5: Polynomials

DRAFT
APPENDIX B. ALGEBRA BASICS 12

B.4 Summation and Products of Sequences

In the previous section, we expressed a univariate nth-degree polynomial
as

an xn + an−1 xn−1 + ... + a2 x2 + a1 x + a0 . (B.26)

In practice, when we are expressing sequences with an indexed variable,
P
we use the summation symbol for our convenience. For instance, we
could rewrite the previous expression as
n
X
ai xi , (B.27)
i=0

where i is the index of summation, and n is the upper bound of summation. For
a simpler example, consider the following:
n
X
xi = x1 + x2 + ... + xn . (B.28)
i=1
Q
Similarly, we use the Capital Pi Notation, , to express products of sequences
with an index variable more concisely. For example,
n
Y
xi = x1 · x2 · . . . · xn . (B.29)
i=1

B.5 Algebra of Sets

Intuitively, a set represents a collection of different and unique elements. In
this section, we will go over basic set theory, discussing the properties and
laws of such sets.

B.5.1 Set Notation

There are several different notations for defining sets; the three most com-
mon ones are listed below:

1. Verbal notation: Let S be the set of letters in the word "algebra."

2. Roster notation: {a, l, g, e, b, r}

3. Set-builder notation: {x | x is a letter in "algebra"}

DRAFT
APPENDIX B. ALGEBRA BASICS 13

Although, the different set notations can be used equivalently, it is some-

times more convenient or appropriate to use one notation over the other.
For example, we can define the set of integers as

Z = {. . . , −2, −1, 0, 1, 2, . . . }, (B.30)

while we write the domain of all positive real numbers as

R>0 = {x ∈ R | x > 0}. (B.31)

(The symbol ∈ means "element of." For example, x ∈ A translates to "x is
an element of set A.")

B.5.2 Set Theory

Now, using the set-builder notation, we can summarize the basic set oper-
ations as follows:

• Union: A ∪ B = {x | x ∈ A or x ∈ B, or both}

• Intersection: A ∩ B = {x | x ∈ A and x ∈ B}

• Difference: A − B = {x | x ∈ A and x ∈
/ B}

• Symmetric difference: A∆B = {x | x ∈

/ A ∩ B}

• Complement: AC = {x | x ∈
/ A}

Figure B.1 provides a visual summary of these operations, given the two
sets A and B, and the universal set Q – a set which contains all possible
elements, including itself.

DRAFT
APPENDIX B. ALGEBRA BASICS 14

Figure B.1: Set Theory

B.5.3 Interval Notation

Another concept related to set theory is the interval notation, which is es-
pecially handy when we are describing real-valued functions. Or in other
words, the interval notation is essentially just a less complicated form of the
set-builder notation when we are working with real numbers and intervals:

Set of Real Numbers Interval Notation

{x ∈ R|a < x < b} (a, b)
{x ∈ R|a ≤ x < b} [a, b)
{x ∈ R| ≤ a} (−∞, a]
{x ∈ R| ≥ a} [a, ∞)

Table B.6: Interval Notation

DRAFT
APPENDIX B. ALGEBRA BASICS 15

B.6 Domain and Range of a Function

Now that we have introduced all the essential notation, we can finally dis-
cuss functions. Intuitively, a function is a relationship between a set of in-
puts and outputs. We can think of a function f as a box that takes in one
input and returns one output, f : x → y.
For a valid function, we require that one input is mapped to one output;
a one-to-many mapping is not allowed. In other words, a function must
always return the same value for a specific input. This is also called one-to-
one mapping. However, note that a function is also valid under the many-
to-one condition; for example, consider the function f (x) = x2 . Here, both
input values x = −2 and x = 2 produce the same output: f (−2) = f (2) =
4.
The domain of a function is defined as the set of all possible input values
for which the function is defined, and the codomain is the set of values that
it returns. For practical reasons, since we often do not have full knowledge
of the codomain, we use the term range (or image), which defines the actual
set of values produced by the function. So, while the codomain is part of
the function definition and defines what possibly comes out, the range is a
subset of the codomain. The following figure provides a visual summary
of these terms given the function f (x) = 2x:

x 1
f f(x)
1 15
2 3 Codomain
2 13
4
3
6 5
4
8
5
10 7 11

Domain Range or Image

Figure B.2: Concept of a function

To better illustrate the notion of a domain, let us take a look at an ex-

DRAFT
APPENDIX B. ALGEBRA BASICS 16

ample and consider the following rational function (a rational function is a

function that consists of the fractional ratio of two polynomials):
√
3−x
f (x) = . (B.32)
x−2
Using interval notation, we define the domain of this function as

(−∞, 2) ∪ (2, 3], (B.33)

since we cannot perform a division by zero, and we cannot take the root of
a negative number. The equivalent set-builder notation of the domain is:

{x ∈ R | x < 2 or 2 < x <= 3}. (B.34)

B.7 Inverse Functions

Before we take a look at inverse function, let us briefly introduce another
important concept that we use frequently in machine learning: function
composition, that is, nesting of two or more functions:

(f ◦ g)(x) = f (g(x)). (B.35)

Now, let g be the inverse function of another function f , so that

• (f ◦ g)(x) = x for all x in the domain of f

• (g ◦ f )(x) = f for all x in the domain of g

Intuitively, the composition of a function and its inverse is equivalent to

the identity function I, where I(x) = x. For example, if we define f (x) =
2x, the inverse function g must be defined so that it satisfies the equation
(f ◦ g)(2) = 2. Thus, g(x) = x2 .
In practice, we use the superscript notation to denote a function inverse,
for example, f −1 . This notation may look confusing at first, and the origin
of this notation may be due to the fact that

f −1 ◦ f = f ◦ f −1 = I (B.36)
looks similar to the following equation:
1
x−1 · x = x · x−1 = x · = 1. (B.37)
x

DRAFT
APPENDIX B. ALGEBRA BASICS 17

However, notice that superscript (−1) does not mean that we take the func-
tion f to the power of (−1):
1
f −1 (x) 6= . (B.38)
f (x)
A good example of inverse functions are exponential and logarithmic func-
tions: logarithms are the inverse of exponential functions and vice versa.
For instance, consider the following exponential function

f (x) = bx , (B.39)
where its inverse is the base b logarithm defined as

f −1 (x) = logb (x). (B.40)

So, we can say that the domain of logarithmic functions is the range of
exponential functions (0, ∞), and the range of logarithmic functions is the
domain of exponential functions (−∞, ∞):

Figure B.3: Domain and range of logarithmic and exponential functions

DRAFT
APPENDIX B. ALGEBRA BASICS 18

B.8 Function Transformations

Lastly, this section introduces the basics of function transformation, which is
a concept that will give us some intuition of how we can modify a function
to our needs. Assume we are given a basis function f (x), and it is decorated
by the scalars a, b, c, and d in the following way:

af (b(x + c)) + d. (B.41)

• a: vertical stretch by a factor of a if a > 1; compression if 0 < a < 1;

flip the graph if a is negative.

• b: horizontal shrinking by a factor of b if b > 1; stretching if 0 < b < 1;

flip the graph if b is negative.

• c: horizontal shift upwards if c is positive; otherwise, shift the func-

tion graph downwards.

• d: vertical shift to the left if d is positive; otherwise, shift the function

graph to the right.

The following above-mentioned concepts are illustrated in the figures be-

low:

DRAFT
APPENDIX B. ALGEBRA BASICS 19

Figure B.4: Function transformations

DRAFT
Bibliography

[Wallace, 2010] Wallace, T. (2010). Beginning and intermediate algebra. Cre-

ative Commons Attribution.

DRAFT 20
Abbreviations and Terms

AMI [Amazon Machine Image]

API [Application Programming Interface]
CNN [Convolutional Neural Network]

DRAFT 21
Index

DRAFT
22