Week 5 Lecture Notes

ML:Neural Networks: Learning

Cost Function
Let's rst de ne a few variables that we will need to use:

a) L= total number of layers in the network

b) sl = number of units (not counting bias unit) in layer l

c) K= number of output units/classes

Recall that in neural networks, we may have many output nodes. We denote hΘ (x)k as being a hypothesis that results in the kth output.

Our cost function for neural networks is going to be a generalization of the one we used for logistic regression.

Recall that the cost function for regularized logistic regression was:

1 m (i) (i) (i) (i) λ n 2

J (θ) = − ∑ [y   log(hθ (x )) + (1 − y )  log(1 − hθ (x ))] + ∑ θj
m i=1 2m j=1

For neural networks, it is going to be slightly more complicated:

m K L−1 sl sl+1
1 (i) (i) λ (l) 2
(i) (i)
J (Θ) = − ∑ ∑[y log((hΘ (x )) ) + (1 − y ) log(1 − (hΘ (x )) )] + ∑ ∑ ∑(Θ )
k k k k j,i
m 2m
i=1 k=1 l=1 i=1 j=1

We have added a few nested summations to account for our multiple output nodes. In the rst part of the equation, between the square brackets,
we have an additional nested summation that loops through the number of output nodes.

In the regularization part, after the square brackets, we must account for multiple theta matrices. The number of columns in our current theta
matrix is equal to the number of nodes in our current layer (including the bias unit). The number of rows in our current theta matrix is equal to the
number of nodes in the next layer (excluding the bias unit). As before with logistic regression, we square every term.

Note:

the double sum simply adds up the logistic regression costs calculated for each cell in the output layer; and

the triple sum simply adds up the squares of all the individual Θs in the entire network.

the i in the triple sum does not refer to training example i

Backpropagation Algorithm
"Backpropagation" is neural-network terminology for minimizing our cost function, just like what we were doing with gradient descent in logistic
and linear regression.

Our goal is to compute:

minΘ J (Θ)

That is, we want to minimize our cost function J using an optimal set of parameters in theta.

In this section we'll look at the equations we use to compute the partial derivative of J(Θ):

∂
J (Θ)
(l)
∂Θ
i,j

In back propagation we're going to compute for every node:

= "error" of node j in layer l

(l)
δ
j

Recall that aj is activation node j in layer l.

(l)
For the last layer, we can compute the vector of delta values with:

(L) (L)
δ = a −y

Where L is our total number of layers and a(L) is the vector of outputs of the activation units for the last layer. So our "error values" for the last
layer are simply the di erences of our actual results in the last layer and the correct outputs in y.

To get the delta values of the layers before the last layer, we can use an equation that steps us back from right to left:

(l) (l) T (l+1) ′ (l)

δ = ((Θ ) δ ) . ∗ g (z )

The delta values of layer l are calculated by multiplying the delta values in the next layer with the theta matrix of layer l. We then element-wise
multiply that with a function called g', or g-prime, which is the derivative of the activation function g evaluated with the input values given by z(l).

The g-prime derivative terms can also be written out as:

′
g (u) = g(u) . ∗ (1 − g(u))

The full back propagation equation for the inner nodes is then:

(l) (l) T (l+1) (l) (l)

δ = ((Θ ) δ ) . ∗ a  . ∗ (1 − a )

A. Ng states that the derivation and proofs are complicated and involved, but you can still implement the above equations to do back propagation
without knowing the details.

We can compute our partial derivative terms by multiplying our activation values and our error values for each training example t:

∂J (Θ) (t)(l) (t)(l+1)

1 m
= ∑ a δ
m t=1 j i
(l)
∂Θ
i,j

This however ignores regularization, which we'll deal with later.

Note: δ and al+1 are vectors with sl+1 elements. Similarly,  a(l) is a vector with sl elements. Multiplying them produces a matrix that is sl+1 by sl
l+1

which is the same dimension as Θ (l) . That is, the process produces a gradient term for every element in Θ (l) . (Actually, Θ (l) has sl + 1 column, so
the dimensionality is not exactly the same).

We can now take all these equations and put them together into a backpropagation algorithm:

Back propagation Algorithm

Given training set {(x(1) , y (1) ) ⋯ (x(m) , y (m) )}

Set Δi,j := 0 for all (l,i,j)

(l)

For training example t =1 to m:

Set a(1) := x
(t)

Perform forward propagation to compute a(l) for l=2,3,…,L

Using y (t) , compute δ

(L) (L) (t)
= a −y

Compute δ using δ
(L−1) (L−2) (2) (l) (l) T (l+1) (l) (l)
,δ ,…,δ = ((Θ ) δ ) . ∗ a  . ∗ (1 − a )

or with vectorization, Δ
(l) (l) (l) (l+1) (l) (l) (l+1) (l) T
Δ := Δ +a δ := Δ +δ (a )
i,j i,j j i

1
If j≠0 NOTE: Typo in lecture slide omits outside parentheses. This version is correct.
(l) (l) (l)
D := (Δ + λΘ )
i,j i,j i,j
m

1
If j=0
(l) (l)
D := Δ
i,j i,j
m

The capital-delta matrix is used as an "accumulator" to add up our values as we go along and eventually compute our partial derivative.

The actual proof is quite involved, but, the Di,j terms are the partial derivatives and the results we are looking for:
(l)

(l)
∂J (Θ)
D = .
i,j
(l)
∂Θ
i,j

Backpropagation Intuition
The cost function is:

m K L−1 sl s l +1
1 (t) (t) λ (l) 2
(t) (t)
J (θ) = − ∑ ∑ [y   log(hθ (x )) + (1 − y )  log(1 − hθ (x ) )] + ∑ ∑ ∑ (θ )
k k k k j,i
m 2m
t=1 k=1 l=1 i=1 j=1

If we consider simple non-multiclass classi cation (k = 1) and disregard regularization, the cost is computed with:

(t) (t) (t) (t)

cost(t) = y   log(hθ (x )) + (1 − y )  log(1 − hθ (x ))

More intuitively you can think of that equation roughly as:

(t) (t) 2
cost(t) ≈ (hθ (x ) − y )

Intuitively, δj is the "error" for aj (unit j in layer l)

(l) (l)

More formally, the delta values are actually the derivative of the cost function:

(l) ∂
δ = cost(t)
j
(l)
∂z
j

Recall that our derivative is the slope of a line tangent to the cost function, so the steeper the slope the more incorrect we are.

Note: In lecture, sometimes i is used to index a training example. Sometimes it is used to index a unit in a layer. In the Back Propagation Algorithm
described here, t is used to index a training example rather than overloading the use of i.

Implementation Note: Unrolling Parameters

With neural networks, we are working with sets of matrices:

(1) (2) (3)

Θ ,Θ ,Θ ,…

(1) (2) (3)

D ,D ,D ,…

In order to use optimizing functions such as "fminunc()", we will want to "unroll" all the elements and put them into one long vector:

1 thetaVector = [ Theta1(:); Theta2(:); Theta3(:); ]

2 deltaVector = [ D1(:); D2(:); D3(:) ]

If the dimensions of Theta1 is 10x11, Theta2 is 10x11 and Theta3 is 1x11, then we can get back our original matrices from the "unrolled" versions as
follows:

1 Theta1 = reshape(thetaVector(1:110),10,11)
2 Theta2 = reshape(thetaVector(111:220),10,11)
3 Theta3 = reshape(thetaVector(221:231),1,11)
4

NOTE: The lecture slides show an example neural network with 3 layers. However, 3 theta matrices are de ned: Theta1, Theta2, Theta3. There
should be only 2 theta matrices: Theta1 (10 x 11), Theta2 (1 x 11).

Gradient Checking
Gradient checking will assure that our backpropagation works as intended.

We can approximate the derivative of our cost function with:

∂ J (Θ + ϵ) − J (Θ − ϵ)
J (Θ) ≈
∂Θ 2ϵ

With multiple theta matrices, we can approximate the derivative with respect to Θ j as follows:

∂ J (Θ 1 , … , Θ j + ϵ, … , Θ n ) − J (Θ 1 , … , Θ j − ϵ, … , Θ n )
J (Θ) ≈
∂Θ j 2ϵ

A good small value for ϵ (epsilon), guarantees the math above to become true. If the value be much smaller, may we will end up with numerical
problems. The professor Andrew usually uses the value ϵ = 10−4 .

We are only adding or subtracting epsilon to the T hetaj matrix. In octave we can do it as follows:
 1 epsilon = 1e-4;
2 for i = 1:n,
3 thetaPlus = theta;
4 thetaPlus(i) += epsilon;
5 thetaMinus = theta;
6 thetaMinus(i) -= epsilon;
7 gradApprox(i) = (J(thetaPlus) - J(thetaMinus))/(2*epsilon)
8 end;
9

We then want to check that gradApprox ≈ deltaVector.

Once you've veri ed once that your backpropagation algorithm is correct, then you don't need to compute gradApprox again. The code to compute
gradApprox is very slow.

Random Initialization
Initializing all theta weights to zero does not work with neural networks. When we backpropagate, all nodes will update to the same value
repeatedly.

Instead we can randomly initialize our weights:

Initialize each Θ ij to a random value between[−ϵ, ϵ]:

(l)

√6
ϵ = −−−−−−−−−−−−−−
√Loutput + Linput

(l)
Θ = 2ϵ rand(Loutput, Linput + 1) − ϵ

1 If the dimensions of Theta1 is 10x11, Theta2 is 10x11 and Theta3 is 1x11.

2
3 Theta1 = rand(10,11) * (2 * INIT_EPSILON) - INIT_EPSILON;
4 Theta2 = rand(10,11) * (2 * INIT_EPSILON) - INIT_EPSILON;
5 Theta3 = rand(1,11) * (2 * INIT_EPSILON) - INIT_EPSILON;
6

rand(x,y) will initialize a matrix of random real numbers between 0 and 1. (Note: this epsilon is unrelated to the epsilon from Gradient Checking)

Why use this method? This paper may be useful: https://web.stanford.edu/class/ee373b/nninitialization.pdf

Putting it Together
First, pick a network architecture; choose the layout of your neural network, including how many hidden units in each layer and how many layers
total.

Number of input units = dimension of features x(i)

Number of output units = number of classes

Number of hidden units per layer = usually more the better (must balance with cost of computation as it increases with more hidden units)

Defaults: 1 hidden layer. If more than 1 hidden layer, then the same number of units in every hidden layer.

Training a Neural Network

1. Randomly initialize the weights

2. Implement forward propagation to get hθ (x(i) )

3. Implement the cost function

4. Implement backpropagation to compute partial derivatives

5. Use gradient checking to con rm that your backpropagation works. Then disable gradient checking.

6. Use gradient descent or a built-in optimization function to minimize the cost function with the weights in theta.

When we perform forward and back propagation, we loop on every training example:

1 for i = 1:m,
2 Perform forward propagation and backpropagation using example (x(i),y(i))
3 (Get activations a(l) and delta terms d(l) for l = 2,...,L

Bonus: Tutorial on How to classify your own images of digits

This tutorial will guide you on how to use the classi er provided in exercise 3 to classify you own images like this:
It will also explain how the images are converted thru several formats to be processed and displayed.

Introduction
The classi er provided expects 20 x 20 pixels black and white images converted in a row vector of 400 real numbers like this

1 [ 0.14532, 0.12876, ...]

Each pixel is represented by a real number between -1.0 to 1.0, meaning -1.0 equal black and 1.0 equal white (any number in between is a shade of
gray, and number 0.0 is exactly the middle gray).

.jpg and color RGB images

The most common image format that can be read by Octave is .jpg using function that outputs a three-dimensional matrix of integer numbers
from 0 to 255, representing the height x width x 3 integers as indexes of a color map for each pixel (explaining color maps is beyond scope).

1 Image3DmatrixRGB = imread("myOwnPhoto.jpg");

Convert to Black & White

A common way to convert color images to black & white, is to convert them to a YIQ standard and keep only the Y component that represents the
luma information (black & white). I and Q represent the chrominance information (color).Octave has a function rgb2ntsc() that outputs a similar
three-dimensional matrix but of real numbers from -1.0 to 1.0, representing the height x width x 3 (Y luma, I in-phase, Q quadrature) intensity for
each pixel.

1 Image3DmatrixYIQ = rgb2ntsc(MyImageRGB);

To obtain the Black & White component just discard the I and Q matrices. This leaves a two-dimensional matrix of real numbers from -1.0 to 1.0
representing the height x width pixels black & white values.
 1 Image2DmatrixBW = Image3DmatrixYIQ(:,:,1);

Cropping to square image

It is useful to crop the original image to be as square as possible. The way to crop a matrix is by selecting an area inside the original B&W image
and copy it to a new matrix. This is done by selecting the rows and columns that de ne the area. In other words, it is copying a rectangular subset
of the matrix like this:

1 croppedImage = Image2DmatrixBW(origen1:size1, origin2:size2);

2

Cropping does not have to be all the way to a square.It could be cropping just a percentage of the way to a squareso you can leave more of the
image intact. The next step of scaling will take care of streaching the image to t a square.

Scaling to 20 x 20 pixels

The classi er provided was trained with 20 x 20 pixels images so we need to scale our photos to meet. It may cause distortion depending on the
height and width ratio of the cropped original photo. There are many ways to scale a photo but we are going to use the simplest one. We lay a
scaled grid of 20 x 20 over the original photo and take a sample pixel on the center of each grid. To lay a scaled grid, we compute two vectors of 20
indexes each evenly spaced on the original size of the image. One for the height and one for the width of the image. For example, in an image of
320 x 200 pixels will produce to vectors like

1 [9 25 41 57 73 ... 313] % 20 indexes

1 [6 16 26 36 46 ... 196] % 20 indexes

Copy the value of each pixel located by the grid of these indexes to a new matrix. Ending up with a matrix of 20 x 20 real numbers.

Black & White to Gray & White

The classi er provided was trained with images of white digits over gray background. Speci cally, the 20 x 20 matrix of real numbers ONLY range
from 0.0 to 1.0 instead of the complete black & white range of -1.0 to 1.0, this means that we have to normalize our photos to a range 0.0 to 1.0 for
this classi er to work. But also, we invert the black and white colors because is easier to "draw" black over white on our photos and we need to get
white digits. So in short, we invert black and white and stretch black to gray.

Rotation of image

Some times our photos are automatically rotated like in our celular phones. The classi er provided can not recognize rotated images so we may
need to rotate it back sometimes. This can be done with an Octave function rot90() like this.

1 ImageAligned = rot90(Image, rotationStep);

Where rotationStep is an integer: -1 mean rotate 90 degrees CCW and 1 mean rotate 90 degrees CW.

Approach
1. The approach is to have a function that converts our photo to the format the classi er is expecting. As if it was just a sample from the training
data set.

2. Use the classi er to predict the digit in the converted image.

Code step by step

De ne the function name, the output variable and three parameters, one for the lename of our photo, one optional cropping percentage (if not
provided will default to zero, meaning no cropping) and the last optional rotation of the image (if not provided will default to cero, meaning no
rotation).

1 function vectorImage = imageTo20x20Gray(fileName, cropPercentage=0, rotStep=0)

2

Read the le as a RGB image and convert it to Black & White 2D matrix (see the introduction).

1 % Read as RGB image

2 Image3DmatrixRGB = imread(fileName);
3 % Convert to NTSC image (YIQ)
4 Image3DmatrixYIQ = rgb2ntsc(Image3DmatrixRGB );
5 % Convert to grays keeping only luminance (Y)
6 % ...and discard chrominance (IQ)
7 Image2DmatrixBW = Image3DmatrixYIQ(:,:,1);
8
Establish the nal size of the cropped image.

1 % Get the size of your image

2 oldSize = size(Image2DmatrixBW);
3 % Obtain crop size toward centered square (cropDelta)
4 % ...will be zero for the already minimum dimension
5 % ...and if the cropPercentage is zero,
6 % ...both dimensions are zero
7 % ...meaning that the original image will go intact to croppedImage
8 cropDelta = floor((oldSize - min(oldSize)) .* (cropPercentage/100));
9 % Compute the desired final pixel size for the original image
10 finalSize = oldSize - cropDelta;
11

Obtain the origin and amount of the columns and rows to be copied to the cropped image.

1 % Compute each dimension origin for croping

2 cropOrigin = floor(cropDelta / 2) + 1;
3 % Compute each dimension copying size
4 copySize = cropOrigin + finalSize - 1;
5 % Copy just the desired cropped image from the original B&W image
6 croppedImage = Image2DmatrixBW( ...
7 cropOrigin(1):copySize(1), cropOrigin(2):copySize(2));
8

Compute the scale and compute back the new size. This last step is extra. It is computed back so the code keeps general for future modi cation of
the classi er size. For example: if changed from 20 x 20 pixels to 30 x 30. Then the we only need to change the line of code where the scale is
computed.

1 % Resolution scale factors: [rows cols]

2 scale = [20 20] ./ finalSize;
3 % Compute back the new image size (extra step to keep code general)
4 newSize = max(floor(scale .* finalSize),1);
5

Compute two sets of 20 indexes evenly spaced. One over the original height and one over the original width of the image.

1 % Compute a re-sampled set of indices:

2 rowIndex = min(round(((1:newSize(1))-0.5)./scale(1)+0.5), finalSize(1));
3 colIndex = min(round(((1:newSize(2))-0.5)./scale(2)+0.5), finalSize(2));
4

Copy just the indexed values from old image to get new image of 20 x 20 real numbers. This is called "sampling" because it copies just a sample
pixel indexed by a grid. All the sample pixels make the new image.

1 % Copy just the indexed values from old image to get new image
2 newImage = croppedImage(rowIndex,colIndex,:);
3

Rotate the matrix using the rot90() function with the rotStep parameter: -1 is CCW, 0 is no rotate, 1 is CW.

1 % Rotate if needed: -1 is CCW, 0 is no rotate, 1 is CW

2 newAlignedImage = rot90(newImage, rotStep);
3

Invert black and white because it is easier to draw black digits over white background in our photos but the classi er needs white digits.

1 % Invert black and white

2 invertedImage = - newAlignedImage;
3

Find the min and max gray values in the image and compute the total value range in preparation for normalization.

1 % Find min and max grays values in the image

2 maxValue = max(invertedImage(:));
3 minValue = min(invertedImage(:));
4 % Compute the value range of actual grays
5 delta = maxValue - minValue;
6

Do normalization so all values end up between 0.0 and 1.0 because this particular classi er do not perform well with negative numbers.

1 % Normalize grays between 0 and 1

2 normImage = (invertedImage - minValue) / delta;

Add some contrast to the image. The multiplication factor is the contrast control, you can increase it if desired to obtain sharper contrast (contrast
only between gray and white, black was already removed in normalization).

1 % Add contrast. Multiplication factor is contrast control.

2 contrastedImage = sigmoid((normImage -0.5) * 5);
3

Show the image specifying the black & white range [-1 1] to avoid automatic ranging using the image range values of gray to white. Showing the
photo with di erent range, does not a ect the values in the output matrix, so do not a ect the classi er. It is only as a visual feedback for the user.

1 % Show image as seen by the classifier

2 imshow(contrastedImage, [-1, 1] );

Finally, output the matrix as a unrolled vector to be compatible with the classi er.
 1 % Output the matrix as a unrolled vector
2 vectorImage = reshape(normImage, 1, newSize(1) * newSize(2));

End function.

1 end;

Usage samples

Single photo

Photo le in myDigit.jpg

Cropping 60% of the way to square photo

No rotationvectorImage = imageTo20x20Gray('myDigit.jpg',60); predict(Theta1, Theta2, vectorImage)

Photo le in myDigit.jpg

No cropping

CCW rotationvectorImage = imageTo20x20Gray('myDigit.jpg',:,-1); predict(Theta1, Theta2, vectorImage)

Multiple photos

Photo les in myFirstDigit.jpg, mySecondDigit.jpg

First crop to square and second 25% of the way to square photo

First no rotation and second CW rotationvectorImage(1,:) = imageTo20x20Gray('myFirstDigit.jpg',100); vectorImage(2,:) =

imageTo20x20Gray('mySecondDigit.jpg',25,1); predict(Theta1, Theta2, vectorImage)

Tips
JPG photos of black numbers over white background

Preferred square photos but not required

Rotate as needed because the classi er can only work with vertical digits

Leave background space around digit. Al least 2 pixels when seen at 20 x 20 resolution. This means that the classi er only really works in a 16 x
16 area.

Play changing the contrast multipier to 10 (or more).

Complete code (just copy and paste)

 1 function vectorImage = imageTo20x20Gray(fileName, cropPercentage=0, rotStep=0)
2 %IMAGETO20X20GRAY display reduced image and converts for digit classification
3 %
4 % Sample usage:
5 % imageTo20x20Gray('myDigit.jpg', 100, -1);
6 %
7 % First parameter: Image file name
8 % Could be bigger than 20 x 20 px, it will
9 % be resized to 20 x 20. Better if used with
10 % square images but not required.
11 %
12 % Second parameter: cropPercentage (any number between 0 and 100)
13 % 0 0% will be cropped (optional, no needed for square images)
14 % 50 50% of available croping will be cropped
15 % 100 crop all the way to square image (for rectangular images)
16 %
17 % Third parameter: rotStep
18 % -1 rotate image 90 degrees CCW
19 % 0 do not rotate (optional)
20 % 1 rotate image 90 degrees CW
21 %
22 % (Thanks to Edwin Frühwirth for parts of this code)
23 % Read as RGB image
24 Image3DmatrixRGB = imread(fileName);
25 % Convert to NTSC image (YIQ)
26 Image3DmatrixYIQ = rgb2ntsc(Image3DmatrixRGB );
27 % Convert to grays keeping only luminance (Y) and discard chrominance (IQ)
28 Image2DmatrixBW = Image3DmatrixYIQ(:,:,1);
29 % Get the size of your image
30 oldSize = size(Image2DmatrixBW);
31 % Obtain crop size toward centered square (cropDelta)
32 % ...will be zero for the already minimum dimension
33 % ...and if the cropPercentage is zero,
34 % ...both dimensions are zero
35 % ...meaning that the original image will go intact to croppedImage
36 cropDelta = floor((oldSize - min(oldSize)) .* (cropPercentage/100));
37 % Compute the desired final pixel size for the original image
38 finalSize = oldSize - cropDelta;
39 % Compute each dimension origin for croping
40 cropOrigin = floor(cropDelta / 2) + 1;
41 % Compute each dimension copying size
42 copySize = cropOrigin + finalSize - 1;
43 % Copy just the desired cropped image from the original B&W image
44 croppedImage = Image2DmatrixBW( ...
45 cropOrigin(1):copySize(1), cropOrigin(2):copySize(2));
46 % Resolution scale factors: [rows cols]
47 scale = [20 20] ./ finalSize;
48 % Compute back the new image size (extra step to keep code general)
49 newSize = max(floor(scale .* finalSize),1);
50 % Compute a re-sampled set of indices:
51 rowIndex = min(round(((1:newSize(1))-0.5)./scale(1)+0.5), finalSize(1));
52 colIndex = min(round(((1:newSize(2))-0.5)./scale(2)+0.5), finalSize(2));
53 % Copy just the indexed values from old image to get new image
54 newImage = croppedImage(rowIndex,colIndex,:);
55 % Rotate if needed: -1 is CCW, 0 is no rotate, 1 is CW
56 newAlignedImage = rot90(newImage, rotStep);
57 % Invert black and white
58 invertedImage = - newAlignedImage;
59 % Find min and max grays values in the image
60 maxValue = max(invertedImage(:));
61 minValue = min(invertedImage(:));
62 % Compute the value range of actual grays
63 delta = maxValue - minValue;
64 % Normalize grays between 0 and 1
65 normImage = (invertedImage - minValue) / delta;
66 % Add contrast. Multiplication factor is contrast control.
67 contrastedImage = sigmoid((normImage -0.5) * 5);
68 % Show image as seen by the classifier
69 imshow(contrastedImage, [-1, 1] );
70 % Output the matrix as a unrolled vector
71 vectorImage = reshape(contrastedImage, 1, newSize(1)*newSize(2));
72 end

Photo Gallery

Digit 2
Digit 6
Digit 6 inverted is digit 9. This is the same photo of a six but rotated. Also, changed the contrast multiplier from 5 to 20. You can
note that the gray background is smoother.
Digit 3
Explanation of Derivatives Used in Backpropagation
We know that for a logistic regression classi er (which is what all of the output neurons in a neural network are), we use the cost function,
J (θ) = −ylog(hθ (x)) − (1 − y)log(1 − hθ (x)) , and apply this over the K output neurons, and for all m examples.

The equation to compute the partial derivatives of the theta terms in the output neurons:
∂J (θ) ∂J (θ) (L) (L)
∂a ∂z
=
(L−1) (L) (L) (L−1)
∂θ ∂a ∂z ∂θ

And the equation to compute partial derivatives of the theta terms in the [last] hidden layer neurons (layer L-1):
∂J (θ) ∂J (θ) (L) (L) (L−1) (L−1)
∂a ∂z ∂a ∂z
=
(L−2) (L) (L) (L−1) (L−1) (L−2)
∂θ ∂a ∂z ∂a ∂z ∂θ

Clearly they share some pieces in common, so a delta term (δ ) can be used for the common pieces between the output layer and the hidden
(L)

layer immediately before it (with the possibility that there could be many hidden layers if we wanted):
∂J (θ) (L)
(L) ∂a
δ =
(L) (L)
∂a ∂z

And we can go ahead and use another delta term (δ ) for the pieces that would be shared by the nal hidden layer and a hidden layer
(L−1)

before that, if we had one. Regardless, this delta term will still serve to make the math and implementation more concise.
∂J (θ) (L) (L) (L−1)
(L−1) ∂a ∂z ∂a
δ =
(L) (L) (L−1) (L−1)
∂a ∂z ∂a ∂z

(L) (L−1)
(L−1) (L) ∂z ∂a
δ = δ
(L−1) (L−1)
∂a ∂z

With these delta terms, our equations become:

∂J (θ) (L)
(L) ∂z
= δ
(L−1) (L−1)
∂θ ∂θ

∂J (θ) (L−1)
(L−1) ∂z
= δ
(L−2) (L−2)
∂θ ∂θ

Now, time to evaluate these derivatives:

Let's start with the output layer:

∂J (θ) (L)
( ) ∂
 ∂J (θ) (L)
(L) ∂z
= δ
(L−1) (L−1)
∂θ ∂θ

∂J (θ) (L)

Using δ , we need to evaluate both partial derivatives.

(L) ∂a
=
(L) (L)
∂a ∂z

Given J (θ) = −ylog(a (L)

) − (1 − y)log(1 − a
(L)
) , where a(L) , the partial derivative is:
= hθ (x))

∂J (θ) 1−y y
= −
(L) (L) (L)
∂a 1−a a

And given a=g(z), where g = 1+e

1
−z
, the partial derivative is:
(L)
∂a (L) (L)
= a (1 − a )
(L)
∂z

So, let's substitute these in for δ :

(L)

∂J (θ) (L)
(L) ∂a
δ =
(L) (L)
∂a ∂z

(L) 1−y y
(L) (L)
δ = ( − )(a (1 − a ))
(L) (L)
1−a a

(L) (L)
δ = a −y

So, for a 3-layer network (L=3),

(3) (3)
δ = a −y

Note that this is the correct equation, as given in our notes.

Now, given z=θ∗input, and in layer L the input is a(L−1) , the partial derivative is:

(L)
∂z (L−1)
= a
(L−1)
∂θ

Put it together for the output layer:

∂J (θ) (L)
(L) ∂z
= δ
(L−1) (L−1)
∂θ ∂θ

∂J (θ)
(L) (L−1)
= (a − y)(a )
(L−1)
∂θ

Let's continue on for the hidden layer (let's assume we only have 1 hidden layer):

∂J (θ) (L−1)
(L−1) ∂z
= δ
(L−2) (L−2)
∂θ ∂θ

Let's gure out δ(L − 1).

Once again, given z=θ∗input, the partial derivative is:

(L)
∂z (L−1)
= θ
(L−1)
∂a

(L−1)

And:
∂a (L−1) (L−1)
= a (1 − a )
(L−1)
∂z

So, let's substitute these in for δ :

(L−1)

(L) (L−1)
(L−1) (L) ∂z ∂a
δ = δ
(L−1) (L−1)
∂a ∂z

(L−1) (L) (L−1) (L−1) (L−1)

δ = δ (θ )(a (1 − a ))

(L−1) (L) (L−1) (L−1) (L−1)

δ = δ θ a (1 − a )

So, for a 3-layer network,

(2) (3) (2) (2) (2)

δ = δ θ a (1 − a )

Put it together for the [last] hidden layer:

∂J (θ) (L−1)
(L−1) ∂z
= δ
(L−2) (L−2)
∂θ ∂θ

∂J (θ) (L) (L−1)

(L) ∂z ∂a (L−2)
= (δ )(a )
(L−2) (L−1) (L−1)
∂θ ∂a ∂z

∂J (θ) (L−1)
(L) (L−1) (L−1) (L−2)
= ((a − y)(θ )(a (1 − a )))(a )
(L−2)
∂θ

NN for linear systems

Introduction
Introduction
The NN we created for classi cation can easily be modi ed to have a linear output. First solve the 4th programming exercise. You can create a new
function script, nnCostFunctionLinear.m, with the following characteristics

There is only one output node, so you do not need the 'num_labels' parameter.

Since there is one linear output, you do not need to convert y into a logical matrix.

You still need a non-linear function in the hidden layer.

The non-linear function is often the tanh() function - it has an output range from -1 to +1, and its gradient is easily implemented. Let g(z)=tanh(z).

The gradient of tanh is g ′ (z) = 1 − g(z) . Use this in backpropagation in place of the sigmoid gradient.
2

Remove the sigmoid function from the output layer (i.e. calculate a3 without using a sigmoid function), since we want a linear output.

Cost computation: Use the linear cost function for J (from ex1 and ex5) for the unregularized portion. For the regularized portion, use the same
method as ex4.

Where reshape() is used to form the Theta matrices, replace 'num_labels' with '1'.

You still need to randomly initialize the Theta values, just as with any NN. You will want to experiment with di erent epsilon values. You will also
need to create a predictLinear() function, using the tanh() function in the hidden layer, and a linear output.

Testing your linear NN

Here is a test case for your nnCostFunctionLinear()

1 % inputs
2 nn_params = [31 16 15 -29 -13 -8 -7 13 54 -17 -11 -9 16]'/ 10;
3 il = 1;
4 hl = 4;
5 X = [1; 2; 3];
6 y = [1; 4; 9];
7 lambda = 0.01;
8
9 % command
10 [j g] = nnCostFunctionLinear(nn_params, il, hl, X, y, lambda)
11
12 % results
13 j = 0.020815
14 g =
15 -0.0131002
16 -0.0110085
17 -0.0070569
18 0.0189212
19 -0.0189639
20 -0.0192539
21 -0.0102291
22 0.0344732
23 0.0024947
24 0.0080624
25 0.0021964
26 0.0031675
27 -0.0064244
28

Now create a script that uses the 'ex5data1.mat' from ex5, but without creating the polynomial terms. With 8 units in the hidden layer and MaxIter
set to 200, you should be able to get a nal cost value of 0.3 to 0.4. The results will vary a bit due to the random Theta initialization. If you plot the
training set and the predicted values for the training set (using your predictLinear() function), you should have a good match.

Deriving the Sigmoid Gradient Function

We let the sigmoid function be σ(x) = 1

1+e
−x

Deriving the equation above yields to ( 1+e1

2 d 1
−x
) −x
ds 1+e

Which is equal to ( 1+e1

2 −x
−x
) e (−1)

1 1 −x
( −x
)( −x
)(−e )
1+e 1+e

−x
1 −e
( −x
)( −x
)
1+e 1+e

σ(x)(1 − σ(x))

Additional Resources for Backpropagation

Very thorough conceptual [example] (https://web.archive.org/web/20150317210621/https://www4.rgu.ac.uk/ les/chapter3%20-%20bp.pdf)

Short derivation of the backpropagation algorithm: http://pandamatak.com/people/anand/771/html/node37.html

Stanford University Deep Learning notes: http://u dl.stanford.edu/wiki/index.php/Backpropagation_Algorithm

Very thorough explanation and proof: http://neuralnetworksanddeeplearning.com/chap2.html