To run this example on a GPU, you need a good GPU. First, it need

at least 1G of GPU RAM and possibly more if your monitor is

Second, when the GPU is connected to the monitor, there is a limit

current GPU can't be used for the monitor while doing

computation. If there wasn't this limit, the screen would freeze

for too long and this look as if the computer froze. User don't

like this. This example hit this limit with medium GPU. When the

Convolutional Neural Networks (CNN) are variants of MLPs which are inspired from

biology. From Hubel and Wiesel's early work on the cat's visual cortex [Hubel68]_,

we know there exists a complex arrangement of cells within the visual cortex.

These cells are sensitive to small sub-regions of the input space, called a

**receptive field**, and are tiled in such a way as to cover the entire visual

field. These filters are local in input space and are thus better suited to

exploit the strong spatially local correlation present in natural images.

Additionally, two basic cell types have been identified: simple cells (S) and

complex cells (C). Simple cells (S) respond maximally to specific edge-like

stimulus patterns within their receptive field. Complex cells (C) have larger

receptive fields and are locally invariant to the exact position of the

stimulus.

The visual cortex being the most powerful "vision" system in existence, it

seems natural to emulate its behavior. Many such neurally inspired models can be

found in the litterature. To name a few: the NeoCognitron [Fukushima]_, HMAX

[Serre07]_ and LeNet-5 [LeCun98]_, which will be the focus of this tutorial.

CNNs exploit spatially local correlation by enforcing a local connectivity pattern between

neurons of adjacent layers. The input hidden units in the m-th layer are

connected to a local subset of units in the (m-1)-th layer, which have spatially

contiguous receptive fields. We can illustrate this graphically as follows:

Imagine that layer **m-1** is the input retina.

In the above, units in layer **m**

have receptive fields of width 3 with respect to the input retina and are thus only

connected to 3 adjacent neurons in the layer below (the retina).

Units in layer **m** have

a similar connectivity with the layer below. We say that their receptive

field with respect to the layer below is also 3, but their receptive field

with respect to the input is larger (it is 5).

The architecture thus

confines the learnt "filters" (corresponding to the input producing the strongest response) to be a spatially local pattern

(since each unit is unresponsive to variations outside of its receptive field with respect to the retina).

As shown above, stacking many such

layers leads to "filters" (not anymore linear) which become increasingly "global" however (i.e

spanning a larger region of pixel space). For example, the unit in hidden

layer **m+1** can encode a non-linear feature of width 5 (in terms of pixel

space).

In CNNs, each sparse filter :math:`h_i` is additionally replicated across the

entire visual field. These "replicated" units form a **feature map**, which

share the same parametrization, i.e. the same weight vector and the same bias.

Weights of the same color are shared, i.e. are constrained to be identical.

Gradient descent can still be used to learn such shared parameters, and

requires only a small change to the original algorithm. The gradient of a

shared weight is simply the sum of the gradients of the parameters being

shared.

Why are shared weights interesting ? Replicating units in this way allows for

features to be detected regardless of their position in the visual field.

greatly reduces the number of free parameters to learn. By controlling model

capacity, CNNs tend to achieve better generalization on vision problems.

Conceptually, a feature map is obtained by convolving the input image with a

linear filter, adding a bias term and then applying a non-linear function. If

we denote the k-th feature map at a given layer as :math:`h^k`, whose filters

are determined by the weights :math:`W^k` and bias :math:`b_k`, then the

feature map :math:`h^k` is obtained as follows (for :math:`tanh` non-linearities):

To form a richer representation of the data, hidden layers are composed of

a set of multiple feature maps, :math:`\{h^{(k)}, k=0..K\}`.

The weights :math:`W` of this layer can be parametrized as a 4D tensor

(destination feature map index, source feature map index, source vertical position index, source horizontal position index)

and

the biases :math:`b` as a vector (one element per destination feature map index).

We illustrate this graphically as follows:

Here, we show two layers of a CNN, containing 4 feature maps at layer (m-1)

and 2 feature maps (:math:`h^0` and :math:`h^1`) at layer m. Pixels (neuron outputs) in

:math:`h^0` and :math:`h^1` (outlined as blue and red squares) are computed

from pixels of layer (m-1) which fall within their 2x2 receptive field in the

layer below (shown

as colored rectangles). Notice how the receptive field spans all four input

feature maps. The weights :math:`W^0` and :math:`W^1` of :math:`h^0` and

:math:`h^1` are thus 3D weight tensors. The leading dimension indexes the

input feature maps, while the other two refer to the pixel coordinates.

The ConvOp

++++++++++

It is meant to replicate the behaviour of scipy.signal.convolve2d. Conceptually,

the ConvOp (once instantiated) takes two symbolic inputs:

Also of note, remark that we use the same weight initialization formula as

with the MLP. Weights are sampled randomly from a uniform distribution in the

range [-1/fan-in, 1/fan-in], where fan-in is the number of inputs to a hidden

unit. For MLPs, this was the number of units in the layer below. For CNNs

however, we have to take into account the number of input feature maps and the

size of the receptive fields.

Another important concept of CNNs is that of max-pooling, which is a form of

Max-pooling is useful in vision for two reasons: (1) it reduces the

computational complexity for upper layers and (2) it provides a form of

translation invariance. To understand the invariance argument, imagine

cascading a max-pooling layer with a convolutional layer. There are 8

directions in which one can translate the input image by a single pixel. If

max-pooling is done over a 2x2 region, 3 out of these 8 possible

configurations will produce exactly the same output at the convolutional

layer. For max-pooling over a 3x3 window, this jumps to 5/8.

Since it provides additional robustness to position, max-pooling is thus a

"smart" way of reducing the dimensionality of intermediate representations.

Max-pooling is done in Theano by way of ``theano.tensor.signal.downsample.max_pool_2d``.

This function takes as input an N dimensional tensor (with N >= 2), a

the tensor.

Note that contrary to most Theano code, the ``max_pool_2d`` operation is a little

*special*. It requires the downscaling factor ``ds`` (tuple of length 2 containing

downscaling factors for image width and height) to be known at graph build

time. This may change in the near future.