maps it (with an *encoder*) to a hidden representation :math:`\mathbf{y} \in [0,1]^{d'}`

Where :math:`s` is a non-linearity such as the sigmoid.

The latent representation :math:`\mathbf{y}`, or **code** is then mapped back (with a *decoder*) into a

**reconstruction** :math:`\mathbf{z}` of same shape as

:math:`\mathbf{x}` through a similar transformation, e.g.:

where ' does not indicate transpose, and

:math:`\mathbf{z}` should be seen as a prediction of :math:`\mathbf{x}`, given the code :math:`\mathbf{y}`.

The weight matrix :math:`\mathbf{W'}` of the reverse mapping may be

optionally constrained by :math:`\mathbf{W'} = \mathbf{W}^T`, which is

an instance of *tied weights*. The parameters of this model (namely

:math:`\mathbf{W}`, :math:`\mathbf{b}`,

:math:`\mathbf{b'}` and, if one doesn't use tied weights, also

:math:`\mathbf{W'}`) are optimized such that the average reconstruction

error is minimized. The reconstruction error can be measured in many ways, depending

on the appropriate distributional assumptions on the input given the code, e.g., using the

traditional *squared error* :math:`L(\mathbf{x}, \mathbf{z}) = || \mathbf{x} - \mathbf{z} ||^2`,

or if the input is interpreted as either bit vectors or vectors of

bit probabilities by the reconstruction *cross-entropy* defined as :

The hope is that the code :math:`\mathbf{y}` is a distributed representation

that captures the coordinates along the main factors of variation in the data

(similarly to how the projection on principal components captures the main factors

of variation in the data).

be a good compression (with small loss) for all :math:`\mathbf{x}`, so learning

input vector.

If there is one linear hidden layer (the code) and

layer is non-linear, the auto-encoder behaves differently from PCA,

autoencoder ( :math:`\mathbf{W}`, :math:`\mathbf{b}` and

:math:`\mathbf{b'}`, since we are using tied weights in this tutorial ):

Note that we pass the symbolic ``input`` to the autoencoder as a

parameter. This is such that later we can concatenate layers of

autoencoders to form a deep network: the symbolic output (the :math:`\mathbf{y}` above) of

the k-th layer will be the symbolic input of the (k+1)-th.

One serious potential issue with auto-encoders is that if there is no other

constraint besides minimizing the reconstruction error,

then an auto-encoder with :math:`n` inputs and an

encoding of dimension at least :math:`n` could potentially just learn

the identity function, for which many encodings would be useless (e.g.,

just copying the input), i.e., the autoencoder would not differentiate

test examples (from the training distribution) from other input configurations.

Surprisingly, experiments reported in [Bengio07]_ nonetheless

suggest that in practice, when trained with

stochastic gradient descent, non-linear auto-encoders with more hidden units

than inputs (called overcomplete) yield useful representations

(in the sense of classification error measured on a network taking this

representation in input). A simple explanation is based on the

observation that stochastic gradient

descent with early stopping is similar to an L2 regularization of the

parameters. To achieve perfect reconstruction of continuous

inputs, a one-hidden layer auto-encoder with non-linear hidden units

(exactly like in the above code)

needs very small weights in the first (encoding) layer (to bring the non-linearity of

the hidden units in their linear regime) and very large weights in the

second (decoding) layer.

With binary inputs, very large weights are

also needed to completely minimize the reconstruction error. Since the

implicit or explicit regularization makes it difficult to reach

large-weight solutions, the optimization algorithm finds encodings which

only work well for examples similar to those in the training set, which is

what we want. It means that the representation is exploiting statistical

regularities present in the training set, rather than learning to

replicate the identity function.

There are different ways that an auto-encoder with more hidden units

than inputs could be prevented from learning the identity, and still

capture something useful about the input in its hidden representation.

One is the addition of sparsity (forcing many of the hidden units to

be zero or near-zero), and it has been exploited very successfully

by many [Ranzato07]_ [Lee08]_. Another is to add randomness in the transformation from

input to reconstruction. This is exploited in Restricted Boltzmann

Machines (discussed later in :ref:`rbm`), as well as in

Denoising Auto-Encoders, discussed below.

Intuitively, a denoising auto-encoder does two things: try to encode the

input (preserve the information about the input), and try to undo the

effect of a corruption process stochastically applied to the input of the

auto-encoder. The latter can only be done by capturing the statistical

dependencies between the inputs. The denoising

auto-encoder can be understood from different perspectives

( the manifold learning perspective,

stochastic operator perspective,

bottom-up -- information theoretic perspective,

top-down -- generative model perspective ), all of which are explained in

[Vincent08].

See also section 7.2 of [Bengio09]_ for an overview of auto-encoders.

In [Vincent08], the stochastic corruption process

consists in randomly setting some of the inputs (as many as half of them)

to zero. Hence the denoising auto-encoder is trying to *predict the corrupted (i.e. missing)

values from the uncorrupted (i.e., non-missing) values*, for randomly selected subsets of

missing patterns. Note how being able to predict any subset of variables

from the rest is a sufficient condition for completely capturing the

joint distribution between a set of variables (this is how Gibbs

sampling works).

In the stacked autoencoder class (:ref:`stacked_autoencoders`) the

weights of the ``dA`` class have to be shared with those of an

corresponding sigmoid layer. For this reason, the constructor of the ``dA`` also gets Theano

variables pointing to the shared parameters. If those parameters are left

to ``None``, new ones will be constructed.

plot the filters (defined by the weight matrix). Bare in mind however,

:ref:`how-to-plot`) so we urge the reader to familiarize himself with

it. Also using the help of PIL library, the following lines of code will