Unsupervised Learning:
Deep Auto-encoder
Unsupervised Learning
“We expect unsupervised learning to become far more important
in the longer term. Human and animal learning is largely
unsupervised: we discover the structure of the world by observing
it, not by being told the name of every object.”
– LeCun, Bengio, Hinton, Nature 2015
Auto-encoder
Usually <784
Compact
NN code representation of
Encoder the input object
28 X 28 = 784
Learn together
NN Can reconstruct
code the original object
Decoder
Recap: PCA
2
Minimize ( 𝑥 − ^
𝑥 )
As close as possible
encode decode
𝑥 𝑐 ^
𝑥
𝑊 𝑊
𝑇
hidden layer
Input layer (linear) output layer
Bottleneck later
Output of the hidden layer is the code
Deep Auto-encoder
• Of course, the auto-encoder can be deep
As close as possible
Output Layer
Input Layer
Layer
Layer
Layer
bottle
Layer
…
Layer
…
Layer
𝑇
𝑊1 𝑊 2 𝑊 2 𝑊 1𝑇
𝑥 Code 𝑥^
Reference: Hinton, Geoffrey E., and Ruslan R. Salakhutdinov. "Reducing the
dimensionality of data with neural networks." Science 313.5786 (2006): 504-507
Deep Auto-encoder
Original
Image
784
784
30
PCA
Deep
Auto-encoder
500
250
250
500
30
1000
1000
784
784
784 784
1000
2
500
784
250
2
250
500
1000
784
More: Contractive auto-encoder
Ref: Rifai, Salah, et al. "Contractive
Auto-encoder auto-encoders: Explicit invariance during
feature extraction.“ Proceedings of the
28th International Conference on
Machine Learning (ICML-11). 2011.
• De-noising auto-encoder
As close as possible
encode decode
𝑐
𝑥 𝑥′ ^
𝑥
Add
noise
Vincent, Pascal, et al. "Extracting and composing robust features
with denoising autoencoders." ICML, 2008.
Deep Auto-encoder - Example
NN
𝑐
Encoder
PCA
32-dim
Pixel -> tSNE
Auto-encoder – Text Retrieval
Vector Space Model Bag-of-word
this 1
is 1
word string:
query “This is an apple” a 0
an 1
apple 1
pen 0
document
…
Semantics are not
considered.
Auto-encoder – Text Retrieval
The documents talking about
the same thing will have close
code.
2 query
125
250
500
2000
Bag-of-word
(document or query)
Auto-encoder –
Similar Image Search
Retrieved using Euclidean distance in pixel intensity space
(Images from Hinton’s slides on Coursera)
Reference: Krizhevsky, Alex, and Geoffrey E. Hinton. "Using very deep
autoencoders for content-based image retrieval." ESANN. 2011.
Auto-encoder –
Similar Image Search
code
8192
1024
4096
2048
256
512
32x32
(crawl millions of images from the Internet)
Retrieved using Euclidean distance in pixel intensity space
retrieved using 256 codes
Auto-encoder As close as
for CNN possible
Deconvolution
Unpooling Convolution
Deconvolution Pooling
Unpooling Convolution
code
Deconvolution Pooling
CNN -Unpooling
14 x 14 28 x 28
Alternative: simply
Source of image :
repeat the values https://leonardoaraujosantos.gitbooks.io/artificial-
inteligence/content/image_segmentation.html
CNN
- Deconvolution
+ +
=
+
Auto-encoder – Pre-training DNN
• Greedy Layer-wise Pre-training again
output 10
500
Target
1000 784 ^
𝑥
W1 ’
1000 1000
W1
Input 784 Input 784 𝑥
Auto-encoder – Pre-training DNN
• Greedy Layer-wise Pre-training again
output 10
500 1000 ^1
𝑎
W2 ’
Target
1000 1000
W2
1000 1000 𝑎
1
fix W1
Input 784 Input 784 𝑥
Auto-encoder – Pre-training DNN
• Greedy Layer-wise Pre-training again
output 10 1000 ^2
𝑎
W3 ’
500 500
W3
Target
2
1000 1000 𝑎
fix W2
1000 1000 𝑎
1
fix W1
Input 784 Input 784 𝑥
Auto-encoder – Pre-training DNN
Find-tune by
• Greedy Layer-wise Pre-training again backpropagation
output 10 output 10
Random
W4
init
500 500
W3
Target
1000 1000
W2
1000 1000
W1
Input 784 Input 784 𝑥
NN
Next ….. code
Decoder
• Can we use decoder to generate something?
NN
Next ….. code
Decoder
• Can we use decoder to generate something?
