Autoencoders: A Brief Introduction
What is an autoencoder?
An autoencoder is a type of neural network that learns to compress and reconstruct data. It
consists of two parts: an encoder and a decoder. The encoder takes an input and transforms it
into a lower-dimensional representation, called a latent code or a bottleneck. The decoder
takes the latent code and tries to reconstruct the original input as closely as possible. The goal
of an autoencoder is to learn the most important features of the data, while discarding the noise
and redundancy.
How does an autoencoder work?
An autoencoder is trained by minimizing the reconstruction error, which is the difference
between the input and the output. The reconstruction error can be measured by different loss
functions, such as mean squared error (MSE) or binary cross-entropy (BCE). The loss function
depends on the type and range of the data. For example, MSE is suitable for continuous data,
while BCE is suitable for binary data. The loss function also determines the activation function
of the output layer. For example, a sigmoid activation function is used for binary data, while a
linear activation function is used for continuous data.
What can an autoencoder do?
An autoencoder can be used for various applications, such as:
• Anomaly Detection: Anomalous datapoints have higher reconstruction error
• Data compression: An autoencoder can reduce the dimensionality of the data, while
preserving the most relevant information. This can save storage space and improve
computational efficiency.
• Data denoising: An autoencoder can remove noise from the data, by learning to ignore
the corrupted or irrelevant parts of the input. This can improve the quality and clarity of
the data.
Types:
1. Vanilla Autoencoder: The basic form of an autoencoder with a simple encoder-decoder
structure. It aims to learn a compressed representation of the input data and then
reconstruct it.
2. Denoising Autoencoder: This type introduces noise to the input data and trains the
model to recover the original, noise-free data. It’s useful for noise reduction and feature
learning1
3. Deep Autoencoder: Composed of multiple layers in both the encoder and decoder,
allowing it to learn more complex representations
4. Undercomplete Autoencoder: The latent space (encoded representation) has a lower
dimensionality than the input, forcing the model to learn the most important features1.
And many more…
Code:
A simple autoencoder code using MSE and BCE loss in case our dataset has both continuous
and categorical data:
import torch
import torch.nn as nn
import torch.optim as optim
# Define the autoencoder model
class Autoencoder(nn.Module):
def __init__(self):
super(Autoencoder, self).__init__()
self.encoder = nn.Sequential(
nn.Linear(10, 6),
nn.ReLU(),
nn.Linear(6, 3)
)
self.decoder = nn.Sequential(
nn.Linear(3, 6),
nn.ReLU(),
nn.Linear(6, 10)
)
def forward(self, x):
encoded = self.encoder(x)
decoded = self.decoder(encoded)
return decoded
# Custom loss function
def custom_loss_function(output, target, alpha=0.5, beta=0.5):
mse_loss = nn.MSELoss()(output[:, :5], target[:, :5]) # Continuous data
bce_loss = nn.BCELoss()(output[:, 5:], target[:, 5:]) # Categorical
data
total_loss = alpha * mse_loss + beta * bce_loss
return total_loss
# Example usage
model = Autoencoder()
optimizer = optim.Adam(model.parameters(), lr=0.001)
# Dummy data
input_data = torch.randn(10, 10)
target_data = torch.randn(10, 10)
# Training loop
for epoch in range(100):
optimizer.zero_grad()
output = model(input_data)
loss = custom_loss_function(output, target_data)
loss.backward()
optimizer.step()
print(f'Epoch {epoch+1}, Loss: {loss.item()}')
Useful link: https://www.analyticsvidhya.com/blog/2021/06/autoencoders-a-gentle-
introduction/
