Autoencoders

Autoencoders learn compact representations by training a network to reconstruct its own input through a narrow bottleneck.

Encode then decode

An autoencoder is a network trained to copy its input to its output through a constrained middle layer. An encoder compresses the input into a low-dimensional code, and a decoder reconstructs the input from that code. Because the code is smaller than the input, the network cannot simply memorize; it must capture the input's essential structure. Training minimizes reconstruction error, typically mean squared error or cross-entropy between input and output.

The bottleneck

Kronos motion — compact footprint

The narrow middle layer, the bottleneck, forces dimensionality reduction. A linear autoencoder with a squared-error loss recovers the same subspace as principal component analysis. With nonlinear activations, an autoencoder learns a nonlinear manifold, capturing structure that linear methods cannot. The learned code is a compressed, feature-rich representation of the data.

python

import torch.nn as nn
encoder = nn.Sequential(nn.Linear(784, 128), nn.ReLU(), nn.Linear(128, 32))
decoder = nn.Sequential(nn.Linear(32, 128), nn.ReLU(), nn.Linear(128, 784))
# loss = mse(decoder(encoder(x)), x)

Variants

Denoising autoencoder: corrupt the input, train to reconstruct the clean version, learning robust features.
Sparse autoencoder: penalize code activations so few units fire, yielding interpretable features.
Contractive autoencoder: penalize sensitivity of the code to input changes.
Convolutional autoencoder: use convolution and transposed convolution for images.

Uses

Autoencoders serve dimensionality reduction, feature learning, and data compression. A widely used application is anomaly detection: train on normal data so the model reconstructs it well, then flag inputs with high reconstruction error as anomalies, since the model never learned to represent them. This suits monitoring sensor streams, where deviations from learned normal behavior signal faults.

Limitations and the generative step

A standard autoencoder's code space has no imposed structure, so points between two valid codes may decode to nonsense, making it a poor generator. The variational autoencoder fixes this by shaping the code space into a smooth probability distribution that can be sampled, turning the autoencoder from a compression tool into a generative model. For pure representation learning, though, the plain autoencoder remains a simple and effective baseline.

Encoder compresses, decoder reconstructs.
The bottleneck forces meaningful compression.
Denoising and sparse variants learn robust features.
Reconstruction error powers anomaly detection.