Variational Autoencoders

VAEs turn the autoencoder into a generative model by mapping inputs to probability distributions over a smooth, sampleable latent space.

From code to distribution

A plain autoencoder maps each input to a single point in latent space, leaving gaps that decode to nonsense. A variational autoencoder (VAE) instead maps each input to a distribution, typically a Gaussian described by a mean and variance. Training encourages these distributions to fill the latent space smoothly, so sampling any point yields a plausible output. This makes the VAE a true generative model: sample from the latent prior, decode, and get a new example.

The evidence lower bound

Kronos motion — space economy

VAEs are trained to maximize a tractable lower bound on the data likelihood, the ELBO, which has two terms. A reconstruction term rewards decoding the latent sample back to the input. A regularization term, the Kullback-Leibler divergence, pulls each input's latent distribution toward a standard normal prior. The balance keeps the latent space both informative and well organized.

The reparameterization trick

Sampling is not differentiable, which would block backpropagation through the latent layer. The reparameterization trick rewrites a sample as z = mu + sigma · epsilon, where epsilon is drawn from a fixed standard normal. Randomness now enters through epsilon, an input, while mu and sigma remain differentiable outputs of the encoder. Gradients flow through mu and sigma, making the whole network trainable by gradient descent.

python

import torch
def reparameterize(mu, logvar):
    std = torch.exp(0.5 * logvar)
    eps = torch.randn_like(std)
    return mu + eps * std   # differentiable sampling

Smooth latent space

Because the KL term packs the encoded distributions against a common prior, the latent space is continuous and interpolatable. Moving smoothly between two latent points produces a smooth morph between the corresponding outputs, and directions in latent space can correspond to interpretable attributes. This structure is what distinguishes a VAE from a plain autoencoder and enables controlled generation.

Strengths and trade-offs

VAEs are stable to train and give an explicit, principled probabilistic model, useful for representation learning and anomaly detection. Their outputs tend to be blurrier than those of GANs or diffusion models, because the Gaussian assumptions and averaging smooth fine detail. VAEs are often combined with other methods or used where a well-structured latent space matters more than photorealism.

Encodes inputs to distributions, not points.
ELBO balances reconstruction and KL regularization.
Reparameterization keeps sampling differentiable.
Smooth latent space enables interpolation and generation.