Monte Carlo Dropout

Monte Carlo dropout estimates uncertainty by keeping dropout active at prediction time and sampling many stochastic forward passes.

Dropout beyond training

Dropout randomly zeros a fraction of a network's units during training to prevent overfitting, and is normally turned off when predicting. Monte Carlo dropout keeps it on at prediction time. Each forward pass then uses a different random subset of the network, so repeated passes on the same input give slightly different outputs. Their spread is an estimate of uncertainty.

The theoretical link

There is a formal argument that a network trained with dropout and weight decay approximates a particular Bayesian model, and that sampling dropout masks at prediction time approximates sampling from the posterior. This gives Monte Carlo dropout a probabilistic interpretation rather than being a mere heuristic, though the approximation is loose.

How to use it

import torch
def mc_predict(net, x, passes=50):
    net.train()  # keep dropout active
    with torch.no_grad():
        ys = torch.stack([net(x) for _ in range(passes)])
    return ys.mean(0), ys.std(0)  # prediction and uncertainty

Strengths and weaknesses

• Almost free: reuses a network already trained with dropout
• Needs no change to training or architecture beyond having dropout layers
• Uncertainty quality depends strongly on the dropout rate
• Often less well-calibrated than deep ensembles

When to reach for it

Monte Carlo dropout is the cheapest way to get some uncertainty from an existing network, useful for a quick sense of where a model is unsure. When uncertainty must be reliable, for a surrogate that guides expensive decisions, deep ensembles or Gaussian processes are usually preferred. Monte Carlo dropout is a convenient first approximation, not a final answer.