PETS: Probabilistic Ensembles with Trajectory Sampling

PETS captures model uncertainty with an ensemble of probabilistic networks and plans with sampling-based model-predictive control.

Two kinds of uncertainty

PETS distinguishes aleatoric uncertainty (inherent environment noise) from epistemic uncertainty (limited data). Handling both is what lets a model-based method match the asymptotic performance of model-free methods while using far fewer samples.

Probabilistic ensemble

Each network outputs a distribution over the next state, typically a Gaussian with predicted mean and variance, capturing aleatoric noise. An ensemble of such networks, each trained on a bootstrap of the data, captures epistemic uncertainty through disagreement between members.

Trajectory sampling

To evaluate a candidate action sequence, PETS propagates many particle trajectories through the ensemble. Each particle is assigned to a bootstrap model, and at each step the next state is drawn from that model's predicted Gaussian. Averaging returns over particles yields a robust estimate that respects both uncertainty types.

# PETS planning = MPC via CEM over action sequences
for iteration in range(cem_iters):
    seqs = sample_action_sequences(mean, std, N)
    returns = [eval_with_particles(ensemble, seq) for seq in seqs]
    elites = top_k(seqs, returns)
    mean, std = fit_gaussian(elites)
execute(mean[0])   # first action only, then replan

Model-predictive control

PETS does not learn an explicit policy. Instead it re-plans at every step: it optimizes an action sequence over a short horizon (using the cross-entropy method), executes only the first action, then re-plans. This receding-horizon control naturally corrects for model error because each decision uses the latest state.

The result was a landmark: on continuous-control benchmarks PETS reached the performance of leading model-free algorithms with roughly an order of magnitude fewer samples, demonstrating that careful uncertainty modeling is central to model-based RL.