Knowledge Distillation

Distillation trains a small student model to mimic a large teacher, transferring its behavior into a cheaper package.

Compressing behavior, not just weights

Knowledge distillation trains a compact student model to reproduce the behavior of a larger, more accurate teacher model. Rather than compressing the teacher weights directly, it transfers what the teacher has learned by training the student on the teacher outputs, so the student can match much of the teacher accuracy at a fraction of its size and cost.

Soft targets carry more information

The key insight is that the teacher full probability distribution, its soft targets, contains richer information than a hard label. When a teacher assigns 0.7 to one class and 0.2 to a similar one, it reveals learned relationships between classes, its dark knowledge. The student trains to match these softened probabilities, produced by dividing the teacher logits by a temperature before the softmax to expose the smaller probabilities.

# Distillation loss combines soft and hard targets
soft = kl_div(log_softmax(student/T), softmax(teacher/T)) * T*T
hard = cross_entropy(student, labels)
loss = alpha * soft + (1 - alpha) * hard

Variants

• Response-based: match the teacher output distribution
• Feature-based: match intermediate representations, not just outputs
• Self-distillation: a model teaches a copy of itself to refine training
• Ensemble distillation: compress many models into one student

Why it matters

Distillation is a primary tool for deploying capable models on constrained hardware, phones, browsers, edge devices, where the teacher would be too slow or large. It complements other compression methods: a distilled student can be further shrunk with quantization and pruning. It also transfers robustness and calibration, and it lets a small model benefit from unlabeled data by learning from the teacher predictions on it.

The student rarely fully matches the teacher, so distillation is a deliberate accuracy-for-efficiency trade, tuned to the deployment budget.