Active Learning for Labeling
Active learning chooses which unlabeled examples to annotate next so a model reaches target accuracy with far fewer labels.
Labeling is the bottleneck
When raw data is abundant but labeling is slow or expensive, the question is not "label everything" but "which examples, if labeled, would help the model most?" Active learning answers this by letting the model itself nominate the examples whose labels would be most informative, so effort concentrates where it changes the model rather than confirming what it already knows.
The loop
- Train a model on the labels gathered so far
- Score the unlabeled pool by an informativeness measure
- Send the highest-scoring examples to annotators
- Add the new labels and retrain
- Repeat until accuracy plateaus or the budget is spent
Selection strategies
Uncertainty sampling picks examples the current model is least confident about, near its decision boundary, since these are where a label most sharpens the model. Query-by-committee trains several models and picks examples they most disagree on. Diversity or representativeness sampling avoids picking a cluster of near-identical uncertain examples by spreading choices across the data. Practical systems combine uncertainty with diversity so the batch is both informative and varied.
The sampling-bias risk
Active learning deliberately builds a non-random labeled set, which creates a hazard: the labeled data no longer reflects the true distribution, so a model evaluated on it, or a later model trained on it as if it were representative, can be misled. The standard guard is to keep a separate, randomly sampled, held-out test set that active learning never touches, so evaluation stays unbiased even as training data is chosen strategically.
When it helps and when it does not
Active learning pays off most when labeling dominates the cost and the informative examples are a small fraction of the pool. It helps little when labels are cheap, when almost every example is informative, or when the model is so weak that its uncertainty estimates are unreliable, in which case its nominations are no better than random. Recognizing these regimes prevents wasted engineering. See data labeling and synthetic data.