SMOTE and Synthetic Oversampling
SMOTE balances classes by creating synthetic minority examples along lines between real minority neighbors.
Beyond copy-paste oversampling
Naive oversampling duplicates minority examples, which lets a model memorize those exact points and overfit. SMOTE, the Synthetic Minority Over-sampling Technique, instead manufactures new, plausible minority examples. It picks a minority point, finds its k minority nearest neighbors, and creates a synthetic point somewhere on the line segment to a randomly chosen neighbor.
How a synthetic point is made
For a minority point x and a chosen neighbor x_nn, the new point is x + lambda (x_nn - x), where lambda is drawn uniformly between zero and one. Because the new point lies inside the convex hull of nearby minority examples, it stays in a region the minority class plausibly occupies, expanding the decision region rather than reinforcing single points.
Variants
- Borderline-SMOTE: synthesize only near the class boundary, where mistakes happen
- ADASYN: generate more synthetic points for minority examples that are harder to learn
- SMOTE-NC: handle mixed numeric and categorical features
- SMOTE + Tomek or ENN: clean overlapping majority points after oversampling
Cautions
SMOTE assumes interpolation between neighbors yields valid examples, which breaks down in high dimensions and with categorical or highly nonlinear features, where synthetic points can land in impossible regions. It can also blur the boundary if minority and majority classes overlap, generating misleading positives. Critically, SMOTE must be applied only to the training folds inside cross-validation; synthesizing before the split leaks information and inflates scores.
SMOTE is one lever among several for class imbalance; class weighting or focal loss is often simpler and, for extreme rarity, framing the task as anomaly detection can beat resampling entirely.