Anomaly Detection
Anomaly detection flags rare, unexpected observations, usually with few or no labeled examples of the anomalies.
Finding the rare and the wrong
Anomaly detection identifies observations that depart from an expected pattern: fraud, equipment faults, network intrusions, or sensor errors. The defining difficulty is that anomalies are rare and diverse, so there are rarely enough labeled examples to train a standard classifier. Most methods therefore model normal behavior and score how poorly a new point fits it.
Families of methods
- Density and distance: flag points in low-density regions, e.g. local outlier factor or kNN distance
- Isolation: flag points that are easy to separate, e.g. isolation forest
- Boundary: learn a tight region enclosing normal data, e.g. one-class SVM
- Reconstruction: train an autoencoder on normal data; large reconstruction error signals an anomaly
- Probabilistic: fit a density (e.g. a Gaussian mixture) and flag low-likelihood points
Point, contextual, and collective
Anomalies come in kinds. A point anomaly is a single odd value. A contextual anomaly is normal in general but odd in context, such as high heating use in summer. A collective anomaly is a sequence that is abnormal as a whole though each element looks normal, common in time series and requiring temporal models.
Evaluation under extreme imbalance
Because positives are rare, accuracy is meaningless. Precision-recall curves, area under the precision-recall curve, and recall at a fixed alarm budget are the honest measures. A threshold on the anomaly score sets the trade-off between missed anomalies and false alarms, and it should be chosen from the operational cost of each error rather than left at a default.
In production, the normal pattern itself drifts, so detectors need periodic recalibration, linking anomaly detection to drift monitoring.