Data Leakage

Data leakage lets information unavailable at prediction time slip into training, inflating scores that collapse in production.

The most common way to fool yourself

Data leakage occurs when a model is trained using information it would not have at prediction time. The model learns to exploit that information, so validation and test scores look excellent, and then performance collapses in deployment. Leakage is subtle, widespread, and responsible for many machine-learning projects that fail after promising evaluations.

Common sources

• Preprocessing on the full dataset: fitting a scaler, imputer, or encoder before splitting leaks test statistics into training.
• Target leakage: a feature that is a proxy for, or computed from, the target (a field only filled in after the outcome is known).
• Train/test contamination: duplicate or near-duplicate records split across sets.
• Temporal leakage: using future information to predict the past when data is time-ordered.
• Group leakage: records from the same patient or device split across train and test.

How to prevent it

Split the data first, then fit all preprocessing on the training portion only and apply it to the rest. Wrap every transform in a pipeline so it is refit inside each cross-validation fold. For time series, split by time and never shuffle. Use grouped splits to keep related records together. Scrutinize any feature that seems too predictive, it is often leakage.

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
pipe = make_pipeline(StandardScaler(), model)   # scaler refit per fold
cross_val_score(pipe, X, y, cv=5)

The tell-tale sign

Suspiciously high performance is the classic symptom, an accuracy far above what the problem should allow. When results look too good, hunt for leakage before celebrating. Honest evaluation depends on a strict boundary between what the model may see during training and what it must predict, enforced by a sealed test set.