Train/Validation/Test Split

Splitting data into training, validation, and test sets keeps model selection honest and the final estimate unbiased.

Three roles for three sets

Sound evaluation partitions the data into three disjoint sets, each with a distinct job. Mixing these roles is the most common way to fool yourself into believing a model is better than it is.

• Training set: used to fit model parameters.
• Validation set: used to tune hyperparameters and choose among models.
• Test set: used exactly once, at the end, to estimate real-world performance.

Why the test set must stay sealed

Every time you look at the test set and change something in response, you leak its information into your choices, and its estimate becomes optimistic. The test set must be untouched during all development. The validation set absorbs the many decisions of model building; the test set is the final, one-shot honest check.

from sklearn.model_selection import train_test_split
X_tmp, X_test, y_tmp, y_test = train_test_split(X, y, test_size=0.2,
                                                stratify=y, random_state=0)
X_tr, X_val, y_tr, y_val = train_test_split(X_tmp, y_tmp, test_size=0.25,
                                            stratify=y_tmp, random_state=0)

Splitting correctly

• Stratify by class so proportions match across splits, vital for imbalanced data.
• For time series, split by time: train on the past, test on the future, never shuffle.
• Keep records from the same entity in one split (grouped split) to prevent leakage.
• With little data, replace a fixed validation set with cross-validation.

Fit preprocessing on training only

Scalers, encoders, imputers, and feature selectors must be fit on the training data and then applied to validation and test data. Fitting them on the full dataset lets test information leak into training, an instance of data leakage that inflates every metric. When data is scarce, cross-validation reuses it while preserving these boundaries.