Feature Scaling

Feature scaling puts numeric features on a comparable range so distance- and gradient-based models are not skewed by units.

Why scale at all

Many algorithms treat features by their magnitude. If one feature ranges 0 to 1 and another 0 to 100000, the large one dominates distances and gradients, drowning out the small one regardless of importance. Feature scaling rewrites features onto a comparable range so each contributes fairly.

Common methods

• Standardization (z-score): subtract the mean, divide by the standard deviation; gives mean 0, variance 1.
• Min-max scaling: rescale linearly to a fixed range such as [0, 1].
• Robust scaling: use the median and interquartile range; resistant to outliers.
• Log or power transforms: compress heavy-tailed, skewed distributions before scaling.

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler().fit(X_train)   # fit on train only
X_train = scaler.transform(X_train)
X_test  = scaler.transform(X_test)

Which models need it

• Need it: k-NN, SVM, k-means, PCA, and neural networks (distance- or gradient-based).
• Do not need it: decision trees, random forests, and gradient boosting (split on thresholds, scale-invariant).
• Regularized linear models need it so the penalty treats features equally.

Fit on training only

The scaler's parameters (mean, standard deviation, min, max) must be computed from the training data and then applied to validation and test data. Computing them on the full dataset leaks information from the test set into training, an instance of data leakage. Wrapping scaling in a pipeline ensures it is refit correctly inside each cross-validation fold.