Feature Selection
Feature selection keeps the informative inputs and drops the rest, reducing overfitting, cost, and complexity.
Fewer, better features
Feature selection chooses a subset of the available inputs to keep. Removing irrelevant or redundant features reduces overfitting, speeds training and prediction, lowers data-collection cost, and makes models easier to interpret. It also mitigates the curse of dimensionality.
Three families of methods
- Filter: score each feature independently (correlation, mutual information, chi-square) and keep the top ones; fast, model-agnostic.
- Wrapper: search over feature subsets using a model's performance (forward selection, backward elimination, recursive feature elimination); accurate but expensive.
- Embedded: selection happens during model fitting (lasso's L1 penalty, tree feature importances).
from sklearn.feature_selection import RFECV
from sklearn.ensemble import RandomForestClassifier
sel = RFECV(RandomForestClassifier(), cv=5, scoring='f1')
sel.fit(X_train, y_train)
kept = X_train.columns[sel.support_]Watch for correlated features
When two features are highly correlated, importance can be split between them or assigned arbitrarily to one. Impurity-based tree importances are biased toward high-cardinality features; permutation importance, which measures the score drop when a feature is shuffled, is more trustworthy but must be computed on held-out data.
Do it inside the fold
Selecting features using the entire dataset, then cross-validating, leaks the test signal and gives optimistic results. Feature selection must sit inside each cross-validation fold, fit only on that fold's training portion. Distinguish selection (choosing existing features) from dimensionality reduction, which creates new combined features instead.
A useful discipline is to weigh what each feature adds against what it costs to collect and maintain. A feature that lifts accuracy by a hair but requires a fragile sensor or a slow query may not be worth keeping. Selection is therefore not only a statistical exercise but an engineering decision about which measurements a deployed system should actually depend on.