Elastic Net
Elastic net blends L1 and L2 penalties, gaining lasso's sparsity and ridge's stability with correlated features.
Combining two penalties
Elastic net minimizes squared error plus a mixture of the lasso and ridge penalties: alpha times [ rho * sum |w_j| + (1-rho)/2 * sum w_j^2 ]. The mixing parameter rho (often called l1_ratio) slides from pure ridge at 0 to pure lasso at 1, while alpha sets the overall strength.
Why blend them
Lasso alone struggles when features come in correlated groups: it keeps one and discards the others, an unstable choice. Ridge keeps all but selects none. Elastic net's L2 term encourages correlated features to be selected or dropped together (the grouping effect), while its L1 term still delivers a sparse model.
from sklearn.linear_model import ElasticNetCV
model = ElasticNetCV(l1_ratio=[.2,.5,.8,1.0], cv=5)
model.fit(X_train, y_train)Tuning two knobs
- alpha controls how much regularization overall.
- l1_ratio controls the balance of sparsity versus grouping.
- Both are chosen jointly by cross-validation over a grid.
- Standardize features first so penalties are comparable.
When it wins
Elastic net is the sensible default for high-dimensional problems with many correlated predictors, such as spectral or sensor data where neighboring channels move together. It generalizes both ridge and lasso, so tuning l1_ratio lets the data decide how much of each behavior you need.
The method is also more stable than lasso when the number of features exceeds the number of samples. Pure lasso can select at most as many features as there are observations, a hard ceiling that elastic net removes because its ridge component allows more features to enter together. This makes it well suited to wide datasets where the true model is sparse but still involves more predictors than rows.