Lasso Regression

Lasso regression adds an L1 penalty to least squares, driving weights to zero so it selects features while it fits.

The L1 penalty

Lasso (least absolute shrinkage and selection operator) minimizes squared error plus alpha times the sum of absolute weights: L(w) = sum (y_i - x_i . w)^2 + alpha * sum |w_j|. The absolute-value penalty has a sharp corner at zero, and that geometry pushes many weights to become exactly zero.

Automatic feature selection

Because lasso zeroes out weights, the fitted model uses only a subset of features. This produces sparse, interpretable models and doubles as a feature selection method. As alpha increases, more weights vanish, so the path from alpha=0 to large alpha traces a sequence of ever-simpler models.

from sklearn.linear_model import LassoCV
model = LassoCV(cv=5).fit(X_train, y_train)
kept = (model.coef_ != 0).sum()   # features retained

Trade-offs

• Sparsity aids interpretation and speeds prediction.
• With correlated features lasso picks one and drops the rest, which can be arbitrary.
• The L1 objective has no closed form; coordinate descent solves it efficiently.
• For grouped correlated features, elastic net is usually better.

Practical notes

Standardize features before fitting so the penalty is fair across scales. Select alpha by cross-validation. When you suspect only a handful of features truly matter, lasso is the natural choice; when many features each contribute a little, prefer ridge, and when both effects are present use elastic net.

Geometrically, the L1 constraint region is a diamond with corners on the axes, and the least-squares contours are most likely to first touch it at a corner, where one coordinate is zero. That corner geometry, absent from ridge's smooth circle, is the reason lasso yields exact zeros and behaves as a built-in feature selector rather than a mere shrinker.