learn — vonhas

Section 5 · Regression

Regularization Details

Executive Summary

Why does regularization work? Three lenses: analytical (smaller weights ⇒ smoother predictions), geometric (the L1 diamond meets the loss contour at axes — zeros pop out; the L2 ball is round so it never quite does), and probabilistic (regularization is a prior on coefficients — Gaussian for L2, Laplacian for L1). Pick λ from the cross-validated dip.

Key Concepts

Code cheat-sheet

import numpy as np
from sklearn.linear_model import RidgeCV, LassoCV
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
 
alphas = np.logspace(-4, 4, 50)
 
ridge = Pipeline([
    ("scaler", StandardScaler()),
    ("ridge",  RidgeCV(alphas=alphas, cv=5)),
]).fit(X_train, y_train)
 
lasso = Pipeline([
    ("scaler", StandardScaler()),
    ("lasso",  LassoCV(alphas=alphas, cv=5, max_iter=10_000)),
]).fit(X_train, y_train)
 
print("Ridge α*:", ridge.named_steps["ridge"].alpha_)
print("Lasso α*:", lasso.named_steps["lasso"].alpha_)
# Pull ranked coefficients out of a fitted regularised model
import pandas as pd
 
coefs = lasso.named_steps["lasso"].coef_
names = X_train.columns
ranked = (
    pd.DataFrame({"feature": names, "coef": coefs})
      .assign(abs_coef=lambda d: d.coef.abs())
      .sort_values("abs_coef", ascending=False)
)
print(ranked.head(10))
print("zeroed out:", (coefs == 0).sum(), "/", len(coefs))

Math

β^ridge=arg⁡min⁡β  ∥y−Xβ∥22s.t.∥β∥22≤s\hat{\beta}_{\text{ridge}} = \arg\min_{\beta}\; \|y - X\beta\|_2^2 \quad \text{s.t.}\quad \|\beta\|_2^2 \le s

Ridge as constrained OLS — minimise the residual, stay inside an L2 ball of radius √s.

β^lasso=arg⁡min⁡β  ∥y−Xβ∥22s.t.∥β∥1≤s\hat{\beta}_{\text{lasso}} = \arg\min_{\beta}\; \|y - X\beta\|_2^2 \quad \text{s.t.}\quad \|\beta\|_1 \le s

LASSO swaps the ball for a diamond. Same optimisation idea, very different geometry at the boundary.

p(β)Ridge=N(0,τ2)⟺p(β)LASSO=Laplace(0,b)p(\beta)_{\text{Ridge}} = \mathcal{N}(0, \tau^2) \quad\Longleftrightarrow\quad p(\beta)_{\text{LASSO}} = \text{Laplace}(0, b)

Equivalent Bayesian priors. The Laplace prior's spike at zero is why L1 produces exact zeros.

λ⋆=arg⁡min⁡λ  CV^ ⁣(β^λ)\lambda^{\star} = \arg\min_{\lambda}\; \widehat{\mathrm{CV}}\!\left(\hat{\beta}_\lambda\right)

Pick λ as the minimiser of cross-validated loss. Empirical, not theoretical — and that's fine.

Self-check

Gotchas

Standardise before everything. All three views — analytical, geometric, probabilistic — assume coefficients live on a comparable scale. Without standardisation the penalty is biased toward features that happen to be measured in small units, and the geometric picture no longer applies.

Don't read absolute coefficient magnitudes as effect sizes. After standardisation, a coefficient of 0.5 means "a one-standard- deviation move in this feature shifts y by half a standard deviation of y" — not "0.5 dollars". Translate back if you need real-world units.

Same α across Ridge and LASSO ≠ same regularisation strength. The penalties have different units. Don't compare them by setting α equal — compare them at their respective CV optima.

LASSO's chosen features are not stable. Among correlated predictors LASSO picks one almost at random. Refit on a slightly different sample and the "important" feature changes. If you need stable rankings, use stability-selection or Elastic Net.

The U-curve can be flat. Sometimes CV error barely moves over a wide α range. That's a signal that regularisation isn't doing much work here — either the model is well-specified already, or features are weak. Either way, don't over-interpret the chosen α.