learn — vonhas

Section 4 · Regression

Bias-Variance & Regularization

Executive Summary

Error decomposes into bias (the model is systematically wrong) and variance (the model wobbles across resamples). Regularization shrinks coefficients to trade a bit of bias for a lot less variance. Ridge (L2) pulls every coefficient toward zero; LASSO (L1) zeroes some — built-in feature selection; Elastic Net mixes the two with an α knob.

Key Concepts

Code cheat-sheet

from sklearn.linear_model import Ridge, Lasso, ElasticNet
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.model_selection import GridSearchCV
 
pipe = Pipeline([
    ("scaler", StandardScaler()),     # MANDATORY before L1/L2
    ("model",  Ridge()),
])
 
grid = GridSearchCV(
    pipe,
    {"model__alpha": [0.001, 0.01, 0.1, 1, 10, 100]},
    cv=5,
    scoring="r2",
).fit(X_train, y_train)
 
print(grid.best_params_)
from sklearn.linear_model import RidgeCV, LassoCV, ElasticNetCV
 
# Built-in CV variants — quicker than full GridSearchCV for the alpha sweep
ridge = RidgeCV(alphas=[0.01, 0.1, 1, 10, 100], cv=5).fit(X_train_s, y_train)
lasso = LassoCV(alphas=None, cv=5, max_iter=10_000).fit(X_train_s, y_train)
enet  = ElasticNetCV(l1_ratio=[0.1, 0.5, 0.9], cv=5, max_iter=10_000).fit(X_train_s, y_train)
from sklearn.feature_selection import RFECV
 
rfe = RFECV(
    estimator=Lasso(alpha=0.01, max_iter=10_000),
    step=1,
    cv=5,
    scoring="r2",
).fit(X_train_s, y_train)
 
print("kept", rfe.n_features_, "features")

Math

E[(y^−y)2]=(Ey^−y)2⏟bias2+E[(y^−Ey^)2]⏟variance+σ2\mathbb{E}[(\hat{y} - y)^2] = \underbrace{(\mathbb{E}\hat{y} - y)^2}_{\text{bias}^2} + \underbrace{\mathbb{E}[(\hat{y} - \mathbb{E}\hat{y})^2]}_{\text{variance}} + \sigma^2

Expected squared error splits into bias², variance, and irreducible noise σ². You can trade the first two; you can't touch the third.

Jridge(β)=∑i=1n(yi−xi⊤β)2+λ∑j=1pβj2J_{\text{ridge}}(\beta) = \sum_{i=1}^{n} (y_i - x_i^\top \beta)^2 + \lambda \sum_{j=1}^{p} \beta_j^2

Ridge cost: OLS plus an L2 penalty on the coefficients. The bigger the λ, the harder the squeeze on β.

Jlasso(β)=∑i=1n(yi−xi⊤β)2+λ∑j=1p∣βj∣J_{\text{lasso}}(\beta) = \sum_{i=1}^{n} (y_i - x_i^\top \beta)^2 + \lambda \sum_{j=1}^{p} |\beta_j|

LASSO swaps the square for absolute value. Same shrinking effect, but the geometry lets coefficients land exactly on zero.

Jenet(β)=∑(yi−xi⊤β)2+λ(α∑∣βj∣+(1−α)∑βj2)J_{\text{enet}}(\beta) = \sum (y_i - x_i^\top \beta)^2 + \lambda \left( \alpha \sum |\beta_j| + (1-\alpha) \sum \beta_j^2 \right)

Elastic Net: convex combination of L1 and L2. α = 1 ⇒ LASSO, α = 0 ⇒ Ridge.

Self-check

Gotchas

Always scale before L1/L2. Without scaling, the penalty isn't fair across features and the resulting coefficient ranking is meaningless. Standardise inside a pipeline so it refits per fold.

LASSO + correlated features = chaos. Among a group of correlated features, LASSO will arbitrarily keep one and zero the rest, and the chosen one can flip across resamples. Use Elastic Net or Ridge if stability matters.

Bump max_iter for Lasso/ElasticNet. Default is 1000. Polynomial expansion, small alpha, or many features and you'll see ConvergenceWarning. Set max_iter=5000 or 10_000 and move on.

Don't compare alphas across Ridge and LASSO. Ridge(alpha=1) and Lasso(alpha=1) are penalising different things — the alpha scales aren't comparable. Search each one's grid independently.

Regularization never improves training error. Of course it doesn't — it's deliberately pulling the fit away from the training data. Its job is the held-out score. If you're judging regularization on training R², you're judging it wrong.