learn — vonhas

Section 3 · Regression

Cross-Validation

Executive Summary

A single train/test split is noisy — the score depends on which rows landed where. k-fold cross-validation averages over k splits to get a stabler estimate, and pairs naturally with hyperparameter search via `GridSearchCV`. Pipelines (`StandardScaler → estimator`) keep scaling inside the fold so the validation rows stay genuinely held-out.

Key Concepts

Code cheat-sheet

from sklearn.model_selection import KFold, cross_val_score, GridSearchCV
from sklearn.preprocessing import StandardScaler, PolynomialFeatures
from sklearn.linear_model import Ridge
from sklearn.pipeline import Pipeline
 
pipe = Pipeline([
    ("poly",   PolynomialFeatures(degree=2, include_bias=False)),
    ("scaler", StandardScaler()),
    ("ridge",  Ridge()),
])
 
kf = KFold(n_splits=5, shuffle=True, random_state=0)
 
grid = GridSearchCV(
    estimator=pipe,
    param_grid={
        "poly__degree": [1, 2, 3],
        "ridge__alpha": [0.01, 0.1, 1, 10, 100],
    },
    cv=kf,
    scoring="neg_mean_squared_error",
    n_jobs=-1,
)
 
grid.fit(X_train, y_train)
print(grid.best_params_, grid.best_score_)
y_pred = grid.predict(X_test)
# Quick k-fold sanity check on a single model
scores = cross_val_score(pipe, X, y, cv=5, scoring="r2")
print(f"R² = {scores.mean():.3f} ± {scores.std():.3f}")

Math

CV^k=1k∑i=1kL ⁣(y foldi,  y^ foldi)\hat{\mathrm{CV}}_k = \frac{1}{k} \sum_{i=1}^{k} \mathcal{L}\!\left(y_{\,\text{fold}_i},\; \hat{y}_{\,\text{fold}_i}\right)

Average loss across k folds. Lower variance estimator than a single hold-out — and roughly the same expected value.

Ltrain(θ)  ↓  ,LCV(θ)  ⌣  as complexity  ↑\mathcal{L}_{\text{train}}(\theta) \;\downarrow\; , \quad \mathcal{L}_{\text{CV}}(\theta) \;\smile\; \text{as complexity}\;\uparrow

Training loss falls, CV loss U-curves. The bottom of the U is where the hyperparameter should sit.

Scoringsklearn:maximize−MSE\text{Scoring}_{\text{sklearn}}: \quad \text{maximize} \quad -\text{MSE}

sklearn maximises whatever you pass to scoring. To minimise MSE you ask for neg_mean_squared_error — it flips the sign so the grid search picks the right winner.

Self-check

Gotchas

shuffle=True matters. Default KFold keeps the original row order. If your data is sorted by date, neighbourhood, or anything else correlated with y, your folds become time-series-like and the scores get weird. Shuffle unless you have a reason not to.

Scaling outside the pipeline leaks. A common bug: scale X once, then run CV. The validation folds were scaled using means computed from themselves. Either put the scaler in the pipeline or do the splitting first and scale per-fold manually.

neg_mean_squared_error not mean_squared_error. sklearn always maximises the scoring function. Pass the neg_ variant for error-like metrics, otherwise the grid will pick the worst model.

LASSO inside CV may not converge. Default max_iter for Lasso / ElasticNet is low. With many features or polynomial expansion, bump it (max_iter=5000 or higher) or you'll get convergence warnings and slightly off coefficients.

CV with grouped data needs GroupKFold. If multiple rows come from the same patient, household, or session, plain k-fold will split the group across train and test — leakage. GroupKFold or StratifiedGroupKFold keeps each group whole.