learn — vonhas

Section 2 · Regression

Data Splits & Polynomial Regression

Executive Summary

Held-out evaluation is non-negotiable: train/test splits stop you from grading the model on the answers it saw during training. Polynomial features let a linear model bend — but bend too far and you overfit. The complexity-vs-error curve is the canonical picture: train error keeps falling, test error U-shapes, the sweet spot is the bottom of that U.

Key Concepts

Code cheat-sheet

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
 
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42
)
 
poly = PolynomialFeatures(degree=2, include_bias=False)
X_train_pf = poly.fit_transform(X_train)
X_test_pf  = poly.transform(X_test)   # transform only — never re-fit
 
lr = LinearRegression().fit(X_train_pf, y_train)
print("test MSE:", mean_squared_error(y_test, lr.predict(X_test_pf)))
from sklearn.model_selection import StratifiedShuffleSplit
 
sss = StratifiedShuffleSplit(n_splits=5, test_size=0.2, random_state=0)
for train_idx, test_idx in sss.split(X, y):
    X_tr, X_te = X.iloc[train_idx], X.iloc[test_idx]
    y_tr, y_te = y.iloc[train_idx], y.iloc[test_idx]

Math

Training error  →  complexity    0\text{Training error} \;\xrightarrow{\;\text{complexity}\;}\; 0

The training-error curve just keeps falling — that's why it's a bad stand-in for real performance.

Test error(complexity)=bias2+variance+σ2\text{Test error}(\text{complexity}) = \text{bias}^2 + \text{variance} + \sigma^2

The test-error U comes from two terms moving in opposite directions as complexity grows. σ² is irreducible noise.

ϕ2(x1,x2)=[ 1,  x1,  x2,  x12,  x1x2,  x22 ]\phi_2(x_1, x_2) = [\, 1,\; x_1,\; x_2,\; x_1^2,\; x_1 x_2,\; x_2^2 \,]

Degree-2 polynomial expansion of two features. Six columns from two — including the interaction x₁x₂.

Self-check

Gotchas

Split before you transform. Fit your scaler / one-hot encoder / log transform on X_train only, then transform the test set. Fitting on the combined dataset leaks test statistics into training.

random_state is your friend. Without it your "the new model is better" comparison might just be a luckier split. Pin the seed and compare apples to apples.

One-hot encoding can blow up your column count. 25 unique neighbourhoods becomes 25 columns (or 24 if you drop one for identifiability). Polynomial features on top of that and you're in the thousands fast — overfitting risk skyrockets.

Drop one dummy if you care about coefficients. Otherwise the encoded columns are perfectly collinear with the intercept and the coefficients are no longer uniquely identified. Pure prediction mostly survives it; interpretation doesn't.

PolynomialFeatures needs a 2-D input. data['x'] is a 1-D Series and will error. Use data[['x']] (double brackets) so it arrives as a DataFrame with shape (n, 1).