Train / validation / test

from sklearn.model_selection import train_test_split
X_tr, X_te, y_tr, y_te = train_test_split(
    X, y, train_size=500, random_state=42)

K-fold cross-validation

Split the training data into K folds; fit on K − 1, score on the one left out; rotate; average. Every row is validated exactly once.

from sklearn.model_selection import KFold, cross_val_score
kf = KFold(5, shuffle=True, random_state=0)
cross_val_score(ols_pipe, X_tr, y_tr, cv=kf, scoring="r2")
# [0.240 0.232 0.319 0.384 0.370]  mean 0.309

bio = pd.read_csv("data/health/biometrics.csv")
# member demographics + 8 chronic conditions
# + 2024 wellness screening (BMI, blood pressure, labs…)
# + 20 lifestyle survey items
# + 2024 encounter counts by category + log 2024 cost
X = pd.get_dummies(features, columns=["sex", "plan", "region"],
                   drop_first=True, dtype=float)
y = np.log1p(cost_2025)        # mean 7.75, sd 1.84
X.shape                        # (8000, 60)

To mimic a small employer group, we fit on 500 members and test on the other 7,500.

Seven pairs have |r| > 0.7: BMI–waist 0.87, HbA1c–glucose 0.87, LDL–total cholesterol 0.85, eGFR–creatinine −0.85…

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LinearRegression

ols = make_pipeline(StandardScaler(), LinearRegression())
ols.fit(X_tr, y_tr)
r2_score(y_tr, ols.predict(X_tr))   # 0.482
r2_score(y_te, ols.predict(X_te))   # 0.292

• Training R² 0.48; test R² 0.29: a large generalization gap
• The 20 survey items have no effect on cost, yet OLS gives them coefficients summing to 0.90 in absolute value
• Correlated blocks get large, offsetting coefficients: weight +0.200, waist −0.183

$$\textbf{Ridge:}\;\; \min_\beta\; \tfrac{1}{2n}\lVert y - \mathbf{X}\beta\rVert^2 + \alpha\lVert\beta\rVert_2^2$$

$$\textbf{Lasso:}\;\; \min_\beta\; \tfrac{1}{2n}\lVert y - \mathbf{X}\beta\rVert^2 + \alpha\lVert\beta\rVert_1$$

$$\textbf{ElasticNet:}\;\; \ldots + \alpha\big(\rho\lVert\beta\rVert_1 + \tfrac{1-\rho}{2}\lVert\beta\rVert_2^2\big)$$

α sets the strength; ρ (l1_ratio) the mix. The intercept is never penalized.

The penalty treats every coefficient alike, but a coefficient’s size depends on its feature’s units. Steps per day (sd 3,529) needs a tiny coefficient; diabetes (sd 0.36) a large one. Without scaling, the penalty barely touches the first and crushes the second.

# ✅ scaler learns mean/sd from each training fold only
lasso = make_pipeline(StandardScaler(), LassoCV(cv=kf))

# ❌ unscaled: same α, different model
Lasso(alpha=0.107).fit(X_tr, y_tr)
# keeps 20 features incl. steps_per_day and two survey items
# test R² 0.322 (scaled: 0.355)

from sklearn.linear_model import LassoCV, RidgeCV, ElasticNetCV

lasso = make_pipeline(StandardScaler(),
                      LassoCV(cv=kf, random_state=0)).fit(X_tr, y_tr)
lasso[-1].alpha_               # 0.107
(lasso[-1].coef_ != 0).sum()   # 13 of 60

ridge = make_pipeline(StandardScaler(),
          RidgeCV(alphas=np.logspace(-2, 4, 60), cv=kf))
enet = make_pipeline(StandardScaler(),
          ElasticNetCV(l1_ratio=[0.1, 0.5, 0.9], cv=kf))

The 1-SE rule picks the largest α within one standard error of the minimum: a simpler model (7 features, test R² 0.329) that’s statistically as good.

Every penalized model has a lower training R² and a higher test R² than OLS. That’s the trade-off working.

Prior utilization enters first and beats every biometric.

With enough data per feature, OLS catches up. Regularization matters most when data is thin relative to the number of parameters:

• Small employer groups, new products, rare coverages
• Many-level factors: vehicle model, ZIP code, diagnosis code
• Interactions: 6 styles × 8 ages × 100 counties is 4,800 cells