{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cell-000",
   "metadata": {},
   "source": [
    "# Lesson 4 — Overfitting, Generalization & Regularization\n",
    "\n",
    "**Companion notebook** for the Lesson 4 slides. You'll watch a polynomial overfit, then\n",
    "regularize a 60-feature health cost model trained on only 500 members with Ridge,\n",
    "Lasso, and ElasticNet, choosing each penalty by cross-validation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-001",
   "metadata": {},
   "outputs": [],
   "source": [
    "def check(label, got, expected, tol=1e-6):\n",
    "    \"\"\"Compare your answer with the expected value; prints ✅ or ❌ (never raises).\"\"\"\n",
    "    import math\n",
    "    try:\n",
    "        if isinstance(expected, float) or isinstance(got, float):\n",
    "            ok = math.isclose(float(got), float(expected), rel_tol=tol, abs_tol=tol)\n",
    "        elif isinstance(expected, (list, tuple)) and expected and isinstance(expected[0], str):\n",
    "            ok = list(got) == list(expected)\n",
    "        elif hasattr(expected, \"__len__\") and not isinstance(expected, str):\n",
    "            import numpy as np\n",
    "            ok = np.allclose(np.asarray(got, dtype=float), np.asarray(expected, dtype=float), rtol=tol, atol=tol)\n",
    "        else:\n",
    "            ok = got == expected\n",
    "    except Exception:\n",
    "        ok = False\n",
    "    print((\"✅ \" if ok else \"❌ \") + f\"{label}: got {got!r}, expected {expected!r}\")\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.linear_model import (ElasticNetCV, Lasso, LassoCV, LinearRegression, RidgeCV)\n",
    "from sklearn.metrics import mean_squared_error, r2_score\n",
    "from sklearn.model_selection import KFold, cross_val_score, train_test_split\n",
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.preprocessing import PolynomialFeatures, StandardScaler"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-002",
   "metadata": {},
   "source": [
    "## 1. Bias and variance with a polynomial\n",
    "\n",
    "Thirty noisy drivers from a U-shaped frequency curve; polynomials of degree 1, 4, 15."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-003",
   "metadata": {},
   "outputs": [],
   "source": [
    "rng = np.random.default_rng(4)\n",
    "\n",
    "\n",
    "def true_curve(a):\n",
    "    return 0.06 * (1.0 + 1.5 * np.exp(-(a - 17) / 5.0) + 0.40 * np.clip((a - 65) / 15, 0, None) ** 2)\n",
    "\n",
    "\n",
    "a = rng.uniform(17, 90, 30)\n",
    "y_poly = true_curve(a) + rng.normal(0, 0.02, 30)\n",
    "grid = np.linspace(17, 90, 300)\n",
    "a_new = rng.uniform(17, 90, 2000)\n",
    "y_new = true_curve(a_new) + rng.normal(0, 0.02, 2000)\n",
    "fig, axes = plt.subplots(1, 3, figsize=(12, 3.2), sharey=True)\n",
    "for ax, deg in zip(axes, [1, 4, 15]):\n",
    "    m = make_pipeline(StandardScaler(), PolynomialFeatures(deg), LinearRegression()).fit(a[:, None], y_poly)\n",
    "    ax.scatter(a, y_poly, s=12)\n",
    "    ax.plot(grid, m.predict(grid[:, None]), color=\"orange\")\n",
    "    ax.set(title=f\"degree {deg}\", ylim=(-0.02, 0.2))\n",
    "    print(f\"degree {deg:2d}: train MSE {mean_squared_error(y_poly, m.predict(a[:, None])):.6f}  \"\n",
    "          f\"new-data MSE {mean_squared_error(y_new, m.predict(a_new[:, None])):.6f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-004",
   "metadata": {},
   "source": [
    "## 2. The member cost dataset\n",
    "\n",
    "Features known at the end of 2024 (demographics, conditions, screening results, a\n",
    "lifestyle survey, and 2024 utilization); target: log(1 + allowed cost in 2025)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-005",
   "metadata": {},
   "outputs": [],
   "source": [
    "enc = pd.read_parquet(\"../data/health/encounters.parquet\")\n",
    "mem = pd.read_csv(\"../data/health/members.csv\")\n",
    "bio = pd.read_csv(\"../data/health/biometrics.csv\")\n",
    "\n",
    "e24 = enc[enc[\"service_date\"].dt.year == 2024]\n",
    "counts = e24.pivot_table(index=\"member_id\", columns=\"service_category\", values=\"encounter_id\",\n",
    "                         aggfunc=\"count\", fill_value=0)\n",
    "counts.columns = [\"n_\" + c.lower().replace(\" \", \"_\") for c in counts.columns]\n",
    "order = [\"n_office_visit\", \"n_specialist\", \"n_lab\", \"n_imaging\", \"n_pharmacy\", \"n_emergency\",\n",
    "         \"n_outpatient_surgery\", \"n_inpatient\"]\n",
    "counts = counts.reindex(mem[\"member_id\"], fill_value=0)[order]\n",
    "cost = enc.assign(year=enc[\"service_date\"].dt.year).pivot_table(\n",
    "    index=\"member_id\", columns=\"year\", values=\"allowed\", aggfunc=\"sum\", fill_value=0\n",
    ").reindex(mem[\"member_id\"], fill_value=0)\n",
    "\n",
    "feat = mem.merge(bio, on=\"member_id\").set_index(\"member_id\").join(counts)\n",
    "feat[\"log_cost_2024\"] = np.log1p(cost[2024])\n",
    "X = pd.get_dummies(feat, columns=[\"sex\", \"plan\", \"region\"], drop_first=True, dtype=float)\n",
    "y = np.log1p(cost[2025])\n",
    "print(X.shape)\n",
    "X_tr, X_te, y_tr, y_te = train_test_split(X, y, train_size=500, random_state=42)\n",
    "kf = KFold(5, shuffle=True, random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-006",
   "metadata": {},
   "source": [
    "---\n",
    "## Your turn 1 — OLS on 500 members\n",
    "\n",
    "Fit `make_pipeline(StandardScaler(), LinearRegression())` on the training members.\n",
    "Store the training R² and the test R²."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-007",
   "metadata": {},
   "outputs": [],
   "source": [
    "ols = r2_train = r2_test = None\n",
    "...  # your code here\n",
    "check(\"OLS training R²\", None if r2_train is None else round(r2_train, 3), 0.482)\n",
    "check(\"OLS test R²\", None if r2_test is None else round(r2_test, 3), 0.292)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-008",
   "metadata": {},
   "source": [
    "## Your turn 2 — the Lasso\n",
    "\n",
    "Fit `make_pipeline(StandardScaler(), LassoCV(cv=kf, random_state=0))`. Store the chosen\n",
    "α, the number of non-zero coefficients, and the test R². Which features survived?\n",
    "What happened to the 20 survey items, and to BMI, weight, and waist?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-009",
   "metadata": {},
   "outputs": [],
   "source": [
    "lasso = None\n",
    "...  # your code here\n",
    "check(\"Lasso alpha\", None if lasso is None else round(lasso[-1].alpha_, 3), 0.107)\n",
    "check(\"non-zero features\", None if lasso is None else int((lasso[-1].coef_ != 0).sum()), 13)\n",
    "check(\"Lasso test R²\", None if lasso is None else round(r2_score(y_te, lasso.predict(X_te)), 3), 0.355)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-010",
   "metadata": {},
   "source": [
    "## Your turn 3 — Ridge and ElasticNet\n",
    "\n",
    "`RidgeCV(alphas=np.logspace(-2, 4, 60), cv=kf)` and\n",
    "`ElasticNetCV(l1_ratio=[0.1, 0.5, 0.9], cv=kf, random_state=0)`, each after a\n",
    "`StandardScaler`. Store both test R² values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-011",
   "metadata": {},
   "outputs": [],
   "source": [
    "ridge_r2 = enet_r2 = None\n",
    "...  # your code here\n",
    "check(\"Ridge test R²\", None if ridge_r2 is None else round(ridge_r2, 3), 0.341)\n",
    "check(\"ElasticNet test R²\", None if enet_r2 is None else round(enet_r2, 3), 0.357)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-012",
   "metadata": {},
   "source": [
    "## Your turn 4 — the one-standard-error rule\n",
    "\n",
    "From `lasso[-1].mse_path_` (alphas × folds), find the largest α whose mean CV error is\n",
    "within one standard error of the minimum. Refit a `Lasso(alpha=...)` pipeline with it.\n",
    "How many features does it keep?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-013",
   "metadata": {},
   "outputs": [],
   "source": [
    "alpha_1se = n_1se = None\n",
    "if lasso is not None:\n",
    "    ...  # your code here\n",
    "check(\"1-SE alpha\", None if alpha_1se is None else round(alpha_1se, 3), 0.264)\n",
    "check(\"1-SE features\", n_1se, 7)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-014",
   "metadata": {},
   "source": [
    "## Your turn 5 — when does regularization matter?\n",
    "\n",
    "Repeat the OLS vs Lasso comparison with 200 training members (`random_state=42`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-015",
   "metadata": {},
   "outputs": [],
   "source": [
    "ols200 = lasso200 = None\n",
    "...  # your code here\n",
    "check(\"OLS test R², 200 members\", None if ols200 is None else round(ols200, 3), 0.115)\n",
    "check(\"Lasso test R², 200 members\", None if lasso200 is None else round(lasso200, 3), 0.331)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}