{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cell-000",
   "metadata": {},
   "source": [
    "# Lesson 5 — Unsupervised Learning & Risk Group Discovery\n",
    "\n",
    "**Companion notebook** for the Lesson 5 slides. You'll turn 180,000 health encounters\n",
    "into one profile per member, cluster the members with K-means, choose k, and translate\n",
    "the clusters into patient archetypes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-001",
   "metadata": {},
   "outputs": [],
   "source": [
    "def check(label, got, expected, tol=1e-6):\n",
    "    \"\"\"Compare your answer with the expected value; prints ✅ or ❌ (never raises).\"\"\"\n",
    "    import math\n",
    "    try:\n",
    "        if isinstance(expected, float) or isinstance(got, float):\n",
    "            ok = math.isclose(float(got), float(expected), rel_tol=tol, abs_tol=tol)\n",
    "        elif isinstance(expected, (list, tuple)) and expected and isinstance(expected[0], str):\n",
    "            ok = list(got) == list(expected)\n",
    "        elif hasattr(expected, \"__len__\") and not isinstance(expected, str):\n",
    "            import numpy as np\n",
    "            ok = np.allclose(np.asarray(got, dtype=float), np.asarray(expected, dtype=float), rtol=tol, atol=tol)\n",
    "        else:\n",
    "            ok = got == expected\n",
    "    except Exception:\n",
    "        ok = False\n",
    "    print((\"✅ \" if ok else \"❌ \") + f\"{label}: got {got!r}, expected {expected!r}\")\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "from sklearn.cluster import KMeans\n",
    "from sklearn.metrics import adjusted_rand_score, silhouette_score\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "enc = pd.read_parquet(\"../data/health/encounters.parquet\")\n",
    "mem = pd.read_csv(\"../data/health/members.csv\")\n",
    "e = enc[enc[\"service_date\"].dt.year == 2024]\n",
    "print(f\"{len(e):,} encounters in 2024, {len(e) / len(mem):.1f} per member\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-002",
   "metadata": {},
   "source": [
    "## 1. From encounters to member profiles\n",
    "\n",
    "One row per member: encounters in each service category, total allowed cost, and the\n",
    "number of months with any care. `reindex` keeps members with no encounters at all."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-003",
   "metadata": {},
   "outputs": [],
   "source": [
    "counts = e.pivot_table(index=\"member_id\", columns=\"service_category\", values=\"encounter_id\",\n",
    "                       aggfunc=\"count\", fill_value=0)\n",
    "counts.columns = [c.lower().replace(\" \", \"_\") for c in counts.columns]\n",
    "cols = [\"office_visit\", \"specialist\", \"lab\", \"imaging\", \"pharmacy\", \"emergency\",\n",
    "        \"outpatient_surgery\", \"inpatient\"]\n",
    "prof = counts.reindex(mem[\"member_id\"], fill_value=0)[cols]\n",
    "prof[\"total_allowed\"] = e.groupby(\"member_id\")[\"allowed\"].sum().reindex(prof.index, fill_value=0)\n",
    "prof[\"active_months\"] = (e.assign(m=e[\"service_date\"].dt.month).groupby(\"member_id\")[\"m\"]\n",
    "                          .nunique().reindex(prof.index, fill_value=0))\n",
    "prof.describe().T.round(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-004",
   "metadata": {},
   "source": [
    "## 2. The curse of dimensionality\n",
    "\n",
    "How much farther is the farthest of 500 random points than the nearest, as the number\n",
    "of features grows?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-005",
   "metadata": {},
   "outputs": [],
   "source": [
    "rng = np.random.default_rng(5)\n",
    "for d in [2, 10, 100, 500]:\n",
    "    pts, q = rng.random((500, d)), rng.random(d)\n",
    "    dist = np.linalg.norm(pts - q, axis=1)\n",
    "    print(f\"{d:4d} features: (farthest - nearest) / nearest = {(dist.max() - dist.min()) / dist.min():.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-006",
   "metadata": {},
   "source": [
    "---\n",
    "## Your turn 1 — prepare the features and choose k\n",
    "\n",
    "1. `X = StandardScaler().fit_transform(np.log1p(prof))`\n",
    "2. For k = 2…10, fit `KMeans(k, n_init=10, random_state=0)` and record the inertia and\n",
    "   the silhouette score (on all members).\n",
    "3. Plot both against k."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-007",
   "metadata": {},
   "outputs": [],
   "source": [
    "X = StandardScaler().fit_transform(np.log1p(prof))\n",
    "inertia = sil = None\n",
    "...  # your code here\n",
    "check(\"inertia, k = 3\", None if inertia is None else round(inertia[3]), 37097)\n",
    "check(\"silhouette, k = 2\", None if sil is None else round(sil[2], 3), 0.408)\n",
    "check(\"silhouette, k = 3\", None if sil is None else round(sil[3], 3), 0.393)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-008",
   "metadata": {},
   "source": [
    "## Your turn 2 — fit k = 3 and name the clusters\n",
    "\n",
    "Cluster numbers are arbitrary. Name the clusters by mean `total_allowed`: lowest\n",
    "\"Healthy\", middle \"Chronic-Maintenance\", highest \"Heavy-Use\". Store a Series `cluster`\n",
    "of names, indexed like `prof`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-009",
   "metadata": {},
   "outputs": [],
   "source": [
    "cluster = None\n",
    "...  # your code here\n",
    "sizes = None if cluster is None else cluster.value_counts()\n",
    "check(\"Healthy members\", None if sizes is None else int(sizes[\"Healthy\"]), 4868)\n",
    "check(\"Chronic-Maintenance members\", None if sizes is None else int(sizes[\"Chronic-Maintenance\"]), 2501)\n",
    "check(\"Heavy-Use members\", None if sizes is None else int(sizes[\"Heavy-Use\"]), 631)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-010",
   "metadata": {},
   "source": [
    "## Your turn 3 — profile the clusters\n",
    "\n",
    "Build a table of mean profile values by cluster, plus each cluster's share of members\n",
    "and share of total 2024 cost. Then draw a z-score heatmap of the centroids:\n",
    "the mean of `X` (standardized log features) within each cluster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-011",
   "metadata": {},
   "outputs": [],
   "source": [
    "heavy_cost_share = None\n",
    "if cluster is not None:\n",
    "    ...  # your code here\n",
    "check(\"Heavy-Use share of cost\", None if heavy_cost_share is None else round(heavy_cost_share, 2), 0.60)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-012",
   "metadata": {},
   "source": [
    "## 4. A radar chart"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-013",
   "metadata": {},
   "outputs": [],
   "source": [
    "if cluster is not None:\n",
    "    Z = pd.DataFrame(X, index=prof.index, columns=prof.columns).groupby(cluster).mean()\n",
    "    ang = np.linspace(0, 2 * np.pi, len(Z.columns), endpoint=False).tolist()\n",
    "    ax = plt.figure(figsize=(5.5, 5)).add_subplot(111, polar=True)\n",
    "    for name in [\"Healthy\", \"Chronic-Maintenance\", \"Heavy-Use\"]:\n",
    "        v = Z.loc[name].tolist()\n",
    "        ax.plot(ang + ang[:1], v + v[:1], label=name)\n",
    "        ax.fill(ang + ang[:1], v + v[:1], alpha=0.1)\n",
    "    ax.set_xticks(ang, [c.replace(\"_\", \"\\n\") for c in Z.columns], fontsize=8)\n",
    "    ax.legend(loc=\"upper right\", bbox_to_anchor=(1.4, 1.1))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-014",
   "metadata": {},
   "source": [
    "## Your turn 4 — are the clusters real?\n",
    "\n",
    "This book was simulated from hidden archetypes, stored in\n",
    "`../data/health/archetypes.csv` (real data never gives you this). Compute the adjusted\n",
    "Rand index between `archetype_2024` and your clusters, and cross-tabulate them.\n",
    "Then compare mean 2025 cost by 2024 cluster: do the groups still separate a year later?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-015",
   "metadata": {},
   "outputs": [],
   "source": [
    "truth = pd.read_csv(\"../data/health/archetypes.csv\").set_index(\"member_id\")\n",
    "ari = None\n",
    "if cluster is not None:\n",
    "    ...  # your code here\n",
    "check(\"adjusted Rand index\", None if ari is None else round(ari, 3), 0.926)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-016",
   "metadata": {},
   "source": [
    "## Your turn 5 — why transform and scale?\n",
    "\n",
    "Refit k = 3 on the raw `prof` values (no log, no scaling) and compute the ARI again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-017",
   "metadata": {},
   "outputs": [],
   "source": [
    "ari_raw = None\n",
    "...  # your code here\n",
    "check(\"ARI without log or scaling\", None if ari_raw is None else round(ari_raw, 3), 0.178)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}