enc = pd.read_parquet("data/health/encounters.parquet")
mem = pd.read_csv("data/health/members.csv")
e = enc[enc["service_date"].dt.year == 2024]

counts = e.pivot_table(index="member_id",
                       columns="service_category",
                       values="encounter_id",
                       aggfunc="count", fill_value=0)
counts.columns = [c.lower().replace(" ", "_")
                  for c in counts.columns]
prof = counts.reindex(mem["member_id"], fill_value=0)

prof["total_allowed"] = (e.groupby("member_id")["allowed"]
                          .sum().reindex(prof.index, fill_value=0))
prof["active_months"] = (e.assign(m=e["service_date"].dt.month)
                          .groupby("member_id")["m"].nunique()
                          .reindex(prof.index, fill_value=0))

180,472 encounters in 2024, 22.6 per member, become an 8,000 × 10 table:

from sklearn.preprocessing import StandardScaler
X = StandardScaler().fit_transform(np.log1p(prof))

1. log1p tames the skew: one member with 133 fills shouldn’t define the pharmacy axis
2. Standardize so each feature has mean 0, sd 1. Otherwise total_allowed (sd $27,718) outweighs everything measured in counts

ARI (adjusted Rand index) compares two labelings: 1 = identical, 0 = chance. We can compute it only because this book was simulated from known archetypes; real data never gives you this.

$$\min_{C_1,\dots,C_k}\;\sum_{j=1}^{k}\sum_{i\in C_j}\lVert \mathbf{x}_i - \boldsymbol{\mu}_j\rVert^2$$

1. Place k centroids (k-means++: spread out at random)
2. Assign each member to the nearest centroid (Euclidean distance)
3. Update each centroid to the mean of its members
4. Repeat 2–3 until nothing moves

from sklearn.cluster import KMeans
km = KMeans(n_clusters=3, n_init=10, random_state=0).fit(X)
km.labels_          # cluster of each member
km.cluster_centers_ # 3 × 10, in standardized units
km.inertia_         # within-cluster sum of squares: 37,097

Elbow: inertia 47,844 → 37,097 → 33,628 for k = 2, 3, 4. The big gain stops after 3.

Silhouette (−1 to 1): how much closer each member is to its own cluster than to the next. 0.408 at k = 2, 0.393 at k = 3, then ≈ 0.21.

Against the hidden archetypes (answer key)

96.9% of members sit on the diagonal; ARI 0.926. The errors sit at the borders: heavy users having a quiet year look chronic.

Against next year’s cost (real-world test)