Course 3 · Lesson 1 Statistical Foundations of Predictive Modeling
Python for Actuarial Students · Course 3
Lesson 1

Statistical Foundations of Predictive Modeling

Ordinary least squares, why it breaks on insurance data, and the log link that every pricing model in this course is built on.

Lesson 1 · OverviewBy the end of this lesson you can…

  • Place an actuarial problem in the supervised or unsupervised paradigm
  • State the Gauss–Markov assumptions and check them with residual plots
  • Explain why skewness, zeros, and non-constant variance break OLS on insurance data
  • Fit a log-linear model in statsmodels and read its coefficients as multiplicative relativities
  • Correct the retransformation bias when you go back from log-dollars to dollars
  • Use heteroscedasticity-consistent standard errors (HC0, HC3)
75 min
Concepts & worked examples
75 min
Lab: severity by service category

Companion notebook: lesson-01-statistical-foundations.ipynb — worked examples and self-checking exercises.

Course 3Two books of business

Motor book · data/motor/

  • Same six-table schema as Course 2, five policy years (2021–2025)
  • 84,346 vehicle-terms · 72,781 car-years · 4,565 liability claims
  • Ready-to-model frame: data/processed/motor_model.parquet
  • Lessons 2, 3, 7, 8: frequency, severity, pure premium, boosting

Health book · data/health/

  • 8,000 members, 375,810 encounters in 2024–2025
  • Eight service categories, from lab tests to inpatient stays
  • Wellness screening results and a lifestyle survey
  • Lessons 1, 4, 5, 6: severity, regularization, clustering, rules

Both books are simulated from a known risk model. Real data never tells you whether your model found the truth; these books do. Throughout the course, you’ll check fitted relativities against the values that generated the data.

Part 1

Supervised & Unsupervised

Two ways to learn from a portfolio.

ParadigmsWhat are you trying to learn?

ParadigmHas a target?Insurance examplesLessons
Supervised · regressionYes, a numberClaim frequency, severity, pure premium, reserves1–4, 7, 8
Supervised · classificationYes, a labelWill this policy lapse? Is this claim fraudulent? Which tier?6
Unsupervised · clusteringNoPatient archetypes, driver segments, territory groupings5
Unsupervised · anomalyNoUnusual billing patterns, data-quality outliers—

Supervised learning fits f(features) ≈ target on examples where the answer is known, then predicts it for new risks.

Unsupervised learning finds structure — groups, directions, outliers — in features alone. Its output must still be interpreted and validated.

WorkflowEvery model in this course follows the same path

QuestionWhat decision will the model support?
→
Target & exposureClaims per car-year? Cost per claim?
→
FeaturesRating factors, transformations
→
FitChoose a distribution and a link
→
ValidateDiagnostics, holdout, business sense
→
DeployRates, rules, reports

Most modeling mistakes happen before the first .fit(): the wrong target (loss ratio instead of pure premium), the wrong exposure, or a feature that leaks the answer.

This lesson is about the Fit box: what the classical linear model assumes, and what to do when insurance data refuses to cooperate.

Part 2

The Classical Linear Model

OLS, its assumptions, and how to check them.

OLSOrdinary least squares

$$y = \mathbf{X}\beta + \varepsilon \qquad \hat\beta = (\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{X}^\top y$$

OLS picks the β that minimizes the sum of squared residuals $\sum_i (y_i - \mathbf{x}_i^\top\beta)^2$.

Gauss–Markov assumptions

  1. Linearity — E[y] is linear in the parameters
  2. Full rank — no feature is an exact combination of others
  3. Exogeneity — E[ε | X] = 0
  4. Homoscedasticity — Var(ε | X) = σ², the same for every row
  5. No autocorrelation — errors are uncorrelated

Under 1–5, OLS is BLUE: the Best (minimum-variance) Linear Unbiased Estimator. Add normal errors and the t-tests and confidence intervals are exact.

Assumption 4 is the one insurance data breaks first. When it fails, the coefficients are still unbiased — but the standard errors are wrong, and so is every p-value built on them.

statsmodelsFitting OLS with a formula

import statsmodels.formula.api as smf

enc = pd.read_parquet("data/health/encounters.parquet")
mem = pd.read_csv("data/health/members.csv")
e = (enc[enc["service_date"].dt.year == 2024]
     .merge(mem, on="member_id", validate="many_to_one"))
e["age10"] = (e["age"] - 40) / 10     # decades from age 40

rhs = ("C(service_category, Treatment('Office Visit'))"
       " + age10 + C(region, Treatment('Suburban'))"
       " + C(plan, Treatment('PPO'))")
ols = smf.ols("allowed ~ " + rhs, data=e).fit()
ols.rsquared          # 0.461
ols.params["age10"]   # 30.7 dollars per decade
  • 180,472 encounters from 2024, one row each
  • C(col, Treatment('level')) makes dummy variables and names the base level
  • Coefficients are additive dollars: an inpatient stay adds $37,722 to the base; a Metro member adds $94.50 to every encounter, from a $55 lab test to a $30,000 admission

An R² of 0.46 looks respectable. The residuals tell a different story.

DiagnosticsRead the residuals, not the R²

OLS residuals versus fitted values fan out with the fitted value, and the normal Q-Q plot has extreme tails
  • Fan shape: residual spread grows with the fitted value — heteroscedasticity
  • Q–Q tails: residuals reach 30 standard deviations — far from normal
  • 3,810 negative fitted values (as low as −$107.51) for costs that can never be negative
  • Additive effects: an HMO discount of $56.80 whether the claim is a lab test or a surgery
Part 3

Why OLS Breaks on Insurance Data

Skewness, zeros, and variance that grows with the mean.

Problem 1Positive skewness

Allowed amounts are strongly right-skewed on the dollar scale and roughly symmetric on the log scale

Mean $441.14 vs median $111.48 · skewness 34.3 · only 11.1% of encounters exceed the mean · largest $303,889

When the mean sits at the 89th percentile, a model of the mean is driven by a handful of large claims. On the log scale the same data is close to symmetric.

Problem 2Zeros and concentration

BookStatisticValue
MotorVehicle-terms with no liability claim94.2%
HealthMembers with zero 2024 cost0.8%
HealthShare of 2024 cost from the top 10% of members68.6%

A normal distribution puts no probability on an exact zero, and is symmetric. Aggregate loss per policy is usually zero — and occasionally enormous.

Two fixes you’ll meet in this course:

  • Split the problem — model how often (frequency, Lesson 2) and how much (severity, Lesson 3) separately
  • Use a distribution with a mass at zero — the Tweedie, Lesson 3

Problem 3Variance grows with the mean

Standard deviation of allowed amount against its mean for eight service categories, close to a line of slope one on log axes
CategoryMeanStd devCV
Lab79670.85
Office Visit167680.40
Pharmacy1693241.92
Specialist3381750.52
Imaging7037431.06
Emergency2,7282,4400.89
Outpatient Surgery6,2824,8040.76
Inpatient37,89540,8791.08

The standard deviation is roughly proportional to the mean (a constant coefficient of variation) — nothing like OLS’s single σ².

Part 4

The Log Transformation

From additive dollars to multiplicative rating factors.

Log-linearModel log(cost): effects become multipliers

$$\ln y = \beta_0 + \beta_1 x_1 + \dots + \varepsilon \;\Longrightarrow\; y = e^{\beta_0}\cdot e^{\beta_1 x_1}\cdots e^{\varepsilon}$$

e["log_allowed"] = np.log(e["allowed"])
loglin = smf.ols("log_allowed ~ " + rhs, data=e).fit()

rel = np.exp(loglin.params)     # relativities
loglin.rsquared                 # 0.397
np.sqrt(loglin.scale)           # residual sd 1.019
  • eβ is a relativity: Metro costs 1.163× Suburban, for every kind of encounter
  • This is exactly how rating plans work: base rate × factor × factor × …
TermFittedHC3 95% CITruth
Region: Metro1.1631.150–1.1751.15
Region: Rural0.8990.887–0.9100.90
Plan: HMO0.8740.865–0.8830.88
Plan: HDHP0.9770.964–0.9890.97
Age, per decade1.0581.054–1.0621.06
Emergency vs Office12.912.6–13.3—
Inpatient vs Office151.7141.8–162.2—

Category relativities have no single “truth”: heavy users’ encounters are more intensive, and the mix differs by category.

PitfallThe retransformation problem

$$e^{\,\mathrm{E}[\ln y]} \neq \mathrm{E}[y] \qquad \text{for lognormal errors: } \mathrm{E}[y] = e^{\mathbf{x}^\top\beta + \sigma^2/2}$$

np.exp(fitted) estimates the median, not the mean. Duan’s smearing estimator multiplies by the average of eresidual:

naive = np.exp(loglin.fittedvalues)
smear = np.mean(np.exp(loglin.resid))          # 1.755

# heteroscedastic? smear within each category
cat = (e.assign(r=np.exp(loglin.resid))
        .groupby("service_category")["r"].mean())
fixed = naive * e["service_category"].map(cat)
Mean allowedActualNaiveOne smearBy category
Pharmacy168.677.3135.6168.7
Office Visit167.4156.8275.1167.4
Emergency2,727.92,053.73,603.92,719.3
Inpatient37,894.924,019.742,150.237,735.9
All441.1292.5513.3440.2

The naive retransform understates total cost by a third. One global smearing factor overshoots because the residual variance differs by category. A pricing model has to get the mean right.

PreviewLog transformation vs log link

Log-linear OLS (this lesson)

$$\mathrm{E}[\ln y] = \mathbf{x}^\top\beta$$

  • Models the mean of log-cost
  • Needs a retransformation to get dollars
  • Can’t handle zeros: ln 0 = −∞

GLM with a log link (Lessons 2–3)

$$\ln \mathrm{E}[y] = \mathbf{x}^\top\beta$$

  • Models the mean in dollars directly — no smearing
  • Keeps multiplicative relativities: Gamma log-link GLM gives Metro 1.175, HMO 0.878
  • Chooses a variance function to match the data

The log link is the backbone of modern pricing: multiplicative rating factors, predictions that are always positive, and a mean that’s right in dollars.

Part 5

Robust Standard Errors

Honest uncertainty when the variance isn’t constant.

HeteroscedasticityThe log model still has it

Log-model residuals versus fitted, and box plots of residual spread by service category from narrow office visits to wide pharmacy
from statsmodels.stats.diagnostic import het_breuschpagan
lm, p, *_ = het_breuschpagan(loglin.resid, loglin.model.exog)
# LM = 21,147.6, p < 1e-300

Residual standard deviation runs from 0.36 for office visits to 1.25 for pharmacy (generics next to specialty drugs). The Breusch–Pagan test rejects constant variance emphatically.

Sandwich estimatorsHC0 and HC3

$$\widehat{\mathrm{Var}}(\hat\beta) = (\mathbf{X}^\top\mathbf{X})^{-1}\,\Big(\sum_i \omega_i\,\hat e_i^2\,\mathbf{x}_i\mathbf{x}_i^\top\Big)\,(\mathbf{X}^\top\mathbf{X})^{-1}$$

  • HC0 (White): ωi = 1 — each row’s own squared residual
  • HC3: ωi = 1 / (1 − hii)² — inflates high-leverage rows; better in small samples
hc3 = smf.ols("log_allowed ~ " + rhs, data=e).fit(cov_type="HC3")
np.exp(hc3.conf_int())    # robust 95% CIs for the relativities
Std errorOLSHC0HC3
Specialist0.01070.00450.0045
Lab0.00880.00490.0049
Inpatient0.03600.03420.0343
Region: Metro0.00540.00540.0054
Age per decade0.00200.00200.0020

Robust SEs are correct, not automatically bigger. Here the base level (office visits) is unusually tight, so category contrasts get smaller SEs. Region, plan, and age hardly move.

Actuarial Lab · 75 minBaseline severity by service category

  1. Load the 2024 encounters, join member age, region, and plan
  2. Fit OLS on raw allowed; list three things the residual plots tell you
  3. Fit the log-linear model; tabulate relativities with HC3 95% intervals
  4. Run Breusch–Pagan; compare OLS, HC0, and HC3 standard errors
  5. Compute category-level smearing factors
  6. Predict the median and mean allowed per encounter for a reference member: age 40, Suburban, PPO
  7. Compare region, plan, and age relativities with the truth

Answer key: reference member

CategoryMedianMean
Lab59.1076.17
Pharmacy74.10161.83
Office Visit151.94162.21
Specialist288.93322.24
Imaging466.39671.65
Emergency1,963.742,600.17
Outpatient Surgery4,815.045,986.45
Inpatient23,045.5636,205.44

Pharmacy’s mean is more than twice its median: the widest spread of any category.

RecapLesson 1 checklist

  • I can say whether a problem is supervised or unsupervised
  • I can list the Gauss–Markov assumptions and what breaks when each fails
  • I read residual-vs-fitted and Q–Q plots before I read R²
  • I know why skew, zeros, and mean-dependent variance defeat OLS
  • I read eβ from a log model as a multiplicative relativity
  • I correct retransformation bias with (category-level) smearing
  • I use cov_type="HC3" when variance isn’t constant
  • I know the difference between a log transform and a log link
Up next
Lesson 2

GLMs for Claim Frequency

The exponential family, Poisson regression with exposure offsets, overdispersion, and incidence rate ratios, on the motor book.