Python for Actuarial Students · Course 2

Lesson 8 · Part A

Pipeline Architecture & Configuration

From a folder of notebooks to one command that rebuilds every number, table, and report from raw data.

Lesson 8 · OverviewCapstone lesson, in two parts

Part A — this deck

Separating extraction, features, calculation, and reporting
Project layout and a command-line entry point
Parameters in TOML (or YAML) configs, not in code
Logging, validation stages, and control totals

≈ 60 min concepts + 45 min refactoring workshop

Part B — the capstone

The end-to-end rating engine brief
Executive Excel report + generated Reveal.js deck
Answer key, rubric, and stretch goals

Project work, at your own pace

Companion notebook: lesson-08a-pipeline-architecture.ipynb — worked examples and self-checking exercises.

MotivationWhy notebooks aren’t a production process

After seven lessons you have seven notebooks. Next quarter, someone must:

Remember which notebooks to run, in which order
Find and change every hard-coded date and file path
Hope no cell was run out of order last time
Copy numbers into Excel and slides by hand

Every manual step is a place for an error that nobody will spot — a stale date, a wrong file, a pasted number from the previous quarter.

A pipeline is code that goes from raw inputs to finished outputs with no manual steps. Run it twice on the same inputs: identical outputs. Run it on new inputs: the new answer.

Part 1

Pipeline Architecture

Four stages, each with one job.

ArchitectureSeparation of concerns

1 · Extractread dumps; validate keys
extract.py

→

2 · Featuresexposure, ABT, territories
features.py

→

3 · Calculatemetrics, indications
calculate.py

→

4 · ReportExcel, slides, charts
report.py

Each stage is a set of functions: DataFrames in, DataFrames out
Only extract reads files; only report writes deliverables
Calculations never format; reports never calculate

Now you can test calculate.indicate() on a three-row DataFrame, swap the Excel report for a dashboard, or point extract at a database instead of CSVs — without touching the other stages.

LayoutThe reference project

rating_pipeline/
├── config.toml           every assumption
├── pyproject.toml        + [project.scripts]
├── README.md
├── src/rating_pipeline/
│   ├── __init__.py
│   ├── cli.py            entry point: runs 1→4
│   ├── extract.py
│   ├── features.py
│   ├── calculate.py
│   ├── report.py
│   └── templates/
│       └── deck.html.j2  Reveal.js template
├── tests/
│   └── test_pipeline.py
└── output/               generated — git-ignored

The Course 1 package layout (Lesson 8), now with a pipeline inside
output/ is disposable: delete it, rerun, get it back
Data lives outside the code repository and is referenced from the config

A complete pipeline like this is about 400 lines, including the slide template — built entirely from what you learned in Lessons 1–7.

Part 2

Configuration

Parameters in a file, not scattered through code.

TOMLOne file of assumptions

# config.toml — every assumption lives here

[paths]
motor = "../../data/motor"
geo = "../../data/geo/nc_geo.gpkg"
output = "output"

[evaluation]
date = 2025-12-31          # a real date, not a string

[territory]
coastal_km = 25

[age_bands]
edges = [17, 25, 35, 50, 65, 75, 120]
labels = ["17-24", "25-34", "35-49", "50-64", "65-74", "75+"]

[indication]
target_loss_ratio = 0.68
annual_trend = 0.04
trend_years = 1.5
full_credibility_claims = 1082
max_change = 0.15

import tomllib                       # standard library, 3.11+

with open("config.toml", "rb") as f:  # binary mode!
    cfg = tomllib.load(f)

cfg["evaluation"]["date"]      # datetime.date(2025, 12, 31)
cfg["indication"]["max_change"]   # 0.15

Changing an assumption is a one-line, reviewable diff in Git
Different runs (quarter-end, a stress test) are different config files — same code
pyproject.toml is TOML too — you already know the syntax

TOML vs. YAMLChoosing a config format

	TOML	YAML
Python support	built-in `tomllib` (read-only)	`pyyaml` package
Syntax	`key = value`, `[sections]`	indentation-based
Types	explicit: strings quoted, real dates	guessed: `no`, `off` → `False`; `yes`, `on` → `True`
Deep nesting	clumsier	natural
Seen in	`pyproject.toml`, Rust, uv	CI pipelines, Kubernetes, dbt

# The same block in YAML
indication:
  target_loss_ratio: 0.68
  annual_trend: 0.04
  max_change: 0.15

YAML’s type guessing has bitten real models: the country code NO (Norway) loads as False. Prefer TOML for parameter files; if you must use YAML, use yaml.safe_load.

PathsResolving paths from the config file

from pathlib import Path

def run(config_path: Path) -> dict[str, Path]:
    with open(config_path, "rb") as f:
        cfg = tomllib.load(f)

    base = config_path.parent
    paths = {k: (base / v).resolve()
             for k, v in cfg["paths"].items()}
    paths["output"].mkdir(parents=True, exist_ok=True)
    eval_date = pd.Timestamp(cfg["evaluation"]["date"])
    params = cfg["indication"]
    ...

Paths in the config are relative to the config file, not to wherever the command was typed
.resolve() turns them into absolute paths — log them so a reviewer knows exactly which files were read
Pass the [indication] table straight into a function with **params

Unpacking a config section into keyword arguments means the TOML keys and the function’s parameter names must match — a free consistency check.

Part 3

Running & Observing

A command-line entry point, logging, and built-in checks.

CLIOne command to run it all

# src/rating_pipeline/cli.py
import argparse, logging

def main() -> None:
    parser = argparse.ArgumentParser(description="Motor rating pipeline")
    parser.add_argument("--config", type=Path, default=Path("config.toml"))
    parser.add_argument("-v", "--verbose", action="store_true")
    args = parser.parse_args()
    logging.basicConfig(
        level=logging.DEBUG if args.verbose else logging.INFO,
        format="%(asctime)s %(levelname)-7s %(name)s: %(message)s",
        datefmt="%H:%M:%S")
    run(args.config)

# pyproject.toml
[project.scripts]
rating-pipeline = "rating_pipeline.cli:main"

$ uv run rating-pipeline --config config.toml
INFO extract: loaded policies        22,185 rows
INFO extract: loaded vehicles        23,993 rows
INFO extract: loaded persons         12,351 rows
INFO extract: loaded policy_drivers  29,803 rows
INFO extract: loaded coverages       63,817 rows
INFO extract: loaded claims           3,354 rows
INFO extract: all primary and foreign keys valid
INFO features: ABT: 23,993 vehicles, 20100.95
               car years, 3,354 claims
INFO features: territories: {'Urban inland': 11658,
               'Rural inland': 7501, 'Coastal': 4834}
INFO rating_pipeline: loss ratio 74.6%,
               proposed change +7.8%
INFO report: wrote output/rating_review.xlsx
INFO report: wrote output/rating_review.html

About 5 seconds, end to end. (Timestamps trimmed.)

Logging`logging`, not `print`

import logging

log = logging.getLogger(__name__)   # one logger per module

log.info("loaded %-15s %8s rows", name, f"{len(df):,}")
log.debug("columns: %s", list(df.columns))   # only with -v
log.warning("%d vehicles outside every county", n_missing)

Every message carries a time, a level, and the module that sent it
Levels filter noise: DEBUG < INFO < WARNING < ERROR
Send the same messages to a file for an audit trail with one extra handler

A run log with row counts, control totals, and file paths is exactly what a peer reviewer or auditor asks for. With logging it’s produced automatically, every run.

ValidationFail loudly, early, and specifically

class DataQualityError(ValueError):
    """Raised when an extract breaks a key constraint."""

# extract.py — before anything else runs
dupes = df.duplicated(key).sum()
if dupes:
    raise DataQualityError(
        f"{name}: {dupes} duplicate primary keys on {key}")

# features.py — control totals after the joins
assert len(abt) == len(t["vehicles"]), "ABT grain changed"
assert abt["n_claims"].sum() == len(t["claims"]), \
    "claims lost or duplicated in ABT"

Lessons 1 and 3’s checks become gates: bad data stops the run, with a message saying exactly what’s wrong
A custom exception class lets callers catch data problems separately from bugs
Control totals after every join (Lesson 3) — now automatic

A pipeline that runs to completion on bad data is worse than one that crashes: it produces a confident, formatted, wrong report.

TestingSmall tests for each stage

def test_full_leap_year_earns_one():
    start = pd.Series(pd.to_datetime(["2024-01-01"]))
    end = pd.Series(pd.to_datetime(["2024-12-31"]))
    assert calendar_exposure(start, end, 2024).iloc[0] \
        == pytest.approx(1.0)

def test_loss_ratio_is_ratio_of_sums():
    abt = pd.DataFrame({"seg": ["A", "A"],
        "exposure": [1.0, 0.1], "n_claims": [0, 1],
        "incurred": [0.0, 500.0],
        "earned_premium": [1000.0, 100.0]})
    seg = experience(abt, ["seg"])
    assert seg.loc[0, "loss_ratio"] == pytest.approx(500 / 1100)

def test_proposed_change_is_capped_and_credibility_weighted():
    ...

Tiny hand-built DataFrames with answers you can verify by hand
Each test pins down one decision from the course: leap years (Lesson 2), ratio of sums (Lesson 4), capping (Lesson 7)
They run in a fraction of a second — no need for the real data

$ uv run pytest -q
...                                  [100%]
3 passed in 0.27s

Workshop · 45 minRefactor your course work into a pipeline skeleton

uv init rating_pipeline --package; add the dependencies
Create the four stage modules and cli.py; register the script in pyproject.toml
Move code from your Lesson 1–7 notebooks into functions in the right stage
Write config.toml; replace every hard-coded path, date, and threshold
Replace print with logging; add the key checks and control totals
uv run rating-pipeline end to end — even if the report stage is still a stub

Search your code for literal numbers and dates (2025, 0.68, 25). Each one is either a genuine constant or a config entry in disguise.

Get the skeleton running first, with stubs. Then fill in each stage and rerun. Never go long without a working end-to-end run.

RecapLesson 8A checklist

I can explain extract → features → calculate → report
Each stage is functions, with I/O only at the edges
All assumptions live in a TOML config
Config paths resolve relative to the config file

One command runs everything
I log with logging, not print
Bad data stops the run with a specific message
Each stage has small, fast tests

Up next

Lesson 8 · Part B

Capstone: An End-to-End Rating Engine & Report

The brief, the deliverables, and how it will be marked.

← Course home Next: Lesson 8 · Part B →

Pipeline Architecture & Configuration

Lesson 8 · OverviewCapstone lesson, in two parts

Part A — this deck

Part B — the capstone

MotivationWhy notebooks aren’t a production process

Pipeline Architecture

ArchitectureSeparation of concerns

LayoutThe reference project

Configuration

TOMLOne file of assumptions

TOML vs. YAMLChoosing a config format

PathsResolving paths from the config file

Running & Observing

CLIOne command to run it all

Logginglogging, not print

ValidationFail loudly, early, and specifically

TestingSmall tests for each stage

Workshop · 45 minRefactor your course work into a pipeline skeleton

RecapLesson 8A checklist

Capstone: An End-to-End Rating Engine & Report

Logging`logging`, not `print`