Python for Actuarial Students · Course 4
Lesson 6

Enterprise Data Stores: SQL & SQLAlchemy

Migrating the motor book from CSV files to a relational database: a schema that enforces the rules, transactions, safe queries, chunked reads, and the ORM, on SQLite and PostgreSQL.

Lesson 6 · OverviewBy the end of this lesson you can…

  • Explain what a database guarantees that a folder of CSVs doesn’t
  • Declare tables with primary keys, foreign keys, and CHECK constraints in SQLAlchemy
  • Load data in one transaction: all of it or none of it
  • Query with bound parameters, never string formatting
  • Process tables larger than memory in chunks, or push the work to the database
  • Choose between the ORM and SQL, and run the same code on SQLite and PostgreSQL
75 min
Concepts & worked examples
75 min
Lab: CSV pipeline → SQL

Companion notebook: lesson-06-databases.ipynb — worked examples and self-checking exercises.

WhyFiles versus a database

QuestionSix CSV filesA relational database
Can a claim point to a vehicle that doesn’t exist?Yes, until someone checksNo: a foreign key refuses it
Two processes write at once?The last one wins; data lostTransactions keep both correct
A load fails half way?Half a fileRolled back: nothing changed
Find one vehicle’s claims in 11,562?Read the whole fileAn index goes straight there
Who may read premiums?Anyone with the folderUsers, roles, permissions
Bigger than memory?Your problemThe database’s job

Course 2 built the pipeline from these files. Production systems keep the same six tables, with the rules written into the schema.

Part 1

A Schema That Enforces the Rules

Say it once, in the database.

SQLAlchemy CoreTables as Python objects

claims = Table("claims", metadata,
    Column("claim_id", String(10), primary_key=True),
    Column("policy_id", String(10),
           ForeignKey("policies.policy_id"), nullable=False),
    Column("vehicle_id", String(10),
           ForeignKey("vehicles.vehicle_id"),
           nullable=False, index=True),
    Column("coverage", String(4), nullable=False),
    Column("accident_date", Date, nullable=False),
    Column("report_date", Date, nullable=False),
    Column("status", String(10), nullable=False),
    Column("paid", Float, nullable=False),
    Column("outstanding", Float, nullable=False),
    CheckConstraint("report_date >= accident_date",
                    name="reported_after_accident"),
    CheckConstraint("paid >= 0 AND outstanding >= 0",
                    name="amounts_nonnegative"),
)
metadata.create_all(engine)       # CREATE TABLE ... for all six
TableRowsKey
persons39,468person_id
policies78,062policy_id → persons
vehicles84,346vehicle_id → policies
policy_drivers105,559(policy, person)
coverages224,988(vehicle, coverage)
claims11,562claim_id → vehicles

The schema is also documentation: anyone can read which column refers to which table, and what values are allowed.

MigrationLoad everything in one transaction

ORDER = [persons, policies, vehicles,          # parents
         policy_drivers, coverages, claims]    # first

with engine.begin() as conn:          # commit at the end,
    for table in ORDER:               # roll back on any error
        for chunk in pd.read_csv(RAW / f"{table.name}.csv",
                                 chunksize=20_000,
                                 parse_dates=DATES[table.name]):
            rows = (chunk.astype(object)
                    .where(chunk.notna(), None)   # NaN -> NULL
                    .to_dict(orient="records"))
            conn.execute(insert(table), rows)
Rows loaded543,985
Time, SQLite4.8 s
Size27.1 MB of CSV → 44.9 MB file
  • Chunks keep memory flat however big the files
  • A missing value must become NULL, not the float NaN
  • The database is bigger than the CSVs: it stores indexes and typed columns, and buys fast lookups with the space

Same answersExposure and claims, in SQL

SELECT strftime('%Y', p.inception_date) AS policy_year,
       COUNT(*) AS vehicles,
       SUM(MAX(0, julianday(MIN(p.expiry_date,
                   COALESCE(p.cancellation_date, p.expiry_date),
                   '2025-12-31'))
                  - julianday(p.inception_date) + 1) / 365.25)
           AS car_years
FROM vehicles v JOIN policies p USING (policy_id)
GROUP BY policy_year;
YearVehiclesCar-years
202115,10914,603.5
202216,28315,707.9
202317,01616,435.6
202417,68917,068.9
202518,2498,964.6
Total84,34672,780.5

pandas gives 72,780.5 too, and the claims query gives 4,565 liability claims ($59.06m incurred) to 31 December 2025: the Course 3 numbers, exactly.

ConstraintsThe database refuses bad data

Attempted insertSQLite says
Claim on VEH999999FOREIGN KEY constraint failed
Reported before the accidentCHECK constraint failed: reported_after_accident
Paid −$50CHECK constraint failed: amounts_nonnegative
Claim id already usedUNIQUE constraint failed: claims.claim_id

One good and one bad claim in the same transaction: afterwards there are still 11,562 claims. Neither was stored.

@event.listens_for(engine, "connect")
def _fk_on(dbapi_conn, _):
    dbapi_conn.execute("PRAGMA foreign_keys = ON")

SQLite ignores foreign keys unless each connection asks. Without the PRAGMA, the orphan claim on VEH999999 was accepted. PostgreSQL always enforces them: insert or update on table “claims” violates foreign key constraint “claims_vehicle_id_fkey”.

Part 2

Querying Safely & at Scale

Parameters, indexes, chunks.

SQL injectionNever build SQL with string formatting

evil = "x' OR '1'='1"          # typed into a search box

# WRONG: the input becomes part of the SQL
conn.execute(text(f"SELECT * FROM policies "
                  f"WHERE policy_id = '{evil}'"))
# ... WHERE policy_id = 'x' OR '1'='1'   -> 78,062 rows

# RIGHT: a bound parameter is only ever a value
conn.execute(text("SELECT * FROM policies "
                  "WHERE policy_id = :pid"), {"pid": evil})
#                                          -> 0 rows

The f-string version returned every policy in the book. With '; DROP TABLE claims; -- it could do worse.

  • Bound parameters (:pid) are sent separately from the SQL; the database never parses them as code
  • SQLAlchemy expressions (select(...).where(...)) always bind their values
  • The same rule applies to dates, amounts, and lists: bind them all

IndexesHow the database finds one vehicle’s claims

SELECT * FROM claims WHERE vehicle_id = 'VEH000005';

SEARCH claims USING INDEX ix_claims_vehicle_id
       (vehicle_id=?)
  • index=True on claims.vehicle_id created ix_claims_vehicle_id: a sorted lookup structure, like a book’s index
  • Primary keys are indexed automatically; foreign-key columns are not, so index the ones you join or filter on
  • Indexes make reads fast and writes a little slower: index what you query

EXPLAIN (PostgreSQL: EXPLAIN ANALYZE) shows the plan. “SCAN” on a big table where you expected “SEARCH” means a missing index.

Bigger than memoryChunk it, or let the database do it

q = "SELECT coverage, annual_premium FROM coverages"

# (a) everything at once
pd.read_sql(text(q), conn).groupby("coverage").sum()

# (b) 10,000 rows at a time, streamed
parts = [chunk.groupby("coverage")["annual_premium"].sum()
         for chunk in pd.read_sql(
             text(q), conn.execution_options(stream_results=True),
             chunksize=10_000)]
pd.concat(parts).groupby(level=0).sum()

# (c) the database aggregates; 3 rows come back
pd.read_sql(text("SELECT coverage, SUM(annual_premium) "
                 "FROM coverages GROUP BY coverage"), conn)
224,988 rowsPeak Python memory
(a) all at once59.2 MB
(b) chunks of 10,0004.3 MB
(c) GROUP BY in SQL3 rows

All three agree: LIAB $92.54m, COLL $27.26m, COMP $6.71m. Chunk memory stays flat as the table grows; the all-at-once version doesn’t.

Summing, counting, and joining are what databases are built for. Pull out aggregates, not raw rows, whenever you can.

Part 3

The ORM, and Production Databases

Objects for application code; the same code on PostgreSQL.

ORMRows as Python objects

class Base(DeclarativeBase):
    pass

class QuoteRecord(Base):
    __tablename__ = "quotes"
    id: Mapped[int] = mapped_column(primary_key=True)
    created_at: Mapped[datetime] = mapped_column(
        UTCDateTime, default=lambda: datetime.now(UTC))
    request: Mapped[dict[str, Any]] = mapped_column(JSON)
    liab_premium: Mapped[float]
    coll_premium: Mapped[float | None]
    total_premium: Mapped[float]
    model_version: Mapped[str] = mapped_column(String(40),
                                               index=True)

with Session(engine) as s:
    s.add(QuoteRecord(request=..., liab_premium=4648.59, ...))
    s.commit()
    rec = s.get(QuoteRecord, 1)
20,000 quote rowsTime
ORM, one object at a time0.76 s
insert(QuoteRecord), one statement0.21 s
  • The ORM suits application code: one quote at a time, typed attributes, relationships
  • Bulk loads and analysis: Core insert() and SQL
  • Both compile to the same SQL; print(stmt) shows it

A bug the tests foundSQLite has no time zones

E  Differing items:
E  {'created_at': '2026-09-30T14:58:24.157592'} !=
E  {'created_at': '2026-09-30T14:58:24.157592Z'}

The quote was created in UTC. Read back from SQLite, its time zone was gone: the same quote now had a different timestamp, and comparing it with an aware one raises TypeError.

class UTCDateTime(TypeDecorator[datetime]):
    """Stored in UTC; always returned time-zone aware."""
    impl = DateTime(timezone=True)
    cache_ok = True

    def process_bind_param(self, value, dialect):
        if value is not None and value.tzinfo is None:
            raise ValueError("naive datetime: attach a "
                             "time zone before storing")
        return value.astimezone(UTC) if value else None

    def process_result_value(self, value, dialect):
        if value is not None and value.tzinfo is None:
            return value.replace(tzinfo=UTC)
        return value

PostgreSQLChange the URL, keep the code

engine = sm.make_engine(
    "postgresql+psycopg://postgres:***@localhost:5433/postgres")
counts = sm.migrate(engine)      # the same function
SQLitePostgreSQL
Rows loaded543,985543,985
Load time4.8 s30.5 s
Car-years72,780.572,780.5
LIAB claims4,5654,565
Foreign keysonly with the PRAGMAalways

What changed: the date arithmetic (julianday becomes date subtraction, MAX/MIN become GREATEST/LEAST). Keep dialect-specific SQL in few, tested places.

PostgreSQL is a server: loading over a network connection is slower than writing a local file, and in exchange you get many concurrent users, permissions, backups, and replication.

Actuarial Lab · 75 minFrom CSV pipeline to SQL

  1. Declare the six tables with keys, foreign keys, and CHECK constraints
  2. Load all six in chunks, in one transaction, with foreign keys switched on
  3. Car-years by policy year and claims by coverage, in SQL; match pandas
  4. Try five bad claims; show a failed transaction leaves nothing behind
  5. An injection-safe policy lookup
  6. Premium by coverage three ways; compare peak memory
  7. An ORM quote log; run everything again on PostgreSQL

Answer key

Rows (all six tables)543,985
Car-years to 2025-12-3172,780.5
LIAB claims4,565
Injection: unsafe / safe78,062 / 0
Premium, LIAB$92.54m

RecapLesson 6 checklist

  • Keys, foreign keys, and constraints are in the schema
  • SQLite connections switch foreign keys on
  • Loads run in one transaction
  • Every value in a query is a bound parameter
  • I index the columns I filter and join on
  • Large tables are chunked or aggregated in the database
  • Timestamps are stored in UTC and read back aware
  • The same code runs on SQLite and PostgreSQL
Up next
Lesson 7

Containerization with Docker

“It works on my machine” solved: multi-stage Docker images, secrets that never enter an image, and the whole platform started with one command.