Migrating the motor book from CSV files to a relational database: a schema that enforces the rules, transactions, safe queries, chunked reads, and the ORM, on SQLite and PostgreSQL.
Companion notebook: lesson-06-databases.ipynb — worked examples and self-checking exercises.
| Question | Six CSV files | A relational database |
|---|---|---|
| Can a claim point to a vehicle that doesn’t exist? | Yes, until someone checks | No: a foreign key refuses it |
| Two processes write at once? | The last one wins; data lost | Transactions keep both correct |
| A load fails half way? | Half a file | Rolled back: nothing changed |
| Find one vehicle’s claims in 11,562? | Read the whole file | An index goes straight there |
| Who may read premiums? | Anyone with the folder | Users, roles, permissions |
| Bigger than memory? | Your problem | The database’s job |
Course 2 built the pipeline from these files. Production systems keep the same six tables, with the rules written into the schema.
Say it once, in the database.
claims = Table("claims", metadata,
Column("claim_id", String(10), primary_key=True),
Column("policy_id", String(10),
ForeignKey("policies.policy_id"), nullable=False),
Column("vehicle_id", String(10),
ForeignKey("vehicles.vehicle_id"),
nullable=False, index=True),
Column("coverage", String(4), nullable=False),
Column("accident_date", Date, nullable=False),
Column("report_date", Date, nullable=False),
Column("status", String(10), nullable=False),
Column("paid", Float, nullable=False),
Column("outstanding", Float, nullable=False),
CheckConstraint("report_date >= accident_date",
name="reported_after_accident"),
CheckConstraint("paid >= 0 AND outstanding >= 0",
name="amounts_nonnegative"),
)
metadata.create_all(engine) # CREATE TABLE ... for all six
| Table | Rows | Key |
|---|---|---|
| persons | 39,468 | person_id |
| policies | 78,062 | policy_id → persons |
| vehicles | 84,346 | vehicle_id → policies |
| policy_drivers | 105,559 | (policy, person) |
| coverages | 224,988 | (vehicle, coverage) |
| claims | 11,562 | claim_id → vehicles |
The schema is also documentation: anyone can read which column refers to which table, and what values are allowed.
ORDER = [persons, policies, vehicles, # parents
policy_drivers, coverages, claims] # first
with engine.begin() as conn: # commit at the end,
for table in ORDER: # roll back on any error
for chunk in pd.read_csv(RAW / f"{table.name}.csv",
chunksize=20_000,
parse_dates=DATES[table.name]):
rows = (chunk.astype(object)
.where(chunk.notna(), None) # NaN -> NULL
.to_dict(orient="records"))
conn.execute(insert(table), rows)
| Rows loaded | 543,985 |
| Time, SQLite | 4.8 s |
| Size | 27.1 MB of CSV → 44.9 MB file |
NULL, not the float NaNSELECT strftime('%Y', p.inception_date) AS policy_year,
COUNT(*) AS vehicles,
SUM(MAX(0, julianday(MIN(p.expiry_date,
COALESCE(p.cancellation_date, p.expiry_date),
'2025-12-31'))
- julianday(p.inception_date) + 1) / 365.25)
AS car_years
FROM vehicles v JOIN policies p USING (policy_id)
GROUP BY policy_year;
| Year | Vehicles | Car-years |
|---|---|---|
| 2021 | 15,109 | 14,603.5 |
| 2022 | 16,283 | 15,707.9 |
| 2023 | 17,016 | 16,435.6 |
| 2024 | 17,689 | 17,068.9 |
| 2025 | 18,249 | 8,964.6 |
| Total | 84,346 | 72,780.5 |
pandas gives 72,780.5 too, and the claims query gives 4,565 liability claims ($59.06m incurred) to 31 December 2025: the Course 3 numbers, exactly.
| Attempted insert | SQLite says |
|---|---|
| Claim on VEH999999 | FOREIGN KEY constraint failed |
| Reported before the accident | CHECK constraint failed: reported_after_accident |
| Paid −$50 | CHECK constraint failed: amounts_nonnegative |
| Claim id already used | UNIQUE constraint failed: claims.claim_id |
One good and one bad claim in the same transaction: afterwards there are still 11,562 claims. Neither was stored.
@event.listens_for(engine, "connect")
def _fk_on(dbapi_conn, _):
dbapi_conn.execute("PRAGMA foreign_keys = ON")
SQLite ignores foreign keys unless each connection asks. Without the PRAGMA, the orphan claim on VEH999999 was accepted. PostgreSQL always enforces them: insert or update on table “claims” violates foreign key constraint “claims_vehicle_id_fkey”.
Parameters, indexes, chunks.
evil = "x' OR '1'='1" # typed into a search box
# WRONG: the input becomes part of the SQL
conn.execute(text(f"SELECT * FROM policies "
f"WHERE policy_id = '{evil}'"))
# ... WHERE policy_id = 'x' OR '1'='1' -> 78,062 rows
# RIGHT: a bound parameter is only ever a value
conn.execute(text("SELECT * FROM policies "
"WHERE policy_id = :pid"), {"pid": evil})
# -> 0 rows
The f-string version returned every policy in the book. With '; DROP TABLE claims; -- it could do worse.
:pid) are sent separately from the SQL; the database never parses them as codeselect(...).where(...)) always bind their valuesSELECT * FROM claims WHERE vehicle_id = 'VEH000005';
SEARCH claims USING INDEX ix_claims_vehicle_id
(vehicle_id=?)
index=True on claims.vehicle_id created ix_claims_vehicle_id: a sorted lookup structure, like a book’s indexEXPLAIN (PostgreSQL: EXPLAIN ANALYZE) shows the plan. “SCAN” on a big table where you expected “SEARCH” means a missing index.
q = "SELECT coverage, annual_premium FROM coverages"
# (a) everything at once
pd.read_sql(text(q), conn).groupby("coverage").sum()
# (b) 10,000 rows at a time, streamed
parts = [chunk.groupby("coverage")["annual_premium"].sum()
for chunk in pd.read_sql(
text(q), conn.execution_options(stream_results=True),
chunksize=10_000)]
pd.concat(parts).groupby(level=0).sum()
# (c) the database aggregates; 3 rows come back
pd.read_sql(text("SELECT coverage, SUM(annual_premium) "
"FROM coverages GROUP BY coverage"), conn)
| 224,988 rows | Peak Python memory |
|---|---|
| (a) all at once | 59.2 MB |
| (b) chunks of 10,000 | 4.3 MB |
| (c) GROUP BY in SQL | 3 rows |
All three agree: LIAB $92.54m, COLL $27.26m, COMP $6.71m. Chunk memory stays flat as the table grows; the all-at-once version doesn’t.
Summing, counting, and joining are what databases are built for. Pull out aggregates, not raw rows, whenever you can.
Objects for application code; the same code on PostgreSQL.
class Base(DeclarativeBase):
pass
class QuoteRecord(Base):
__tablename__ = "quotes"
id: Mapped[int] = mapped_column(primary_key=True)
created_at: Mapped[datetime] = mapped_column(
UTCDateTime, default=lambda: datetime.now(UTC))
request: Mapped[dict[str, Any]] = mapped_column(JSON)
liab_premium: Mapped[float]
coll_premium: Mapped[float | None]
total_premium: Mapped[float]
model_version: Mapped[str] = mapped_column(String(40),
index=True)
with Session(engine) as s:
s.add(QuoteRecord(request=..., liab_premium=4648.59, ...))
s.commit()
rec = s.get(QuoteRecord, 1)
| 20,000 quote rows | Time |
|---|---|
| ORM, one object at a time | 0.76 s |
insert(QuoteRecord), one statement | 0.21 s |
insert() and SQLprint(stmt) shows itE Differing items:
E {'created_at': '2026-09-30T14:58:24.157592'} !=
E {'created_at': '2026-09-30T14:58:24.157592Z'}
The quote was created in UTC. Read back from SQLite, its time zone was gone: the same quote now had a different timestamp, and comparing it with an aware one raises TypeError.
class UTCDateTime(TypeDecorator[datetime]):
"""Stored in UTC; always returned time-zone aware."""
impl = DateTime(timezone=True)
cache_ok = True
def process_bind_param(self, value, dialect):
if value is not None and value.tzinfo is None:
raise ValueError("naive datetime: attach a "
"time zone before storing")
return value.astimezone(UTC) if value else None
def process_result_value(self, value, dialect):
if value is not None and value.tzinfo is None:
return value.replace(tzinfo=UTC)
return value
engine = sm.make_engine(
"postgresql+psycopg://postgres:***@localhost:5433/postgres")
counts = sm.migrate(engine) # the same function
| SQLite | PostgreSQL | |
|---|---|---|
| Rows loaded | 543,985 | 543,985 |
| Load time | 4.8 s | 30.5 s |
| Car-years | 72,780.5 | 72,780.5 |
| LIAB claims | 4,565 | 4,565 |
| Foreign keys | only with the PRAGMA | always |
What changed: the date arithmetic (julianday becomes date subtraction, MAX/MIN become GREATEST/LEAST). Keep dialect-specific SQL in few, tested places.
PostgreSQL is a server: loading over a network connection is slower than writing a local file, and in exchange you get many concurrent users, permissions, backups, and replication.
| Rows (all six tables) | 543,985 |
| Car-years to 2025-12-31 | 72,780.5 |
| LIAB claims | 4,565 |
| Injection: unsafe / safe | 78,062 / 0 |
| Premium, LIAB | $92.54m |
“It works on my machine” solved: multi-stage Docker images, secrets that never enter an image, and the whole platform started with one command.