Enterprise Data Stores: SQL & SQLAlchemy
Migrating the motor book from CSV files to a relational database: a schema that enforces the rules, transactions, safe queries, chunked reads, and the ORM, on SQLite and PostgreSQL.
Lesson 6 · OverviewBy the end of this lesson you can…
- Explain what a database guarantees that a folder of CSVs doesn’t
- Declare tables with primary keys, foreign keys, and CHECK constraints in SQLAlchemy
- Load data in one transaction: all of it or none of it
- Query with bound parameters, never string formatting
- Process tables larger than memory in chunks, or push the work to the database
- Choose between the ORM and SQL, and run the same code on SQLite and PostgreSQL
Companion notebook: lesson-06-databases.ipynb — worked examples and self-checking exercises.
WhyFiles versus a database
| Question | Six CSV files | A relational database |
|---|---|---|
| Can a claim point to a vehicle that doesn’t exist? | Yes, until someone checks | No: a foreign key refuses it |
| Two processes write at once? | The last one wins; data lost | Transactions keep both correct |
| A load fails half way? | Half a file | Rolled back: nothing changed |
| Find one vehicle’s claims in 11,562? | Read the whole file | An index goes straight there |
| Who may read premiums? | Anyone with the folder | Users, roles, permissions |
| Bigger than memory? | Your problem | The database’s job |
Course 2 built the pipeline from these files. Production systems keep the same six tables, with the rules written into the schema.
A Schema That Enforces the Rules
Say it once, in the database.
SQLAlchemy CoreTables as Python objects
claims = Table("claims", metadata,
Column("claim_id", String(10), primary_key=True),
Column("policy_id", String(10),
ForeignKey("policies.policy_id"), nullable=False),
Column("vehicle_id", String(10),
ForeignKey("vehicles.vehicle_id"),
nullable=False, index=True),
Column("coverage", String(4), nullable=False),
Column("accident_date", Date, nullable=False),
Column("report_date", Date, nullable=False),
Column("status", String(10), nullable=False),
Column("paid", Float, nullable=False),
Column("outstanding", Float, nullable=False),
CheckConstraint("report_date >= accident_date",
name="reported_after_accident"),
CheckConstraint("paid >= 0 AND outstanding >= 0",
name="amounts_nonnegative"),
)
metadata.create_all(engine) # CREATE TABLE ... for all six
| Table | Rows | Key |
|---|---|---|
| persons | 39,468 | person_id |
| policies | 78,062 | policy_id → persons |
| vehicles | 84,346 | vehicle_id → policies |
| policy_drivers | 105,559 | (policy, person) |
| coverages | 224,988 | (vehicle, coverage) |
| claims | 11,562 | claim_id → vehicles |
The schema is also documentation: anyone can read which column refers to which table, and what values are allowed.
MigrationLoad everything in one transaction
ORDER = [persons, policies, vehicles, # parents
policy_drivers, coverages, claims] # first
with engine.begin() as conn: # commit at the end,
for table in ORDER: # roll back on any error
for chunk in pd.read_csv(RAW / f"{table.name}.csv",
chunksize=20_000,
parse_dates=DATES[table.name]):
rows = (chunk.astype(object)
.where(chunk.notna(), None) # NaN -> NULL
.to_dict(orient="records"))
conn.execute(insert(table), rows)
| Rows loaded | 543,985 |
| Time, SQLite | 4.8 s |
| Size | 27.1 MB of CSV → 44.9 MB file |
- Chunks keep memory flat however big the files
- A missing value must become
NULL, not the floatNaN - The database is bigger than the CSVs: it stores indexes and typed columns, and buys fast lookups with the space
Same answersExposure and claims, in SQL
SELECT strftime('%Y', p.inception_date) AS policy_year,
COUNT(*) AS vehicles,
SUM(MAX(0, julianday(MIN(p.expiry_date,
COALESCE(p.cancellation_date, p.expiry_date),
'2025-12-31'))
- julianday(p.inception_date) + 1) / 365.25)
AS car_years
FROM vehicles v JOIN policies p USING (policy_id)
GROUP BY policy_year;
| Year | Vehicles | Car-years |
|---|---|---|
| 2021 | 15,109 | 14,603.5 |
| 2022 | 16,283 | 15,707.9 |
| 2023 | 17,016 | 16,435.6 |
| 2024 | 17,689 | 17,068.9 |
| 2025 | 18,249 | 8,964.6 |
| Total | 84,346 | 72,780.5 |
pandas gives 72,780.5 too, and the claims query gives 4,565 liability claims ($59.06m incurred) to 31 December 2025: the Course 3 numbers, exactly.
ConstraintsThe database refuses bad data
| Attempted insert | SQLite says |
|---|---|
| Claim on VEH999999 | FOREIGN KEY constraint failed |
| Reported before the accident | CHECK constraint failed: reported_after_accident |
| Paid −$50 | CHECK constraint failed: amounts_nonnegative |
| Claim id already used | UNIQUE constraint failed: claims.claim_id |
One good and one bad claim in the same transaction: afterwards there are still 11,562 claims. Neither was stored.
@event.listens_for(engine, "connect")
def _fk_on(dbapi_conn, _):
dbapi_conn.execute("PRAGMA foreign_keys = ON")
SQLite ignores foreign keys unless each connection asks. Without the PRAGMA, the orphan claim on VEH999999 was accepted. PostgreSQL always enforces them: insert or update on table “claims” violates foreign key constraint “claims_vehicle_id_fkey”.
Querying Safely & at Scale
Parameters, indexes, chunks.
SQL injectionNever build SQL with string formatting
evil = "x' OR '1'='1" # typed into a search box
# WRONG: the input becomes part of the SQL
conn.execute(text(f"SELECT * FROM policies "
f"WHERE policy_id = '{evil}'"))
# ... WHERE policy_id = 'x' OR '1'='1' -> 78,062 rows
# RIGHT: a bound parameter is only ever a value
conn.execute(text("SELECT * FROM policies "
"WHERE policy_id = :pid"), {"pid": evil})
# -> 0 rows
The f-string version returned every policy in the book. With '; DROP TABLE claims; -- it could do worse.
- Bound parameters (
:pid) are sent separately from the SQL; the database never parses them as code - SQLAlchemy expressions (
select(...).where(...)) always bind their values - The same rule applies to dates, amounts, and lists: bind them all
IndexesHow the database finds one vehicle’s claims
SELECT * FROM claims WHERE vehicle_id = 'VEH000005';
SEARCH claims USING INDEX ix_claims_vehicle_id
(vehicle_id=?)
index=Trueonclaims.vehicle_idcreatedix_claims_vehicle_id: a sorted lookup structure, like a book’s index- Primary keys are indexed automatically; foreign-key columns are not, so index the ones you join or filter on
- Indexes make reads fast and writes a little slower: index what you query
EXPLAIN (PostgreSQL: EXPLAIN ANALYZE) shows the plan. “SCAN” on a big table where you expected “SEARCH” means a missing index.
Bigger than memoryChunk it, or let the database do it
q = "SELECT coverage, annual_premium FROM coverages"
# (a) everything at once
pd.read_sql(text(q), conn).groupby("coverage").sum()
# (b) 10,000 rows at a time, streamed
parts = [chunk.groupby("coverage")["annual_premium"].sum()
for chunk in pd.read_sql(
text(q), conn.execution_options(stream_results=True),
chunksize=10_000)]
pd.concat(parts).groupby(level=0).sum()
# (c) the database aggregates; 3 rows come back
pd.read_sql(text("SELECT coverage, SUM(annual_premium) "
"FROM coverages GROUP BY coverage"), conn)
| 224,988 rows | Peak Python memory |
|---|---|
| (a) all at once | 59.2 MB |
| (b) chunks of 10,000 | 4.3 MB |
| (c) GROUP BY in SQL | 3 rows |
All three agree: LIAB $92.54m, COLL $27.26m, COMP $6.71m. Chunk memory stays flat as the table grows; the all-at-once version doesn’t.
Summing, counting, and joining are what databases are built for. Pull out aggregates, not raw rows, whenever you can.
The ORM, and Production Databases
Objects for application code; the same code on PostgreSQL.
ORMRows as Python objects
class Base(DeclarativeBase):
pass
class QuoteRecord(Base):
__tablename__ = "quotes"
id: Mapped[int] = mapped_column(primary_key=True)
created_at: Mapped[datetime] = mapped_column(
UTCDateTime, default=lambda: datetime.now(UTC))
request: Mapped[dict[str, Any]] = mapped_column(JSON)
liab_premium: Mapped[float]
coll_premium: Mapped[float | None]
total_premium: Mapped[float]
model_version: Mapped[str] = mapped_column(String(40),
index=True)
with Session(engine) as s:
s.add(QuoteRecord(request=..., liab_premium=4648.59, ...))
s.commit()
rec = s.get(QuoteRecord, 1)
| 20,000 quote rows | Time |
|---|---|
| ORM, one object at a time | 0.76 s |
insert(QuoteRecord), one statement | 0.21 s |
- The ORM suits application code: one quote at a time, typed attributes, relationships
- Bulk loads and analysis: Core
insert()and SQL - Both compile to the same SQL;
print(stmt)shows it
A bug the tests foundSQLite has no time zones
E Differing items:
E {'created_at': '2026-09-30T14:58:24.157592'} !=
E {'created_at': '2026-09-30T14:58:24.157592Z'}
The quote was created in UTC. Read back from SQLite, its time zone was gone: the same quote now had a different timestamp, and comparing it with an aware one raises TypeError.
class UTCDateTime(TypeDecorator[datetime]):
"""Stored in UTC; always returned time-zone aware."""
impl = DateTime(timezone=True)
cache_ok = True
def process_bind_param(self, value, dialect):
if value is not None and value.tzinfo is None:
raise ValueError("naive datetime: attach a "
"time zone before storing")
return value.astimezone(UTC) if value else None
def process_result_value(self, value, dialect):
if value is not None and value.tzinfo is None:
return value.replace(tzinfo=UTC)
return value
PostgreSQLChange the URL, keep the code
engine = sm.make_engine(
"postgresql+psycopg://postgres:***@localhost:5433/postgres")
counts = sm.migrate(engine) # the same function
| SQLite | PostgreSQL | |
|---|---|---|
| Rows loaded | 543,985 | 543,985 |
| Load time | 4.8 s | 30.5 s |
| Car-years | 72,780.5 | 72,780.5 |
| LIAB claims | 4,565 | 4,565 |
| Foreign keys | only with the PRAGMA | always |
What changed: the date arithmetic (julianday becomes date subtraction, MAX/MIN become GREATEST/LEAST). Keep dialect-specific SQL in few, tested places.
PostgreSQL is a server: loading over a network connection is slower than writing a local file, and in exchange you get many concurrent users, permissions, backups, and replication.
Actuarial Lab · 75 minFrom CSV pipeline to SQL
- Declare the six tables with keys, foreign keys, and CHECK constraints
- Load all six in chunks, in one transaction, with foreign keys switched on
- Car-years by policy year and claims by coverage, in SQL; match pandas
- Try five bad claims; show a failed transaction leaves nothing behind
- An injection-safe policy lookup
- Premium by coverage three ways; compare peak memory
- An ORM quote log; run everything again on PostgreSQL
Answer key
| Rows (all six tables) | 543,985 |
| Car-years to 2025-12-31 | 72,780.5 |
| LIAB claims | 4,565 |
| Injection: unsafe / safe | 78,062 / 0 |
| Premium, LIAB | $92.54m |
RecapLesson 6 checklist
- Keys, foreign keys, and constraints are in the schema
- SQLite connections switch foreign keys on
- Loads run in one transaction
- Every value in a query is a bound parameter
- I index the columns I filter and join on
- Large tables are chunked or aggregated in the database
- Timestamps are stored in UTC and read back aware
- The same code runs on SQLite and PostgreSQL
Containerization with Docker
“It works on my machine” solved: multi-stage Docker images, secrets that never enter an image, and the whole platform started with one command.