{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cell-000",
   "metadata": {},
   "source": [
    "# Lesson 8 (Part A) — Pipeline Architecture & Configuration\n",
    "\n",
    "**Companion notebook** for the Lesson 8A slides. The building blocks of a pipeline —\n",
    "configuration, path handling, logging, validation gates, and a command-line interface —\n",
    "tried out one at a time before you assemble them in your project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-001",
   "metadata": {},
   "outputs": [],
   "source": [
    "def check(label, got, expected, tol=1e-6):\n",
    "    \"\"\"Compare your answer with the expected value; prints ✅ or ❌ (never raises).\"\"\"\n",
    "    import math\n",
    "    try:\n",
    "        if isinstance(expected, float) or isinstance(got, float):\n",
    "            ok = math.isclose(float(got), float(expected), rel_tol=tol, abs_tol=tol)\n",
    "        elif isinstance(expected, (list, tuple)) and expected and isinstance(expected[0], str):\n",
    "            ok = list(got) == list(expected)\n",
    "        elif hasattr(expected, \"__len__\") and not isinstance(expected, str):\n",
    "            import numpy as np\n",
    "            ok = np.allclose(np.asarray(got, dtype=float), np.asarray(expected, dtype=float), rtol=tol, atol=tol)\n",
    "        else:\n",
    "            ok = got == expected\n",
    "    except Exception:\n",
    "        ok = False\n",
    "    print((\"✅ \" if ok else \"❌ \") + f\"{label}: got {got!r}, expected {expected!r}\")\n",
    "import argparse\n",
    "import logging\n",
    "import tomllib\n",
    "from pathlib import Path\n",
    "\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-002",
   "metadata": {},
   "source": [
    "## 1. Configuration in TOML\n",
    "`tomllib.loads` parses a string; `tomllib.load` reads a file opened in **binary** mode.\n",
    "Dates become real `datetime.date` objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-003",
   "metadata": {},
   "outputs": [],
   "source": [
    "CONFIG_TEXT = '''\n",
    "[paths]\n",
    "motor = \"../data/motor\"\n",
    "output = \"../output\"\n",
    "\n",
    "[evaluation]\n",
    "date = 2025-12-31\n",
    "\n",
    "[indication]\n",
    "target_loss_ratio = 0.68\n",
    "annual_trend = 0.04\n",
    "trend_years = 1.5\n",
    "full_credibility_claims = 1082\n",
    "max_change = 0.15\n",
    "'''\n",
    "cfg = tomllib.loads(CONFIG_TEXT)\n",
    "print(cfg[\"evaluation\"][\"date\"], type(cfg[\"evaluation\"][\"date\"]))\n",
    "cfg[\"indication\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-004",
   "metadata": {},
   "source": [
    "## 2. Resolving paths relative to the config file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-005",
   "metadata": {},
   "outputs": [],
   "source": [
    "config_path = Path(\"../config.toml\")          # imagine the file lives in the project root\n",
    "base = config_path.parent\n",
    "paths = {k: (base / v).resolve() for k, v in cfg[\"paths\"].items()}\n",
    "paths"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-006",
   "metadata": {},
   "source": [
    "## 3. Logging instead of `print`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-007",
   "metadata": {},
   "outputs": [],
   "source": [
    "logging.basicConfig(level=logging.INFO, format=\"%(levelname)-7s %(name)s: %(message)s\", force=True)\n",
    "log = logging.getLogger(\"rating_pipeline.extract\")\n",
    "log.info(\"loaded %-10s %8s rows\", \"policies\", f\"{22_185:,}\")\n",
    "log.debug(\"only shown with --verbose\")\n",
    "log.warning(\"%d vehicles outside every county\", 12)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-008",
   "metadata": {},
   "source": [
    "## 4. A command-line interface with `argparse`\n",
    "In a script, `parse_args()` reads the real command line; here we pass a list to try it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-009",
   "metadata": {},
   "outputs": [],
   "source": [
    "parser = argparse.ArgumentParser(description=\"Motor rating pipeline\")\n",
    "parser.add_argument(\"--config\", type=Path, default=Path(\"config.toml\"))\n",
    "parser.add_argument(\"-v\", \"--verbose\", action=\"store_true\")\n",
    "parser.parse_args([\"--config\", \"quarterly.toml\", \"-v\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-010",
   "metadata": {},
   "source": [
    "---\n",
    "## Your turn 1 — a validation gate\n",
    "\n",
    "Write `check_keys(tables)` that raises `DataQualityError` with a clear message if any\n",
    "table has duplicate primary keys, or if any claim's `policy_id` is missing from\n",
    "`policies`. `KEYS` gives each table's primary key."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-011",
   "metadata": {},
   "outputs": [],
   "source": [
    "class DataQualityError(ValueError):\n",
    "    \"\"\"Raised when an extract breaks a key constraint.\"\"\"\n",
    "\n",
    "\n",
    "KEYS = {\"policies\": [\"policy_id\"], \"claims\": [\"claim_id\"]}\n",
    "\n",
    "\n",
    "def check_keys(tables: dict[str, pd.DataFrame]) -> None:\n",
    "    ...  # your code here\n",
    "\n",
    "\n",
    "def raises_dq_error(tables) -> bool:\n",
    "    try:\n",
    "        check_keys(tables)\n",
    "    except DataQualityError:\n",
    "        return True\n",
    "    return False\n",
    "\n",
    "\n",
    "good = {\"policies\": pd.DataFrame({\"policy_id\": [\"P1\", \"P2\"]}),\n",
    "        \"claims\": pd.DataFrame({\"claim_id\": [\"C1\"], \"policy_id\": [\"P1\"]})}\n",
    "dup = {**good, \"policies\": pd.DataFrame({\"policy_id\": [\"P1\", \"P1\"]})}\n",
    "orphan = {**good, \"claims\": pd.DataFrame({\"claim_id\": [\"C1\"], \"policy_id\": [\"P9\"]})}\n",
    "check(\"good data passes\", raises_dq_error(good), False)\n",
    "check(\"duplicate key caught\", raises_dq_error(dup), True)\n",
    "check(\"orphan claim caught\", raises_dq_error(orphan), True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-012",
   "metadata": {},
   "source": [
    "## Your turn 2 — pass config straight into a function\n",
    "\n",
    "Write `indicate(loss_ratio, claims, **params)` so it can be called with\n",
    "`**cfg[\"indication\"]`. Trend the loss ratio, compare with the target, weight by\n",
    "credibility $Z = \\min(1, \\sqrt{\\text{claims}/\\text{full\\_credibility\\_claims}})$, and cap."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-013",
   "metadata": {},
   "outputs": [],
   "source": [
    "def indicate(loss_ratio, claims, target_loss_ratio, annual_trend, trend_years,\n",
    "             full_credibility_claims, max_change):\n",
    "    ...  # your code here\n",
    "\n",
    "\n",
    "check(\"Coastal 17-24\", indicate(0.775, 132, **cfg[\"indication\"]), 0.0729, tol=1e-3)\n",
    "check(\"capped at +15%\", indicate(1.186, 2000, **cfg[\"indication\"]), 0.15)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}