This is the actual architecture, not a hand-wave. Each stage is re-runnable, idempotent, and refuses to start if the upstream stage failed. The validator hard-fails the pipeline on schema drift before bad data reaches modeling. The walk-forward harness invokes the leakage gate before the first model fit. The holdout 2025 season is read exactly once.
From the build log (spring 2026), kept current where it matters: the sealed 2025 season was subsequently opened exactly once — a frozen-weights replay published after the season on the 2025 proof page. The pipeline and gates described below still run on every commit.
Six stages, all idempotent. scripts/ingest_raw.py calls the validator inline; if validation fails, ingest exits non-zero and the freeze step refuses to start.
(1) validate_raw.py hard-fails ingest on schema drift. (2) walk_forward_splits invokes validate_feature_columns(live=True) at split-time, before any model fit. (3) holdout_seasons=(2025,) is honored by both walk-forward and season-rollover splits, AND by the runner's pre-filter.
Every stage writes a deterministic output. Re-running ingest produces the same parquets (modulo nflverse releases). Re-running freeze produces the same slices. Re-running the harness produces bit-identical fold predictions (XGBoost tree_method=hist + random_state=0; sklearn Ridge with random_state=0).
Every nflverse source falls into one of four schemas. Knowing which one a source belongs to determines how we filter it for slicing.
| Kind | Defining trait | Filter rule | Example sources |
|---|---|---|---|
| ASeason + Week | Has season AND week columns; no nflverse_game_id |
season == X AND (week ≥ min_week OR week IS NULL) |
pbp, player_stats, team_stats, schedules, snap_counts, officials, ngs_*, rosters_weekly, injuries, pfr_advstats_*, depth_charts_legacy, ff_opportunity_weekly |
| BSeason only | Has season but no week (annual snapshot) |
season == X |
rosters |
| CGame-keyed | Has nflverse_game_id as authoritative join (with or without season/week) |
nflverse_game_id IN slice.game_ids |
participation, ftn_charting |
| DTimestamp-keyed | Has dt ISO8601 column instead of season/week |
dt ∈ [Aug 1 of season X, Mar 1 of X+1) |
depth_charts_timestamped (post-2025 nflverse schema) |
| ZAll-time | No season key (reference tables) | Not slice-able; read whole | players, teams, combine, draft_picks, contracts, ff_playerids |
The freeze produces two physically-separate slice directories. Both are read-only post-freeze; the AutoResearch loop is contractually forbidden from writing to them.
Drives the AutoResearch loop's keep-or-revert decision. Every change to features, model class, hyperparameters, anything — gets evaluated here. Walk-forward CV across 14 folds (weeks 9-22), per fold the model trains on everything strictly earlier and predicts the target week.
The sacred ship-gate. The AutoResearch loop NEVER reads this. Belt + suspenders: run_baseline.py drops season == 2025 from the input frame BEFORE the harness; walk_forward_splits filters again; holdout_seasons is honored by both split flavors.
Per-source filtering uses one of four functions depending on the source's kind: _filt_season_week (predicate-based, captures wk 0/23/null), _filt_season_only, _filt_nflverse_game_id, _filt_dt_year. Two invariants enforced: (1) every source filters cleanly for at least one slice; (2) row counts match the raw season-predicate count exactly — no silent leak from PBP-only pair sets.
The model that ships to POST /api/snapscore/predict runs the LIVE pipeline. A research model can use the full training surface for upper-bound studies, but it doesn't serve traffic.
Released within ~24h of game completion or known pre-kickoff. Safe to use as features in the production pipeline.
Three surfaces: every column in participation.parquet (FTN end-of-regular-season backfill), PBP postgame mirrors, post-game telemetry.
Per @realfrankbrank 2025-09-02 (FTN): "we will backfill each year at the end of the regular season." 2025 participation lands ~Feb 2026 post-Super-Bowl. The leakage validator runs at split-time via validate_feature_columns(feature_columns, live=True); any pipeline using a forbidden column raises LeakageError before training starts. 16 anti-leakage tests pin the contract; one is schema-driven against the actual participation.parquet file.
Schema drift catches added/removed/dtype-changed columns. Coverage invariants catch missing seasons + weeks + key uniqueness violations. In-season currency catches a stale ingest at game-day. Each layer is independently tested with planted-regression smoke tests.
Diff against data/raw/_expected_schemas.json (committed baseline). Any column that disappears, or whose dtype changes, fails the ingest.
Per-source row count, per-season presence, per-week-set match. Plus PBP (game_id, play_id) uniqueness and depth-charts dual-schema invariant.
If today is in the NFL game window (Sept-Feb), 14 live-critical sources MUST have rows for the current season AND latest-week lag ≤ 2 weeks behind today. Otherwise predictions go stale.
Pre-kickoff inputs (current injuries, current depth charts, current Vegas lines, weather forecast, inactives list) are moving targets. Every prediction binds to a single (target_game_id, as_of_ts) tuple with per-source freshness guarantees. M_serve scope; spec'd in M0 architecture.
| Live source | Provider | max_staleness | Rationale |
|---|---|---|---|
injuries | Adam's ESPN/PFR scrape | 4h | NFL official injury reports finalize Fri 4 PM ET; can shift Sat AM |
depth_charts | nflverse depth_charts_timestamped | 24h | Daily updates; rare same-day swings unless injury cascade |
vegas_lines | Sportsbook API (TBD) | 1h | Lines move fast near kickoff; sharp money typically lands T-3h |
weather_forecast | NWS / Tomorrow.io | 6h | Forecast accuracy stabilizes inside 6h of kickoff |
inactives | NFL inactives feed | 1.5h | Officially announced 90 min before kickoff |
rosters_weekly | nflverse | 7d | Updates Wed; covers the gameweek |
schedules (Vegas opening) | nflverse schedules | 24h | Opening line + venue + opponent — slow-changing |
pbp_lagged / ngs_lagged | nflverse | 48h | Prior-week stats; landed by Tue AM |
A snapshot is valid iff every source's actual_staleness_seconds ≤ max_staleness_seconds. Otherwise the loader raises StaleSnapshotError and the prediction is blocked. POST /api/snapscore/predict returns 503 instead of guessing.
17 boundary tests pin this contract via synthetic frames. Every fit instantiates a fresh model — no parameters carry across folds. The imputer + scaler fit on each fold's train rows separately; no global statistics leak.
| Invariant | What's tested | Status |
|---|---|---|
| train never contains target week | train.filter(season == S AND week == W).is_empty() | ✓ pinned |
| train never contains future seasons | train.filter(season > S).is_empty() | ✓ pinned |
| train never contains future weeks same season | train.filter(season == S AND week > W).is_empty() | ✓ pinned |
| train DOES contain prior seasons | positive coverage check | ✓ pinned |
| train DOES contain prior weeks of same season | positive coverage check | ✓ pinned |
| train sizes monotonically grow fold-over-fold | walk-forward shape | ✓ pinned |
| min_train_weeks gate filters early folds | no fold fires with insufficient train | ✓ pinned |
| season_filter restricts predict seasons | arg-handling correctness | ✓ pinned |
| holdout_seasons excluded from train AND predict | quarantine integrity | ✓ pinned |
| leakage gate fires at split-time on blocked column | raises before any fit | ✓ pinned |
| training mode permits everything (live=False) | gate is gated on live flag | ✓ pinned |
| schema-driven leakage coverage from disk | every participation column blocked | ✓ pinned |
Every architectural decision is logged with a date and a reason. Future readers see the why, not just the what.
| Date | Decision | Why |
|---|---|---|
| 2026-05-08 | Migrate from nfl_data_py to nflreadpy | Upstream parquet renames broke 2025 ingest under the archived nfl_data_py |
| 2026-05-08 | Two-tier eval (validation 2024 wks 9+ + holdout 2025) | Validation drives the loop; holdout is sacred ship-gate |
| 2026-05-08 | Split depth_charts at ingest | nflverse changed schema mid-2025; one parquet would mix two key formats |
| 2026-05-09 | Predicate-based filtering (not pair-set) for Kind A | Codex C1: PBP wks 1-22 missed NGS wk 0/23 + officials wk 23, leak risk |
| 2026-05-09 | Add validate_raw.py 3-layer validator | Catch silent nflverse regressions at ingest, not at modeling |
| 2026-05-09 | Holdout min_week=0 (full season including wk 0/23) | Ensure NGS preseason aggregates can never leak to training |
| 2026-05-09 | Commit _expected_schemas.json baseline | Anyone cloning the repo can verify their fresh ingest matches our committed baseline |
| 2026-05-09 | XGBoost target-aware objective (poisson for TDs) | Poisson predicts non-negative rate; fixes Ridge "negative TD" sMAPE inflation |
| 2026-05-09 | Add team_changed_since_last_game indicator (codex F4) | Lets model down-weight prior-team signal on free-agency / trades |
| 2026-05-09 | Block play_id + nflverse_game_id in leakage gate (codex F1) | Even though these are normally join keys, the validator's contract is "no FEATURE column from participation" |
| 2026-05-09 | Remove min_target_value auto-default (codex F2) | Label-based filter introduced selection bias; default cohort is now unfiltered |