Snapcount Methodology, v1.0

1.Scoreboards

1.1 The internal yardstick: TD-free, position-pure slate Spearman

The north-star internal metric is weekly ranking quality: within each (position × week) slate, rank players by a projected score, and measure the Spearman rank correlation against the realized ordering, alongside Top-N hit rate and start/sit accuracy. The score is deliberately touchdown-free:

QB score = pass_yd × 0.04 + rush_yd × 0.10
RB score = (rush_yd + rec_yd) × 0.10 + receptions
WR score = rec_yd × 0.10 + receptions
TE score = rec_yd × 0.10 + receptions

Per-position slate sizes (Top-N): QB 12 · RB 24 · WR 36 · TE 12

Three design choices, each load-bearing:

• TD-free internally. Touchdowns are high-variance, low-frequency events; putting them in the modeling target pollutes the volume-signal loop the model actually learns from. TDs are handled by dedicated calibrated probability models (§3.5) and, prospectively, an additive expected-TD layer (§7).
• Position-pure. Truth assigns each player one modal position and ranks within it (WRs against WRs). An earlier mixed-pool variant ranked all ~289 skill players together, which inflated Spearman (~0.744) by rewarding easy cross-position ordering and made the recency baseline look artificially strong. The two truth constructions were reconciled (per-slate Spearman 1.000 on identical scores); the divergence was position-purity, not the yardstick.
• Un-game-able. Ranking cannot be floor-gamed the way banded "closeness" metrics were — an earlier closeness headline (~50% within-band) was shown to be floor-gamed (honest ~19%), which is why the ranking metric replaced it.

A result "counts" only under the robustness contract of snapcore/eval/ranking_eval.py: beats the recency baseline in ≥3 of 4 seasons (2021–2024), beats a flat constant, and survives a label-shuffle placebo.

1.2 Fantasy-points variants (the public-facing candidate)

Because the market is judged in fantasy points, a parallel scoreboard (snapcore/eval/fantasy_points_eval.py) mirrors the same position-pure slate-Spearman machinery on actual fantasy points — half-PPR primary, standard and full-PPR secondary (interceptions/fumbles/2-pt conversions excluded from truth: no predictor in the stack projects them, documented). How much do TDs reorder reality? Spearman between TD-free truth and fantasy truth:

So a TD-blind predictor concedes real headroom (most at QB and TE) on fantasy truth — which is exactly what the prospective E[TD] layer (§7) is measured against. The internal modeling metric stays TD-free; the fantasy-points scoreboard is the candidate public yardstick.

1.3 Benchmarks are fenced, never blended

RotoWire (via Sleeper's public API, company="rotowire" on the wire) and ESPN's weekly projections are scored as benchmarks only. They are structurally fenced in code (BENCHMARK_ONLY flag, separate registry, an assert_not_benchmark() guard on every feature path): no third-party projection is ever an input, feature, anchor, or blend component of our predictions. The same rule holds for betting prop lines (eval-only since 2026-06-08). Game-level environment (spread/total from schedules) is the one market signal that is feature-legal — and it tested dead for within-position ranking anyway (§6).

2.Data and the anti-leakage doctrine

2.1 Sources

All ingestion comes from the nflverse open-data ecosystem via nflreadpy (play-by-play, weekly box scores, rosters, depth charts, schedules with pregame lines, NGS weekly aggregates, draft, contracts, trades). Raw pulls are schema-validated (drift detection against expected schemas) before any feature build. No paid data feeds are in the model today (§5.3).

2.2 Feature layers

Features are built in eight composable layers, each contributing named columns; the canonical entry point composes them and then passes every column list through the anti-leakage gate:

2.3 The block-list: what can never reach a live prediction

The validator (snapcore/eval/leakage.py) is mandatory on every live feature build and fail-closed. It block-lists five surfaces:

• 1 · The participation feed, in its entirety. FTN charting (routes, coverage types, pressure) is backfilled only at end-of-regular-season; in-season these values are NULL. Every column in participation.parquet is blocked — including identifier keys, so the surface has zero silent-leak area.
• 2 · Play-by-play post-game mirrors. The same charting data re-embedded in PBP rows after backfill (was_pressure, route, defense_coverage_type). Blocked at both surfaces.
• 3 · Post-game telemetry. Measured-after-the-fact context: temp, wind, series_result, fixed_drive_result. (Weather forecasts are a separate, allowed feed.)
• 4 · Outcome columns. epa, wpa, yards_gained, complete_pass — the targets themselves or trivially derived from them.
• 5 · Current-week true-opportunity surfaces. Actual same-week snap share and raw expected-value opportunity columns. Lagged derivatives are allowed only after renaming to explicit prior-window features (e.g. snap_share_lag3) — the rename forces the lag to be visible and auditable.

The doctrine is precise to the column name: per-play time_to_throw (participation, post-game charted) is blocked, while avg_time_to_throw (NGS weekly aggregate, released within ~24h of games and available pregame the following week) is allowed — and there is a unit test asserting exactly that distinction.

3.Models, per season regime

3.1 The core: walk-forward opportunity projection (W5+, RB/WR/TE)

The production ranking model decomposes each player-week into projected volume × shrunken efficiency (snapcore/models/opportunity_model.py). Per position, the supervised volume targets and feature sets are:

Volume targets      QB: pass att + rush att   RB: carries + targets   WR: targets   TE: targets

CANONICAL_FEATURES  QB: lag_att, lag_car, lag_snap_share, depth_chart_order
                    RB: lag_car, lag_car_share, lag_tgt, lag_tgt_share, lag_snap_share, depth_chart_order
                    WR: lag_tgt, lag_tgt_share, lag_snap_share, depth_chart_order, lag_airyds
                    TE: lag_tgt, lag_tgt_share, lag_snap_share, depth_chart_order, lag_airyds

Every lagged feature is an exponentially-decayed trailing window computed strictly on prior weeks: shift(1).ewm_mean(half_life=2.5) within (player, season) — the shift(1) guarantees week W reads only weeks < W. The training panel spans seasons 2018–2024; evaluation seasons are 2021–2024.

Each volume is projected by its own walk-forward regression. For the ranking lane the estimator is a standardized RidgeCV (alphas 0.1, 1, 10, 100, 1000) — the feature set is small and collinear by design, so a regularized linear model selects among proxies rather than hand-stacking them. Missing feature cells are imputed with training-set medians (computed on pre-eval rows only). The training mask for a season-S, weeks-9–18 target is:

train:   (season < S)  OR  (season == S AND week ≤ 8)     — with week ≥ 2 and a non-null lag guard
predict: (season == S) AND (week ∈ eval_weeks)

Projected volumes are recombined through the position's TD-free efficiency decomposition (yards-per-attempt / per-carry / per-target and catch rate), producing proj_score. An oracle variant that keeps actual volumes measures the achievable ceiling (§5.2).

For the displayed yards median (the number on the board, not the ranking), the estimator is a gradient-boosted quantile median — HistGradientBoostingRegressor(max_iter=300, max_depth=3, learning_rate=0.05, l2_regularization=1.0, random_state=0, loss="quantile", quantile=0.5) — a frozen configuration, adopted after paired MAE wins over the ridge base (RB −1.19 / WR −0.94 / TE −0.21 displayed yards, pooled sign-test p ≈ 2e-20 over 120 cells) with interval coverage unchanged. This is an L1-versus-L2 structural improvement, not a modeling breakthrough, and we quote it as such.

On top of the model score, the ranking lane applies a walk-forward blend with the recency baseline (models/recency_blend.py, RB/WR/TE only; blend weight re-selected per position and season on strictly prior data). Current canonical pooled Spearman (2021–2024, W9–18, own population):

Honest headline: the gap to the market narrowed and WR is statistically unresolved — that is parity-adjacent, not a win. QB is absent from this table by design (§3.3).

3.2 Efficiency shrinkage (empirical Bayes)

Trailing per-opportunity efficiency mean-reverts; carrying a hot streak forward hurts ranking. Each player's trailing efficiency is therefore shrunk toward a position prior:

shrunk_eff = w · trailing_eff + (1 − w) · prior        w = n / (n + k)

where n is the trailing opportunity count and k is chosen by argmin reconstruction-MAE on an out-of-sample calibration cohort (each season's weeks 1–8), then frozen before scoring weeks 9–18. Low-n players (noisy ratios) are pulled hard toward the prior; high-n players keep their own signal. Shrinkage is applied to RB/WR/TE only: it measurably hurts QB ranking, so QB efficiency stays raw — a finding from the Phase-2 validation, kept as a hard rule.

3.3 QB: no model is claimed

Every QB ranking model built to date fails the season-consistency gate (0/4 seasons vs the recency baseline for the promoted-model family; one later side-flag candidate resolved NULL under paired testing against the stronger ewma5 incumbent — mean lift fails Bonferroni, 0/4 seasons, unstable weights). QB therefore serves recency_ewma5 — an exponentially weighted average of prior TD-free scores — and the projection grid leaves the QB model score deliberately NULL. We publish QB numbers under this baseline and do not represent them as model output.

3.4 The early-season regime: W1–4 (promoted 2026-07-12)

The W5+ production training mask is illegal for W1–4 targets — it would train on the very weeks being predicted. The early lane therefore uses a strict mask, and Week 1 gets its own construction:

• W1 — the model-free book. QB = recency_ewma5; RB/WR/TE = last_season_mean_post, the player's prior-season mean TD-free score including playoff games (playoffs happened before W1 and are invisible to the regular recency windows — including them is free, legal information). Backtest: the book scores ALL 0.571 vs 0.550 for the serving-family recency baseline (+0.021, matched population, 16 position-season cells, 2021–2024). Five research iterations (§6) tried to beat it with models; all failed the gate; the book is what we serve.
• W2–4 — model_early. The full opportunity pipeline under the strict (season < S) mask — no same-season rows at all. Backtest (bucket ALL): 0.545 / 0.594 / 0.588 at W2/W3/W4 vs 0.540 / 0.579 / 0.577 for the recency baseline. Honest caveat, recorded at promotion: at W2 the prior-season book scores 0.558 on its own population, but no W2-specific candidate cleared the gate — the promoted construction mirrors the harness's validated book-to-model transition exactly rather than cherry-picking a per-week argmax.
• W5+ — unchanged; the promotion was proven byte-identical on late weeks (a 2024 W9–18 regeneration diffed clean, zero diffs).
• Seal discipline. The early lane refuses any request for seasons ≥ 2025 with a hard error: 2025 Week 1 is the program's pre-registered one-shot holdout (§4.4).

The promotion also closed a latent leakage path: before it, an early-week request could reach the W9+ production mask. The W1 numbers were independently hand-traced to raw stats (agreement to 1e-9) during adversarial verification. Known caveat, recorded: the W1–4 boom/bust and interval columns are not yet early-week-validated.

3.5 Touchdown models: distributions, then calibration

TDs are never regressed as counts (symmetric-percentage error on 0-vs-0.4 outcomes is a math artifact, not signal). Instead, multinomial XGBoost (multi:softprob) predicts a distribution over discrete outcomes, binned per target from the empirical distribution:

passing TDs:    classes [0, 1, 2, 3+]   E[Y] values [0, 1, 2, 3.3]
rushing TDs:    classes [0, 1, 2+]      E[Y] values [0, 1, 2.2]
receiving TDs:  classes [0, 1, 2+]      E[Y] values [0, 1, 2.2]

E[TD] = Σ class_value × P(class)

Two calibration fixes were merged 2026-07-03 after a three-fold bootstrap verification (all folds p < .001, coefficients re-derived independently to 4 decimal places):

• Per-position predicted fractions replace a uniform rule that ignored a 4× spread in any-TD base rates (QB .82 / RB .33 / WR .27 / TE .20). Any-TD predicted fractions: QB .75, RB .50, WR .075, TE .05; multi-TD: QB .75, RB .50, WR .15, TE .20.
• Per-position Platt maps (logistic regression, C=1e6) map predicted TD count to P(any TD); fit on a 2021–2023 + 2024 W1–8 calibration surface (n = 13,885 rows), applied frozen. Multi-TD keeps the raw distribution (its calibration error was already .008–.021; recalibration tested null).

Recorded caveats: the QB any-TD fraction .75 is grid-capped below its .82 base rate; the multi-TD board expands 186 → 571 predicted rows (QB-dominated); 2022 any-TD precision is .535 on the thinnest calibration fold.

3.6 Conformal intervals: the floor–ceiling range

Every displayed projection is a 70%-target central interval on yards (snapcore/serving/projection_intervals.py):

median  = walk-forward gbm_q50 projected yards (§3.1)
raw_lo  = median − σ        raw_hi = median + σ        σ = lagged trailing SD of the player's yards
floor   = clip(raw_lo − q̂_lo, 0)                       ceiling = raw_hi + q̂_hi

The asymmetric offsets (q̂_lo, q̂_hi) are conformal corrections calibrated walk-forward on prior-season out-of-sample residuals (each median is itself an out-of-sample projection, so no eval rows leak into calibration). Target coverage 0.70 (α = 0.30); measured coverage post-merge:

The per-player width is the honest product form of the boom/bust research: a steady every-down role gets a tight band, a volatile role a wide one.

3.7 Boom / bust probabilities (leave-one-season-out)

Per player-week, calibrated P(boom) and P(bust) on the TD-free score (snapcore/models/boom_bust.py): boom = deviation from the pregame expectation in the top tercile of the per-position residual distribution, bust = bottom tercile (terciles, not quintiles — TE slates carry only ~12 players, and a ~33% base rate keeps calibration bins populated). Tercile thresholds are fit only on calibration seasons, never eval rows. The classifier is a standardized logistic regression on lagged volatility/role features (air yards, target share, snap share, depth-chart order, trailing residual volatility, trailing score CV); its output is calibrated by a Platt / isotonic / beta calibrator selected by a pre-declared 0.6·Brier + 0.4·ECE criterion, fit on out-of-fold predictions, leave-one-season-out over 2021–2024. Scoring follows a reliability contract (AUC ≥ 0.60, ECE ≤ 0.05, BSS > 0) against climatology and a naive-volatility floor. Positioning is explicit: this is a confidence layer, not an accuracy edge — an adversarial re-check showed much of the raw bust signal is mean-reversion a naive variance ranker also captures.

3.8 Cell-scoped exception: the RB-receptions count head

One cell (RB receptions spike/bust probabilities) is served by a negative-binomial count head: a Poisson-loss gradient-boosted mean on the RB feature set, NB2 dispersion via method-of-moments, spike probability from the NB CDF, shipped only behind a strict-prior isotonic calibration map (verified ECE 0.021). Gate evidence: paired lift over the incumbent provider in 3/3 seasons (p = .0215 / .0025 / .009; pooled 0.6206 vs 0.5105 spike AUC, Δ +0.110, p = 0.0015). Adoption conditions are recorded and enforceable: the incumbent runs in parallel through 2026 with auto-fallback on any trailing-8-week regression, and the head is scoped to this single cell — it was tested for TE receptions and made things significantly worse (−0.152 spike AUC), which is exactly why cell-scoping is a rule and not a vibe.

4.Validation protocol

4.1 Walk-forward cross-validation, and nothing else

The only valid evaluation shape for this domain (snapcore/eval/harness.py::walk_forward_splits): to score predictions at (season S, week W), train on (season < S) OR (season == S AND week < W), predict on (season == S AND week == W), iterate chronologically. Random k-fold is invalid — it trains on future weeks of the same season. Feature columns are validated against the leakage block-list before the first fold yields.

4.2 What it takes for a change to count

• Season-consistency gate: beat the baseline in ≥3 of 4 seasons (2021–2024). Pooled means alone are never accepted — weekly cells are few (4 W1 slates per position-season), so we demand consistency instead of averages.
• Label-shuffle placebo: re-score against shuffled truth; the signal must collapse to chance. Survivorship of this probe is quoted with every result.
• Paired significance: per-cell sign tests and Wilcoxon on identical populations, quoted verbatim — including when they are unflattering (many honest results in §6 carry sign_p 0.4–1.0 and are treated accordingly).
• Placebo baselines: flat-constant and naive-variance comparators guard against floor-gaming — the failure mode that demoted three of five early "champion" models (§4.3).
• No degradation elsewhere: early-week changes must leave W5–18 byte-identical (verified by regeneration diffs), and additive serving changes must leave existing flows untouched.

4.3 Adversarial verification

Before any promotion, an independent verification pass — instructed to refute, not confirm — re-derives the numbers and attacks the construction. This step has teeth; its record includes:

• Demoting 3 of 5 method-search "champion" models as floor-gaming or fragile noise (a flat constant beat them across seasons); the associated "+4% clean-live" claim was publicly retracted in the internal record.
• Correcting a claimed 4/4-season WR result to an honest 3/4.
• Re-anchoring the entire headroom narrative when the review found the comparison baseline (ewma3) was weak: against the honest ewma5 anchor, our measured edges shrink to ~+0.006–0.008 Spearman RB/WR/TE and the "% of oracle headroom captured" drops to ~3% (§5.2). The corrected numbers replaced the old ones everywhere.
• Hand-tracing the W1 promotion's numbers to raw stats (1e-9 agreement) and independently re-proving W5+ byte-identity.

4.4 The sealed-2025 protocol and pre-registration

• 2025 is sealed. No training, tuning, or feature selection ever touched the 2025 season (one narrow, dated exception: a one-time Week 5 spot-validation, recorded in the repo history). After the season ended, the frozen 2016–2024 model was replayed over 2025 once and every number published — the 2025 proof. Framing discipline: that artifact is a holdout replay generated after the fact, and is labeled as such. It is not, and is never described as, a live-season track record.
• 2025 Week 1 remains untouched even by the Week-1 research program — it is that program's pre-registered one-shot holdout, opened only by founder sign-off, and the code refuses 2025 requests with a hard error until then.
• W1-2026 pre-registration: the plan of record is to publish the timestamped Week-1 2026 prediction set before kickoff, so the first live-season claim is verifiable by anyone from public artifacts rather than asserted afterward.

5.Honest limitations

5.1 We trail the market on yards-ranking, at every position, today

The public market scoreboard (market_scoreboard.json, rendered on the 2024 replay page) grades us against RotoWire and ESPN on identical player populations — a player-week counts only if truth, our board, and every provider cover it. 2024 W9–18, 40 graded cells:

Readings, verbatim from the internal tracker: we are #1 at zero positions today. QB is a statistically resolved loss and 3–4× the size of any other gap. TE is the closest to unresolved (sign_p 0.344; we already take 3–4 of 10 weekly slates). These all-common numbers run higher than the own-population numbers elsewhere in this document because the common population drops market-uncovered fringe players — compare only within one table, never across scoreboards. The claim rule is encoded in the artifact itself: "best at a position" requires beating every provider on the identical population, with paired tests agreeing, over consecutive graded periods. By that rule we claim nothing today.

5.2 The free-data ceiling is real and measured

Headroom map (pooled Spearman, recency → our model → opportunity oracle, W9–18): RB .675 → .688 → .932; WR .643 → .666 → .876; TE .574 → .609 → .861; QB .474 → .457 → .751 (the model loses to recency at QB, hence §3.3). Against the corrected, honest recency anchor (ewma5), our model's edge is ~+0.006–0.008 Spearman at RB/WR/TE — roughly 3% of the oracle headroom, not the 5–12% an earlier weak-anchor comparison suggested. The ceiling was stress-tested: a richer gradient-boosted configuration plus every additional free lagged signal we could construct (team pace, pass rate, red-zone shares) added approximately nothing, replicated per position. Free public data is exhausted for this problem on both the ranking and boom/bust axes; the honest description of our current edge is "beats naive baselines, trails paid-analyst market projections."

5.3 What paid data would change (and the gate on buying it)

The oracle gap decomposes mostly into this-week volume information we cannot see in free data. The #1 scoped paid lever is in-season route participation (charted, e.g. FTN/PFF-class feeds), with an evidence bar attached: a Stage-A partial-correlation probe grades it +.251 (TE) / +.186 (RB) against residuals, and the expected gain is +0.03–0.08 Spearman on WR/TE. The spend rule is pre-registered: a paid sample must beat the free RotoWire benchmark on WR/TE (> 0.687 / > 0.636 on the own-population yardstick) in ≥3 of 4 seasons — beating our own recency baseline is not enough — and a $0 public-tracking-data pre-validation is mandatory before any vendor contract is signed.

5.4 Other stated limits

• QB: no model claimed; the served QB number is a recency baseline (§3.3), and QB is our largest market gap (§5.1).
• Early weeks: the market still leads us at W1–4 on fantasy truth (RotoWire ~0.606–0.609 vs our best early constructions ~0.514–0.599 depending on bucket and layer; §7).
• Rookies: the W1 book has 0% rookie coverage by construction (no prior season); today only the draft-capital research features even emit for W1 rookies. Providers do rank rookies; on rookie-inclusive populations we forfeit those rows.
• Truth definition: the fantasy scoreboard's truth excludes INT/fumbles/2-pt (unprojected by any predictor in the stack); a truth including INTs would slightly favor providers who project them at QB.
• No live-season track record yet. Everything here is backtest + frozen holdout replay; the 2026 season is the first live test, pre-registered per §4.4.

6.The improvement loop (ledger excerpts)

Every research iteration appends a verdict — with numbers — to a ledger; NULL verdicts are recorded with the same detail as promotions, and "never re-quote as support" flags are attached to near-misses. Excerpts from the Week-1 2026 program (five iterations, 2026-07-11 → 07-12) and the preceding loop campaign:

Selected reverts from the wider campaign, kept on the record so they stay dead: ranking-loss objectives (significantly worse than the squared-loss default); removal of an "obviously wrong" /3.0 divisor (significantly negative — it was beneficial implicit shrinkage); RB share pooling (significantly negative, Wilcoxon .017); the TE transfer of the RB count head (−0.152 spike AUC); player-level red-zone share features for TDs (null beyond lagged TD rate + volume); and four consecutive linear book-tilt grids (A2 tilt, A3 discount, A4 delta, A5 z-tilt — all ≈ net zero; the idea is retired). The QB side-flag that briefly suggested a promotable QB model resolved NULL under paired re-testing and is quarantined.

7.Roadmap

• The W4 window. Four independent feature families (market, roster, coaching, contract-timing) each show their only early-season lift at Week 4 — the earliest week where offseason context beats trailing lags. A dedicated W4-window blend iteration is the queued candidate, rather than more single-family W1 attempts.
• Rookies. Nobody in the current stack ranks W1 rookies (the book cannot, by construction); draft-capital features are the seed of a rookie-inclusive scoring cut — likely the honest W1 differentiator.
• Stakes and playoffs. Late-season motivation distortion (resting starters, clinched seeds) and the playoffs are recognized regimes without validated models yet — dashed on the regime diagram, deliberately unserved until they earn a seat.
• The E[TD] layer (pending adoption). On fantasy-points truth, adding the calibrated E[TD] term to the TD-free center measures +0.055 (W9–18) / +0.064 (W2–4) / +0.123 (W1) slate Spearman, 4/4 seasons, sign_p ≈ 0, in every scoring variant, placebo-clean — the largest single measured mover available. It is not yet promoted: house rules require an independent adversarial verification plus an ablation separating TD information from the richer feature surface it rides on (part of the lift is generic player-quality signal entering through the TD term). And we explicitly do not claim it beats RotoWire: that comparison reads +0.008, 3/4 seasons, sign_p 0.048, loses 2022, and flips above 0.05 under standard scoring — hint-grade, not receipt-grade, and the 2026-07-12 research memo says exactly that.
• Fantasy-points public scoreboard. Candidate adoption of half-PPR fantasy points as the public accuracy yardstick (the market's own truth), with the internal modeling metric staying TD-free.
• Paid route participation under the §5.3 validate-before-spend gate.