We run autonomous research loops on every modeling milestone. One file the agent edits, one metric, one fixed budget, git keep-or-revert as the only acceptance gate. Below is the complete log of the first such loop — 17 experiments that tuned the per-player Bayesian shrinkage on QB passing TDs, lifting the composite metric +25.5% over the published baseline. No cherry-picked results. No deleted runs.
Archive — spring 2026 research log. This documents the MAE/shrinkage-era loops. The research program has since moved to the weekly ranking scoreboard and calibrated ranges — current methodology lives on the methodology page, receipts on the 2025 proof page.
The principle generalizes beyond LLM training: single mutable file, single metric, fixed budget, git keep-or-revert. We adopted it for SnapScore modeling lanes. Every milestone with a directed search space — hyperparameter sweeps, feature ablation, ensemble blend tuning — runs through this harness.
Stable infrastructure (feature pipeline, eval harness, metrics) lives in snapcore/. The agent edits exactly one file per experiment — model logic, hyperparameters, structural changes. Diffs are tiny and reviewable.
Single number. No metric-shopping mid-loop. For the M6.2 sweep, the locked metric was composite = recall_pos − 2·|ECE − 0.05| — rewards tail-class recall, penalizes drift from the 5% calibration target.
M6.2: 25-second per-experiment budget (one full walk-forward eval). Total loop budget: 30 experiments. We hit "no improvement for 5 consecutive runs" at experiment 17 and called it.
If composite beats current best → git commit. If not → git checkout HEAD -- experiment.py and revert. No subjective "this seems better." Every commit in the research dir corresponds to a kept experiment; every reverted file goes only into results.tsv.
The M6.2 shipped framework (per-player Dirichlet-multinomial shrinkage on top of XGB) had its v1 hyperparameters chosen by hand: k=8, w=0.5, uniform prior weights. The autoresearch loop tested whether those settings were locally optimal — and if not, what's better.
| Element | Value |
|---|---|
| Hypothesis | v1 settings are too conservative on the cohort prior, killing positive-class recall |
| Single metric | composite = P(argmax≥2 | truth≥2) − 2·|ECE − 0.05| |
| Search space | prior_strength_k ∈ [1, 25], model_trust_w ∈ [0.3, 1.0], optional per-class prior weights |
| Budget | ~25 sec/experiment, 30-experiment loop cap |
| Baseline (target to beat) | v1 composite 0.3189 |
| Min-bar success | composite > 0.3689 (baseline + 0.05) |
| Final winner | exp 15: k=15, w=0.5, prior_weights=[1, 1, 1.3, 4] · composite 0.4002 |
Each dot is one experiment. Gold dots are kept (committed), gray dots are reverted. The faint gold line is the running best.
Same data that lives in snapcore/research/m6_2_shrinkage_sweep/results.tsv in our repo. No deletions.
| # | Experiment | k | w | Prior weights | Composite | Recallpos | ECE | Δ vs prev best | Outcome |
|---|---|---|---|---|---|---|---|---|---|
| 1 | v1-baseline-repro |
8.0 | 0.5 | uniform |
0.3189 | 0.366 | 0.0736 | +0.0000 | KEEP |
| 2 | exp02-pure-xgb |
8.0 | 1.0 | uniform |
0.2627 | 0.3922 | 0.1147 | -0.0562 | revert |
| 3 | exp03-w07-k8 |
8.0 | 0.7 | uniform |
0.3082 | 0.3791 | 0.0854 | -0.0107 | revert |
| 4 | exp04-w03-k8 |
8.0 | 0.3 | uniform |
0.2576 | 0.3464 | 0.0944 | -0.0613 | revert |
| 5 | exp05-w05-k4 |
4.0 | 0.5 | uniform |
0.3061 | 0.366 | 0.08 | -0.0128 | revert |
| 6 | exp06-w05-k15 |
15.0 | 0.5 | uniform |
0.3387 | 0.366 | 0.0636 | +0.0198 | KEEP |
| 7 | exp07-w05-k25 |
25.0 | 0.5 | uniform |
0.3351 | 0.3529 | 0.0589 | -0.0036 | revert |
| 8 | exp08-w05-k18 |
18.0 | 0.5 | uniform |
0.3315 | 0.3529 | 0.0607 | -0.0072 | revert |
| 9 | exp09-w05-k12 |
12.0 | 0.5 | uniform |
0.3321 | 0.366 | 0.067 | -0.0066 | revert |
| 10 | exp10-boost-c3plus |
15.0 | 0.5 | [1.00,1.00,1.00,2.00] |
0.3432 | 0.366 | 0.0614 | +0.0045 | KEEP |
| 11 | exp11-boost-c3plus-3x |
15.0 | 0.5 | [1.00,1.00,1.00,3.00] |
0.3593 | 0.3791 | 0.0599 | +0.0161 | KEEP |
| 12 | exp12-boost-c3plus-4x |
15.0 | 0.5 | [1.00,1.00,1.00,4.00] |
0.3798 | 0.3922 | 0.0562 | +0.0205 | KEEP |
| 13 | exp13-boost-c3plus-5x |
15.0 | 0.5 | [1.00,1.00,1.00,5.00] |
0.3776 | 0.3987 | 0.0606 | -0.0022 | revert |
| 14 | exp14-boost-c2-c3plus |
15.0 | 0.5 | [1.00,1.00,1.50,4.00] |
0.3802 | 0.4052 | 0.0625 | +0.0004 | KEEP |
| 15 | exp15-c2-light |
15.0 | 0.5 | [1.00,1.00,1.30,4.00] |
0.4002 | 0.4052 | 0.0525 | +0.0200 | KEEP |
| 16 | exp16-c2-lighter |
15.0 | 0.5 | [1.00,1.00,1.20,4.00] |
0.3874 | 0.3987 | 0.0556 | -0.0128 | revert |
| 17 | exp17-c0-downweight |
15.0 | 0.5 | [0.70,1.00,1.30,4.00] |
0.3845 | 0.4052 | 0.0603 | -0.0157 | revert |
| Metric | M6.2 v1 (shipped earlier) | M6.2-tuned (after sweep) | Δ |
|---|---|---|---|
| QB passing_tds composite | 0.3189 | 0.4002 | +25.5% |
| QB ECE | 0.0736 | 0.0525 | closer to 0.05 target |
| QB class 3+ recall (multi-TD) | 0.086 | 0.224 | +160% |
| QB Brier | 0.7269 | 0.7318 | small cost, calibration gain |
Production code at snapcore/models/shrinkage.py updated with new defaults
(prior_strength_k=15, model_trust_w=0.5, plus target-aware
prior_class_weights for 4-class TD targets). Three-class TD targets
(RB/WR/TE) got the k improvement but kept uniform prior weights pending
their own sweeps.
| Loop | Goal | Status |
|---|---|---|
| m7_closing_line_validation | Independently validate the "closing line is #1 predictor" claim before our data lane commits to building a closing-line ingest pipeline | 2 of 3 hypotheses answered. H1 (is nflverse's spread_line the closing line?) — yes, ~80% confidence. H2 (is closing line a big predictor in our pipeline?) — surprisingly small: 0.35 yards MAE lift. H3 (does line-movement add signal?) queued. |
| m6.2 RB / WR / TE TD sweeps | Adapt the QB-tuned shrinkage hyperparameters to the 3-class TD targets (RB/WR/TE) which have different class distributions | Queued. Will run when a clear leverage gain is justified — first the closing-line / DvP / injury-severity higher-impact work. |