Model Evaluation

Track Record

Every prediction below was graded against the real result — including backtested rounds, which were generated blind, using only data available before that race, never the race's own outcome. This is what separates a model from a claim about a model.

Graded Races

Calibration

Every driver-race win% prediction across every graded race, bucketed by predicted win% and plotted against how often drivers in that bucket actually won. The dashed line is perfect calibration — a point on it means when the model said "X%," it was right about X% of the time. Points above the line mean the model was underconfident in that range; below means overconfident. Bubble size shows how many predictions fed that point — small bubbles are a handful of races and should be read as noisy, not as a verdict.

Calibration matters more than raw accuracy for a model that reports a probability rather than a single guess: a model can pick the wrong winner every time and still be well-calibrated (if the driver it gave 20% to wins about 1 time in 5), and it can pick the right winner often while being badly calibrated (if everything it's confident about is systematically over- or under-stated). Accuracy asks "did it guess right" — calibration asks "can you trust the number it gave you."

How this is scored

Winner hit rate — how often the driver the model gave the highest win probability actually won. Brier score — mean squared error between each driver's predicted win probability and whether they actually won (0 = perfect, lower is better; a model that always predicts uniformly across ~20 drivers scores much worse than one that concentrates probability sensibly). Podium hits — how many of the model's predicted top 3 actually finished top 3. "Backtest" rounds were generated after the fact but deliberately blind — restricted to only the data that would have existed before that race — so they count as a fair test of the model, not hindsight.