The short answer
A probability model is well calibrated when its predictions match reality: of all the cases it gives a 30% chance, about 30% happen. AUC measures how well it ranks cases; the Brier score and expected calibration error measure how accurate the probabilities themselves are.
At a glance
- Term
- Model calibration (AUC, Brier score)
- Also known as
- calibration · AUC · Brier score · expected calibration error
- Category
- Method & analysis
- Where it matters
- Company and founder due diligence
Model calibration in practice
AUC ranges from 0.5 (no better than chance) to 1.0 (perfect ranking). The Brier score is the mean squared error of the probabilities; Brier skill compares it with always predicting the base rate. Expected calibration error is the average gap between predicted and observed rates.
Why model calibration matters
- Good ranking alone is not enough if you act on the probability itself.
- Showing the base rate next to each number keeps it in perspective.
Model calibration in MarketPrior
MarketPrior shows each forecast with an 80% interval and the historical base rate, and publishes its calibration only as out-of-time backtest results (for follow-on: AUC 0.83, Brier skill 0.20, calibration error about 0.01). Companies outside the population the model was trained on get no forecast rather than a guess.