The short answer
An out-of-time backtest evaluates a predictive model only on periods after the data it was trained on, the way it would be used in real life. A rolling version retrains and tests year by year, so every score reflects a genuine prediction of the future.
At a glance
- Term
- Out-of-time backtest
- Also known as
- rolling backtest · temporal validation
- Category
- Method & analysis
- Where it matters
- Company and founder due diligence
Out-of-time backtest in practice
Random train/test splits leak information from the future into training and flatter a model. Testing out of time removes that leak and gives a more honest estimate of real-world performance.
Why out-of-time backtest matters
- It is the minimum standard for trusting any forecast number.
- It exposes drift: models that worked in one period and not the next.
Out-of-time backtest in MarketPrior
MarketPrior’s outcome forecast — the probability of a new Form D within 18 months, and of no further Form D within 24 months — is evaluated on a rolling yearly out-of-time backtest over 2015–2024 using the complete public SEC Form D data sets, about 301,000 held-out US company-years. Backtest results: AUC 0.83 for follow-on, Brier skill 0.20, calibration error about 0.01 and 3.3× lift in the top decile. On startup-like companies alone the AUC is lower (0.73).
The outcome is a Form D filing, not company success: a company can raise without filing a Form D. Backtest figures are not guarantees.