Why backtests lie
View details →
Backtesting & Strategy Validation
A strategy that looks perfect in backtest often fails in production. The difference is usually bias—data the model wouldn't have had, or assumptions that don't hold. Proper validation catches it.
- Backtest = run your strategy on historical data. But the past isn't the future.
- Survivorship bias: testing on data that only includes survivors (e.g. stocks that still exist) inflates returns.
- Lookahead bias: using information you wouldn't have had at decision time. Fatal.
- Purged K-fold: time-aware cross-validation. Train on past, test on future. No data leakage.
- Cost-aware: include fees, slippage, and execution reality. Paper returns ≠ live returns.
Real-world example
A trading strategy shows 40% annual returns in backtest
You ran it on 5 years of price data. It looks amazing. You deploy with real money.
- Survivorship bias: Your dataset only had stocks that survived. Bankrupt companies were excluded. Real universe would have included losers. Returns drop 10–15%.
- Lookahead: Your "signal" used that day's closing price—but you'd trade at open. You had information you wouldn't have had. Remove it: returns collapse.
- Costs: Backtest ignored commissions, slippage, and spread. Add 0.1% per trade × 200 trades/year = 20% drag. Returns halve.
- Purged K-fold: Standard cross-validation shuffled data—future leaked into past. Time-aware split: train 2019–2021, test 2022. Out-of-sample performance much worse.
- The "40% strategy" might be 8% after proper validation. Or negative.
Backtest results are hypotheses. Validation turns them into defensible estimates.
Purged K-fold and time-aware validation
View details →
What good validation looks like
View details →
Common mistakes
View details →
Building a strategy that needs proper validation?