Backtesting and Multiple Comparison Bias [/ˈbakktestɪŋ ənd ˈməltəpəl kəmˈpɛrəsən ˈbaɪəs/] n - Backtesting is the nearest thing quantitative trading possesses to a flight simulator: it records how a strategy would have fared in the past, given the assumptions you chose to make. A backtest that has been optimized, however, is no simulator at all. It is a tale you composed to feel well about the future.
Multiple comparison bias is the heart of the matter. Test one hundred strategies and five will show statistical significance at the five percent level by chance alone. Report only the winners, and the backtest becomes a p-hacked advertisement. This is why most published strategies perish in live trading. The selection process contrives a survivorship bias that the final report never shows:
Multiple comparison p-values
The reported idea looks singular. It is not. It won a contest of coin flips.
Of the Defenses
The chief defense is pre-registration. Fix the hypothesis, the features, the target, the rules, and the evaluation metric before you look upon the data; then run the test once. If the result is not significant, go on to other work. The moment you begin adjusting parameters to rescue a Sharpe ratio, you are fitting to noise.
Out-of-sample testing is the least that may be asked: reserve a period the model never sees during development. Walk-forward testing is stronger still, re-estimating and re-evaluating through time. See Cross-Validation in Finance for purged, embargoed, and combinatorial methods that remove data near events and ward against information leakage. Each layer makes the backtest harder to deceive.
Transaction cost assumptions must be conservative. A backtest that presumes midprice fills and no market impact is not wrong; it is irrelevant. Use realistic spread estimates, slippage models, and commission schedules. A strategy that scarcely works with perfect fills will not work in production. See Market Microstructure and Execution for why fills are never perfect.
Data snooping creeps in by subtle doors: choosing a universe on past performance, using adjusted prices before the adjustment date, selecting features from the full sample. Each leaks the future into the past. The only safeguard is discipline about timestamps.
Report the distribution of backtests, not merely the best. If you tried twenty variations and the finest Sharpe is 1.5, the true expected Sharpe is likely lower. The False Discovery Rate and White’s Reality Check adjust for the search itself; they are more honest than a single t-statistic.
What a Clean Test Proves
A clean backtest is a necessary condition, not a sufficient one. Markets change, and a strategy that passed every trial may yet fail because the world turned. The backtest proves the model understood the past. Live trading proves whether it understands the future.