Skip to content
Statistical Arbitrage / Backtesting and Validation

The Overfitting Trap: Multiple Testing and the Probability of False Discovery

Backtest overfitting is the central methodological failure in quantitative finance. A researcher who runs 20 backtests on the same dataset and selects the best result will almost always find one that achieves a 5% significance threshold — not because the strategy is genuine, but because at the 5% false positive rate, one in 20 pure noise strategies clears the bar. Every experienced quant has encountered this trap; few hedge funds have institutional procedures to avoid it.

This note explains the mechanics of false discovery under multiple testing, formalises the probability of backtest overfitting, and introduces Bailey and López de Prado's third law of backtesting as the governing principle for honest strategy research. Understanding why repeated backtests are methodologically equivalent to data-mining is prerequisite to applying the corrected statistical tools (PSR, DSR, CPCV) in the rest of this topic.
Learning objectives. By the end of this note you should be able to:
  • explain why a single out-of-sample backtest on a dataset used for model selection is still biased;
  • state the expected number of false discoveries at a 5% significance level as a function of the number of trials;
  • compute the minimum Sharpe ratio required to achieve a given false discovery rate given NN trials;
  • state and apply Bailey-López de Prado's third law of backtesting;
  • identify the seven sins of backtesting.

In this lesson

    1. The mechanics of data-mining
    1. The probability of backtest overfitting
    1. The minimum required Sharpe ratio
    1. The third law of backtesting
    1. The seven sins of backtesting

Before you start

The full lesson, with worked examples and exercise solutions, is for members.

Read the full lesson

New to SelfMadeQuant.com? See membership options or browse the Statistical Arbitrage syllabus.