Skip to content
Statistical Arbitrage / Backtesting and Validation

Combinatorial Purged Cross-Validation

Purged kk-fold cross-validation estimates the out-of-sample Sharpe ratio from a single backtest path — one realisation of the strategy's performance over the evaluation period. A single Sharpe ratio number has high variance: a strategy that genuinely earns SR = 0.6 might backtest anywhere from SR = −0.5 to SR = 1.8 on a 3-year sample due to sampling noise. Choosing a strategy by its single-path Sharpe ratio in the face of this variance almost guarantees selecting an over-performer, leading to disappointed OOS performance.
Combinatorial Purged Cross-Validation (CPCV) solves this by generating a large number of test paths through the combinatorial structure of all ways to partition NN blocks of data into kk test folds. Rather than a single SR estimate, CPCV delivers a distribution of Sharpe ratios across ϕ(N,k)\phi(N, k) paths — a much more reliable summary of strategy quality. It also enables the Probability of Backtest Overfitting (PBO) and defeats selective disclosure.
Learning objectives. By the end of this note you should be able to:
  • state the CPCV construction procedure and identify the role of NN and kk;
  • compute the number of paths ϕ(N,k)=(Nk)k/N\phi(N, k) = \binom{N}{k}k/N;
  • explain why CPCV has lower variance than WF or simple purged CV;
  • compute the PBO from the CPCV path distribution;
  • describe how CPCV defeats selective disclosure of backtest results.

In this lesson

    1. The single-path variance problem
    1. CPCV construction
    1. Variance reduction
    1. Probability of Backtest Overfitting
    1. Defeating selective disclosure

Before you start

The full lesson, with worked examples and exercise solutions, is for members.

Read the full lesson

New to SelfMadeQuant.com? See membership options or browse the Statistical Arbitrage syllabus.