A backtest that shows a good Sharpe ratio in-sample is almost worthless without validation: the same data used to develop a strategy can be used to fabricate a spurious track record by iterating over model variants until one looks good. In the words of López de Prado : "Every backtest result must be reported in conjunction with all the trials involved in its production." Without the number of trials, it is impossible to assess whether a strategy is genuinely skilled or merely the product of data-mining.
This topic covers the statistical infrastructure for honest backtesting: purged cross-validation to prevent look-ahead leakage, combinatorial purged CV to estimate the distribution of Sharpe ratios across paths, the probabilistic and deflated Sharpe ratios to correct for non-normality and multiple testing, and the practical realities of realistic backtesting including transaction costs, borrow, and survivorship bias.
By the end of this topic you should be able to:
- explain why standard k-fold CV is invalid for financial time series and what purging and embargoing fix;
- implement purged k-fold CV and set appropriate embargo lengths;
- compute the Combinatorial Purged CV (CPCV) backtest and interpret the resulting distribution of Sharpe ratios;
- compute the Probabilistic Sharpe Ratio (PSR) and the Deflated Sharpe Ratio (DSR) and interpret them as false discovery metrics;
- describe the main sources of backtest inflation: transaction costs, survivorship bias, look-ahead bias;
- apply bet sizing to translate a predicted probability into a portfolio weight.
In this lesson
- Contents
- The big picture
The full lesson is for members.
Read the full lessonNew to SelfMadeQuant.com? See membership options or browse the Statistical Arbitrage syllabus.