The best of a hundred unprofitable strategies can have an attractive historical Sharpe ratio. Correct accounting and purged validation do not remove that selection effect. This lesson treats the research search itself as evidence: what was tried, how the winner was selected and which observations remain available for confirmation.
Learning objectives. By the end of this note you should be able to:
- calculate a simple familywise false-positive probability;
- identify hidden trials beyond model hyperparameters;
- specify a reproducible trial register and selection rule;
- distinguish multiplicity adjustments from a guarantee of economic validity;
- design a final assessment that preserves the meaning of the held-out result.
In this lesson
- Why the winner looks better than the population
- The search is wider than the model
- A trial register that another researcher can read
- Confirmation is an operational boundary
- Deflated Sharpe and overfitting diagnostics
- Topic assessment: design before results
Before you start
The full lesson, with worked examples and exercise solutions, is for members.
Read the full lessonNew to SelfMadeQuant.com? See membership options or browse the Quantitative Research, Data and Backtesting syllabus.