Skip to content

How to Test a Trading Edge for Overfitting

Original question

How can I tell whether my trading strategy has discovered a genuine market inefficiency or is simply overfitted?
Treat the result as evidence of a trading hypothesis. Look for a mechanism that predicts when the signal should work, repeat the evaluation on data untouched by the search, and test whether the result survives costs, related specifications and realistic sizing. A profitable backtest alone cannot identify an inefficiency, especially after many attempted signals.

Ask the discovery to predict something else

  1. Record the hypothesis and predictions before the next evaluation. List the tested variants and choose a benchmark with relevant risk exposures.
  2. Recreate the historical opportunity set. Use dated inputs, feasible fills, costs, borrow and financing. Inspect the trades responsible for most gains.
  3. Assess candidate selection with forward validation and a holdout. Use search-aware evidence such as deflated Sharpe where its assumptions are appropriate. If estimating uncertainty by resampling, preserve temporal dependence and consider whether candidate selection needs to be repeated inside the procedure.
  4. Check adjacent windows, periods, assets and cost levels. Make the interpretation conditional on the mechanism. An event-driven rule need not work in assets where that event never happens.
  5. Replicate against a new date range, independent data source or shadow execution. Estimate returns with conservative costs and report uncertainty alongside point estimates.

What a narrow parameter peak is telling you

You test 80 signals and find that a 19-day reversal rule earns net Sharpe 1.4 in development. Its neighbours at 17, 18, 20 and 21 days earn 0.1 to 0.3. Nearly all of its P&L comes from one month and two stocks. A later untouched sample yields net Sharpe 0.2. That pattern gives weak evidence for the exact lookback and calls for a closer inspection of the contributing trades.

Now suppose a preregistered event rule earns smaller but positive out-of-sample returns in two independent universes. Returns are concentrated where liquidity pressure is highest, and disappear after the event window as predicted. That is more consistent with the proposed mechanism. You would still check how much exposure to market, size, volatility and other known risks explains the result.

A mechanism should imply evidence besides a positive backtest. A short-horizon reversal hypothesis tied to temporary price pressure might predict a stronger response after large liquidity shocks and a decay once orders normalise. Decide how to identify those events before checking the next sample. A nearby parameter that fails can matter for diagnosis, but choosing a new window after seeing that failure is another trial. Keep a log of the search so the final assessment reflects what actually happened.

You can inspect the rules and code in Strategy Lab when choosing a baseline for the next evaluation.

References and next steps