Skip to content

How to Audit a Backtest for Look-Ahead and Survivorship Bias

Original question

How do I design a backtesting framework that avoids look-ahead bias, survivorship bias, and unrealistic transaction cost assumptions?

Make each decision from a point-in-time snapshot, including the securities eligible on that date. Give orders a separate execution event, apply explicit fill and cost rules, and persist enough information to replay each simulated trade. Test the simulator on small cases where you already know the expected holdings, fills, cash and P&L.

What information could the strategy see?

  1. Store both an observation timestamp and an availability timestamp. For a quarterly report, the fiscal period belongs to the observation timestamp and the release time belongs to availability. Preserve vintages where the strategy uses revised series.
  2. Build a dated security master with identifiers, trading eligibility, corporate actions and delistings. Use dated membership when trading a historical index universe, and model delisting proceeds where the data supports them.
  3. Separate the portfolio target, orders and fills. Given a desired position, account for pending orders and allowed trading hours. Record fill size, execution price and fee separately. Decide how the simulator treats unfilled orders and halts.
  4. Reconcile cash, holdings and equity after every event. Keep an order-to-fill trail and compare gross P&L with net P&L. Stress costs at higher spreads and lower liquidity.

For a signed fill of qq shares at price PP, cash changes by −qP−fee-qP-\text{fee}. If the order crosses a spread, the simulated fill price should reflect that. For strategies with short exposure, apply borrow costs and include borrow availability in the eligible universe.

A company releases earnings on Tuesday at 16:15. Its fiscal period ended weeks earlier, and Tuesday's regular market close was 16:00. A regular-session strategy that reads the filing on Tuesday evening first gets an opportunity to trade at Wednesday's open. Suppose that open has a $20.00 bid and a $20.04 ask. A 500-share immediate buy at the ask, assuming enough liquidity, costs $10,020 before fees. A $1 commission raises the cash debit to $10,021. A model that buys 500 shares at the $20.02 midpoint omits $10 of spread cost.

Write this case as a fixture. Assert that the simulator creates zero Tuesday fills, records 500 Wednesday shares and changes cash by $10,021. Add a version with a halted Wednesday session to confirm that the order stays pending or expires according to your policy.

Make the replay falsifiable. Add one filing published after the decision timestamp and assert that it does not change the earlier signal. Repeat with a company that later delisted and check that it remains eligible on historical dates when it actually belonged to the universe. These tests cover two distinct leaks. Even a perfect as-of join cannot bring a missing delisted company back into a universe constructed from current members. Keep both the dated membership and the dated inputs.

Use the order book tool to see how price and quantity affect execution when an order crosses the spread.

Give the simulation a clear ordering when two events share a recorded timestamp. A report timestamp rounded to the minute and a decision taken at the start of that minute may look simultaneous even though the report had not arrived. Use an explicit ingestion timestamp or a conservative availability delay in that case. Keep a decision log with both the source record and its availability time. It should let you explain why an observation was admitted without assuming that anything on the same calendar day was known at the open.

References and next steps