A backtest cannot be reproduced
Question
Two researchers run the same model code and obtain different backtest results. What would you compare before attributing the difference to randomness?
Short answer
Compare dataset versions, point-in-time snapshots, universe definitions, time zones, corporate actions, preprocessing fits, split boundaries, label horizons, execution assumptions and software versions. Then inspect RNG seeds and hardware nondeterminism. Record a reproducible experiment manifest and deterministic reference examples. Matching code and a shared seed are insufficient if the underlying information set or execution policy differs.
Original question and answer written by SelfMadeQuant.com.
See the hint and the full worked solution in the question bank (free account).
Open the worked solution