In-sample performance uses observations that helped fit or select a model. Out-of-sample performance uses observations that were unseen by the corresponding fit and selection decisions. For a historical simulation of deployment, split chronologically, tune on forward validation folds, and reserve the latest period for a final evaluation. Exclude training labels that have yet to finish at each fit date.
Assign each block one job
Define the timing first. If you compute features at a close, enter at the next open, and hold five sessions, identify the exact open at which each label ends. Then apply three chronological roles:
- Fit model parameters, feature scalers and imputers on the training period.
- Choose specifications on one or more later validation periods. With many dates, use walk-forward splits for more than one assessment.
- Once the choices are fixed, fit the final model using the eligible training and validation history. Evaluate on the untouched test block.
Repeat any fitting transformations inside each fold. Trailing feature windows for the first test row may use earlier observations already known at that date. Forecast outcomes are a separate matter. A training label that reaches into the test period must be omitted from the fit at that boundary.
A five-day label at a year boundary
Imagine daily data from 2016 through 2025. Use 2016–2020 to fit, 2021–2023 for model selection with yearly walk-forward folds, and 2024–2025 for the final evaluation. The strategy predicts the next five close-to-close returns. If the final model is fitted just after the last 2023 close, rows whose five-session outcomes extend into 2024 have yet to resolve and must be omitted.
Suppose a linear model earns a net validation Sharpe of 0.6 and a tree model earns 0.8. You choose the tree before opening the final holdout. If the tree earns 0.3 there, report 0.3 as the final test result. Switching to the linear model because it did better in 2024–2025 turns those years into a second validation block.
A single test period covers only the regimes it happened to contain. Show calendar subperiod results and the number of trades, alongside the full-period metric, to help distinguish an isolated episode from broad performance.
The clean split is on information availability, not just the date stamped on a feature row. For instance, if a 29 December observation predicts returns through 5 January, it cannot be a training label for a model first used on 2 January. Drop the boundary row from training or choose a later cutoff. If you adjust model selection after reading the final score, that sample has become part of development. Keep the old result and define a new evaluation period for the changed procedure.
References and next steps
- Scikit-learn TimeSeriesSplit documentation documents chronological folds, an optional gap, and constraints on comparing fold metrics.
- How to Run Walk-Forward Optimisation Without Peeking