Time-series cross-validation for trading models: walk-forward splits, purging and baselines

A random train/test split can answer whether an estimator interpolates among rows from a pooled dataset. It does not automatically answer whether a trading process trained in the past would work in the future. Financial observations may share market events, overlapping labels and a changing relationship between features and outcomes.

This article builds an expanding-window experiment with three validation blocks and one untouched final block. It also explains where a fixed gap helps, where purging requires actual event times, and why repeatedly choosing the best result consumes evidence.

How does walk-forward validation work?

In an expanding-window design, each successive training set includes more eligible history. The validation set stays later in time. A rolling-window design instead limits the training history to a chosen recent span. That span is a modelling choice, and selecting it is part of the research search.

Our data have 1,800 synthetic return intervals indexed from 0 through 1,799. Feature construction uses a twenty-one-session warm-up. Predictions are submitted before open t; target return y_t finishes at open t+1. A training label is eligible only if it finished strictly before the current model-fitting boundary.

The first three rows choose settings. The fourth is reserved for one final evaluation after that choice. Each fitted model stays frozen within its evaluation block, while feature rows can incorporate newly available observations. Observing yesterday's completed return is different from refitting parameters using tomorrow's target.

What is purging, and when is a gap insufficient?

Suppose a label at decision 897 measures a five-session outcome ending at 902. A validation block starts before open 900. That training outcome is still incomplete when validation begins, so it cannot be used for a model fitted at that boundary.

Under the strict pre-open convention, even a label ending exactly at 900 is unavailable to that pre-open fit. The eligibility rule is:

For fixed one-session targets on a regular grid, this becomes a simple excluded row. For event labels with variable holding periods, inspect each event's actual interval and information-release time. Dropping an arbitrary fixed number of rows may leave an unfinished training label in place.

Purging removes training observations whose label intervals conflict with the evaluation information boundary. Embargo rules are often discussed for designs where training can occur after an evaluation block. Our strictly forward-only experiment never trains on later observations, so it does not add a generic post-validation embargo and claim that this solves every dependence problem.

A twenty-session lookback is also not automatically a twenty-session gap requirement. A feature at the start of validation can legitimately use earlier observations. Sharing historical inputs creates dependence that may matter for uncertainty estimates; it is not inherently future information. Inspect the label interval and the feature's availability rather than equating every overlap with leakage.

Where should scaling and feature selection happen?

Inside each fold, fit data-dependent transformations on that fold's training rows. Apply the learned transformation to its validation rows. The next fold gets its own fitted transformation.

For a ridge regression, our lab estimates the training means and standard deviations, standardises the training matrix, fits coefficients, and applies those same means and scales to validation. The validation distribution is allowed to look different. Replacing it with an all-data scale would change what the model knew at fitting time.

The same rule applies to imputation, feature selection, PCA and thresholds learned from quantiles. A deterministic feature definition and a fitted transformation are different operations; document both.

How do you select a financial ML model fairly?

The experiment declares four ridge penalties and four shallow-tree configurations in advance. It selects one configuration per family using the mean validation mean-squared error across the three equal-size blocks. Zero and training-mean forecasts remain fixed reference baselines.

For predicted return f(x) and target y, the evaluation loss is:

Equal-size folds make the arithmetic mean of their MSEs equal to the pooled per-observation MSE. With unequal blocks, decide whether the intended weighting is by observations, duration or another research objective. An unexplained average can change which setting wins.

The selected ridge penalty is 0.1 under the lab's mean-squared-loss objective. The selected tree has maximum depth three and a minimum leaf size of forty. These are results of this synthetic experiment, not recommended default settings for financial datasets.

We then refit those chosen configurations on all 1,178 eligible feature rows before the final boundary. The immediately preceding unfinished label remains excluded. Settings and coefficients stay fixed during the 600 final decisions.

Why does another test consume the holdout?

Suppose the final ridge result disappoints, so you try more penalties, change the signal definition and inspect the same final block again. That block has now influenced the revised research design. Calling the second result “out of sample” does not undo the information flow.

For uncertainty, ordinary independent-observation formulas may be inappropriate when errors or strategy returns are serially dependent. Block-based resampling or other time-series methods require their own assumptions. Our single synthetic path does not report a significance claim or a probability that the strategy will succeed.

Common questions

References

Until then, peace.