Skip to content

How to Test Whether a Trading Feature Adds Predictive Information

Original question

How do I identify which features genuinely contribute predictive information rather than simply fitting historical market noise?
Start with a small, credible baseline and run out-of-sample ablations. Refit the same model on each chronological fold with and without a feature or related feature group. Compare performance on the same dates, check that the gain recurs across folds, and track how the feature changes turnover and net returns if the forecast drives trades.

Ask what the feature adds to a fixed baseline

Fix the target and baseline features first. Define related families such as momentum at several horizons, volatility, volume and fundamentals. For fold kk, fit both the baseline and the candidate model independently using only the eligible training observations. With loss L\mathcal L, measure
Δk=L(yk,y^kbase)−L(yk,y^kcandidate).\Delta_k=\mathcal L(y_k,\widehat y_k^{\mathrm{base}})- \mathcal L(y_k,\widehat y_k^{\mathrm{candidate}}).

Positive values indicate lower error with the candidate. Preserve the same data eligibility and test rows in both models. If removing an input also removes a missing-data constraint, quantify the benefit from more coverage separately. Fit imputation and selection within each fold.

Inspect differences by date as well as aggregate folds. In a stock panel, same-date observations can share market shocks, so a confidence interval built by treating each stock-day as independent can be misleading. When estimating uncertainty, group by date and consider resampling consecutive blocks of dates. Use the final holdout after settling on the feature set.
You compare a momentum-only ridge model with one that also uses a volatility family. Across four walk-forward blocks, test mean squared error improves by 1.1%, 0.6%, 0.8% and declines by 0.2%. A version with two nearly identical trailing-volatility features gives the same results as either individually. Both may represent the same information, so you could test the cheaper of the two on later data. If the model trades more frequently, also calculate whether the modest forecast gain survives higher turnover.

Now consider an earnings surprise feature that looks important during training but contributes only in one held-out week. Inspect what dates supplied the gain. If a single data revision or one stock accounts for it, the next check is its availability and behaviour across independent earnings events.

A feature family can help even if none of its individual members gets a high importance score. Nearby momentum windows share information, and a tree may assign most of the split credit to whichever one it happened to choose first. Compare a model with the whole family against one without it on the same folds. If the gain is mostly in one market episode, look at the prediction errors by date. This is more informative for retention than a ranking produced by the full fitted sample.

Compare errors in paired observations. For each validation date, compute baseline loss minus candidate loss on the same assets and horizon. A positive average says the candidate improved that score on those dates. Examine the distribution of paired differences and account for dependence if labels overlap. A result driven by one day should lead you back to that day's feature availability and outcomes. For a trading decision, repeat the paired comparison on net performance with the same execution rules, since the feature can improve forecasts without changing feasible trades.

References and next steps