Feature screening before validation

You select the 20 signals most correlated with returns over the entire dataset, then cross-validate a model using those signals. Why is the result optimistic?

Feature selection has already used validation labels, so the subsequent cross-validation does not assess the full selection process independently. Repeat screening within each training fold, then score on its untouched validation observations. Apply the same discipline to transformations, missingness filters influenced by labels and dimensionality choices. A final held-out period remains valuable after repeated development experiments.