Tuning after repeated holdout checks
Question
You inspect the final holdout after every model change and keep the best result. Is it still a final holdout?
Short answer
No. Its feedback is influencing model development, creating adaptive selection even if the model never directly fits those labels. Treat the inspected period as development data. Freeze a selection procedure and evaluate on genuinely unseen data, or use a prospective paper-trading period. Keep an experiment log so the number and nature of attempted changes remain visible.
Original question and answer written by SelfMadeQuant.com.
See the hint and the full worked solution in the question bank (free account).
Open the worked solution