A complex model loses its advantage
Question
Boosting beats ridge in validation but loses in the next year. How would you investigate without choosing a winner retrospectively?
Short answer
Use the next year as diagnostic evidence, not a new tuning leaderboard for the same performance claim. Inspect regime shifts, feature availability, concentration, costs, calibration and sensitivity to complexity. Compare the predeclared training procedures across additional chronological folds and reserve a new outer evaluation. Simpler-model robustness is a testable explanation, not something established by one failed year.
Original question and answer written by SelfMadeQuant.com.
See the hint and the full worked solution in the question bank (free account).
Open the worked solution