Early stopping consumes validation data
Question
You stop boosting when validation error is lowest and report that minimum as an unbiased performance estimate. What is the issue?
Short answer
The validation data selected the stopping iteration, so the minimum is optimistically selected. Use that window for stopping but evaluate the resulting training procedure on an independent outer period. Repeat the stopping rule within each historical fold, with preprocessing and any label purging fitted appropriately. Early stopping is a hyperparameter-selection mechanism even if its implementation is automatic.
Original question and answer written by SelfMadeQuant.com.
See the hint and the full worked solution in the question bank (free account).
Open the worked solutionLessons behind this question
- Finance-aware validation
- Cross-Validation and Information Criteria
- Maximum Likelihood Estimation: Score and Log-Likelihood
Start with a free lesson: Basic Statistics and Regression is part of a free topic. Create a free account, no card needed, to read it in full with worked solutions.
Read it free