Skip to content

Early stopping consumes validation data

Question

You stop boosting when validation error is lowest and report that minimum as an unbiased performance estimate. What is the issue?

Short answer

The validation data selected the stopping iteration, so the minimum is optimistically selected. Use that window for stopping but evaluate the resulting training procedure on an independent outer period. Repeat the stopping rule within each historical fold, with preprocessing and any label purging fitted appropriately. Early stopping is a hyperparameter-selection mechanism even if its implementation is automatic.

Original question and answer written by SelfMadeQuant.com.

See the hint and the full worked solution in the question bank (free account).

Open the worked solution

Lessons behind this question

Start with a free lesson: Basic Statistics and Regression is part of a free topic. Create a free account, no card needed, to read it in full with worked solutions.

Read it free