Shrinkage chosen using tomorrow's data

Question

You choose the ridge penalty that minimises error on next quarter's test set and report that same error. What is wrong?

Short answer

The test set has been used for model selection, so its reported error is no longer an untouched assessment of the selected procedure. Select the penalty inside historical training and validation windows, then evaluate once on an outer holdout or use nested chronological folds. If the quarter has already influenced decisions, treat it as development data and obtain a new evaluation period.

Original question and answer written by SelfMadeQuant.com.

See the hint and the full worked solution in the question bank (free account).

Open the worked solution

Lessons behind this question

Ridge, Lasso, and Elastic Net
The Normal Equations: Closed-Form Least Squares
Counting and Combinatorics

Start with a free lesson: Normal Distribution is part of a free topic. Create a free account, no card needed, to read it in full with worked solutions.

Read it free