Bias and variance under regularisation
Question
Why can a regularised model with worse training error outperform unrestricted least squares on future observations?
Short answer
Shrinkage restricts how strongly coefficients respond to the training sample. It generally introduces estimation bias while reducing variance, potentially lowering expected prediction error. The balance depends on signal strength, noise, feature dependence and sample size. Choose regularisation using out-of-sample validation rather than training error. An observed improvement in one window does not guarantee a permanent advantage.
Original question and answer written by SelfMadeQuant.com.
See the hint and the full worked solution in the question bank (free account).
Open the worked solutionLessons behind this question
- Ridge, Lasso, and Elastic Net
- The Normal Equations: Closed-Form Least Squares
- Covariance and Correlation
Start with a free lesson: Expectation, Variance and Covariance is part of a free topic. Create a free account, no card needed, to read it in full with worked solutions.
Read it free