Bayesian interpretation of ridge
Question
With Gaussian observation errors and an isotropic zero-mean Gaussian coefficient prior, explain the connection between MAP estimation and ridge.
Short answer
The negative log likelihood is proportional to squared residual error divided by the error variance. The negative log prior is proportional to the squared coefficient norm divided by the prior variance. Their sum is a ridge objective, with penalty strength equal to error variance divided by prior variance when the loss is an unnormalised residual sum of squares. Different averaging conventions change this numerical relationship.
Original question and answer written by SelfMadeQuant.com.
See the hint and the full worked solution in the question bank (free account).
Open the worked solutionLessons behind this question
- Ridge, Lasso, and Elastic Net
- The Normal Equations: Closed-Form Least Squares
- Conditional Probability, Total Probability, and Bayes' Rule
Start with a free lesson: Normal Distribution is part of a free topic. Create a free account, no card needed, to read it in full with worked solutions.
Read it free