Complete separation in logistic regression
Question
A linear boundary perfectly separates all training labels. Why might unregularised logistic coefficients fail to converge to a finite estimate?
Short answer
Along a separating direction, increasing coefficient magnitudes continually improves the likelihood while probabilities approach zero or one. The likelihood supremum can therefore occur only at infinite coefficient norms. Use a regularised fit or an appropriate bias-reduction method and investigate leakage or a small sample. Numerical convergence messages alone do not establish that a finite unregularised maximum exists.
Original question and answer written by SelfMadeQuant.com.
See the hint and the full worked solution in the question bank (free account).
Open the worked solutionLessons behind this question
- Ridge, Lasso, and Elastic Net
- The Normal Equations: Closed-Form Least Squares
- Simple Linear Regression: OLS Estimators and Geometry
Start with a free lesson: Normal Distribution is part of a free topic. Create a free account, no card needed, to read it in full with worked solutions.
Read it free