Skip to content

Complete separation in logistic regression

Question

A linear boundary perfectly separates all training labels. Why might unregularised logistic coefficients fail to converge to a finite estimate?

Short answer

Along a separating direction, increasing coefficient magnitudes continually improves the likelihood while probabilities approach zero or one. The likelihood supremum can therefore occur only at infinite coefficient norms. Use a regularised fit or an appropriate bias-reduction method and investigate leakage or a small sample. Numerical convergence messages alone do not establish that a finite unregularised maximum exists.

Original question and answer written by SelfMadeQuant.com.

See the hint and the full worked solution in the question bank (free account).

Open the worked solution

Lessons behind this question

Start with a free lesson: Normal Distribution is part of a free topic. Create a free account, no card needed, to read it in full with worked solutions.

Read it free