Stable softmax by row

Question

Implement row-wise softmax for a finite nonempty matrix of class logits. Why does subtracting each row's maximum preserve the probabilities?

Short answer

Multiplying every numerator and the denominator by the same exponential factor leaves each ratio unchanged. Subtracting the maximum makes exponentials at most one, preventing overflow.

Original question and answer written by SelfMadeQuant.com.

See the hint and the full worked solution in the question bank (free account).

Open the worked solution

Lessons behind this question

The Supervised Learning Problem and the ML Pipeline
Matrix Algebra: Operations, Rules, and the Transpose
Kernel, Image, and the Rank-Nullity Theorem

Start with a free lesson: Basic Statistics and Regression is part of a free topic. Create a free account, no card needed, to read it in full with worked solutions.

Read it free