Backpropagation as the Chain Rule

An MLP with several hidden layers has potentially millions of parameters spread across every layer's weight matrix and bias vector. Gradient descent needs the gradient of the loss with respect to every one of them. This note explains backpropagation: not a new optimisation algorithm, but an efficient way of computing that entire gradient — for every parameter, in every layer — using exactly two passes through the network, by systematically applying the multivariate chain rule from the output layer backward to the input.