Training Deep Networks: Initialization, Batch Normalization, Dropout, and Adam

Exercise 4 of the derivation lesson identified the mechanism: as the backward pass's delta is propagated through more layers, its magnitude tends to shrink (or, in less common cases, grow) multiplicatively at every step. In a genuinely deep network, this compounds into the vanishing/exploding gradients problem, which stalled the field for years. This note surveys the four practical techniques that made training truly deep networks reliable: careful weight initialization, batch normalization, dropout regularisation, and the Adam optimiser.