Training Deep Networks: Initialization, Batch Normalization, Dropout, and Adam

Exercise 4 of the derivation lesson identified the mechanism: as the backward pass's delta is propagated through more layers, its magnitude tends to shrink (or, in less common cases, grow) multiplicatively at every step. In a genuinely deep network, this compounds into the vanishing/exploding gradients problem, which stalled the field for years. This note is Topic 6's practical training toolkit survey — the four techniques that made deep networks trainable: careful weight initialization, batch normalization, dropout regularisation, and the Adam optimiser. Dedicated optimiser comparisons, regularisation discipline, and loss-curve diagnosis continue in Training Deep Networks.