SGD, Momentum, RMSProp, and Adam as Optimization Choices

The previous lesson named Adam as the default optimiser and gave its update rule, but a quant research pipeline still needs to know why that default exists — and when to override it. Backpropagation supplies a noisy, high-dimensional gradient at every mini-batch; the optimiser decides how much of that gradient to trust, how fast to move, and whether parameters with different curvature histories should share the same step size. This note compares mini-batch SGD, momentum, RMSProp, and Adam as distinct answers to those questions, so an experiment log can record an optimiser choice as a deliberate modelling decision rather than a framework default.