SGD, momentum, RMSProp, Adam — what each one fixes over plain gradient descent
Backprop only computes gradients — the optimizer decides how to use them to update weights. Picking the wrong optimizer or learning rate is the single most common reason a model "doesn't train," and understanding the failure modes each optimizer was designed to fix matters more than knowing their names.
Plain SGD takes a step directly opposite the current gradient — like walking downhill by only looking at the slope under your feet, which oscillates badly in narrow valleys. Momentum adds inertia so you keep rolling in a consistent direction instead of zig-zagging. RMSProp gives each parameter its own adaptive step size based on how large its recent gradients have been. Adam combines both ideas: it remembers direction (momentum) and scales per-parameter (adaptive rate).
This matters when different parameters need very different effective learning rates — common in deep nets where early and late layers see very different gradient scales.
Adam is the default choice for most deep learning because it needs little tuning and adapts per-parameter — but plain SGD with momentum still generalizes better for some vision tasks. Neither is strictly better; the tradeoff is "Adam converges faster, SGD+momentum sometimes generalizes better."
Warmup exists because early in training, weights are random and gradients are large/noisy — a full-size learning rate can cause an unstable early update the model never recovers from, especially with Adam's adaptive denominator or in transformers with layer norm. Decay later in training lets the model settle into a sharper minimum instead of oscillating around it.