NexForay

Gradient Descent & optimizers

The algorithm that trains every neural network

Why

This is the training algorithm. Everything before this was building up to it. Gradient Descent is how weights get updated, how models improve over epochs, how a random initialization becomes a useful model — including why Adam beats vanilla SGD in most cases.

Intuition

Imagine you're blindfolded on a hilly landscape and want to reach the lowest point. You can only feel the slope under your feet. Strategy: take a small step in the downhill direction, feel the slope again, take another step. Repeat. That's gradient descent. The learning rate is your step size — too large and you overshoot valleys; too small and you take forever.

Explanation
The update rule

At each step, every parameter moves a small amount opposite to its gradient:

w ← w − η · ∂L/∂w where: w = current parameter value η = learning rate (step size, e.g. 0.001) ∂L/∂w = gradient of loss w.r.t. this parameter

If ∂L/∂w is positive (increasing w increases loss) → subtract → decrease w → reduce loss. The sign always works out correctly.

Three variants — batch, mini-batch, stochastic
Batch GD: compute gradient on ALL training data → one update accurate but slow, can't fit large datasets in memory Stochastic GD: compute gradient on ONE random sample → update (SGD) fast, noisy, can escape local minima, poor GPU util Mini-batch GD: compute gradient on a BATCH (e.g. 32 or 256 samples) balance of accuracy and speed — standard in practice "SGD" in frameworks usually means mini-batch SGD

Smaller batch = noisier gradients (can help generalization). Larger batch = more stable, better GPU utilization, may converge to sharp minima that generalize worse.

Learning rate — the most important hyperparameter
Too high (η too large)
Steps overshoot the minimum. Loss oscillates or diverges. NaN loss is often caused by exploding gradients amplified by a high learning rate.
Too low (η too small)
Training is very slow. Risk of getting stuck in a poor local minimum or plateau. Takes too many epochs to converge.

Common practice: use a learning rate scheduler — start high, decay over time (step decay, cosine annealing, warmup + decay). Warmup is critical for Transformers.

Adam optimizer — why it's the default

Vanilla SGD uses the same learning rate for every parameter. Adam adapts the learning rate per parameter using running estimates of the gradient's mean and variance:

m_t = β₁·m_{t-1} + (1-β₁)·g_t ← 1st moment: running mean v_t = β₂·v_{t-1} + (1-β₂)·g_t² ← 2nd moment: running mean of grad² w ← w − η · m̂_t / (√v̂_t + ε) m̂_t, v̂_t are bias-corrected estimates β₁ = 0.9, β₂ = 0.999, ε = 1e-8 (typical defaults)
  • Parameters with consistently large gradients get a smaller effective LR (denominator grows)
  • Parameters with small or noisy gradients get a relatively larger effective LR
  • This adaptive behavior makes Adam robust to different parameter scales and sparse gradients
  • AdamW adds weight decay directly to weights (not via gradient) — fixes L2 regularization behavior and is preferred for Transformers
← Previous Next →