Reading loss curves and tuning the knobs that decide whether a correctly-wired network actually trains well
A model can have the right architecture and a correct loss function and still train badly — too high a learning rate diverges, too low wastes compute crawling toward a minimum, and the wrong batch size changes both gradient noise and wall-clock time. Training dynamics is the practical layer between "the math is right" and "the model actually converges," and it's where most day-to-day debugging time goes. Interviewers use this to separate candidates who've only read about architectures from ones who've actually trained models.
Picture the loss as a landscape and training as walking downhill. The learning rate is your step size — too large and you overshoot the valley and bounce around or diverge; too small and you crawl, and can stall out in a flat region. Batch size controls how noisy your view of the terrain is at each step: small batches give a jittery, noisy gradient estimate that can help you escape shallow local dips; large batches give a smoother, more accurate estimate but see fewer distinct "views" of the landscape per epoch.
Warmup avoids destabilizing the model with large updates while weights are still near their random initialization and optimizer moment estimates (in Adam) haven't stabilized yet. Decaying the learning rate later lets the model take smaller, more precise steps as it approaches a good minimum instead of oscillating around it.
Larger batches produce a lower-variance gradient estimate, which generally tolerates a larger learning rate — a common heuristic is to scale the learning rate roughly linearly with batch size. Very large batches can also generalize slightly worse in some settings, since the noise from small-batch SGD acts as an implicit regularizer that helps escape sharp minima. Gradient accumulation (summing gradients over several small forward/backward passes before one optimizer step) is the standard way to simulate a larger batch when GPU memory is the limiting factor.
Training loss falling while validation loss rises is the classic overfitting signature — the fix is more data, augmentation, regularization, or fewer epochs. Both curves staying high and flat points to underfitting: too little model capacity, too low a learning rate, or a bug in the data pipeline or loss computation. A loss that suddenly spikes mid-training usually means the learning rate is too high, a batch contains a numerical outlier, or gradients are exploding — gradient clipping and a lower learning rate are the first things to try.