Dropout, batch/layer norm, weight decay, and early stopping — the toolkit that keeps big models from overfitting or destabilizing
Deep networks have enough capacity to memorize their training data, so left unchecked they overfit. Normalization layers separately solve a training-stability problem — keeping activations in a well-behaved range as they flow through many layers. Production models fail in exactly these two ways: they memorize noise, or training diverges/stalls.
Regularization (dropout, weight decay, early stopping) fights overfitting by making the model work with a handicap or by cutting training short before it starts memorizing noise. Normalization (batch norm, layer norm) fights internal instability by re-centering and re-scaling activations at every layer, so gradients don't explode or vanish as they pass through a deep stack.
Dropout forces the network to not rely on any single neuron (co-adaptation), since that neuron might be zeroed next step — it's an implicit ensemble of many thinned sub-networks trained jointly. At inference you want the full, deterministic network — dropout would just add noise to a prediction that should be reproducible, and the 1/(1-p) scaling during training already accounts for the expected magnitude so no correction is needed at test time.
Transformers use layer norm (not batch norm) because sequences have variable length and are often processed with small or per-token batches (e.g. autoregressive decoding one token at a time) — batch statistics would be unstable or meaningless in that setting, while layer norm's per-example statistics work regardless of batch size or sequence length.
Penalizes large weights, which biases the model toward simpler decision boundaries and reduces variance. Note: with Adam, naively adding the L2 term to the gradient interacts badly with the adaptive denominator — AdamW decouples weight decay from the gradient-based update, which is why AdamW is now standard.
Monitor validation loss during training; stop (or checkpoint and revert) once it stops improving for a set number of epochs ("patience"), even if training loss keeps dropping. It's a free regularizer — no architecture change, no extra hyperparameter search — because the gap between falling train loss and rising val loss is the overfitting signal itself.