Batch normalization vs layer normalization — which is preferred in Transformers?medium
Answer
BatchNorm normalizes across the batch dimension — problematic for variable-length sequences and small batches.
Explanation
BatchNorm normalizes across the batch dimension — problematic for variable-length sequences and small batches. LayerNorm normalizes across the feature dimension for each sample independently — no batch dependency. Transformers use LayerNorm. Pre-LN (normalize before attention) is more stable than post-LN during training for very deep models.
Follow-upWhen would you choose one approach over the other?