Transformers have no inherent notion of sequence order — self-attention alone is permutation-invariant, so without positional information "dog bites man" and "man bites dog" would look identical. Positional encodings are added (summed) to token embeddings before the first layer to inject that order.

Transformers have no inherent notion of sequence order. Positional encodings are added (summed) to token embeddings before the first layer. Original paper uses sinusoidal functions (fixed), computed from position and dimension with no training needed. Modern models use learned positional embeddings, treated as trainable parameters like any other weight. RoPE (Rotary Position Embedding) encodes position by rotating Q and K vectors rather than adding to the embedding — better for long contexts, used in LLaMA and Qwen. ALiBi takes a different approach: instead of touching the embeddings at all, it adds a position-based bias directly to the attention scores, which lets it generalize more gracefully to sequences longer than anything seen in training.