How do token embeddings combine with positional embeddings in a Transformer?

How do token embeddings combine with positional embeddings in a Transformer?medium

Type: conceptual
Topic: do-token-embeddings-combine-with-positional-embeddings-in
Frequency: common
Tags: deep-learning, how, token, embeddings, combine, with

Answer

Transformers have no inherent notion of sequence order. Positional encodings are added (summed) to token embeddings before the first layer.

Explanation

Transformers have no inherent notion of sequence order. Positional encodings are added (summed) to token embeddings before the first layer. Original paper uses sinusoidal functions (fixed). Modern models use learned positional embeddings. RoPE (Rotary Position Embedding) encodes position by rotating Q and K vectors — better for long contexts, used in LLaMA and Qwen.

Follow-upCan you give a production example?