What's the difference between attention, self-attention, and cross-attention?medium
Attention is the general idea of one sequence dynamically focusing on relevant parts of another sequence. Self-attention is the special case where a sequence attends to itself — every token attends to other tokens in the same input. Cross-attention is when a decoder attends to a separate encoder's output, as opposed to attending to its own previous tokens.
A classic example of plain attention: in translation, a French decoder attends to English encoder outputs — two different sequences. Self-attention instead means words in an English sentence attend to other words in that same sentence. Inside a full encoder-decoder transformer, all three show up together: encoder self-attention lets input tokens attend to other input tokens; decoder self-attention lets generated tokens attend to previously generated tokens (with causal masking); and cross-attention is the bridge where the decoder's tokens attend to the encoder's output representations, which is how the decoder actually incorporates the source sentence while generating the target. GPT-style LLMs simplify this picture considerably — they're decoder-only, so there's no separate encoder and no cross-attention at all, just causal self-attention throughout.