Attention Vs Self Attention Cross Attention

Deep Learning interview question · Medium difficulty

What's the difference between attention, self-attention, and cross-attention?medium

Status
✓
Type
conceptual
Topic
attention-vs-self-attention-cross-attention
Frequency
common
Tags
attention, self-attention, cross-attention, encoder-decoder
Answer

Attention is the general idea of one sequence dynamically focusing on relevant parts of another sequence. Self-attention is the special case where a sequence attends to itself — every token attends to other tokens in the same input. Cross-attention is when a decoder attends to a separate encoder's output, as opposed to attending to its own previous tokens.

Explanation

A classic example of plain attention: in translation, a French decoder attends to English encoder outputs — two different sequences. Self-attention instead means words in an English sentence attend to other words in that same sentence. Inside a full encoder-decoder transformer, all three show up together: encoder self-attention lets input tokens attend to other input tokens; decoder self-attention lets generated tokens attend to previously generated tokens (with causal masking); and cross-attention is the bridge where the decoder's tokens attend to the encoder's output representations, which is how the decoder actually incorporates the source sentence while generating the target. GPT-style LLMs simplify this picture considerably — they're decoder-only, so there's no separate encoder and no cross-attention at all, just causal self-attention throughout.

Follow-upWhy doesn't a decoder-only GPT-style model need cross-attention?
Follow-upIn an encoder-decoder model like T5, what would go wrong if cross-attention were removed?
Follow-upIs decoder self-attention causally masked the same way in an encoder-decoder model as in a GPT-style decoder-only model?