Every position looking at every other position — the architecture behind modern LLMs
RNNs process a sequence one step at a time, so training can't be parallelized across the sequence dimension and information from far-back tokens has to survive many sequential updates to still matter. Self-attention lets every position directly look at every other position in a single step — no long chain of hidden-state updates to preserve information across, and every step is independent enough to compute in parallel on a GPU. That combination is why transformers scale so much better than RNNs, which is the direct answer to "why did transformers replace RNNs for most NLP tasks?"
Self-attention is a soft, differentiable lookup: each token asks a question (its query), every token in the sequence offers an answer (its key), and the token retrieves a weighted blend of everyone's value based on how well each key matches its query. A word like "it" ends up attending heavily to whichever earlier noun it actually refers to, without anyone hand-coding that rule — the weighting is learned.
The output for each position is a weighted sum of all value vectors, where the weights come from how similar that position's query is to every key in the sequence — so every token gets a representation informed by the whole sequence, not just its neighbors.
A single attention head learns one notion of "relevant" — multi-head attention runs several attention operations in parallel with different learned projections, so one head might track syntactic dependencies while another tracks coreference, and their outputs are concatenated and projected back down. Since attention itself has no built-in sense of order — it would produce the same output for a shuffled sequence — positional encodings (sinusoidal or learned) are added to the input embeddings so the model can tell position 1 from position 50.
The original transformer used an encoder (bidirectional self-attention over the full input) feeding a decoder (causal self-attention plus cross-attention into the encoder output) — a good fit for translation, where the whole source sentence is available upfront. Most modern LLMs are decoder-only: causal self-attention where each token can only attend to earlier tokens, trained to predict the next token, which is simpler to scale and naturally supports open-ended generation.