Attention Vs Context Compression

LLMs interview question · Medium difficulty

Is an LLM's long-context handling done through attention, or something else?medium

Status
✓
Type
conceptual
Topic
attention-vs-context-compression
Frequency
common
Tags
attention, context-window, summarization, architecture
Answer

They're different layers. Compression, summarization, and retrieval happen before the model runs, to decide what goes into the context window. Self-attention then runs inside the transformer over whatever ended up in that window, deciding which of the included tokens matter most for the next prediction.

Explanation

Self-attention helps the model decide which parts of the currently available context are most relevant when generating the next token — for example, weighting "GitHub Pages," "private repo," and "custom domain" more heavily than an unrelated older topic. But attention only operates over tokens that are already present in the context window; it does not by itself compress a long conversation down to fit. If the full conversation is too long, preprocessing happens first — older messages get summarized, selected, or retrieved as relevant memory — and that already-compressed information is what gets inserted into the context. So the flow is: long conversation → recent messages kept mostly raw → older messages summarized or selectively retrieved → final context window assembled → transformer self-attention operates over that context → answer generated. Compression decides what's included; attention decides how much each included part matters.

Follow-upWhat would happen if you fed a self-attention model a conversation longer than its context window with no preprocessing?
Follow-upWhich stage — compression or attention — would you optimize first if long-context answers were missing older details?