Is an LLM's long-context handling done through attention, or something else?medium
They're different layers. Compression, summarization, and retrieval happen before the model runs, to decide what goes into the context window. Self-attention then runs inside the transformer over whatever ended up in that window, deciding which of the included tokens matter most for the next prediction.
Self-attention helps the model decide which parts of the currently available context are most relevant when generating the next token — for example, weighting "GitHub Pages," "private repo," and "custom domain" more heavily than an unrelated older topic. But attention only operates over tokens that are already present in the context window; it does not by itself compress a long conversation down to fit. If the full conversation is too long, preprocessing happens first — older messages get summarized, selected, or retrieved as relevant memory — and that already-compressed information is what gets inserted into the context. So the flow is: long conversation → recent messages kept mostly raw → older messages summarized or selectively retrieved → final context window assembled → transformer self-attention operates over that context → answer generated. Compression decides what's included; attention decides how much each included part matters.