Attention lets a model dynamically focus on the most relevant input tokens while generating or understanding each token, instead of relying on one fixed compressed vector for the entire input. That removed the main bottleneck of early encoder-decoder models: for each output token, the decoder can look back at all input tokens and weight them by relevance, rather than reading everything through a single fixed-size representation.

In early encoder-decoder models, the encoder compressed the full input sentence into a single context vector — fine for short sentences, but lossy for long ones. Attention changed the shape of the problem: for each output word, the model computes importance scores over all input tokens and focuses on the relevant ones. While translating a sentence about Amit preparing for an AI interview, the model focuses on "Amit" when generating the token related to Amit, and shifts focus to "preparing" when generating the token related to preparing. This was a breakthrough for several concrete reasons: it reduced information loss on long sequences, improved translation and summarization quality, helped the model capture long-range dependencies it previously lost, made behavior more interpretable through inspectable attention weights, and — most consequentially — became the foundation the Transformer architecture was built on.