What does the feed-forward network inside a transformer block actually do?medium
Attention mixes information across tokens — deciding which tokens should talk to each other and pulling in relevant context. The feed-forward network (FFN) then transforms each token's representation independently, applying the same linear → activation → linear transformation to every position separately, without any cross-token mixing.
Attention answers "which tokens should talk to each other?"; the FFN answers "how should each token's representation be transformed after receiving that context?" — they're complementary, not redundant. A typical FFN is a linear layer, an activation function, and another linear layer, applied identically at every token position but with no interaction between positions during that step. This gives the model extra capacity to transform and store complex patterns beyond what the attention mixing step provides, and in modern LLMs the feed-forward layers often account for a large portion of total model parameters — attention gets most of the conceptual attention in explanations, but FFN layers are doing a lot of the actual parameter-weighted computation.