Feedforward Network In Transformer

Deep Learning interview question · Medium difficulty

What does the feed-forward network inside a transformer block actually do?medium

Status
✓
Type
conceptual
Topic
feedforward-network-in-transformer
Frequency
common
Tags
feed-forward-network, transformer-block, architecture
Answer

Attention mixes information across tokens — deciding which tokens should talk to each other and pulling in relevant context. The feed-forward network (FFN) then transforms each token's representation independently, applying the same linear → activation → linear transformation to every position separately, without any cross-token mixing.

Explanation

Attention answers "which tokens should talk to each other?"; the FFN answers "how should each token's representation be transformed after receiving that context?" — they're complementary, not redundant. A typical FFN is a linear layer, an activation function, and another linear layer, applied identically at every token position but with no interaction between positions during that step. This gives the model extra capacity to transform and store complex patterns beyond what the attention mixing step provides, and in modern LLMs the feed-forward layers often account for a large portion of total model parameters — attention gets most of the conceptual attention in explanations, but FFN layers are doing a lot of the actual parameter-weighted computation.

Follow-upWhy does the FFN need to be applied identically at every position rather than having position-specific weights?
Follow-upIf attention layers were removed and only FFN layers remained, what would the model lose?
Follow-upWhy do FFN layers hold such a large share of a modern LLM's total parameters?