Is Multi Head Attention And Why Is It Better Than Single H

NLP interview question · Medium difficulty

What is multi-head attention and why is it better than single-head?medium

Type
conceptual
Topic
is-multi-head-attention-and-why-is-it-better-than-single-h
Frequency
common
Tags
nlp, what, multi, head, attention, and
Answer

Single-head computes one attention distribution — one way of relating tokens.

Explanation

Single-head computes one attention distribution — one way of relating tokens. Multi-head runs h parallel attention functions with different learned projections, then concatenates. Each head can attend to different relationship types (syntactic, semantic, positional). For sarcasm or contract parsing, some heads focus on negation, others on named entities, others on long-range dependencies.

Follow-upCan you give a production example?
← What does bidirectionality add in a Bi-LSTM text classifier? You fine-tuned BERT for sarcasm detection. What layers did you modify? →