Multi-Query and Grouped-Query Attention
Multi-query attention shares a single set of keys and values across all attention heads, shrinking the memory needed to cache them during generation.
The generation memory problem
During autoregressive generation, a transformer caches the keys and values of all previous tokens so it does not recompute them at each step. This key-value cache grows with sequence length and with the number of attention heads, and moving it from memory becomes the main cost of long generations. Multi-query attention (MQA) attacks this directly by changing how many key and value projections the heads use.
Sharing keys and values
In standard multi-head attention, each head has its own query, key, and value projections. MQA keeps a separate query projection per head, preserving the diversity of what each head looks for, but shares one key projection and one value projection across all heads. The key-value cache therefore holds a single head's worth of keys and values instead of one per head, cutting its size by roughly the number of heads.
- Query heads stay independent, so representational diversity is largely retained
- The key-value cache shrinks dramatically, easing the memory bottleneck
- Faster generation because less data moves per step
- A small quality drop can occur versus full multi-head attention
Grouped-query as the middle ground
Grouped-query attention (GQA) interpolates between the extremes. Instead of one shared key-value pair (MQA) or one per head (standard attention), it partitions the query heads into a few groups, each sharing one key-value pair. With, say, eight groups over many heads, GQA recovers most of the quality of full attention while keeping most of the memory savings of MQA. It has become the default in many large models for this reason.
Practical role
MQA and GQA are architectural choices aimed squarely at inference efficiency for decoder-only models, complementary to systems optimizations like flash attention. Flash attention speeds up the attention computation itself; grouped-query attention reduces how much key-value data must be stored and moved. Combined, they let large models generate long sequences with far lower memory and latency than the original multi-head design.