StackScholar

Transformers

KV Cache

Storing past keys and values during autoregressive decoding so each step is O(n) instead of O(n²).

Why interviewers ask about it

Size = 2·L·n·h_kv·d_k·batch·bytes. At batch 32 and 8k context a 70B model's MHA cache is ~687 GB - more than the weights. This is the entire motivation for GQA.

Related terms

Grouped-Query Attention (GQA)

Multiple query heads sharing one key/value head, typically 8 KV heads for 32–64 query heads.

PagedAttention

Storing the KV cache in fixed-size blocks with a page table, as in OS virtual memory.

This term is part of the free AI/ML Engineer interview preparation module - browse the full glossary for every definition.