Transformers
KV Cache
Storing past keys and values during autoregressive decoding so each step is O(n) instead of O(n²).
Why interviewers ask about it
Size = 2·L·n·h_kv·d_k·batch·bytes. At batch 32 and 8k context a 70B model's MHA cache is ~344 GB - more than the weights. This is the entire motivation for GQA.
Related terms
This term is part of the free AI/ML Engineer interview preparation module - browse the full glossary for every definition.