PagedAttention

LLMs

PagedAttention

Storing the KV cache in fixed-size blocks with a page table, as in OS virtual memory.

Why interviewers ask about it

Cuts KV fragmentation from 60–80% to under 4%, and enables prefix sharing so one long system prompt is stored once for thousands of concurrent users.

KV Cache

Storing past keys and values during autoregressive decoding so each step is O(n) instead of O(n²).

Continuous Batching

Evicting finished sequences and admitting new ones every decode step instead of holding a fixed batch.

This term is part of the free AI/ML Engineer interview preparation module - browse the full glossary for every definition.