LLMs
Speculative Decoding
A small draft model proposes k tokens, the large model verifies them in one pass, and modified rejection sampling accepts or resamples.
Why interviewers ask about it
Provably exact, not approximate. Works because decode is memory-bandwidth-bound, so verifying 5 tokens costs almost the same as 1. Helps latency at low concurrency, can hurt throughput under saturation.
Related terms
This term is part of the free AI/ML Engineer interview preparation module - browse the full glossary for every definition.