Transformers

Scaled Dot-Product Attention

softmax(QKᵀ/√d_k)V - a soft dictionary lookup where Q asks, K advertises and V delivers.

Why interviewers ask about it

The √d_k exists because Var(q·k) = d_k for unit-variance components; unscaled logits saturate softmax and its gradient vanishes.

An IO-aware exact attention algorithm that tiles Q/K/V into SRAM and uses online softmax so the n×n score matrix never reaches HBM.

This term is part of the free AI/ML Engineer interview preparation module - browse the full glossary for every definition.