69 terms across 8 areas. Each entry gives the definition and the interview angle — the specific point an interviewer will push on once you have given the textbook answer.
The decomposition of expected prediction error into irreducible noise, squared bias (how far the average model is from truth) and variance (how much the model moves across training sets).
Interview angle
The expectation is over resampled training sets, not test points. Say that and you separate yourself from candidates reciting a definition.
The distribution of the sample mean of i.i.d. variables approaches a normal distribution as sample size grows, regardless of the underlying distribution.
Interview angle
It is a statement about sample means, not about raw data. Candidates routinely misstate this as 'my data becomes normal'.
Choosing parameters that maximise the probability of the observed data under the model.
Interview angle
Every loss function is a negative log-likelihood: Gaussian noise gives MSE, Bernoulli gives cross-entropy. Choosing a loss is choosing a noise model.
Maximising the posterior P(θ|data) ∝ P(data|θ)P(θ) — maximum likelihood with a prior.
Interview angle
A Gaussian prior is exactly L2 regularisation; a Laplace prior is L1. λ = σ²/τ² makes the correspondence exact.
Factorisation A = UΣVᵀ that exists for any matrix: a rotation, an axis-aligned scaling, then another rotation.
Interview angle
PCA is the SVD of the mean-centred data matrix. Never form the covariance matrix explicitly — it squares the condition number.
The ratio σ_max/σ_min of a matrix's singular values; a measure of how elongated the loss surface is.
Interview angle
Feature standardisation, BatchNorm/LayerNorm and Adam's per-parameter scaling are all conditioning fixes. Framing them that way sounds senior.
Estimating an expectation under p while sampling from q, by reweighting each sample by p(x)/q(x).
Interview angle
PPO's clipped ratio is a variance-bounded importance weight. Off-policy evaluation of a recommender is the same idea. Always mention the infinite-variance failure mode and effective sample size.
Any information present at training time that will not be available at prediction time — from the target, from the future, from preprocessing fitted on the full dataset, or from entity overlap across splits.
Interview angle
0.99 AUC on a business problem is a leakage alarm, not a success. Adversarial validation and single-feature AUC screens are the detectors to name.
Estimating generalisation by repeatedly training on some folds and evaluating on the held-out fold.
Interview angle
Plain k-fold assumes exchangeable rows. Time needs TimeSeriesSplit with purging, grouped data needs GroupKFold. The rule: the validation split must simulate the production split.
The property that predicted probabilities match observed frequencies — among all cases scored 0.7, about 70% are positive.
Interview angle
Independent of discrimination: a model can have AUC 1.0 and be badly miscalibrated. Essential whenever a score feeds an expected-value decision. Fix with Platt, isotonic or temperature scaling on held-out data.
Binned average of |accuracy − confidence|, weighted by bin size.
Interview angle
Calibration drifts before discrimination does, which makes ECE an excellent early drift alarm in production.
Area under the precision–recall curve; equivalently average precision.
Interview angle
ROC-AUC's FPR denominator is dominated by the huge negative class under imbalance and barely moves. PR-AUC's baseline is the positive rate, so judge it as lift over prevalence.
Synthetic Minority Over-sampling: creates new minority examples by interpolating between a point and its k nearest minority neighbours.
Interview angle
Frequently hurts in high dimensions, where interpolated points fall in majority territory. Always measure PR-AUC before and after, and remember it destroys calibration.
Shapley-value feature attributions — each feature's average marginal contribution across all inclusion orderings.
Interview angle
The unique attribution satisfying local accuracy, missingness and consistency. But it explains the *model*, not the world — interventions require a causal design.
The drop in a held-out metric when a feature's values are randomly shuffled.
Interview angle
The honest alternative to impurity importance, which is biased toward high-cardinality features and computed on training data. Breaks down with correlated features — use grouped permutation.
Building each training row from only the feature values that existed at that row's timestamp.
Interview angle
The structural defence against temporal leakage, and the main reason feature stores exist. Say 'what did we know about entity X at time T?'.
Computing gradients by applying the chain rule in reverse topological order over the computation graph, caching intermediates.
Interview angle
∂L/∂W = xᵀg needs the forward activation, which is exactly why activation memory dominates training and why gradient checkpointing exists.
Exponential decay of gradient magnitude through depth, caused by repeatedly multiplying by Jacobians with small spectral norm.
Interview angle
Sigmoid's derivative caps at 0.25, so 20 layers multiply by at most 0.25²⁰ ≈ 10⁻¹². ReLU mitigates; residual connections solve.
y = x + F(x), giving ∂y/∂x = I + ∂F/∂x — a gradient path that passes through no weight matrices.
Interview angle
ResNet's insight was that deep plain networks had higher *training* error: an optimisation failure, not overfitting. Every transformer block uses the same idea.
Weight variance 2/fan_in, compensating for ReLU zeroing roughly half its inputs.
Interview angle
Xavier (2/(fan_in+fan_out)) is for symmetric activations like tanh. Using Xavier with ReLU in a deep network causes slow signal decay.
Normalisation across the feature dimension within each sample, independent of batch composition.
Interview angle
Batch-independence is why transformers use it: variable sequence lengths and single-token autoregressive decoding make BatchNorm unusable.
LayerNorm without mean-centring and without a bias — scale only.
Interview angle
Ablations showed re-centring contributes almost nothing. ~10–15% faster at identical quality; used by Llama, Mistral and Gemma.
x = x + Sublayer(LN(x)) — normalisation inside the residual branch, leaving the residual stream unnormalised.
Interview angle
Creates an uninterrupted identity path to layer 0, which removed the warmup fragility of the original post-norm transformer and enabled 100+ layer models.
Adam with weight decay applied directly to the parameter update rather than added to the gradient.
Interview angle
Under plain Adam, L2 in the loss gets divided by √v, so parameters with large gradients receive the least decay — the opposite of intended. Never decay biases or norm parameters.
Ramping the learning rate from ~0 over the first few hundred to few thousand steps.
Interview angle
Adam's second-moment estimate is effectively a one-sample estimate at step 1; a small v produces an enormous update that can permanently damage an untrained network.
Running matmuls in 16-bit while keeping master weights and optimiser state in fp32.
Interview angle
fp16 overflows at 65,504 and needs loss scaling; bf16 has fp32's exponent range and does not. A 7B model needs ~14 GB of weights but ~112 GB to train — optimiser state dominates.
Discarding intermediate activations during the forward pass and recomputing them during backward.
Interview angle
Trades roughly 30% extra compute for a large activation-memory reduction. The trade works because recompute is compute-bound and storage is memory-bound.
softmax(QKᵀ/√d_k)V — a soft dictionary lookup where Q asks, K advertises and V delivers.
Interview angle
The √d_k exists because Var(q·k) = d_k for unit-variance components; unscaled logits saturate softmax and its gradient vanishes.
Storing past keys and values during autoregressive decoding so each step is O(n) instead of O(n²).
Interview angle
Size = 2·L·n·h_kv·d_k·batch·bytes. At batch 32 and 8k context a 70B model's MHA cache is ~344 GB — more than the weights. This is the entire motivation for GQA.
Multiple query heads sharing one key/value head, typically 8 KV heads for 32–64 query heads.
Interview angle
Not an accuracy improvement — a serving-economics decision. 8× smaller KV cache means ~8× more concurrent requests per GPU at near-identical quality.
Rotating Q and K by position-proportional angles so the attention score depends only on the relative offset (m − n).
Interview angle
Applies to Q and K only, never V. Extendable beyond the trained length via position interpolation, NTK-aware scaling or YaRN.
An IO-aware exact attention algorithm that tiles Q/K/V into SRAM and uses online softmax so the n×n score matrix never reaches HBM.
Interview angle
It is exact and does not reduce FLOPs — it eliminates memory traffic. Calling it an approximation is the classic mistake (that is Linformer/Performer).
Subword tokenisation that starts from bytes and iteratively merges the most frequent adjacent pair.
Interview angle
Non-English text can cost 3–5× more tokens; digit tokenisation inconsistency hurts arithmetic; trailing whitespace changes completions. All downstream cost estimates should be in tokens.
Compute-optimal training scales parameters and tokens roughly equally — about 20 tokens per parameter, with C ≈ 6ND.
Interview angle
It optimises *training* compute only. Production models deliberately overtrain far past it (Llama-3-8B at ~1,900 tokens/param) because inference dominates lifetime cost.
Freezing pretrained W and learning ΔW = (α/r)·BA with B initialised to zero, training ~0.1–2% of parameters.
Interview angle
B = 0 means training starts exactly at the pretrained model. Targeting all linear layers matters more than increasing rank — most tutorials get this wrong.
LoRA over a 4-bit NF4-quantised frozen base, with double quantisation and paged optimisers.
Interview angle
Saves memory, not compute — typically ~30% slower per step than LoRA. Compute still happens in bf16 after on-the-fly dequantisation.
Optimising preference pairs directly using the closed-form optimal KL-constrained policy, eliminating the reward model and the RL loop.
Interview angle
The partition function Z(x) cancels because it is identical for chosen and rejected. Two models instead of four, one main hyperparameter instead of twelve — but it is offline and cannot exceed its preference data.
SFT, then a Bradley–Terry reward model on human preference pairs, then PPO with a KL penalty to the reference policy.
Interview angle
It trains on human *rankings*, not human-written answers. The KL term is what prevents reward hacking; omitting it in an explanation is a common tell.
A small draft model proposes k tokens, the large model verifies them in one pass, and modified rejection sampling accepts or resamples.
Interview angle
Provably exact, not approximate. Works because decode is memory-bandwidth-bound, so verifying 5 tokens costs almost the same as 1. Helps latency at low concurrency, can hurt throughput under saturation.
Storing the KV cache in fixed-size blocks with a page table, as in OS virtual memory.
Interview angle
Cuts KV fragmentation from 60–80% to under 4%, and enables prefix sharing so one long system prompt is stored once for thousands of concurrent users.
Evicting finished sequences and admitting new ones every decode step instead of holding a fixed batch.
Interview angle
Static batching wastes ~80% of slot-steps because output lengths are long-tailed — one 1,000-token request makes 31 finished slots spin.
Retrieving relevant documents at query time and conditioning generation on them.
Interview angle
It is a search problem with a generation step attached — 80% of the effort belongs in retrieval. Measure retrieval and generation separately or you cannot tell which is failing.
Combining dense vector retrieval with sparse lexical retrieval (BM25), usually fused by reciprocal rank fusion.
Interview angle
Embeddings fail on rare exact tokens (error codes, IDs, names) because those fragments carry little semantic signal. BM25 fails on paraphrase. The failure modes are complementary.
Fusing ranked lists by summing 1/(k + rank) across systems, with k typically 60.
Interview angle
Rank-based, so it needs no score normalisation — robust when BM25 scores are unbounded and cosine is in [−1, 1]. The safer default over weighted score fusion.
Scoring (query, document) pairs jointly with full attention between them, applied to the top 20–100 candidates.
Interview angle
Hybrid search improves recall; reranking improves precision at the top. Diagnose recall@50 first — if it is low, no reranker can help.
Hierarchical Navigable Small World: a multi-layer proximity graph for approximate nearest-neighbour search.
Interview angle
Best recall/latency but ~1.5× vector memory. At 100M×768 that is ~330 GB, which is why IVF-PQ (~10 GB) is standard at that scale.
Splitting a vector into m subvectors and quantising each with its own 256-centroid codebook, storing m bytes per vector.
Interview angle
8–64× compression with distances from lookup tables. Pair with an exact rescoring stage over the top ~200 to recover most of the lost accuracy.
Embedding small units (sentences) for precise matching while returning the larger parent section to the model.
Interview angle
Resolves the chunk-size trade-off directly — precision from small embeddings, context from large returns. The single most effective chunking upgrade in most systems.
The fraction of atomic claims in a generated answer that are supported by the retrieved context.
Interview angle
Measured by decomposing the answer into claims and verifying each against the context, usually with an LLM judge. Validate the judge against human labels before trusting it.
Text in untrusted content that the model interprets as instructions, because LLMs have no architectural separation between instructions and data.
Interview angle
No prompt-level defence is reliable. Design so a successful injection cannot cause harm: least privilege, drop irreversible capabilities after untrusted reads, enforce limits in code.
Exposing typed functions the model can invoke, with a JSON schema describing parameters.
Interview angle
A tool description is a prompt. Say what it does, when to use it and when NOT to. Return errors as observations so the agent self-corrects, and shape results aggressively.
observe → decide (choose a tool) → act → observe result → repeat until a stop condition.
Interview angle
Bound steps, tokens, wall-clock and cost, and detect repeated identical calls. If you can draw the flowchart, write the flowchart instead of using an agent.
Any difference between how a feature is computed at training time and at inference time.
Interview angle
Detect by logging serving features and re-scoring them offline with the training pipeline. That one practice catches implementation, freshness and source skew simultaneously.
A system providing point-in-time correct training-set generation and a low-latency online store serving identical values.
Interview angle
Adopt when multiple teams share features or you need point-in-time joins across many entities — not on day one. A shared transform library plus Redis covers most small teams.
Running a new model on real traffic without serving its output, comparing scores and latency against the incumbent.
Interview angle
Answers 'does it break?' at zero user risk — including latency regressions offline evaluation cannot see. Only a powered A/B answers 'is it better?'.
Ramping a new version through 1% → 5% → 25% → 100% with automated guardrail thresholds.
Interview angle
Automate the kill thresholds so rollback needs no human, and make rollback a config flag against a warm artefact — not a redeploy.
A binned measure of distribution shift between a reference and current sample. <0.1 stable, 0.1–0.25 moderate, >0.25 significant.
Interview angle
PSI measures effect size, which is why it is the operational alert. KS p-values fire on nearly every feature every day at 10M rows.
A change in P(Y|X) — the same inputs now imply a different outcome.
Interview angle
The dangerous one, because input-drift dashboards stay green while the model is systematically wrong. Adversarial domains guarantee it by design.
Testing model properties rather than exact outputs: invariance, directional expectation, and minimum functionality.
Interview angle
You cannot assert predict(x) == 0.73, so assert 'more failed payments must increase churn risk'. Catches regressions an aggregate AUC threshold hides completely.
A cheap model handling confident cases, escalating only the uncertain band to an expensive model.
Interview angle
With an 88% early-exit rate, a 2 ms + 180 ms cascade averages ~23 ms. Large latency and cost wins with tunable accuracy loss.
Training a small student to match a large teacher's soft output distribution, usually with a temperature-scaled KL term.
Interview angle
Distil on *unlabelled production data* so the student trains on the true serving distribution. Often 90–95% of teacher quality at 1–5% of the cost.
P(data at least this extreme | H₀ true).
Interview angle
Not P(H₀ | data), not the probability the result is chance, and not a measure of effect size. Lead with the effect size and confidence interval instead.
The probability of detecting a true effect of a given size. Sample size n ≈ 16·p(1−p)/δ² per arm at 80% power, α = 0.05.
Interview angle
Halving the minimum detectable effect quadruples the required sample. That quadratic is why small model gains are so expensive to validate.
Repeatedly checking an experiment and stopping when it reaches significance.
Interview angle
Daily checks for two weeks push a nominal 5% α above 20%. Fix with group-sequential boundaries or always-valid confidence sequences.
Controlled-experiment Using Pre-Experiment Data: regressing out each user's pre-period metric value to reduce variance.
Interview angle
Cuts variance 30–50% for retention-style metrics — equivalent to a free 30–50% more traffic. Naming it is a strong senior signal.
Observed traffic split differing significantly from the intended split.
Interview angle
The first check when an experiment result looks odd. An SRM invalidates the whole test — investigate before interpreting any metric.
Mixing results from two rankers into one list and attributing clicks back to the contributing system.
Interview angle
Removes between-user variance, needing 10–100× less traffic than a conventional A/B for ranking comparisons. A strong thing to name in search/RecSys interviews.
Higher-ranked items receive more clicks regardless of relevance.
Interview angle
Training on raw clicks makes the model reproduce the existing ranker. Correct with an examination model (position feeds only the propensity term, dropped at serving) or inverse propensity weighting.
A procedure controlling the false discovery rate across many comparisons by comparing sorted p-values to (k/m)·q.
Interview angle
Bonferroni controls family-wise error and is badly over-conservative for correlated product metrics. BH tolerates positive dependence and preserves power.
Knowing what a term means gets you past the first question. The lessons cover the derivation, the trade-off and the follow-up.
Browse the curriculum