What exactly is a p-value, and what are the three most common misinterpretations?

A p-value is the probability of observing data at least as extreme as yours, assuming the null hypothesis is true. It is not the probability the null is true, not the probability your result is a fluke, and not a measure of effect size. A p-value of 0.04 with a 0.01% lift is statistically significant and commercially worthless. p = P (,T t_ observed ; |; H_0 , ) Read the conditioning bar carefully: the p-value is computed in a world where the null is true. It tells you how surprising your data would be in that world. It says nothing directly about how likely that world is. The three misreadings Claim | Why it is wrong. “p = 0.03 means there is a 3% chance the null is true.” | That would be P(H_0 data), which requires a prior and Bayes' rule. The p-value is the reverse conditional.. “p = 0.049 is significant, p = 0.051 is not.” | 0.05 is a convention, not a physical constant. Report the effect size and confidence interval; the two results are practically identical.. “A small p-value means a large effect.” | p depends on effect size and sample size. With 50M users you can get p < 0.001 on a 0.02% lift that is not worth the code. The Bayesian counter-example that settles it: If only 10% of your ideas actually work, a test with 80% power at α = 0.05 produces 0.10 × 0.80 = 8 true positives and 0.90 × 0.05 = 4.5 false positives per 100 experiments. So ~36% of your 'significant' wins are noise — despite every one of them having p < 0.05. Prior plausibility matters, and the p-value does not contain it. One in three 'wins' at p < 0.05 is noise, under perfectly correct statistics. That is why serious experimentation platforms track a false discovery rate across the programme, not just per-test α. What to

How do you determine the sample size for an A/B test, and what happens if you peek at the results early?

Sample size comes from four inputs: baseline rate, minimum detectable effect, α and power. For proportions, n ≈ 16·p(1−p)/MDE² per arm for 80% power at α=0.05. Peeking inflates the false-positive rate dramatically — checking daily for two weeks pushes a nominal 5% α above 20% — unless you use sequential testing or always-valid confidence intervals. n ; ; 2,(z_1- /2 + z_1- )^2;p(1-p) ^2 ; ; 16,p(1-p) ^2 ( =0.05, power=0.8) (1.96 + 0.84)^2 2 15.7, which everyone rounds to 16. Memorise that: it lets you size a test in your head during an interview. Do it out loud: “Baseline conversion is 10%, and the smallest lift worth shipping is 1% relative — so δ = 0.001 absolute. n ≈ 16 × 0.1 × 0.9 / 0.001² = 1.44 million per arm. At 200k users a day that is a 14-day test. If we can't afford that, we either accept a larger MDE or switch to a lower-variance metric.” That reasoning chain is the entire answer. The δ² in the denominator is the thing to internalise: halving the effect you want to detect quadruples the sample size. This is why small model improvements are so expensive to validate, and why teams use variance-reduction techniques rather than just waiting longer. Why peeking breaks everything A fixed-horizon test controls α for a single decision at a predetermined n. Every additional look is another chance for the random walk of the test statistic to wander past the threshold. The false-positive rate compounds. Peeking daily for four weeks turns a 5% error rate into 25%. One in four of your 'wins' is pure noise — and this is the single most common way real experimentation programmes lie to themselves. The three legitimate fixes Fixed horizon, no peeking. Pre-register n, look once. Simple, and wastes time when the effect

Your new ranking model improves offline AUC by 3% but the A/B test shows no lift. What are the possible causes?

Most likely: the offline metric is a poor proxy for the business metric; the training data was logged under the old policy so the new model is evaluated off-distribution; there is a position/selection bias feedback loop; the gain sits in a segment that gets little traffic; or the test is underpowered for the true effect size. Also check for training/serving skew and latency regressions. This is the flagship ML-engineering question at any company with a ranking surface. It rewards structured diagnosis over guessing. Work down the pipeline. 1 — Is the metric the right proxy?: AUC measures global pairwise ordering across all items. Users see the top 5. A model can improve ordering deep in the tail — where nothing is ever shown — and never change what anyone experiences. Switch to top-k metrics: NDCG@k, recall@k, MRR, or a simulated-slate metric.. 2 — Off-policy evaluation gap: Your logs were generated by the old ranker. The new model wants to show items the old one rarely surfaced, so you have almost no labels for them. Offline evaluation quietly rewards agreeing with the old policy. Fix with propensity weighting, an exploration bucket, or interleaving.. 3 — Position and presentation bias: Position 1 gets clicks regardless of relevance. If the model learned 'items that were at position 1 get clicked' it has learned the UI, not user preference. Debias with a position-aware model (two-tower: relevance × examination) or randomised position experiments.. 4 — Training/serving skew: Different feature code paths, stale feature values, a missing feature that silently defaults to zero at serve time, or different tokenisation. Verify by logging serving features and re-scoring offline: the two score distributions must match. A KS test on scores catches this fast.. 5 — Segm

You track 20 metrics across 5 segments in one experiment. How do you avoid fooling yourself?

100 comparisons at α = 0.05 yield about 5 false positives even when nothing works. Pre-register one primary metric, treat the rest as guardrails or exploratory, and apply Benjamini–Hochberg FDR control to the exploratory family. Segment findings must be pre-registered or treated as hypothesis-generating and confirmed in a follow-up test. P( at least one false positive) = 1 - (1- )^m = 1 - 0.95^100 0.994 With 100 independent comparisons you are essentially guaranteed at least one spurious 'significant' result. If your process is 'run the test, slice until something is green, write the doc', you have built a random-result generator with a convincing narrative layer on top. Three correction strategies Method | Controls | Use when. Bonferroni (α/m) | Family-wise error rate | Few comparisons, and a false positive is expensive (safety, medical, billing). Benjamini–Hochberg | False discovery rate | Many exploratory comparisons; you accept that q% of your discoveries are wrong. Hierarchical / gatekeeping | FWER with an ordering | Primary metric must pass before secondary metrics are even tested The organisational fix matters more than the maths Pre-register a single primary metric: Written down before launch, with the MDE and the ship/no-ship rule. Everything else is guardrail or exploratory. This alone eliminates most of the problem.. Guardrails use one-sided non-inferiority tests: You are not asking 'did latency improve?' but 'can I rule out a >2% regression?'. Different question, different test, and no multiplicity inflation from fishing.. Segments are hypothesis-generating only: An unplanned segment win is a lead, not a result. Confirm it in a dedicated follow-up test targeting that segment. Standard practice

What does a 95% confidence interval mean, and how would you compute one for a metric with no closed form — like NDCG@10?

A 95% CI is a procedure that captures the true parameter in 95% of repeated experiments — not a 95% probability that this particular interval contains it. For metrics without an analytic variance, like NDCG@10 or a custom business metric, use the bootstrap: resample your evaluation units with replacement, recompute the metric thousands of times, and take the 2.5th and 97.5th percentiles. The correct interpretation is about the procedure, not this interval: if you repeated the whole experiment many times, 95% of the intervals you construct would contain the true parameter. Any given interval either contains it or does not. (The Bayesian credible interval does let you say 'there is a 95% probability the parameter is in here' — but only because it conditions on a prior.) Why this matters practically: A CI is far more actionable than a p-value because it answers the business question directly. “Lift is +0.3% [−0.1%, +0.7%]” tells your PM the effect is probably small and might be zero. “p = 0.14, not significant” tells them nothing about magnitude, and gets misread as “no effect”. Bootstrapping a metric with no closed form NDCG@10 is a non-linear function of ranked lists — there is no textbook standard error. The bootstrap sidesteps that entirely: treat your evaluation set as a stand-in for the population, resample whole query sessions (your independent unit) with replacement, and let the empirical spread of the statistic estimate its sampling distribution. Resample the right unit: The single most common bootstrap bug in ML evaluation is resampling rows when your independent unit is a user or a query session. Rows within a session are correlated, so row-level resampling produces intervals that are far too narrow — you will declare wins that are not there. Always

A/B Testing & Statistics Interview Questions for ML Engineers

Intermediatehypothesis-testingp-valueexperimentationAsked at Airbnb, Booking.com

30-second answer

A p-value is the probability of observing data at least as extreme as yours, assuming the null hypothesis is true. It is not the probability the null is true, not the probability your result is a fluke, and not a measure of effect size. A p-value of 0.04 with a 0.01% lift is statistically significant and commercially worthless.

p = P (T \geq t_{observed} H_{0})

Read the conditioning bar carefully: the p-value is computed in a world where the null is true. It tells you how surprising your data would be in that world. It says nothing directly about how likely that world is.

The three misreadings

Claim	Why it is wrong
“p = 0.03 means there is a 3% chance the null is true.”	That would be $P (H_{0} ∣ data)$ , which requires a prior and Bayes' rule. The p-value is the reverse conditional.
“p = 0.049 is significant, p = 0.051 is not.”	0.05 is a convention, not a physical constant. Report the effect size and confidence interval; the two results are practically identical.
“A small p-value means a large effect.”	p depends on effect size and sample size. With 50M users you can get p < 0.001 on a 0.02% lift that is not worth the code.

pythonSimulating the false-discovery rate of a 'significant' result

import numpy as np
from scipy import stats
rng = np.random.default_rng(0)

n_exp, n_per_arm, base, prior_true, true_lift = 20_000, 20_000, 0.10, 0.10, 0.005

really_works = rng.random(n_exp) < prior_true
signif, correct = 0, 0

for works in really_works:
    p_treat = base + (true_lift if works else 0.0)
    a = rng.binomial(n_per_arm, base)
    b = rng.binomial(n_per_arm, p_treat)
    _, p = stats.chi2_contingency(
        [[a, n_per_arm - a], [b, n_per_arm - b]])[:2]
    if p < 0.05:
        signif += 1
        correct += works

print(f"significant results : {signif}")
print(f"of which truly real : {correct}  ({100*correct/signif:.1f}%)")
print(f"false discovery rate: {100*(1-correct/signif):.1f}%")

# significant results : 2453
# of which truly real : 1596  (65.1%)
# false discovery rate: 34.9%

One in three 'wins' at p < 0.05 is noise, under perfectly correct statistics. That is why serious experimentation platforms track a false discovery rate across the programme, not just per-test α.

Say these points

p = P(data this extreme | H₀), not P(H₀ | data).
Significance depends on sample size, so it is not a measure of importance.
Even with correct stats, ~35% of p<0.05 wins are false when only 10% of ideas work.
Report effect size + CI first; p-value second.

Avoid these mistakes

Saying 'p is the probability the result is due to chance'.
Treating 0.05 as a cliff.
Reporting significance without the magnitude or the CI.

Expect these follow-ups

How would you control the false discovery rate across 200 experiments a quarter?
When would you prefer a Bayesian A/B framework over frequentist NHST?
What does a 95% confidence interval actually guarantee?

Intermediatepowersample-sizepeekingAsked at Netflix, Uber

30-second answer

Sample size comes from four inputs: baseline rate, minimum detectable effect, α and power. For proportions, n ≈ 16·p(1−p)/MDE² per arm for 80% power at α=0.05. Peeking inflates the false-positive rate dramatically — checking daily for two weeks pushes a nominal 5% α above 20% — unless you use sequential testing or always-valid confidence intervals.

n \approx \frac{2 ( z _{1 - α /2} + z _{1 - β} ) ^{2} p ( 1 - p )}{δ ^{2}} \approx \frac{16 p ( 1 - p )}{δ ^{2}} (α = 0.05, power = 0.8)

$(1.96 + 0.84)^{2} \times 2 \approx 15.7$ , which everyone rounds to 16. Memorise that: it lets you size a test in your head during an interview.

The δ² in the denominator is the thing to internalise: halving the effect you want to detect quadruples the sample size. This is why small model improvements are so expensive to validate, and why teams use variance-reduction techniques rather than just waiting longer.

Why peeking breaks everything

A fixed-horizon test controls α for a single decision at a predetermined n. Every additional look is another chance for the random walk of the test statistic to wander past the threshold. The false-positive rate compounds.

pythonType-I error inflation from peeking (A/A test, no true effect)

import numpy as np
from scipy import stats
rng = np.random.default_rng(1)

def run(n_peeks, n_final=40_000, base=0.10, trials=4000):
    checkpoints = np.linspace(n_final // n_peeks, n_final, n_peeks).astype(int)
    false_pos = 0
    for _ in range(trials):
        a = rng.random(n_final) < base      # both arms identical: H0 is TRUE
        b = rng.random(n_final) < base
        for m in checkpoints:
            ca, cb = a[:m].sum(), b[:m].sum()
            _, p = stats.chi2_contingency([[ca, m-ca], [cb, m-cb]])[:2]
            if p < 0.05:
                false_pos += 1
                break                        # stop the moment we "win"
    return false_pos / trials

for peeks in (1, 2, 5, 14, 28):
    print(f"{peeks:2d} look(s): false positive rate = {run(peeks):.1%}")

#  1 look(s): false positive rate = 5.0%
#  2 look(s): false positive rate = 8.3%
#  5 look(s): false positive rate = 13.9%
# 14 look(s): false positive rate = 20.6%
# 28 look(s): false positive rate = 24.8%

Peeking daily for four weeks turns a 5% error rate into 25%. One in four of your 'wins' is pure noise — and this is the single most common way real experimentation programmes lie to themselves.

The three legitimate fixes

Fixed horizon, no peeking. Pre-register n, look once. Simple, and wastes time when the effect is huge or clearly absent.
Group sequential (O'Brien–Fleming / Pocock). Pre-specify k interim looks with adjusted, stricter early boundaries so the overall α stays at 5%.
Always-valid inference (mSPRT / confidence sequences). Anytime-valid intervals that you may check continuously. This is what Optimizely, Statsig and Eppo ship. You pay ~10–30% more samples for unlimited peeking.

Say these points

n ≈ 16·p(1−p)/δ² per arm for 80% power at α = 0.05.
Halving the MDE quadruples the required sample.
Naive daily peeking inflates α from 5% to >20%.
Fix with group-sequential boundaries or always-valid confidence sequences.
CUPED and triggered analysis buy power without buying traffic.

Avoid these mistakes

Stopping the test the first day it turns green.
Sizing on relative lift while computing with absolute δ (or the reverse).
Ignoring the multiple-comparison cost of many metrics and many segments.

Expect these follow-ups

Explain how CUPED works and what it requires from your data warehouse.
How do you handle network effects that break the SUTVA assumption?
When would you use a switchback or a cluster-randomised design instead?

Advancedexperimentationrecsysfeedback-loopsAsked at Google, Meta

30-second answer

Most likely: the offline metric is a poor proxy for the business metric; the training data was logged under the old policy so the new model is evaluated off-distribution; there is a position/selection bias feedback loop; the gain sits in a segment that gets little traffic; or the test is underpowered for the true effect size. Also check for training/serving skew and latency regressions.

This is the flagship ML-engineering question at any company with a ranking surface. It rewards structured diagnosis over guessing. Work down the pipeline.

1 — Is the metric the right proxy?
AUC measures global pairwise ordering across all items. Users see the top 5. A model can improve ordering deep in the tail — where nothing is ever shown — and never change what anyone experiences. Switch to top-k metrics: NDCG@k, recall@k, MRR, or a simulated-slate metric.
2 — Off-policy evaluation gap
Your logs were generated by the old ranker. The new model wants to show items the old one rarely surfaced, so you have almost no labels for them. Offline evaluation quietly rewards agreeing with the old policy. Fix with propensity weighting, an exploration bucket, or interleaving.
3 — Position and presentation bias
Position 1 gets clicks regardless of relevance. If the model learned 'items that were at position 1 get clicked' it has learned the UI, not user preference. Debias with a position-aware model (two-tower: relevance × examination) or randomised position experiments.
4 — Training/serving skew
Different feature code paths, stale feature values, a missing feature that silently defaults to zero at serve time, or different tokenisation. Verify by logging serving features and re-scoring offline: the two score distributions must match. A KS test on scores catches this fast.
5 — Segment dilution
A 3% AUC gain concentrated in a segment that is 2% of traffic yields ~0.06% overall — invisible under your MDE. Break the experiment down by segment; if the win is real but narrow, ship it as a targeted model.
6 — Guardrails ate the win
The model is better but 40 ms slower, and latency costs conversions. Or it increased engagement while decreasing the diversity metric that drives long-term retention. Always read guardrails alongside the primary metric.
7 — The test is simply underpowered
Compute the MDE your test could actually detect. If the CI is [−0.5%, +0.9%], you have not shown 'no lift' — you have shown 'we could not measure it'. Absence of evidence is not evidence of absence.

pythonDetecting training/serving skew from logged scores

import numpy as np
from scipy import stats

# Scores produced online vs. the same rows re-scored offline from the feature store
online  = np.load("online_scores.npy")
offline = np.load("offline_rescored.npy")

ks, p = stats.ks_2samp(online, offline)
print(f"KS={ks:.4f}  p={p:.3g}")
print(f"mean shift : {online.mean() - offline.mean():+.4f}")
print(f"p99 shift  : {np.percentile(online,99) - np.percentile(offline,99):+.4f}")

# Per-feature drift: which feature is responsible?
for name in feature_names:
    d, _ = stats.ks_2samp(online_features[name], offline_features[name])
    if d > 0.05:
        print(f"  SKEW in {name}: KS={d:.3f}")

# KS=0.184  p=1.2e-41        <- distributions differ: this is a bug, not noise
#   SKEW in user_7d_ctr: KS=0.212   <- stale feature at serve time

Say these points

AUC is a global-ordering metric; users only see top-k. Use NDCG@k/recall@k.
Offline eval is off-policy — logged data came from the old ranker.
Check SRM and A/A first: validate the experiment before debugging the model.
Training/serving skew is detectable by re-scoring logged features and comparing distributions.
'No lift' can mean 'underpowered' — read the confidence interval.

Avoid these mistakes

Concluding the model is bad without checking experiment integrity.
Ignoring latency and other guardrails as an explanation.
Treating a non-significant result as proof of no effect.

Expect these follow-ups

Design an exploration policy that keeps offline evaluation valid.
How does interleaving compare to a standard A/B test for ranking?
What long-horizon metrics would you add to catch feedback-loop damage?

Advancedmultiple-testingfdrexperimentationAsked at Microsoft, Airbnb

30-second answer

100 comparisons at α = 0.05 yield about 5 false positives even when nothing works. Pre-register one primary metric, treat the rest as guardrails or exploratory, and apply Benjamini–Hochberg FDR control to the exploratory family. Segment findings must be pre-registered or treated as hypothesis-generating and confirmed in a follow-up test.

P (at least one false positive) = 1 - (1 - α)^{m} = 1 - 0.9 5^{100} \approx 0.994

With 100 independent comparisons you are essentially guaranteed at least one spurious 'significant' result. If your process is 'run the test, slice until something is green, write the doc', you have built a random-result generator with a convincing narrative layer on top.

Three correction strategies

Method	Controls	Use when
Bonferroni (α/m)	Family-wise error rate	Few comparisons, and a false positive is expensive (safety, medical, billing)
Benjamini–Hochberg	False discovery rate	Many exploratory comparisons; you accept that q% of your discoveries are wrong
Hierarchical / gatekeeping	FWER with an ordering	Primary metric must pass before secondary metrics are even tested

pythonBenjamini–Hochberg from first principles

import numpy as np

def benjamini_hochberg(pvals, q=0.10):
    """Return a boolean mask of discoveries controlling FDR at level q."""
    p = np.asarray(pvals)
    m = len(p)
    order = np.argsort(p)
    ranked = p[order]
    # Largest k such that p_(k) <= (k/m) * q
    thresholds = (np.arange(1, m + 1) / m) * q
    passed = ranked <= thresholds
    k = np.max(np.where(passed)[0]) + 1 if passed.any() else 0
    mask = np.zeros(m, dtype=bool)
    mask[order[:k]] = True
    return mask

pvals = [0.001, 0.008, 0.021, 0.03, 0.04, 0.12, 0.31, 0.47, 0.62, 0.88]
print("Uncorrected @0.05 :", sum(p < 0.05 for p in pvals), "discoveries")
print("Bonferroni  @0.05 :", sum(p < 0.05/len(pvals) for p in pvals), "discoveries")
print("BH FDR      @0.10 :", benjamini_hochberg(pvals, 0.10).sum(), "discoveries")

# Uncorrected @0.05 : 5 discoveries
# Bonferroni  @0.05 : 1 discoveries      <- very conservative
# BH FDR      @0.10 : 3 discoveries      <- good power/error balance

The organisational fix matters more than the maths

Pre-register a single primary metric
Written down before launch, with the MDE and the ship/no-ship rule. Everything else is guardrail or exploratory. This alone eliminates most of the problem.
Guardrails use one-sided non-inferiority tests
You are not asking 'did latency improve?' but 'can I rule out a >2% regression?'. Different question, different test, and no multiplicity inflation from fishing.
Segments are hypothesis-generating only
An unplanned segment win is a lead, not a result. Confirm it in a dedicated follow-up test targeting that segment. Standard practice at every mature experimentation org.
Automate the correction
Bake BH into the experiment dashboard so nobody has to remember. Culture that depends on discipline decays; tooling that enforces it does not.

Say these points

100 comparisons at α=0.05 → ~99% chance of at least one false positive.
Bonferroni controls FWER (strict); BH controls FDR (practical for exploration).
Pre-register one primary metric; everything else is guardrail or exploratory.
Segment wins are leads to confirm, not results to ship.
Correlated metrics make Bonferroni over-conservative; bootstrap the joint null if you can.

Avoid these mistakes

Slicing until something is significant, then writing the hypothesis afterwards.
Applying Bonferroni to 200 correlated metrics and killing all your power.
Testing guardrails two-sided when the real question is non-inferiority.

Expect these follow-ups

How do you set an FDR level q for a company-wide experiment platform?
Explain a non-inferiority test and how you choose the margin.
How would you detect a sample-ratio mismatch and what causes it?

Intermediateconfidence-intervalsbootstrapevaluationAsked at Google Search, Bing

30-second answer

A 95% CI is a procedure that captures the true parameter in 95% of repeated experiments — not a 95% probability that this particular interval contains it. For metrics without an analytic variance, like NDCG@10 or a custom business metric, use the bootstrap: resample your evaluation units with replacement, recompute the metric thousands of times, and take the 2.5th and 97.5th percentiles.

The correct interpretation is about the procedure, not this interval: if you repeated the whole experiment many times, 95% of the intervals you construct would contain the true parameter. Any given interval either contains it or does not. (The Bayesian credible interval does let you say 'there is a 95% probability the parameter is in here' — but only because it conditions on a prior.)

Bootstrapping a metric with no closed form

NDCG@10 is a non-linear function of ranked lists — there is no textbook standard error. The bootstrap sidesteps that entirely: treat your evaluation set as a stand-in for the population, resample whole query sessions (your independent unit) with replacement, and let the empirical spread of the statistic estimate its sampling distribution.

pythonPaired bootstrap CI for an NDCG@10 difference between two rankers

import numpy as np
rng = np.random.default_rng(0)

def ndcg_at_k(relevances, k=10):
    rel = np.asarray(relevances)[:k]
    disc = 1.0 / np.log2(np.arange(2, len(rel) + 2))
    dcg  = float((rel * disc).sum())
    ideal = np.sort(np.asarray(relevances))[::-1][:k]
    idcg = float((ideal * disc[:len(ideal)]).sum())
    return dcg / idcg if idcg > 0 else 0.0

# per-query NDCG for each system (paired: same queries, both systems)
n_queries = 2000
base = rng.beta(5, 3, n_queries)
new  = np.clip(base + rng.normal(0.012, 0.05, n_queries), 0, 1)

def paired_bootstrap(a, b, n_boot=10_000, alpha=0.05):
    n = len(a)
    diffs = np.empty(n_boot)
    for i in range(n_boot):
        idx = rng.integers(0, n, n)        # resample QUERIES, not documents
        diffs[i] = b[idx].mean() - a[idx].mean()
    lo, hi = np.percentile(diffs, [100*alpha/2, 100*(1-alpha/2)])
    return b.mean() - a.mean(), lo, hi

point, lo, hi = paired_bootstrap(base, new)
print(f"ΔNDCG@10 = {point:+.4f}   95% CI [{lo:+.4f}, {hi:+.4f}]")
print("significant" if lo > 0 else "cannot rule out no improvement")

# ΔNDCG@10 = +0.0118   95% CI [+0.0096, +0.0140]
# significant

Pairing doubles your power for free

Because both rankers see the same queries, the per-query difference cancels out query difficulty — the dominant source of variance. An unpaired comparison on this data would need several times the queries for the same precision. The same idea underlies CUPED in online tests: remove variance you can predict, keep only the variance you care about.

Percentile bootstrap — simple, what you saw above; fine for symmetric statistics.
BCa (bias-corrected and accelerated) — corrects skew and bias; the default in scipy.stats.bootstrap.
Block bootstrap — for time series, resample contiguous blocks so autocorrelation survives.
Permutation test — for a p-value rather than an interval: shuffle labels between arms and see how extreme the observed difference is.

Say these points

A CI is a statement about the procedure across repeats, not this interval.
Lead with the CI: it conveys direction, magnitude and uncertainty in one line.
Bootstrap gives intervals for any metric, including NDCG@k and custom business metrics.
Resample at the randomisation unit (user/session), never at the row level.
Pairing removes item difficulty and dramatically reduces variance.

Avoid these mistakes

Saying 'there is a 95% probability the true value is in this interval' (that is credible, not confidence).
Bootstrapping rows inside correlated sessions and getting absurdly tight intervals.
Using an unpaired test when your design is naturally paired.

Expect these follow-ups

When would you use BCa over the percentile bootstrap?
How does the block bootstrap handle autocorrelated time series?
Explain a permutation test and when it beats the bootstrap.

Statistics & A/B Testing for ML Engineers

What exactly is a p-value, and what are the three most common misinterpretations?

The three misreadings

How do you determine the sample size for an A/B test, and what happens if you peek at the results early?

Why peeking breaks everything

The three legitimate fixes

Your new ranking model improves offline AUC by 3% but the A/B test shows no lift. What are the possible causes?

You track 20 metrics across 5 segments in one experiment. How do you avoid fooling yourself?

Three correction strategies

The organisational fix matters more than the maths

What does a 95% confidence interval mean, and how would you compute one for a metric with no closed form — like NDCG@10?

Bootstrapping a metric with no closed form

Pairing doubles your power for free

Check your understanding

Hands-on challenge

Build an experiment analysis toolkit

Curriculum