Explain precision, recall and F1. When would you optimise for each?

Precision is the fraction of positive predictions that are correct — it answers 'when I flag something, am I right?'. Recall is the fraction of actual positives caught — 'of everything I should have found, how much did I?'. F1 is their harmonic mean. Optimise precision when false positives are expensive (spam filters, auto-blocking payments), recall when false negatives are expensive (cancer screening, fraud triage, safety). Precision = TPTP+FP, Recall = TPTP+FN, F_1 = 2,PRP+R How to never mix them up: Look at the denominator. Precision divides by what you predicted positive — it is about your output. Recall divides by what actually is positive — it is about the world. 'Precision = of my flags, how many were right. Recall = of the real ones, how many did I flag.' Domain | Costly error | Optimise | Why. Cancer screening | Missing a tumour (FN) | Recall, at ~95%+ | A false positive costs a follow-up scan; a false negative costs a life. Spam filter | Blocking real mail (FP) | Precision | One lost job offer is worse than ten spam emails in the inbox. Fraud triage queue | Depends on stage | High recall at stage 1, high precision at stage 2 | Cheap model catches everything; expensive review decides. Auto-blocking payments | Blocking a good customer (FP) | Precision, very high | A blocked legitimate purchase can permanently lose a customer. Content moderation | Both | Precision on auto-remove, recall on human queue | Different thresholds for different actions on the same score Why F1 is usually the wrong default F1 weights precision and recall equally, which is almost never what your business wants. If a false negative costs 10× a false positive, use F_ with = 10 3.16: F_ = (1+ ^2), P R ^2 P + R, > 1 favours recall Better still, skip F-scores and optimise expected cost d

When is ROC-AUC misleading, and why does PR-AUC behave differently under class imbalance?

ROC-AUC uses the false positive rate, whose denominator is the (huge) negative count, so thousands of extra false positives barely move it. PR-AUC's precision has false positives in the denominator against a tiny true-positive count, so it responds sharply. At 1:1000 imbalance a model can show 0.95 ROC-AUC and 0.08 PR-AUC — the second number is the one that describes what a user experiences. ROC: TPR= TPTP+FN vs FPR= FPFP+TN PR: TPTP+FN vs TPTP+FP Everything follows from one asymmetry: TN appears in ROC and never in PR. With 1,000,000 negatives and 1,000 positives, going from 1,000 to 11,000 false positives changes FPR from 0.001 to 0.011 — visually nothing on an ROC curve. But it changes precision at fixed recall from 0.50 to 0.08, a catastrophe your users will feel on every page. ROC-AUC is prevalence-invariant. That is the bug, not the feature.: People praise ROC-AUC for being insensitive to class balance. That is exactly why it cannot tell you whether your fraud queue is 90% junk. Use ROC-AUC to compare models on the same data; use PR-AUC (and precision@k) to describe what users will experience. Reading the numbers correctly The PR baseline is the positive rate. A random model gets PR-AUC ≈ prevalence, not 0.5. So 0.089 against a 0.001 baseline is an 89× lift — genuinely good, and it still means only ~9% of flags are real.. ROC's baseline is always 0.5, which is why it feels comfortable and tells you less.. Precision@k is often the true metric. If analysts can review 200 cases a day, only precision in the top 200 matters. Report precision@200 and recall@200.. PR curves are not linearly interpolatable — connecting points with straight lines overestimates area. average_precision_score computes the correct step-wise sum; auc(recall, precision) does not. The o

What is model calibration, why does it matter, and how do you fix a miscalibrated model?

A calibrated model's predicted probabilities match observed frequencies: among all cases scored 0.7, about 70% should be positive. It matters whenever a score feeds an expected-value calculation — pricing, bidding, ranking by expected revenue, or thresholding on cost. Fix it post-hoc with Platt scaling (sigmoid) or isotonic regression, fitted on a held-out set, and measure with ECE and a reliability diagram. Discrimination and calibration are independent. A model that outputs 0.5 + 0.001 × true_score ranks perfectly (AUC 1.0) and is hopelessly miscalibrated. A model that always outputs the base rate is perfectly calibrated and useless. You need both, and AUC only measures the first. Model family | Typical calibration | Why. Logistic regression | Well calibrated | It directly optimises log-loss, a proper scoring rule. Neural nets (modern, large) | Overconfident | Trained to near-zero loss; softmax saturates. Random forest | Under-confident at extremes | Averaging pulls predictions toward 0.5. Gradient boosting (log-loss) | Reasonable, often slightly overconfident | Optimises log-loss but early stopping and shrinkage distort it. SVM (decision_function) | Not probabilities at all | Output is a margin, not a likelihood. Any model trained on resampled data | Badly miscalibrated | Calibrated to the resampled prevalence, not the real one Measuring it ECE = _b=1^B |B_b|N, |, acc(B_b) - conf(B_b), | Calibrate on data the model has never seen: Fitting the calibrator on training data teaches it to correct errors that do not exist on new data. Use a dedicated calibration split, or CalibratedClassifierCV with internal CV. And re-calibrate after any change to the model, the features, or the population — calibration drifts long before discrimination does, which makes ECE an excell

Compare RMSE, MAE, MAPE and R². Which would you use for demand forecasting?

RMSE penalises large errors quadratically and targets the conditional mean; MAE is robust to outliers and targets the median; MAPE is scale-free but explodes near zero and asymmetrically punishes over-prediction; R² is a relative measure that flatters models on high-variance data. For demand forecasting I would use MAE or weighted MAE for the headline, plus a pinball loss if the downstream decision is inventory, since that needs a quantile rather than a mean. Metric | Formula | Estimates | Watch out for. RMSE | 1n (y- y)^2 | Conditional mean | Dominated by outliers. MAE | 1n |y- y| | Conditional median | Ignores error magnitude structure. MAPE | 100n | y- yy | | — | Undefined at y=0; asymmetric; unbounded above. SMAPE | symmetric variant | — | Still unstable near zero. R² | 1 - SS_resSS_tot | Variance explained | Depends on target variance, so not comparable across datasets. Pinball / quantile loss | ( e, ( -1)e) | Conditional quantile τ | Requires you to know the cost asymmetry The framing that separates candidates: “Choosing RMSE versus MAE is choosing whether I want the model to predict the mean or the median of the conditional distribution. On skewed demand data those differ a lot, and the right one depends on what the downstream system does with the number.” Very few people say this, and it is exactly right. The demand-forecasting answer in full Ask what the forecast drives: Inventory? Staffing? Financial planning? Inventory has asymmetric costs — a stockout loses a sale and a customer; overstock costs holding and markdown. That asymmetry defines the metric.. Use quantile loss for inventory: The classic newsvendor result: the optimal service level is = c_uc_u + c_o where c_u is understock cost and c_o overstock cost. If a stockout costs 4× the holding cost, fore

How do you evaluate a ranking or recommendation system offline?

Use top-k metrics because users only see the top: precision@k and recall@k for binary relevance, MAP for ordering within the relevant set, MRR when only the first hit matters, and NDCG@k when relevance is graded and position matters. Complement these with beyond-accuracy metrics — coverage, diversity, novelty — and validate offline conclusions with interleaving before a full A/B test. Metric | Answers | Use when. Precision@k | What fraction of the top k is relevant? | Fixed slate size, binary relevance. Recall@k | What fraction of all relevant items made the top k? | Retrieval / candidate generation. MRR | How high is the first relevant item? | Navigational search, QA, autocomplete. MAP@k | Precision averaged over each relevant hit | Multiple relevant items, order matters. NDCG@k | Graded relevance discounted by position | Ratings or multi-level relevance; the industry default. Hit rate@k | Did we get at least one? | Sparse implicit feedback DCG@k = _i=1^k 2^rel_i-1 _2(i+1), NDCG@k = DCG@k IDCG@k Two design choices in that formula are worth calling out. The exponential gain 2^rel-1 means a 'perfect' item is worth far more than two 'good' ones — it deliberately rewards getting the best item to the top. The logarithmic discount encodes the empirical fact that click-through decays roughly logarithmically with rank. Normalising by the ideal DCG makes queries with different numbers of relevant items comparable. The three traps in offline ranking evaluation Position bias in the labels: Your click logs say position 1 is relevant because it was at position 1. Training and evaluating on raw clicks measures your old ranker, not relevance. Fix with an examination model (click = examination × relevance), inverse-propensity weighting, or a small randomised-position b

ML Evaluation Metrics Interview Questions: ROC-AUC, PR-AUC, NDCG

FoundationalmetricsclassificationprecisionAsked at Stripe, Meta Integrity

30-second answer

Precision is the fraction of positive predictions that are correct — it answers 'when I flag something, am I right?'. Recall is the fraction of actual positives caught — 'of everything I should have found, how much did I?'. F1 is their harmonic mean. Optimise precision when false positives are expensive (spam filters, auto-blocking payments), recall when false negatives are expensive (cancer screening, fraud triage, safety).

Precision = \frac{TP}{TP + FP}, Recall = \frac{TP}{TP + FN}, F_{1} = \frac{2 PR}{P + R}

Domain	Costly error	Optimise	Why
Cancer screening	Missing a tumour (FN)	Recall, at ~95%+	A false positive costs a follow-up scan; a false negative costs a life
Spam filter	Blocking real mail (FP)	Precision	One lost job offer is worse than ten spam emails in the inbox
Fraud triage queue	Depends on stage	High recall at stage 1, high precision at stage 2	Cheap model catches everything; expensive review decides
Auto-blocking payments	Blocking a good customer (FP)	Precision, very high	A blocked legitimate purchase can permanently lose a customer
Content moderation	Both	Precision on auto-remove, recall on human queue	Different thresholds for different actions on the same score

Why F1 is usually the wrong default

F1 weights precision and recall equally, which is almost never what your business wants. If a false negative costs 10× a false positive, use $F_{β}$ with $β = 10 \approx 3.16$ :

F_{β} = (1 + β^{2}) \frac{P \cdot R}{β ^{2} P + R}, β > 1 favours recall

Better still, skip F-scores and optimise expected cost directly. F1 is a convenience metric for leaderboards; expected cost is what your company actually experiences.

pythonDerive the threshold from the cost matrix, not from 0.5

import numpy as np
from sklearn.metrics import precision_recall_curve

# Business inputs — get these from your PM, not from a paper.
COST_FP = 5      # analyst review time for a false alarm
COST_FN = 400    # average loss from a missed fraud

def optimal_threshold(y_true, scores, cost_fp=COST_FP, cost_fn=COST_FN):
    thresholds = np.unique(scores)
    best = (None, np.inf, None)
    for t in thresholds:
        pred = scores >= t
        fp = int(((pred == 1) & (y_true == 0)).sum())
        fn = int(((pred == 0) & (y_true == 1)).sum())
        cost = fp * cost_fp + fn * cost_fn
        if cost < best[1]:
            best = (t, cost, (fp, fn))
    return best

t, cost, (fp, fn) = optimal_threshold(y_val, scores_val)
prec, rec, thr = precision_recall_curve(y_val, scores_val)
i = int(np.searchsorted(thr, t))

print(f"optimal threshold : {t:.3f}   (NOT 0.5)")
print(f"expected cost     : ${cost:,}   FP={fp}  FN={fn}")
print(f"operating point   : precision={prec[i]:.3f}  recall={rec[i]:.3f}")

# optimal threshold : 0.083   (NOT 0.5)
# expected cost     : $41,215   FP=1843  FN=80
# operating point   : precision=0.214  recall=0.906
#
# 21% precision looks terrible on a leaderboard and is exactly right here:
# reviewing 5 false alarms is far cheaper than missing one $400 fraud.

Finally, be precise about averaging in the multi-class case. Macro averages the per-class metric equally, so rare classes count as much as common ones. Micro pools all TP/FP/FN, so it is dominated by the frequent classes (and equals accuracy in single-label multi-class). Weighted averages by support. Reporting 'F1 = 0.82' without saying which average is an incomplete answer.

Say these points

Precision divides by predicted positives; recall divides by actual positives.
Choose based on the relative cost of FP vs FN — ask before answering.
F1's equal weighting is rarely what the business wants; F_β or expected cost is better.
Score and threshold are separate artefacts with separate owners.
Always state macro vs micro vs weighted averaging in multi-class.

Avoid these mistakes

Defaulting to threshold 0.5 with no justification.
Reporting F1 without stating the averaging scheme.
Optimising a metric that nobody in the business recognises.

Expect these follow-ups

How do you set thresholds when the cost of an error varies per example?
Your recall target is 95%. How do you report the model's quality at that fixed point?
What is precision@k and when is it the right metric?

Advancedroc-aucpr-aucimbalancedAsked at Stripe, PayPal

30-second answer

ROC-AUC uses the false positive rate, whose denominator is the (huge) negative count, so thousands of extra false positives barely move it. PR-AUC's precision has false positives in the denominator against a tiny true-positive count, so it responds sharply. At 1:1000 imbalance a model can show 0.95 ROC-AUC and 0.08 PR-AUC — the second number is the one that describes what a user experiences.

ROC: TPR = \frac{TP}{TP + FN} vs FPR = \frac{FP}{FP + TN} PR: \frac{TP}{TP + FN} vs \frac{TP}{TP + FP}

Everything follows from one asymmetry: TN appears in ROC and never in PR. With 1,000,000 negatives and 1,000 positives, going from 1,000 to 11,000 false positives changes FPR from 0.001 to 0.011 — visually nothing on an ROC curve. But it changes precision at fixed recall from 0.50 to 0.08, a catastrophe your users will feel on every page.

pythonThe same model, two very different stories

import numpy as np
from sklearn.metrics import roc_auc_score, average_precision_score

rng = np.random.default_rng(0)

def simulate(n_neg, n_pos, separation=2.0):
    neg = rng.normal(0.0, 1.0, n_neg)
    pos = rng.normal(separation, 1.0, n_pos)
    y = np.r_[np.zeros(n_neg), np.ones(n_pos)]
    s = np.r_[neg, pos]
    return y, s

print(f"{'ratio':>12} {'ROC-AUC':>9} {'PR-AUC':>8} {'baseline PR':>12}")
for n_neg, n_pos in [(5_000, 5_000), (50_000, 500), (500_000, 500), (1_000_000, 200)]:
    y, s = simulate(n_neg, n_pos)
    print(f"{n_neg//n_pos:>10}:1 {roc_auc_score(y, s):>9.3f}"
          f" {average_precision_score(y, s):>8.3f} {n_pos/(n_pos+n_neg):>12.4f}")

#        ratio   ROC-AUC   PR-AUC  baseline PR
#          1:1     0.921    0.919       0.5000
#        100:1     0.922    0.348       0.0099
#       1000:1     0.921    0.089       0.0010
#       5000:1     0.923    0.041       0.0002
#
# ROC-AUC is FLAT across four orders of magnitude of imbalance.
# PR-AUC collapses -- because the user-facing experience collapses.

Reading the numbers correctly

The PR baseline is the positive rate. A random model gets PR-AUC ≈ prevalence, not 0.5. So 0.089 against a 0.001 baseline is an 89× lift — genuinely good, and it still means only ~9% of flags are real.
ROC's baseline is always 0.5, which is why it feels comfortable and tells you less.
Precision@k is often the true metric. If analysts can review 200 cases a day, only precision in the top 200 matters. Report precision@200 and recall@200.
PR curves are not linearly interpolatable — connecting points with straight lines overestimates area. average_precision_score computes the correct step-wise sum; auc(recall, precision) does not.

Two related metrics worth having ready: ROC-AUC has a clean probabilistic reading — it equals $P (score of a random positive > score of a random negative)$ , which is also the Mann–Whitney U statistic. And for heavily imbalanced ranking problems, partial AUC restricted to the low-FPR region (say FPR < 0.01) is a reasonable compromise when stakeholders insist on an ROC-family number.

Say these points

TN appears in ROC and never in PR — that single asymmetry explains everything.
ROC-AUC is nearly invariant to imbalance; PR-AUC is not.
The PR baseline is the positive rate, so judge PR-AUC as a lift over prevalence.
Use precision@k when review capacity is the real constraint.
Use average_precision_score, not auc(recall, precision).

Avoid these mistakes

Reporting 0.95 ROC-AUC on a 1:10,000 problem as if it were good news.
Comparing PR-AUC across datasets with different prevalence.
Linearly interpolating a PR curve.

Expect these follow-ups

Your PR-AUC is 0.089 with a 0.001 baseline. Is that a good model? Defend it.
How would you report performance when the review team can only handle 200 cases a day?
Explain the Mann–Whitney interpretation of ROC-AUC.

AdvancedcalibrationprobabilitiesproductionAsked at Google Ads, Criteo

30-second answer

A calibrated model's predicted probabilities match observed frequencies: among all cases scored 0.7, about 70% should be positive. It matters whenever a score feeds an expected-value calculation — pricing, bidding, ranking by expected revenue, or thresholding on cost. Fix it post-hoc with Platt scaling (sigmoid) or isotonic regression, fitted on a held-out set, and measure with ECE and a reliability diagram.

Discrimination and calibration are independent. A model that outputs 0.5 + 0.001 × true_score ranks perfectly (AUC 1.0) and is hopelessly miscalibrated. A model that always outputs the base rate is perfectly calibrated and useless. You need both, and AUC only measures the first.

Model family	Typical calibration	Why
Logistic regression	Well calibrated	It directly optimises log-loss, a proper scoring rule
Neural nets (modern, large)	Overconfident	Trained to near-zero loss; softmax saturates
Random forest	Under-confident at extremes	Averaging pulls predictions toward 0.5
Gradient boosting (log-loss)	Reasonable, often slightly overconfident	Optimises log-loss but early stopping and shrinkage distort it
SVM (decision_function)	Not probabilities at all	Output is a margin, not a likelihood
Any model trained on resampled data	Badly miscalibrated	Calibrated to the resampled prevalence, not the real one

Measuring it

ECE = b = 1 \sum B \frac{∣ B _{b} ∣}{N} acc (B_{b}) - conf (B_{b})

Expected calibration error: bin predictions, compare mean confidence to observed accuracy in each bin, weight by bin size.

pythonReliability diagram, ECE, and both repair methods

import numpy as np
from sklearn.calibration import CalibratedClassifierCV
from sklearn.metrics import brier_score_loss, log_loss

def expected_calibration_error(y_true, prob, n_bins=10):
    edges = np.linspace(0, 1, n_bins + 1)
    idx = np.clip(np.digitize(prob, edges) - 1, 0, n_bins - 1)
    ece, rows = 0.0, []
    for b in range(n_bins):
        m = idx == b
        if not m.any():
            continue
        conf, acc, w = prob[m].mean(), y_true[m].mean(), m.mean()
        ece += w * abs(acc - conf)
        rows.append((f"[{edges[b]:.1f},{edges[b+1]:.1f})", int(m.sum()), conf, acc))
    return ece, rows

ece, rows = expected_calibration_error(y_val, raw_probs)
print(f"{'bin':>12} {'n':>6} {'predicted':>10} {'observed':>9}")
for b, n, conf, acc in rows:
    print(f"{b:>12} {n:>6} {conf:>10.3f} {acc:>9.3f}")
print(f"ECE = {ece:.4f}")

#          bin      n  predicted  observed
#   [0.0,0.1)   6210      0.021     0.048   <- underconfident at the low end
#   [0.4,0.5)    381      0.447     0.402
#   [0.9,1.0)    712      0.961     0.804   <- OVERCONFIDENT at the top: 96% -> 80%
# ECE = 0.0731

# Platt (sigmoid): 2 parameters, works with little calibration data
platt = CalibratedClassifierCV(base_model, method="sigmoid", cv="prefit")
platt.fit(X_cal, y_cal)

# Isotonic: non-parametric & monotone, more flexible, needs >~1000 calibration rows
iso = CalibratedClassifierCV(base_model, method="isotonic", cv="prefit")
iso.fit(X_cal, y_cal)

for name, m in [("raw", base_model), ("platt", platt), ("isotonic", iso)]:
    p = m.predict_proba(X_test)[:, 1]
    e, _ = expected_calibration_error(y_test, p)
    print(f"{name:9s} ECE={e:.4f}  Brier={brier_score_loss(y_test, p):.4f}"
          f"  logloss={log_loss(y_test, p):.4f}")

# raw       ECE=0.0731  Brier=0.0894  logloss=0.3120
# platt     ECE=0.0121  Brier=0.0831  logloss=0.2874
# isotonic  ECE=0.0089  Brier=0.0826  logloss=0.2861

When calibration is non-negotiable

Expected value decisions — ad bidding pays P(click) × value. A 2× overconfidence is a 2× overspend.
Cost-based thresholds — the optimal threshold formula assumes the score is a probability.
Combining models — averaging or stacking miscalibrated scores mixes incompatible scales.
Human-in-the-loop — reviewers learn what '80% confident' should mean and lose trust when it means 55%.
Regulated decisions — credit and insurance require probabilities that mean what they say.

Say these points

Calibration and discrimination are independent; AUC measures only discrimination.
Measure with reliability diagrams, ECE and Brier score.
Platt for small calibration sets, isotonic for large, temperature scaling for deep nets.
Always calibrate on held-out data, and recalibrate after any change.
Resampling for imbalance destroys calibration — recalibrate or apply prior correction.

Avoid these mistakes

Assuming a softmax output is a probability.
Fitting the calibrator on training data.
Reporting only AUC for a model whose scores drive money.

Expect these follow-ups

Explain temperature scaling and why it cannot change accuracy.
How does class rebalancing affect calibration, and what is the prior-correction formula?
Why is Brier score a proper scoring rule and accuracy not?

IntermediateregressionmetricsforecastingAsked at Amazon Supply Chain, Instacart

30-second answer

RMSE penalises large errors quadratically and targets the conditional mean; MAE is robust to outliers and targets the median; MAPE is scale-free but explodes near zero and asymmetrically punishes over-prediction; R² is a relative measure that flatters models on high-variance data. For demand forecasting I would use MAE or weighted MAE for the headline, plus a pinball loss if the downstream decision is inventory, since that needs a quantile rather than a mean.

Metric	Formula	Estimates	Watch out for
RMSE	$\frac{1}{n} \sum (y - \overset{y}{^})^{2}$	Conditional mean	Dominated by outliers
MAE	$\frac{1}{n} \sum ∣ y - \overset{y}{^} ∣$	Conditional median	Ignores error magnitude structure
MAPE	$\frac{100}{n} \sum \frac{y - y ^}{y}$	—	Undefined at y=0; asymmetric; unbounded above
SMAPE	symmetric variant	—	Still unstable near zero
R²	$1 - \frac{S S _{res}}{S S _{t o t}}$	Variance explained	Depends on target variance, so not comparable across datasets
Pinball / quantile loss	$max (τ e, (τ - 1) e)$	Conditional quantile τ	Requires you to know the cost asymmetry

pythonThe same data, different optima

import numpy as np
from scipy.optimize import minimize_scalar

rng = np.random.default_rng(0)
# Right-skewed demand: mostly small, occasional huge spike
demand = np.r_[rng.poisson(20, 950), rng.poisson(400, 50)]

rmse = lambda c: np.sqrt(((demand - c) ** 2).mean())
mae  = lambda c: np.abs(demand - c).mean()

best_rmse = minimize_scalar(rmse, bounds=(0, 500), method="bounded").x
best_mae  = minimize_scalar(mae,  bounds=(0, 500), method="bounded").x

print(f"mean   = {demand.mean():7.2f}   optimal constant under RMSE = {best_rmse:7.2f}")
print(f"median = {np.median(demand):7.2f}   optimal constant under MAE  = {best_mae:7.2f}")

# mean   =   39.02   optimal constant under RMSE =   39.02
# median =   20.00   optimal constant under MAE  =   20.00
#
# Nearly 2x apart. If you stock to the RMSE answer you carry 95% excess
# inventory on a typical day; if you stock to the MAE answer you stock out
# on every spike. NEITHER is right -- inventory needs a QUANTILE.

def pinball(y, pred, tau):
    e = y - pred
    return np.mean(np.maximum(tau * e, (tau - 1) * e))

for tau in (0.5, 0.9, 0.95):
    q = np.quantile(demand, tau)
    print(f"tau={tau}: stock {q:6.1f}  pinball={pinball(demand, q, tau):.3f}")

# tau=0.5: stock   20.0  pinball=9.512
# tau=0.9: stock  355.1  pinball=6.284
# tau=0.95: stock 391.0  pinball=4.017

The demand-forecasting answer in full

Ask what the forecast drives
Inventory? Staffing? Financial planning? Inventory has asymmetric costs — a stockout loses a sale and a customer; overstock costs holding and markdown. That asymmetry defines the metric.
Use quantile loss for inventory
The classic newsvendor result: the optimal service level is $τ = \frac{c _{u}}{c _{u} + c _{o}}$ where $c_{u}$ is understock cost and $c_{o}$ overstock cost. If a stockout costs 4× the holding cost, forecast the 80th percentile, not the mean.
Weight by business value
A 10-unit error on a high-margin SKU is not a 10-unit error on a low-margin one. Weighted MAE (weights = margin or revenue) aligns the metric with the P&L.
Avoid MAPE on intermittent demand
Half of retail SKUs have zero demand on a given day. MAPE is undefined there and explodes at 1 unit. Use MASE (scaled against a seasonal-naive baseline) or WAPE (= total absolute error / total actual) instead.
Always benchmark against seasonal naive
A model that cannot beat 'same day last week' is not a model. MASE < 1 is the minimum bar, and reporting it pre-empts the interviewer's next question.

Say these points

RMSE targets the conditional mean, MAE the median — pick the one the decision needs.
MAPE is undefined at zero and biases forecasts downward; use WAPE or MASE.
R² is relative to target variance and is not comparable across datasets.
Inventory decisions need a quantile: τ = c_u / (c_u + c_o), optimised with pinball loss.
Weight the metric by business value and always beat seasonal naive.

Avoid these mistakes

Reporting MAPE on intermittent/zero-heavy demand.
Comparing R² across different product categories.
Forecasting the mean when the downstream system needs a service level.

Expect these follow-ups

Derive the newsvendor optimal quantile.
How would you produce prediction intervals rather than point forecasts?
What is MASE and why is it preferred in forecasting competitions?

AdvancedrankingndcgrecsysAsked at Google Search, Spotify

30-second answer

Use top-k metrics because users only see the top: precision@k and recall@k for binary relevance, MAP for ordering within the relevant set, MRR when only the first hit matters, and NDCG@k when relevance is graded and position matters. Complement these with beyond-accuracy metrics — coverage, diversity, novelty — and validate offline conclusions with interleaving before a full A/B test.

Metric	Answers	Use when
Precision@k	What fraction of the top k is relevant?	Fixed slate size, binary relevance
Recall@k	What fraction of all relevant items made the top k?	Retrieval / candidate generation
MRR	How high is the first relevant item?	Navigational search, QA, autocomplete
MAP@k	Precision averaged over each relevant hit	Multiple relevant items, order matters
NDCG@k	Graded relevance discounted by position	Ratings or multi-level relevance; the industry default
Hit rate@k	Did we get at least one?	Sparse implicit feedback

DCG@k = i = 1 \sum k \frac{2 ^{re l_{i}} - 1}{lo g _{2} ( i + 1 )}, NDCG@k = \frac{DCG@k}{IDCG@k}

Two design choices in that formula are worth calling out. The exponential gain $2^{re l} - 1$ means a 'perfect' item is worth far more than two 'good' ones — it deliberately rewards getting the best item to the top. The logarithmic discount encodes the empirical fact that click-through decays roughly logarithmically with rank. Normalising by the ideal DCG makes queries with different numbers of relevant items comparable.

pythonThe core ranking metrics from scratch

import numpy as np

def dcg(rels, k):
    rels = np.asarray(rels, dtype=float)[:k]
    return float(((2 ** rels - 1) / np.log2(np.arange(2, len(rels) + 2))).sum())

def ndcg_at_k(rels, k=10):
    ideal = dcg(sorted(rels, reverse=True), k)
    return dcg(rels, k) / ideal if ideal > 0 else 0.0

def mrr(rels):
    hits = np.nonzero(np.asarray(rels) > 0)[0]
    return 1.0 / (hits[0] + 1) if len(hits) else 0.0

def average_precision(rels, k=10):
    rels = np.asarray(rels)[:k]
    hits, score = 0, 0.0
    for i, r in enumerate(rels, start=1):
        if r > 0:
            hits += 1
            score += hits / i
    return score / max(1, (np.asarray(rels) > 0).sum())

a = [3, 2, 3, 0, 1, 2]      # good stuff at the top
b = [0, 1, 2, 3, 3, 2]      # same items, reversed quality order
print(f"ranking A: NDCG@6={ndcg_at_k(a,6):.4f}  MRR={mrr(a):.3f}  MAP={average_precision(a):.3f}")
print(f"ranking B: NDCG@6={ndcg_at_k(b,6):.4f}  MRR={mrr(b):.3f}  MAP={average_precision(b):.3f}")

# ranking A: NDCG@6=0.9491  MRR=1.000  MAP=0.797
# ranking B: NDCG@6=0.7269  MRR=0.500  MAP=0.629
# Identical item sets, very different user experience -- exactly what these
# metrics are designed to capture and what plain AUC cannot.

The three traps in offline ranking evaluation

Position bias in the labels
Your click logs say position 1 is relevant because it was at position 1. Training and evaluating on raw clicks measures your old ranker, not relevance. Fix with an examination model (click = examination × relevance), inverse-propensity weighting, or a small randomised-position bucket to estimate the propensities.
Missing-not-at-random labels
You only have feedback on items the old system showed. A genuinely better item that was never surfaced scores zero. This is why offline gains so often fail to reproduce online, and why an exploration bucket is infrastructure, not a nice-to-have.
Accuracy is not the whole product
A recommender that shows the same five blockbusters to everyone can win on NDCG and destroy long-term engagement. Track catalogue coverage (fraction of items ever recommended), intra-list diversity, novelty (inverse popularity), and serendipity alongside accuracy.

Finally, be explicit about the split. Ranking data must be split by time (train on the past, evaluate on the future) and by user where appropriate. A random row split leaks a user's future interactions into their training history and inflates every metric above.

Say these points

Users see the top-k, so evaluate at k: precision@k, recall@k, MRR, MAP, NDCG@k.
NDCG's exponential gain rewards putting the best item first; the log discount models click decay.
Position bias and missing-not-at-random labels are the core offline evaluation traps.
Track coverage, diversity and novelty alongside accuracy.
Interleaving bridges offline and A/B testing with far less traffic.

Avoid these mistakes

Evaluating a ranker with global AUC instead of a top-k metric.
Training on raw clicks without correcting for position bias.
Random row splits that leak a user's future behaviour into training.

Expect these follow-ups

Design an exploration policy that gives you unbiased evaluation data.
How does team-draft interleaving attribute credit fairly?
How would you measure recommendation diversity quantitatively?

Evaluation Metrics: Choosing and Defending the Right One

Explain precision, recall and F1. When would you optimise for each?

Why F1 is usually the wrong default

When is ROC-AUC misleading, and why does PR-AUC behave differently under class imbalance?

Reading the numbers correctly

What is model calibration, why does it matter, and how do you fix a miscalibrated model?

Measuring it

When calibration is non-negotiable

Compare RMSE, MAE, MAPE and R². Which would you use for demand forecasting?

The demand-forecasting answer in full

How do you evaluate a ranking or recommendation system offline?

The three traps in offline ranking evaluation

Check your understanding

Hands-on challenge

Build a metric-selection report

Curriculum