ML System DesignExpert20 min read5 questions
Search, Ranking & Feed Design Case Studies
Four more worked designs covering the surfaces most companies actually build: search, feeds, ads and moderation — each with the trap the interviewer is waiting for.
Covers: Learning to rank, search architecture, feed ranking, ads CTR prediction, content moderation at scale
Recommendation is the classic ML system design question, but search ranking, feed ranking, ads and moderation come up just as often — and each has a distinctive trap. This lesson works through those traps so you recognise them under time pressure.
Filter
0/5 mastered
Expertsearchlearning-to-ranksystem-designAsked at Amazon, Etsy
30-second answer
Query understanding (spell correction, intent, attribute extraction), retrieval (inverted index for lexical + vector index for semantic, fused), then learning-to-rank over the candidates using query, product, query-product and behavioural features. Optimise NDCG offline but the real objective is conversion, and the key trap is position bias in the click logs used for training.
Pipeline
| Stage | What it does | Latency |
|---|---|---|
| Query understanding | Spell correction, tokenisation, intent, attribute extraction | ~15 ms |
| Retrieval | BM25 over the inverted index + ANN over embeddings, fused | ~30 ms |
| Filtering | In-stock, region, price band, category constraints | ~5 ms |
| Ranking (LTR) | Score ~500 candidates with a GBDT/DNN ranker | ~40 ms |
| Re-ranking | Business rules, diversity, sponsored blending | ~10 ms |
pythonFeature groups for a learning-to-rank model
def ltr_features(query, product, user, context):
return {
# --- Query features (same for all products in a request) ---
"query_length": len(query.tokens),
"query_frequency": query.log_frequency, # head vs tail query
"query_has_brand": int(query.brand is not None),
"query_intent": query.intent_id, # browse | specific | comparison
# --- Product features (precomputable) ---
"price": product.price,
"price_percentile_in_category": product.price_pct,
"rating": product.rating,
"n_reviews_log": product.log_reviews,
"conversion_rate_30d": product.cvr_30d, # strong, but a feedback loop
"in_stock": int(product.in_stock),
"days_to_deliver": product.delivery_days,
# --- Query-product interaction (the real signal) ---
"bm25_title": bm25(query, product.title),
"bm25_description": bm25(query, product.description),
"embedding_cosine": cosine(query.vector, product.vector),
"exact_attr_matches": count_attr_matches(query.attributes, product.attributes),
"brand_match": int(query.brand == product.brand),
"historical_ctr_for_query": ctr_lookup(query.id, product.id), # sparse but potent
# --- User personalisation ---
"user_category_affinity": user.affinity.get(product.category, 0.0),
"user_price_band_match": price_band_match(user.avg_order_value, product.price),
"previously_viewed": int(product.id in user.recent_views),
# --- Context ---
"is_mobile": int(context.device == "mobile"),
"position": context.position, # TRAINING ONLY -- see position bias below
}pythonTwo-tower position-bias correction
import torch, torch.nn as nn
class UnbiasedRanker(nn.Module):
"""click = P(examined | position) x P(relevant | query, product).
Position feeds ONLY the examination tower and is dropped at serving."""
def __init__(self, n_features, n_positions=50):
super().__init__()
self.relevance = nn.Sequential(
nn.Linear(n_features, 256), nn.ReLU(),
nn.Linear(256, 64), nn.ReLU(), nn.Linear(64, 1))
self.examination = nn.Embedding(n_positions, 1) # learned position propensity
def forward(self, features, positions=None):
rel = self.relevance(features)
if positions is None: # SERVING: relevance only
return rel
return rel + self.examination(positions) # TRAINING: additive in logit space
# After training, examination(0) >> examination(9): the model has absorbed
# the position effect into a separate parameter, leaving `relevance` clean.
# Serving with relevance alone is what breaks the feedback loop.- Head vs tail queries. The top 1,000 queries may be 40% of traffic — cache their results and hand-tune them. Tail queries need semantic retrieval because lexical matching fails on unusual phrasing.
- Zero-result queries are a conversion killer. Fall back progressively: relax filters, drop the least important token, then semantic-only retrieval, and always show something with an explanation.
- Sponsored blending. Ads must be ranked jointly with organic results under a relevance floor, or you degrade the experience for short-term revenue. Model it as an auction with a quality score.
- Freshness and availability must be near-real-time: showing an out-of-stock product at position 1 is worse than not showing it at all.
Say these points
- Query understanding → hybrid retrieval → LTR → re-rank, with a ~100 ms budget.
- E-commerce optimises conversion, not pure relevance; attributes should become filters.
- Position bias is the central trap — correct with an examination model or IPW.
- Serve with the relevance tower only; position exists solely for training.
- Handle head/tail queries, zero-results and sponsored blending explicitly.
Avoid these mistakes
- Training on raw clicks with position as a feature and serving it too.
- Ignoring availability and delivery time in a conversion-optimising ranker.
- Treating tail queries with the same lexical-only retrieval as head queries.
Expect these follow-ups
- How would you evaluate a ranking change with far less traffic than an A/B needs?
- How do you personalise without creating a filter bubble on a shopping site?
- How would you handle a merchant keyword-stuffing their titles?
Expertfeedrankingsystem-designAsked at Meta, LinkedIn
30-second answer
There is no query, so intent must be inferred from context and history; recency matters enormously; the objective is multi-dimensional (engagement, satisfaction, creator ecosystem health, integrity); and the feedback loop is severe because the feed shapes the behaviour you learn from. Architecture is still a funnel — candidate sources, a multi-task ranker, then a re-ranking layer applying diversity, integrity and pacing rules.
| Search | Feed | |
|---|---|---|
| User intent | Explicit in the query | Inferred from history and context |
| Recency | Usually secondary | Often dominant |
| Candidate pool | Matches the query | Everything the user could see |
| Objective | Relevance → conversion | Multi-objective, with integrity constraints |
| Session effects | Weak | Strong — the feed shapes the next session |
| Diversity | Nice to have | Essential — a monotonous feed kills retention |
pythonBlending candidate sources with per-source quotas
CANDIDATE_SOURCES = {
"following": {"quota": 400, "weight": 1.00}, # people they follow
"collaborative": {"quota": 300, "weight": 0.90}, # similar-user signals
"content_similar": {"quota": 200, "weight": 0.85}, # similar to recent engagement
"trending_network": {"quota": 150, "weight": 0.80}, # popular in their graph
"exploration": {"quota": 100, "weight": 0.60}, # deliberately outside the bubble
"fresh_creators": {"quota": 50, "weight": 0.70}, # ecosystem health
}
def generate_candidates(user, ctx):
out, seen = [], set()
for name, cfg in CANDIDATE_SOURCES.items():
for item in SOURCES[name].fetch(user, ctx, limit=cfg["quota"]):
if item.id in seen or item.id in user.recently_seen:
continue
seen.add(item.id)
item.source, item.source_weight = name, cfg["weight"]
out.append(item)
return out
# The exploration and fresh_creators quotas are DELIBERATE inefficiency.
# Without them the feed converges on what the model already believes, the
# creator ecosystem starves, and you never learn about new interests.pythonValue model with negative signals and pacing
def feed_value(preds, item, user, weights):
"""Positive engagement minus explicit dissatisfaction, plus ecosystem terms."""
positive = (weights["click"] * preds["p_click"]
+ weights["dwell"] * preds["p_long_dwell"]
+ weights["like"] * preds["p_like"]
+ weights["comment"] * preds["p_comment"]
+ weights["share"] * preds["p_share"])
# Negative signals are weighted HEAVILY -- one hide is worth many likes
negative = (weights["hide"] * preds["p_hide"]
+ weights["report"] * preds["p_report"]
+ weights["unfollow"] * preds["p_unfollow"])
score = positive - negative
# Freshness decay -- feeds are recency-sensitive in a way search is not
score *= 0.5 ** (item.age_hours / weights["freshness_half_life_hours"])
# Pacing: dampen sources the user has already seen a lot of this session
score *= 1.0 / (1.0 + weights["pacing"] * user.session_counts.get(item.source, 0))
# Integrity is a HARD constraint, not a weight
if item.integrity_score < INTEGRITY_FLOOR:
return float("-inf")
return score
WEIGHTS = {"click":0.3,"dwell":1.0,"like":0.5,"comment":1.5,"share":2.0,
"hide":-8.0,"report":-40.0,"unfollow":-25.0,
"freshness_half_life_hours":18,"pacing":0.35}
# Note the asymmetry: one report outweighs 20 shares. That asymmetry is
# what keeps engagement optimisation from degrading the product.Two implementation notes worth raising: near-real-time features matter far more here than in search — what the user engaged with 30 seconds ago is highly predictive, so you need a streaming feature path with sub-minute latency. And precompute where possible: a partial feed can be built asynchronously after each session and refreshed on open, so the p99 request latency is a cache read rather than a full ranking pass.
Say these points
- No query means intent is inferred; recency and diversity matter far more.
- Blend multiple candidate sources with deliberate exploration and fresh-creator quotas.
- Multi-task prediction with product-tuned weights; integrity as a hard constraint.
- Weight negative signals (hide, report, unfollow) heavily and asymmetrically.
- Feedback loops are severe — use exploration, holdbacks and long-horizon metrics.
Avoid these mistakes
- Optimising a single engagement metric.
- No exploration quota, so the feed collapses onto the model's priors.
- Judging success on per-session engagement rather than 30-day retention.
Expect these follow-ups
- How would you measure whether the feed is creating a filter bubble?
- How do you handle a brand-new user with no history at all?
- How would you rank when 40% of candidates are less than an hour old?
ExpertadsctrcalibrationAsked at Google Ads, Meta Ads
30-second answer
Predict calibrated P(click) for every eligible ad, then rank by expected value — bid × pCTR (× pConversion for conversion campaigns) — inside a second-price-style auction. The defining constraints are extreme scale (millions of QPS, billions of sparse features), a hard sub-50 ms budget, and an absolute requirement for calibrated probabilities because those probabilities directly determine what advertisers pay.
pythonAuction mechanics — where the prediction is consumed
def run_auction(eligible_ads, context, reserve_price=0.50):
"""Generalised second price with a quality score."""
scored = []
for ad in eligible_ads:
p_click = ctr_model.predict(ad, context) # MUST be calibrated
quality = quality_score(ad) # landing page, relevance, history
# eCPM: what this ad is worth per 1000 impressions
ecpm = ad.bid * p_click * quality * 1000
if ecpm >= reserve_price:
scored.append((ecpm, ad, p_click, quality))
scored.sort(reverse=True, key=lambda t: t[0])
if not scored:
return None
winner_ecpm, winner, p_click, quality = scored[0]
runner_up_ecpm = scored[1][0] if len(scored) > 1 else reserve_price
# GSP: the winner pays the minimum needed to have beaten the runner-up
price = (runner_up_ecpm / 1000) / max(p_click * quality, 1e-9) + 0.01
return {"ad": winner, "price": min(price, winner.bid), "p_click": p_click}
# If pCTR is 20% too high, `price` is systematically wrong for every advertiser.
# Calibration error here is not a modelling detail -- it is billing accuracy.Handling extreme sparsity and scale
| Challenge | Approach |
|---|---|
| Billions of sparse features (user × ad × context) | Hashing to fixed buckets + learned embeddings; feature crosses |
| Sub-50 ms with thousands of candidates | Two-stage: cheap filter/retrieval, then a ranker on the top few hundred |
| Millions of QPS | Horizontal sharding; embedding tables in a distributed parameter server |
| Rapid distribution change (new campaigns hourly) | Online/incremental learning with frequent model refresh |
| Cold-start ads | Content and advertiser-level priors + a guaranteed exploration budget |
| Delayed conversions | Delayed-feedback models that account for the censoring window |
pythonCalibration monitoring in production
import numpy as np
def calibration_report(predicted, observed_clicks, n_bins=20):
"""The report an ads team looks at every single day."""
edges = np.percentile(predicted, np.linspace(0, 100, n_bins + 1))
rows, total_err = [], 0.0
for i in range(n_bins):
m = (predicted >= edges[i]) & (predicted < edges[i + 1])
if m.sum() < 100:
continue
pred, obs, n = predicted[m].mean(), observed_clicks[m].mean(), int(m.sum())
ratio = pred / max(obs, 1e-9)
total_err += abs(pred - obs) * n
rows.append((f"{edges[i]:.4f}-{edges[i+1]:.4f}", n, pred, obs, ratio))
print(f"{'bucket':>18} {'n':>9} {'pred':>8} {'obs':>8} {'ratio':>7}")
for b, n, p, o, r in rows:
flag = " <-- MISCALIBRATED" if not 0.9 <= r <= 1.1 else ""
print(f"{b:>18} {n:>9,} {p:>8.4f} {o:>8.4f} {r:>7.2f}{flag}")
print(f"weighted absolute calibration error: {total_err/len(predicted):.5f}")
# bucket n pred obs ratio
# 0.0100-0.0180 412,883 0.0142 0.0139 1.02
# 0.0900-0.1400 98,221 0.1103 0.0891 1.24 <-- MISCALIBRATED
# 0.1400-0.9000 41,002 0.2214 0.1702 1.30 <-- MISCALIBRATED
#
# Over-prediction concentrated in the HIGH-pCTR bucket -- exactly the ads that
# win auctions. Advertisers are being overcharged on the inventory that matters.Close with the business framing: “The system's objective is long-run marketplace health — advertiser ROI, user experience and platform revenue together. Over-predicting CTR raises short-term revenue and destroys advertiser trust; showing irrelevant ads raises impressions and destroys user retention. So I'd carry guardrails on advertiser-reported ROAS and on ad-load-adjusted user engagement alongside the revenue metric.”
Say these points
- pCTR feeds an auction, so calibration is a billing-accuracy requirement.
- Rank by expected value (bid × pCTR × quality), price by second-price mechanics.
- Handle sparsity with hashing plus learned embeddings and feature crosses.
- Monitor calibration per bucket daily — miscalibration concentrates in high-pCTR ads.
- Guard revenue with advertiser-ROI and user-experience metrics.
Avoid these mistakes
- Optimising AUC and never checking calibration.
- Ignoring delayed conversions when constructing labels.
- No exploration budget, so new ads can never earn signal.
Expect these follow-ups
- How would you handle a brand-new advertiser with no history?
- How do you prevent a feedback loop where only proven ads ever get shown?
- How would you detect click fraud?
Expertmoderationsafetysystem-designAsked at Meta, TikTok
30-second answer
A tiered pipeline: hash matching for known violating content (instant, exact), fast multimodal classifiers on everything, escalation of uncertain or high-reach content to stronger models, and human review for the ambiguous and appealable tail. The defining constraints are asymmetric error costs that differ per policy, adversarial evasion, cultural and linguistic context, and a legally required appeals path.
| Tier | Coverage | Latency | Action |
|---|---|---|---|
| Hash matching (PhotoDNA/PDQ) | 100% of media | < 10 ms | Auto-remove known violating content |
| Fast classifiers | 100% of posts | < 100 ms | Auto-remove at high confidence; score everything |
| Heavy multimodal models | ~2% (uncertain or high-reach) | ~2 s | Refine the decision |
| Human review | ~0.1% | Minutes to hours | Ambiguous, appealed, and novel cases |
| Proactive sweeps | Retroactive | Batch | Re-scan history when a policy or model changes |
pythonPer-policy thresholds driven by asymmetric costs
from dataclasses import dataclass
@dataclass
class PolicyConfig:
name: str
auto_remove_threshold: float
human_review_threshold: float
demote_threshold: float | None
fn_cost_multiplier: float # how much worse a MISS is than a false positive
appealable: bool = True
POLICIES = [
PolicyConfig("csam", 0.30, 0.05, None, fn_cost_multiplier=10_000,
appealable=False),
PolicyConfig("terrorism", 0.45, 0.15, None, fn_cost_multiplier=5_000),
PolicyConfig("spam", 0.85, 0.60, 0.40, fn_cost_multiplier=1),
PolicyConfig("hate_speech", 0.95, 0.55, 0.70, fn_cost_multiplier=3),
PolicyConfig("misinformation", 0.99, 0.60, 0.65, fn_cost_multiplier=2),
]
def decide(scores, reach_estimate):
"""Reach-weighted: a post about to be seen by 5M people deserves more scrutiny."""
actions = []
for p in POLICIES:
s = scores.get(p.name, 0.0)
# High-reach content gets a LOWER bar for human review
review_threshold = p.human_review_threshold * (1.0 if reach_estimate < 10_000
else 0.6)
if s >= p.auto_remove_threshold:
actions.append(("remove", p.name, s))
elif p.demote_threshold and s >= p.demote_threshold:
actions.append(("demote", p.name, s))
elif s >= review_threshold:
actions.append(("human_review", p.name, s))
return actions or [("allow", None, 0.0)]
# Note the CSAM threshold is 0.30 -- deliberately low. False positives are
# reviewed by humans; false negatives are unacceptable at any rate.Scale arithmetic and the adversarial reality
pythonDoes this fit in a budget?
POSTS_PER_DAY = 1_000_000_000
qps_avg = POSTS_PER_DAY / 86_400
qps_peak = qps_avg * 2.5
print(f"avg {qps_avg:,.0f} QPS, peak {qps_peak:,.0f} QPS")
# avg 11,574 QPS, peak 28,935 QPS
# Tier 1: a small multilingual encoder, ~4 ms on GPU, batch 64
tier1_gpus = qps_peak * 0.004 / 64 * 64 # throughput-bound estimate
print(f"tier-1 GPUs (with 2x headroom): {tier1_gpus*2:,.0f}")
# Tier 2: 2% escalation to a heavy multimodal model at ~250 ms
tier2_qps = qps_peak * 0.02
print(f"tier-2 QPS: {tier2_qps:,.0f} -> GPUs: {tier2_qps*0.25/8*2:,.0f}")
# Human review capacity
review_rate = 0.001
reviews_per_day = POSTS_PER_DAY * review_rate
reviewers = reviews_per_day / (60 / 1.5 * 8) # 1.5 min/item, 8h shifts
print(f"reviews/day: {reviews_per_day:,.0f} reviewers needed: {reviewers:,.0f}")
# reviews/day: 1,000,000 reviewers needed: 3,125
#
# THAT number -- thousands of reviewers -- is why the threshold policy is
# an operational decision, not just a modelling one. Every 0.1% shift in
# the human-review rate is a million reviews and thousands of staff.- Appeals are a legal requirement in many jurisdictions, and also your best source of false-positive labels. Track appeal-overturn rate per policy — a rising rate means your threshold is too aggressive.
- Language and cultural coverage. A model trained mostly on English will systematically over-flag some dialects and under-flag others. Measure per-language and per-region precision and recall separately; aggregate numbers hide serious harm.
- Reviewer welfare is a genuine design constraint: rotate exposure, blur by default, cap graphic-content hours. Mentioning it shows you understand this is a socio-technical system.
- Policy changes require backfill. When a policy is updated, historical content must be re-scanned. Build the batch re-scoring path from the start.
Say these points
- Tier: hash match → fast classifier → heavy model → human review, with per-policy thresholds.
- Error-cost asymmetry differs per policy; one global threshold is wrong.
- Weight scrutiny by reach — high-reach content gets a lower review bar.
- Human review capacity is the real constraint; a 0.1% shift is thousands of staff.
- Adversarial evasion, per-language fairness, appeals and backfill are all design requirements.
Avoid these mistakes
- A single threshold across all policy categories.
- Reporting aggregate precision/recall that hides per-language failures.
- No plan for re-scanning history after a policy change.
Expect these follow-ups
- How would you detect a coordinated inauthentic-behaviour campaign?
- How do you measure moderation quality when 'correct' is contested?
- How would you handle a viral post that is borderline under policy?
Advancedscalesystem-designarchitectureAsked at Google, Meta
30-second answer
Identify what actually breaks first rather than saying 'add more machines'. Walk through each component and state whether it scales horizontally, vertically or not at all: stateless serving scales trivially, the feature store becomes the bottleneck, the vector index may no longer fit in memory, training data volume forces sampling, and the human-in-the-loop components hit a hard staffing ceiling. Then propose the specific change for the first thing to break.
| Component | Scales how | Breaks at scale because |
|---|---|---|
| Stateless model serving | Horizontally, trivially | Rarely the bottleneck |
| Feature store reads | Horizontally with sharding | Hot keys; p99 tail from fan-out |
| Vector index | Poorly past memory limits | Must shard or compress (IVF-PQ, DiskANN) |
| Training data | Requires sampling | Cannot train on 90B events; need stratified sampling |
| Embedding tables | Distributed parameter server | Single-host memory limit |
| Model refresh | Fixed cost, higher frequency needed | Retraining cadence cannot keep up |
| Human review | Linearly with headcount | Hard cost and hiring ceiling |
| Logging and monitoring | Sampling required | Storage and query cost explode |
pythonFind the breaking point quantitatively
def scale_analysis(current, multiplier=100):
limits = {
"serving_qps": {"current": current["qps"], "limit_per_node": 2_000},
"feature_store_qps": {"current": current["qps"] * 12, "limit_per_node": 50_000},
"vector_index_gb": {"current": current["vectors"] * 768 * 4 / 1e9,
"limit_per_node": 400},
"training_rows": {"current": current["events_per_day"] * 90,
"limit_per_node": 5e9},
"human_reviews": {"current": current["qps"] * 86_400 * 0.001,
"limit_per_node": 320}, # per reviewer per day
}
print(f"{'component':22s} {'now':>14} {'at 100x':>14} {'nodes/staff':>12}")
for name, v in limits.items():
scaled = v["current"] * multiplier
needed = scaled / v["limit_per_node"]
flag = " <-- RETHINK" if needed > 200 else ""
print(f"{name:22s} {v['current']:>14,.0f} {scaled:>14,.0f} {needed:>12,.0f}{flag}")
scale_analysis({"qps": 500, "vectors": 50_000_000, "events_per_day": 100_000_000})
# component now at 100x nodes/staff
# serving_qps 500 50,000 25
# feature_store_qps 6,000 600,000 12
# vector_index_gb 154 15,360 38
# training_rows 9,000,000,000 900,000,000,000 180 <-- RETHINK
# human_reviews 43,200 4,320,000 13,500 <-- RETHINK
#
# Serving is fine. TRAINING DATA and HUMAN REVIEW are what break.
# The answer is stratified sampling + far more aggressive automation,
# not more GPUs.The standard responses per bottleneck
- Training data explosion → stratified and importance sampling (keep all positives, downsample easy negatives), plus a logQ correction so probabilities stay calibrated.
- Feature store hot keys → cache the head entities locally in the serving process, batch and parallelise reads, and precompute the expensive aggregates.
- Vector index too large → IVF-PQ compression, shard by tenant or region, or move to DiskANN and serve from SSD.
- Embedding tables too large → distributed parameter server, hashing tricks, or frequency-based pruning of rare ids.
- Human review ceiling → raise automation thresholds where error cost permits, prioritise by reach and risk rather than reviewing uniformly, and use active learning so reviewed items are the most informative ones.
- Monitoring cost → sample logs, aggregate at write time, and keep full fidelity only for a small random slice plus all errors.
Say these points
- Name what breaks first and why, then the fix, then the next constraint.
- Stateless serving is almost never the bottleneck.
- At extreme scale, training data volume and human-in-the-loop capacity break first.
- Do the arithmetic — it turns a hand-wave into a defensible answer.
- Be able to answer the reverse question about far less data too.
Avoid these mistakes
- Answering 'add more replicas' for every component.
- Ignoring non-compute constraints like reviewer headcount and label budget.
- Not doing the arithmetic when the numbers are available.
Expect these follow-ups
- How would you shard a vector index across 20 nodes while keeping recall?
- How does stratified sampling affect calibration, and how do you correct it?
- What breaks first if latency budget drops from 200 ms to 20 ms?
Showing 5 of 5 questions for ml-system-design-search-and-ranking.
Check your understanding
5 questions · no sign-up, nothing stored
Hands-on challenge
Build it — this is what you talk about in a deep-dive round.
Design review: critique and improve a real system
Take a public architecture write-up (YouTube recommendations, Pinterest search, Meta feed ranking) and analyse it as though you were on the design review.
Requirements
- Summarise the design in the 8-step framework and identify which constraint dominates.
- Identify at least three decisions you would question, with the trade-off each involves.
- Do the back-of-envelope arithmetic for their stated scale and check whether the design is consistent.
- Propose one concrete improvement and describe how you would A/B test it.
Stretch goals
- Write the position-bias correction for their ranking stage and explain how you would validate it.
- Design the exploration policy that would keep their offline evaluation unbiased.